KR102011430B1 - Method and apparatus for parallel processing image - Google Patents

Abstract

The present disclosure relates to a method and apparatus for encoding / decoding an image including a plurality of perspective images. An encoding method of an image according to the present disclosure may include: identifying a processing order of an encoding target perspective image among the plurality of perspective images, determining whether the identified processing sequence is the same as a predetermined processing sequence, and the identified If the processing sequence is not the same as the predetermined processing sequence, the method may include converting the encoding target perspective image.

Description

Image parallel processing method and apparatus {METHOD AND APPARATUS FOR PARALLEL PROCESSING IMAGE}

The present disclosure relates to a method and apparatus for encoding / decoding an image.

Specifically, the present disclosure relates to a method and apparatus for encoding / decoding a virtual reality image, and more particularly, to a method and apparatus for parallel processing of encoding / decoding of a virtual reality image.

Recently, broadcast services having a high definition (HD) and ultra high definition (UHD) resolution (1920x1080 or 3840x2160) have been expanded not only in Korea but also worldwide. Users are getting used to high-resolution, high-definition video, and in response, many organizations are speeding up the development of next-generation video devices.

Currently, the Moving Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG) have jointly completed the standardization of High Efficiency Video Coding (HEVC), the next generation video codec. According to this, the compression efficiency for the image including the UHD video shows twice the compression efficiency compared to H.264 / AVC. As the resolution of images increases and technology is developed, interest in realistic images such as stereoscopic images and omnidirectional (360 degree) video as well as 2D images is increasing. As a result, many companies are launching devices capable of playing head-mounted displays or realistic images.

An object of the present disclosure is to provide a method and apparatus for improving the encoding / decoding efficiency of an image.

Another technical problem of the present disclosure is to provide a method and apparatus for improving the encoding / decoding efficiency of a virtual reality image.

Another technical problem of the present disclosure is to provide a method and apparatus for parallel processing of encoding / decoding of a virtual reality image.

Another technical problem of the present disclosure is to provide a method and apparatus for improving the encoding / decoding speed of an image having a plurality of perspectives, such as a polyhedral 360 degree VR image.

Technical problems to be achieved in the present disclosure are not limited to the above-mentioned technical problems, and other technical problems not mentioned above will be clearly understood by those skilled in the art from the following description. Could be.

According to the present disclosure, a method of encoding an image including a plurality of perspective images includes: identifying a processing sequence of an encoding target perspective image among the plurality of perspective images, wherein the identified processing sequence corresponds to a predetermined processing sequence; And determining whether or not they are the same, and converting the encoding target perspective image when the identified processing sequence is not the same as the predetermined processing sequence.

In the encoding method according to the present disclosure, the processing sequence is identified by a processing starting position and a processing progress direction, and the processing starting position of the predetermined processing sequence is the upper left position of the perspective image, and the processing progress of the predetermined processing sequence is performed. The direction may be clockwise.

In the encoding method according to the present disclosure, the transform may include at least one of a symmetric transform and a motion transform on the encoding target perspective image.

In the encoding method according to the present disclosure, the symmetric transformation may be performed when the processing progress direction of the identified processing sequence is different from the processing progress direction of the predetermined processing sequence.

In the encoding method according to the present disclosure, the symmetric transformation may be one of left and right symmetry transformation, vertical symmetry transformation, and diagonal symmetry transformation.

In the encoding method according to the present disclosure, the rotation transformation may be performed when the processing start position of the identified processing sequence is different from the processing start position of the predetermined processing sequence.

In the encoding method according to the present disclosure, the rotation transformation may be a transformation for rotating the encoding target perspective image such that the processing start position of the encoding target perspective image is the same as the processing start position of the predetermined processing sequence.

In the encoding method according to the present disclosure, the transforming of the target image to be encoded may be performed. First converting the encoding target perspective image, determining whether a processing order of the first transformed encoding target perspective image is the same as the predetermined processing sequence, and processing the first transformed encoding target perspective image If the order is not the same as the predetermined processing sequence, the method may include converting the first transformed encoding target perspective image.

In the encoding method according to the present disclosure, the first transform may be one of a symmetric transform and a shift transform, and the second transform may be the other.

According to the present disclosure, an apparatus for encoding an image including a plurality of perspective images includes a transducer, wherein the transducer identifies a processing sequence of an encoding target perspective image among the plurality of perspective images and performs the identified processing. It may be configured to determine whether the order is the same as the predetermined processing order, and convert the encoding target perspective image when the identified processing order is not the same as the predetermined processing order.

According to the present disclosure, a method of decoding an image including a plurality of perspective images includes: identifying a processing sequence of a decoding target perspective image among the plurality of perspective images, wherein the identified processing sequence is determined by a predetermined processing sequence; And determining whether or not they are the same, and converting the decoding target perspective image when the identified processing order is not the same as the predetermined processing order.

In the decoding method according to the present disclosure, the processing sequence is identified by a processing start position and a processing progress direction, and the processing start position of the predetermined processing sequence is the upper left position of the perspective image, and the processing progress of the predetermined processing sequence is performed. The direction may be clockwise.

In the decoding method according to the present disclosure, the transform may include at least one of a symmetric transform and a shift transform on the decoding target perspective image.

In the decoding method according to the present disclosure, the symmetric transformation may be performed when the processing progress direction of the identified processing order is different from the processing progress direction of the predetermined processing order.

In the decoding method according to the present disclosure, the symmetric transformation may be one of symmetric transformation, vertical symmetry transformation, and diagonal symmetry transformation.

In the decoding method according to the present disclosure, the rotation conversion may be performed when the processing start position of the identified processing sequence is different from the processing start position of the predetermined processing sequence.

In the decoding method according to the present disclosure, the rotation transformation may be a transformation for rotating the decoding target perspective image such that the processing start position of the decoding target perspective image is the same as the processing start position of the predetermined processing sequence.

In the decoding method according to the present disclosure, the step of converting the decoding target perspective image. First converting the decoding target perspective image, determining whether the processing order of the first transformed decoding target perspective image is the same as the predetermined processing order, and the first transformed decoding target perspective image If the processing order is not the same as the predetermined processing order, the method may include performing a second transformation on the first transformed decoding target perspective image.

In the decoding method according to the present disclosure, the first transform may be one of a symmetric transform and a shift transform, and the second transform may be the other.

According to the present disclosure, an apparatus for decoding an image including a plurality of perspective images includes a transducer, wherein the transducer identifies a processing sequence of a decoding target perspective image among the plurality of perspective images, and the identified processing. It may be configured to determine whether the order is the same as a predetermined processing order, and convert the decoding target perspective image when the identified processing order is not the same as the predetermined processing order.

The computer-readable recording medium according to the present disclosure may store a bitstream generated by the method and / or apparatus for encoding an image according to the present disclosure.

According to the present disclosure, a method and apparatus for improving encoding / decoding efficiency of an image may be provided.

In addition, according to the present disclosure, a method and apparatus for improving encoding / decoding efficiency of a virtual reality image may be provided.

In addition, according to the present disclosure, a method and apparatus for parallel processing of encoding / decoding of a virtual reality image may be provided.

In addition, according to the present disclosure, a method and apparatus for improving the encoding / decoding speed of an image having a plurality of perspectives, such as a polyhedral 360 degree VR image, can be provided.

The effects obtainable in the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned above may be clearly understood by those skilled in the art from the following description. will be.

1 illustrates various types of cameras for capturing 360 degree video.
2 is a diagram exemplarily illustrating an image in which a 360 degree video is projected on a 3D space.
3 is a diagram illustrating a structure of a video encoding apparatus according to the present disclosure.
4 is a diagram illustrating a structure of an image decoding apparatus according to the present disclosure.
5 is a diagram for exemplarily describing a divergent camera image and a converged camera image.
6 is an exemplary diagram for describing segmentation of an image.
7 is a diagram for explaining a WPP processing procedure in an image.
8 is a diagram for explaining tile-based parallel processing.
9 is a diagram illustrating a development view for encoding / decoding a 360 degree omnidirectional video.
FIG. 10 is a diagram for explaining a developed view when a 360 degree omnidirectional image is projected onto a cube. FIG.
FIG. 11 is a diagram for describing various methods for developing an omnidirectional image projected on a cube in two dimensions.
12 is an exemplary diagram for explaining dependencies between CTUs belonging to adjacent perspectives.
FIG. 13 is a diagram for describing an end point of decoding / decoding according to a deployment method.
14 is a diagram for explaining a difference in image processing time according to a developed view.
15 is a conceptual diagram for describing image parallel processing according to the present disclosure.
16 is a flowchart for explaining image parallel processing according to the present disclosure.
17 is a diagram illustrating a processing sequence of an omnidirectional image on a cube.
FIG. 18 is a diagram illustrating an expanded view of the omnidirectional image of FIG. 17. FIG.
19 is an example of a 360-degree VR image in the form of a cube exploded view.
20 is a diagram for explaining a step (step 1) of encoding / decoding the video of FIG. 19 in parallel.
21 is a diagram for explaining a step (step 2) of encoding / decoding the video of FIG. 19 in parallel.
22 is a diagram for explaining a step (step 3) of encoding / decoding the video of FIG. 19 in parallel.
23 is an exemplary diagram for explaining a negative / decoding direction of a perspective.
FIG. 24 is a diagram for describing an encoding / decoding procedure for two non-neighboring perspectives in a cube omnidirectional image.
FIG. 25 is a diagram for describing a decoding / decoding procedure of two perspectives facing each other in a cube omnidirectional image. FIG.
FIG. 26 is a diagram for explaining a difference according to FIGS. 25A and 25B.
FIG. 27 is a diagram for describing encoding / decoding according to FIG. 25B.
FIG. 28 is a diagram for describing encoding / decoding of perspectives that may refer to two or more perspectives.
FIG. 29 is a diagram for describing a method of dividing one perspective into two regions and encoding / decoding each region. FIG.
30 is a diagram for explaining a coding / decoding process procedure when referring to two or more perspectives.
FIG. 31 is an exemplary diagram for describing an encoding / decoding procedure of a cube omnidirectional image. FIG.
32 is a diagram illustrating a configuration of an encoding device according to the present disclosure.
33 is a diagram illustrating a configuration of a decoding apparatus according to the present disclosure.
34 is an exemplary diagram for explaining a method of converting a processing order of a perspective.
35 is a diagram illustrating 33 spatial prediction modes.
36 is an exemplary diagram for describing spatial prediction of a current perspective with reference to a prediction mode of a neighbor perspective.
37 is a diagram for explaining a difference between prediction modes of a current block and a reference block.
38 is a diagram for explaining a method of analyzing a directional prediction mode of a reference block.
FIG. 39 is a diagram illustrating eight cases of directional prediction modes according to a method of developing a perspective. FIG.
40 is a diagram for explaining a position of a reference sample used for spatial prediction of a cube omnidirectional image.
FIG. 41 is a view for explaining a method of parallel processing a cube omnidirectional image utilizing continuity on a developed view.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present disclosure. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention.

In the following description of embodiments of the present disclosure, when it is determined that a detailed description of a known structure or function may obscure the subject matter of the present disclosure, a detailed description thereof will be omitted. In the drawings, parts irrelevant to the description of the present disclosure are omitted, and like reference numerals designate like parts.

In the present disclosure, when a component is "connected", "coupled" or "connected" with another component, it is not only a direct connection, but also an indirect connection in which another component exists in the middle. It may also include. In addition, when a component "includes" or "having" another component, it means that it may further include another component, without excluding the other component unless otherwise stated. .

In the present disclosure, terms such as first and second are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance between the components unless specifically mentioned. Accordingly, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and likewise, a second component in one embodiment may be referred to as a first component in another embodiment. It may also be called.

In the present disclosure, components that are distinguished from each other are for clearly describing each feature, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated into one hardware or software unit, or one component may be distributed and formed into a plurality of hardware or software units. Therefore, even if not mentioned otherwise, such integrated or distributed embodiments are included in the scope of the present disclosure.

In the present disclosure, components described in various embodiments are not necessarily required components, and some may be optional components. Therefore, an embodiment composed of a subset of components described in an embodiment is also included in the scope of the present disclosure. In addition, embodiments including other components in addition to the components described in the various embodiments are included in the scope of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

The present disclosure relates to parallel processing of images. Specifically, the present disclosure relates to an apparatus for increasing processing speed by encoding / decoding an image having various viewpoints in parallel. The image having various viewpoints may be, for example, a 360 degree video, an omnidirectional video, a polyhedral 360 degree VR image, or the like. In encoding / decoding of an image according to the present disclosure, it may be used that reference can be made in various directions within or between images. In the present disclosure, a perspective may refer to a position of a camera or a direction of a camera that captures video.

Conventional 2D video has a fixed viewpoint. However, omni-directional video or 360-degree video refers to video that the user can see at a desired point in time. Omni-directional video or 360-degree video can be obtained by photographing all directions of 360 degrees from a point using multiple cameras, fisheye lenses or reflectors and then projecting onto polyhedrons or spheres in three-dimensional space. In the omnidirectional video or 360 degree video, only an area corresponding to the viewpoint viewed by the user may be played.

1 illustrates various types of cameras for capturing 360 degree video.

FIG. 1A illustrates a plurality of cameras, and FIG. 1B illustrates a fisheye lens. (C) of FIG. 1 illustrates a reflector.

2 is a diagram exemplarily illustrating an image in which a 360 degree video is projected on a 3D space.

2A illustrates an example of projecting a 360 degree image onto a polyhedron in a three-dimensional space. 2B shows an example in which a 360 degree image is projected onto a sphere in a three-dimensional space. In the example of FIG. 2, the shaded portion indicates the position of the 360-degree image. That is, the shaded portion of FIG. 2 may be an area corresponding to the viewpoint viewed by the user.

In order to compressively encode omnidirectional video (360 degree video), a method and apparatus for compression encoding 2D video may be used.

3 and 4 illustrate the structures of an image encoding apparatus and a decoding apparatus according to the present disclosure.

The image encoding apparatus of FIG. 3 may generate a bitstream by receiving and encoding an input image. The image encoding apparatus includes a reconstructed picture buffer 301, an inter prediction unit 302, an intra prediction unit 303, a switch 304, a subtractor 305, a transform unit 306, a quantization unit 307, and an encoding. The unit 308 may include an inverse quantization unit 309, an inverse transform unit 310, and / or an adder 311.

The input image may be encoded in block units. The current block may be intra prediction or inter prediction based on an already decoded / decoded image.

When the current block is predicted in the screen, the switch 304 may be switched to the in-screen predictor 303. The intra prediction unit 303 may predict the current block by referring to an already encoded / decoded area within the current picture. When the current block is predicted inter screen, the switch 304 may be switched to the inter screen predictor 302. The inter prediction unit 302 may predict the current block by referring to at least one picture that is already encoded / decoded and stored in the reconstructed picture buffer 301.

The prediction block of the current block generated by the intra prediction or the inter prediction may be transmitted to the subtractor 305. The subtractor 305 may generate a difference block between the current block and the prediction block of the current block. The transform block 306 and / or the quantization block 307 may be performed on the generated difference block. The residual coefficients, which have been transformed and / or quantized, may be encoded by the encoder 308. The encoder 308 may perform, for example, entropy encoding based on a probability of occurrence of a symbol.

Inverse quantization and / or inverse transform in the inverse transform unit 310 may be performed in the transform and / or quantized difference block. The inverse quantized and / or inversely transformed differential blocks may be sent to the adder 311. The adder 311 may reconstruct the current block by adding the received difference block to the prediction block of the current block. The reconstructed picture including the reconstructed block may be stored in the reconstructed picture buffer 301.

The image decoding apparatus of FIG. 4 may generate a reconstructed image by receiving and decoding a bitstream. The video decoding apparatus includes a decoder 401, an inverse quantizer 402, an inverse transform unit 403, an adder 404, an inter prediction unit 405, an intra prediction unit 406, a switch 407, and / Alternatively, the reconstructed picture buffer 408 may be included.

The decoder 401 may generate a difference block of the current block by decoding the bitstream. The decoder 401 may perform entropy decoding based on, for example, a probability of occurrence of a symbol. The difference block may include transformed and / or quantized difference coefficients.

An inverse quantization in the inverse quantization unit 402 and / or an inverse transformation in the inverse transform unit 403 may be performed on the difference block including the transform and / or quantized difference coefficients. The inverse quantized and / or inversely transformed differential blocks may be sent to the adder 404. The adder 404 may reconstruct the current block by adding the received difference block to the prediction block of the current block. The reconstructed picture including the reconstructed block may be stored in the reconstructed picture buffer 408.

The prediction block of the current block input to the adder 404 may be generated based on an image that is already decoded. When the current block is predicted in the screen, the switch 407 may be switched to the in-screen predictor 406. The intra prediction unit 406 may predict the current block by referring to an already encoded / decoded area within the current picture. When the current block is predicted inter screen, the switch 407 may be switched to the inter screen predictor 405. The inter prediction unit 405 may predict the current block by referring to at least one picture that is already encoded / decoded and stored in the reconstructed picture buffer 408.

The encoding / decoding method according to the present disclosure may be applied to immersive media images in addition to omnidirectional (360 degree) video. The sensational media image may include a divergent camera image, a convergent camera image, and the like.

5 is a diagram for exemplarily describing a divergent camera image and a converged camera image.

The divergent camera image may include images W1, W2, and W3 of various directions obtained by using a plurality of cameras C1, C2, and C3. The converged camera image may include images W, W2, W3, and W4 of a specific direction obtained by using the plurality of cameras C1 to C4.

The video encoding apparatus may compress a still image or a video. The video encoding apparatus may perform frame-based compression, which compresses the entire image in one processing unit, or block-based compression, which divides and processes the image in specific block units. An image encoding / decoding apparatus that performs frame-based compression has difficulty in applying a prediction technique and requires a large amount of memory and computation. Therefore, most video encoding / decoding apparatuses perform block-based compression.

For example, block-based compression may be performed by dividing an image into coding tree units (CTUs), which are large block units. CTUs may be encoded / decoded in sequential scanning order. The sequential scan order may be a raster scan order or a Z scan order. For example, the CTU may be divided into coding units (CUs) which are smaller block units and then encoded / decoded. This method is advantageous in terms of compression efficiency since the block size can be more precisely adjusted according to the characteristics of the image.

6 is an exemplary diagram for describing segmentation of an image.

6 (a) shows a state in which an image is divided by CTU. Referring to FIG. 6A, each CTU may be encoded / decoded in order of CTU A to CTU K according to a sequential scanning order.

The block-based video compression apparatus may reduce the amount of information to be transmitted by using the correlation between neighboring blocks in the image, thereby increasing compression efficiency. Referring to (a) of FIG. 6, at the time when the encoding / decoding of the CTU F begins, the encoding / decoding of the neighboring CTUs B, C, D, and E is completed. Therefore, the compression efficiency of the CTU F can be increased by using information of the neighbor CTU. This may mean that decoding of CTU B, C, D, and E should be completed in order to decode the CTU F.

6B illustrates an example of recursively dividing the CTU. As described above, the CTU may be divided into a plurality of CUs, where recursive partitioning may be applied. For example, quad tree splitting can be used for recursive splitting. As in the example shown in FIG. 6B, the CTU F may be divided into four CUs. In this case, the division flag indicating whether to divide the CTU F may be 1. Each partitioned CU may be encoded / decoded in a Z scan order. In addition, each CU may be further divided into four CUs. Or, each CU can be broken at the corresponding depth. The partition flag of each CU may indicate whether to partition the corresponding CU. As such, the recursive segmentation may be stopped by reaching the segmentation flag value or the maximum allowable segmentation depth specified in the image. As shown in (b) of FIG. 6, the CTU F may be divided into four CUs, and the first, second and third CUs may not be divided in the Z scan order. In addition, the fourth CU F1 may be further divided into four CUs. The first, second and third CUs may not be divided in the Z scan order among the four CUs generated by dividing the CU F1. In addition, the fourth CU F2 may be further divided into four CUs. Through the division structure illustrated in FIG. 6B, the size of the prediction block can be adjusted in the image, thereby improving the efficiency of prediction.

Wavefront parallel processing (WPP) and / or tile based encoding / decoding may be used to improve the encoding / decoding speed of an image. Wavefront technology starts parallel processing when the dependence on pixel information is resolved. Referring to FIG. 6A, when the CTU D completes the decoding / decoding of B, which is the CTU of the upper row, the dependency on pixel information may be resolved. Therefore, the prediction algorithm using the information of the CTU B can be applied. Similarly, CTU H can resolve the dependence on pixel information when the CTU E's encoding / decoding of the top row is completed. Therefore, the prediction algorithm using the information of the CTU E can be applied. However, even if the image is encoded / decoded in parallel in consideration of the pixel information dependency, entropy coding is not executed in parallel but executed in sequential scanning order. Therefore, entropy coding has become a bottleneck for parallel processing in encoding / decoding. This problem can be solved by the application of WPP. WPP can additionally support parallel encoding / decoding for entropy coding. To this end, by generating a sub bitstream in the row unit of each CTU and transmitting the position of the corresponding sub bitstream to the slice header, parallel entropy coding is possible for each sub bitstream. In other words, entropy coding can be used independently for each row through WPP.

In WPP, in order to minimize the deterioration in encoding efficiency of entropy coding and simultaneously support parallel encoding / decoding, the lower row acquires an entropy context model and starts encoding / decoding at the time when the second block of the upper row is completed. The current block and neighboring blocks are highly correlated. Therefore, it is more advantageous to increase the compression efficiency of the decoder / decoder by obtaining the updated context information from the neighboring block and initializing the context model of the current row than using the fully initialized context model in each row.

7 is a diagram for explaining a WPP processing procedure in an image.

The intra-picture block of FIG. 7 means CTU. The number in the block means the processing order of each CTU. Arrows in FIG. 7 indicate that the context information of the top row is obtained to initialize the context model of the current row.

In FIG. 7, the shaded CTU represents a CTU in which encoding and / or decoding is completed. In addition, unshaded CTUs represent CTUs that have not yet started encoding and / or decoding.

8 is a diagram for explaining tile-based parallel processing.

As shown in FIG. 8, one picture may be divided into several tiles (eg, six tiles) having a rectangular shape. Each tile may be encoded / decoded into areas independent of each other. CTUs included in each tile may be encoded / decoded in sequential scanning order. That is, in tile-based parallel processing, blocks located at a boundary of a tile may be encoded or decoded without using information such as pixel values or motion vectors of blocks belonging to neighboring tiles. Thus, when using tiles, similarly to slices, each tile area can be simultaneously encoded or decoded.

The tile division information may be signaled through, for example, a picture parameter set (PPS). The tile splitting information may include information about a starting point of each tile. Alternatively, the tile division information may include information regarding one or more vertical borders and / or one or more horizontal borders that divide the picture. Alternatively, the tile splitting information may include information about splitting the picture into tiles of equal size.

9 is a diagram illustrating a development view for encoding / decoding a 360 degree omnidirectional video.

As shown in FIG. 9, an image of a 360-degree omnidirectional video may be processed by being projected in an unfolded view of a polyhedron or an elongated isometric (Equirectangular) form. When the image is played back, the image may be projected onto a polyhedron or sphere, and the user may play the image in the form of viewing the image from the center of the polyhedron or sphere. For example, as shown in FIG. 2, the user's viewpoint may be assumed to be in the center of a polyhedron or sphere.

FIG. 10 is a diagram for explaining a developed view when a 360 degree omnidirectional image is projected onto a cube. FIG.

As shown in the example of FIG. 10, when a 360-degree omnidirectional image is projected onto a cube, information of the image may be stored for six planes P1, P2, P3, P4, P5, and P6. In addition, when the image is reproduced, as shown in the example of FIG. 10A, each image of six surfaces may be reproduced by projecting onto a cube in a three-dimensional space. Here, the viewer's position is at the center of the cube, and each side may be a position and / or a direction (perspective) of the eye view. For example, in the example shown in Figure 10, if perspective P4 is the front of the cube, perspective P1 is the top of the cube, then perspective P2 is the back of the cube, perspective P3 is the cube's left, perspective P5 is the cube's right, perspective P6 May correspond to the bottom of the cube.

The omnidirectional image projected on the polyhedron can be expressed in various developments depending on how the polyhedron is developed. FIG. 11 is a diagram for describing various methods for developing an omnidirectional image projected on a cube in two dimensions.

For example, as in the example shown in FIG. 11, there may be eleven deployment schemes. Alternatively, the developed view obtained by symmetry of the developed view shown in FIG. 11 up / down / left / right may be treated as another form of developed view.

In the case of independently encoding / decoding each perspective, the developed view illustrated in FIG. 11 may have little meaning other than information about where the image is located. However, if the encoding / decoding process is performed in the sequential scanning order using the continuity of the image information between perspectives, the encoding / decoding rate may be different depending on the form of the developed view.

In the following, the difference of the decoding / decoding rate according to the development will be described in detail.

12 is an exemplary diagram for explaining dependencies between CTUs belonging to adjacent perspectives. In Fig. 12, each of A, B, C, D, E, and F represents the positions of the perspectives that are developed on the developed view, and the thick lines represent the boundaries between the perspectives. Small squares inside each perspective represent blocks (eg, CTUs) that are units of image processing.

When the video is encoded / decoded through WPP parallel processing, CTU e1 of Perspective E has a dependency on CTU b1 of Perspective B and CTU d2 of Perspective D. Therefore, the CTU b1 of the Perspective B and the CTU d2 of the Perspective D must be completed to decode / decode the CTU e1 of the Perspective E. For the same reason, the CTU c1 of the Perspective C and the CTU e2 of the Perspective E must be completed to decode / decode the CTU f1 of the Perspective F.

In the case of the omnidirectional image, information may not exist at a specific position of the image according to the development view. For example, the image of the perspective D shown in FIG. 12 may not exist according to the developed view. However, even if there is no perspective D image according to the existing WPP parallel processing method, the encoding / decoding of the perspective E may be started after the encoding / decoding of the CTU b1 of the perspective B is completed. Therefore, the time of sub / decoding processing of the entire image is not different from that of the perspective D image. However, if there is no image of the perspective F on the developed view, when the encoding / decoding process of the image of the perspective E is finished, the encoding / decoding of the entire image may be completed.

FIG. 13 is a diagram for describing an end point of decoding / decoding according to a deployment method.

FIG. 13A illustrates a case where the last perspective in the sub / decoding sequence is at the D position of FIG. 12. FIG. 13B is a diagram illustrating a case in which the last perspective is at position E of FIG. 12. In Figs. 13A and 13B, thick lines indicate boundaries between perspectives. In addition, the small squares in each perspective represent CTUs. In addition, the number inside each CTU indicates the encoding / decoding order of the corresponding CTU.

On the assumption that the encoding / decoding processing time of each CTU is the same, it can be seen that in the case of FIG. 13A, when the 23rd CTU is processed, encoding / decoding of the entire image is completed. In addition, in case (b) of FIG. 13, when processing the 28 th CTU, it may be confirmed that encoding / decoding of the entire image is completed. As described above, when using the WPP method of performing the decoding / decoding in the order of the upper left to the lower right, even when the number of perspectives included in the image is the same, it can be seen that the timing at which the encoding / decoding of the entire image is completed is different. . In other words, the position (or deployment method) of the perspective may affect the speed of the decoding / decoding process.

The encoding / decoding process order using WPP may be determined by image information dependency. That is, the processing of the CTU can be started when two CTUs in the upper row are processed. Assuming that each CTU's encoding / decoding processing time is the same, the delay per row becomes the time to process two CTUs due to the above dependency. In addition, since the processing of the CTU can be started only when one CTU adjacent to the left is processed, the delay per column becomes the time to process one CTU.

14 is a diagram for explaining a difference in image processing time according to a developed view.

Referring to FIG. 14A, each viewpoint (perspective) has 5 × 5 CTUs. The last time point is located in the third row of the fourth row, and the last CTU is located in the 20th row and the 15th column. As mentioned above, there are two CTU delays per row and one CTU delay per column. Therefore, the last CTU is processed at (2 × 20 (row) -1) + (1 × 15 (column) -1). As a result, in FIG. 14A, it can be seen that the last CTU is completed at the 53rd time. Therefore, when the processing time of one CTU is t (CTU), the time required for processing the entire image according to the development of FIG. 14A becomes 53 * t (CTU).

FIG. 14B is a diagram illustrating another developed view of an omnidirectional image having 5 × 5 CTUs per view. Referring to FIG. 14B, the CTU located at the lower right of the developed view exists in two places of the 15th row, the 5th column, the 10th row, and the 20th column. In each case, when the processing time is calculated, the CTU of the 15th row and the 5th column is processed at (2 × 15 (row) − 1) + (1 × 5 (column) −1) = 33rd. The CTU of the 10th row and the 20th column is processed at (2 × 10 (row)-1) + (1 × 20 (column) -1) = 38th. Therefore, the time required for processing the entire image is 38 * t (CTU). As described above, it can be seen that there is a difference in the processing speed of the WPP depending on how the developed view of the cube of the same size is developed.

Delays in processing time caused by dependencies between adjacent CTUs can be mitigated by applying tile-based techniques. In other words, each face (that is, an area having different perspectives) is treated as an independent tile to be encoded / decoded in parallel. When each perspective is processed in parallel with independent tiles, the processing time of the entire image may be shortened to the processing time of one perspective (eg, 25 * t (CTU) in the example of FIG. 14). Tile-based parallel encoding / decoding can be usefully applied to an image that a user watches at a certain distance. However, in the case of a realistic image such as a 360-degree VR image, the user can watch a part of the image at a very close position, and thus, the user must have a much larger resolution than the existing images. Particularly, in the case of a 360-degree VR image, immersion is very important, and thus high image quality is required such that pixels are not visible during image reproduction.

Using tile-based techniques, each tile is processed independently, reducing the reference area available for prediction and lowering compression efficiency. Therefore, it may not be suitable to apply the tile-based technology to the 360 degree omnidirectional image which requires a great compression efficiency.

Therefore, encoding / decoding using the dependencies between perspectives is important for parallel processing of a 360 degree VR image. Referring to FIGS. 13 and 14, various development methods may be applied to process 360-degree VR images, and parallel processing may be performed using dependencies between adjacent perspectives on each development view. However, if only the continuity between perspectives in the developed view is taken into consideration, there is a limit in shortening the entire image processing time. For example, a developed view of a 360-degree VR image is made by unfolding three-dimensionally connected perspectives as shown in FIG. 10. Therefore, on the polyhedron, there are adjacent perspectives as many as the number of sides of the perspective. That is, the number of adjacent perspectives available on a polyhedron is greater than the number of adjacent perspectives available on a development view.

Hereinafter, a method and apparatus for parallel encoding / decoding of an image in consideration of characteristics of a 360 degree VR image will be described. Specifically, according to the present disclosure, in the image having a plurality of perspectives such as 360 degree video (omnidirectional video), the parallel processing of encoding / decoding may be more efficiently processed by using one or more continuities between each perspective. Can be.

As in the example shown in FIG. 10A, the 360 degree omnidirectional image is stored in a developed view of a polyhedron with several perspectives, such as a cube. In the development view, as in the example shown in Fig. 10C, the perspective P1 is adjacent to one perspective. In practice, however, as in the example shown in Fig. 10A, each perspective is adjacent to four perspectives in the horizontal direction. This means that the encoding / decoding of the image information may be performed in a direction different from the sequential scanning direction of the existing upper left to lower right. In addition, since there is no continuity between images between perspectives that do not neighbor in three-dimensional space, there is no dependency. That is, since the neighboring perspectives do not refer to each other, even if the encoding / decoding proceeds in parallel, the prediction efficiency does not decrease. Compression / decoding parallel processing is performed by taking advantage of the fact that successive perspectives exist in multiple directions and that the decoding and decoding directions can be performed in multiple directions, and that there is one or more perspectives with no correlation between images. Speed up processing while minimizing losses.

15 is a conceptual diagram for describing image parallel processing according to the present disclosure.

16 is a flowchart for explaining image parallel processing according to the present disclosure.

17 is a diagram illustrating a processing sequence of an omnidirectional image on a cube.

FIG. 18 is a diagram illustrating an expanded view of the omnidirectional image of FIG. 17. FIG.

 Surfaces in which the same alphabet is written in FIGS. 15, 17, and 18 refer to points of time that do not refer to each other. For example, A and A 'represent time points not referenced to each other. B and B 'represent viewpoints not referenced to each other, except for A and A' viewpoints. C and C 'represent viewpoints not referenced to each other among the viewpoints except for the above viewpoints. Time points that do not refer to each other may be encoded / decoded in parallel.

In FIG. 15, an arrow connecting a viewpoint and a viewpoint represents a reference relationship between viewpoints. For example, an arrow connecting the view point A and the view point B means that the view point A refers to a time point B when decoding / decoding the view point B. The number of threads may be determined by the number of time points that can be encoded / decoded in parallel. For example, if the number of time points that can be encoded / decoded in parallel is N, N or more threads may be used.

Referring to FIG. 15, the encoding / decoding method according to the present disclosure will be described. Referring to FIG. 15A, first, one or more viewpoints that may be encoded / decoded simultaneously with the viewpoint A and the viewpoint A may be encoded / decoded in parallel. For example, since the viewpoint A and the viewpoint A 'are not continuous, they can be processed in parallel. Referring to FIG. 15B, after some or all of the time points A and A ′ processed in FIG. 15A are encoded / decoded, the time points A. A ′ are referred to. One or more viewpoints that can be encoded / decoded simultaneously with the viewpoint B and the viewpoint B may be encoded / decoded in parallel. For example, the viewpoint B and the viewpoint B 'are not continuous, and thus may be processed in parallel. Referring to FIG. 15C, in the same manner, one or more viewpoints that can be encoded / decoded simultaneously with the viewpoint C and the viewpoint C may be encoded / decoded in parallel. In this way, the remaining views can also be encoded / decoded by combining the points that can be encoded / decoded in parallel.

The method described with reference to FIG. 15 may be represented by the flowchart of FIG. 16.

In step S1610 of FIG. 16, it is possible to start encoding / decoding of one perspective or two or more perspectives that are not neighboring.

In step S1620, encoding / decoding of the perspective in which encoding / decoding has started may proceed. Encoding / decoding may be performed in coding units (eg, CTU units) included in each perspective.

In step S1630, it may be determined whether encoding / decoding for all perspectives is completed. If encoding / decoding for all perspectives has been completed (Yes in step S1630), video encoding / decoding can be ended.

If the encoding / decoding for all the perspectives is not completed (No in step S1630), in step S1640, it may be determined whether there is a neighbor perspective in which the encoding / decoding is in progress or whose dependencies are resolved by the completed perspective. If there is no neighbor perspective in which the dependency is resolved (No in step S1640), in step S1620, the encoding / decoding for the perspective in which the decoding / decoding is in progress may continue.

If there is a neighbor perspective in which the dependency is resolved (Yes in step S1640), then in step S1650, encoding / decoding of the neighbor perspective can be started.

Subsequently, in step S1620, encoding / decoding for a plurality of perspectives in which encoding / decoding is started may proceed. Steps S1620 to S1650 may be repeated until the encoding / decoding of all the perspectives is completed.

19 is an example of a 360-degree VR image in the form of a cube exploded view.

20 to 22 are diagrams for describing an operation of encoding / decoding a video of FIG. 19 in parallel. In the following description with reference to FIGS. 20 to 22, symbols representing each perspective are the same as symbols of the corresponding perspective shown in FIG. 18. Arrows shown in FIGS. 21 and 22 indicate reference relationships between perspectives.

First, as shown in FIG. 20, perspectives A and / or A 'which are not adjacent to each other may be negative / decoded in parallel.

Thereafter, as illustrated in FIG. 21, perspective B and / or B 'may be encoded / decoded in parallel with reference to perspective A and / or A'. Encoding / decoding of perspective B and / or B 'can be initiated once the dependencies with perspective A and / or A' are resolved. That is, the encoding / decoding of the perspectives B and / or B 'may be started when the encoding / decoding of the perspectives A and / or A' is performed in part or in whole. As shown in FIG. 21, perspective B and / or B 'may be encoded / decoded with reference to perspective A and / or A'. Perspectives B and / or B 'may be decoded / decoded from the portion where the dependency is resolved according to the encoding / decoding of the perspectives A and / or A' at the boundary portion adjacent to the perspectives A and / or A ', respectively.

For example, the top boundary of Perspective B is adjacent to the left boundary of Perspective A. Thus, if the block of the second row, first column of perspective A is negative / decoded, the block / decoding of the block of the first row, first column of perspective B can be started. In this case, the encoding / decoding of the perspective B referring to the perspective A may be performed according to the raster scan order.

Also, for example, the bottom border of perspective B is adjacent to the left border of perspective A '. Thus, if the block of the second row, first column of perspective A 'is negative / decoded, the block / decoding of the block of the last row, last column of perspective B can be started. In this case, the encoding / decoding of perspective B referring to perspective A 'may be performed according to an inverse raster scan order (eg, order from bottom right to top left).

When encoding / decoding Perspective B and / or B 'with reference to Perspective A and / or A', the information of the referenced block may be used as is, or after appropriate modification depending on the time point at which the current decoding / decoding is in progress. May be used.

Then, as shown in FIG. 22, perspective C and / or C 'may be encoded / decoded with reference to perspectives A, A', B and / or B 'that have been partially / decoded for some or all. . In FIG. 22, a case of encoding / decoding perspective C and / or C 'with reference to perspective A and / or A' is illustrated. However, the present invention is not limited thereto, and for example, perspective C and / or C 'may be encoded / decoded with reference to perspective B and / or B'. Or perspective C and / or C 'may be encoded / decoded with reference to one or more perspectives of perspectives A, A', B and B '. In this case, the referenced block may be referenced after being appropriately modified according to the point in time at which encoding / decoding is currently in progress.

In the case of a cube, encoding / decoding of a 360 degree VR image may be terminated in three steps shown in FIGS. 20 to 22. However, in the case of octahedrons or icosahedrons, there may be perspectives that have not yet been decoded or decoded. Therefore, the above process can be repeated until the encoding / decoding of all perspectives is completed.

As described above, each perspective of the omnidirectional image has a plurality of perspectives and continuity. Also, each perspective may not have continuity with at least one perspective. The present disclosure may use the same to encode / decode perspectives of an omnidirectional image in parallel.

In addition, in addition to the method described above, applying the WPP-based technology to each perspective, it can be processed faster in parallel while maintaining a high compression efficiency. However, as described with reference to FIG. 7, if each row is processed in parallel in the direction of the upper left block to the lower right block of each perspective, the negative side of the perspective until the dependence of the upper left block of each perspective is resolved. Decryption cannot be started.

23 is an exemplary diagram for explaining a negative / decoding direction of a perspective.

Referring to FIG. 23, even when the perspective P2 and P4 are parallelized / decoded in parallel in the cube omnidirectional image using the WPP-based technique, in order to start the WPP-based encoding / decoding of the perspective P6, the encoding / decoding of the perspective P4 is almost impossible. Should be completed. In other words, two CTUs of the last row of the perspective P4 must be negative / decoded before starting / decoding of the perspective P6.

However, in the omnidirectional image, all sides of each perspective are connected to other perspectives. Therefore, by adjusting the negative / decoding sequence of the perspective P6, the negative / decoding of the perspective P6 can be started more quickly. For example, as in the example shown in FIG. 23, when the encoding / decoding of the perspective P2 is performed from the lower right block to the upper left block, the dependency of the lower left CTU of the perspective P6 may be solved when only two CTUs of the perspective P2 are encoded / decoded. have. When the dependency of the lower left CTU of the perspective P6 is resolved, the WPP for the perspective P6 may be progressed from the lower left block to the upper right block.

When proceeding with WPP-based encoding / decoding, it is possible to utilize pixel information of neighboring perspectives, and also to initialize the entropy context model by importing updated entropy contexts in neighboring perspectives. Indicates the direction to get and initialize the entropy context. As shown in FIG. 23, reference of information between neighboring perspectives is possible.

Hereinafter, the encoding / decoding method according to the present disclosure will be described in more detail with reference to FIG. 23.

First, one may begin to decode / decode a perspective or two or more perspectives that are not neighboring. Within each perspective, encoding / decoding may be performed in the order described with reference to FIG. 7. For example, perspectives P2 and P4 are not adjacent to each other, so that the encoding / decoding can be started in parallel. At this time, in order to more efficiently encode / decode, the perspectives P2 and P4 may be encoded / decoded in different orders.

FIG. 24 is a diagram for describing an encoding / decoding procedure for two non-neighboring perspectives in a cube omnidirectional image. Small blocks in FIGS. 24 (b) and 24 (c) refer to image blocks (eg, CTUs) constituting the perspectives P4 and P2, and the numbers in the blocks indicate the processing order of the blocks. In addition, the dotted arrow shows the processing direction of one row in each perspective, and each block (CTU) can update and use the entropy context of the previous column of the same row.

As shown in Fig. 24A, the perspectives P2 and P4 are not adjacent to each other and thus are not referenced to each other. Therefore, perspectives P2 and P4 can be independently encoded / decoded.

As shown in (b) of FIG. 24, for example, the perspective P4 may be encoded / decoded in the order from the upper left to the lower right according to a normal raster scan order. At this time, the normal raster scan order can also be applied to the perspective P2. Alternatively, as illustrated in (c) of FIG. 24, the encoding / decoding may be performed in the order of the upper left to the upper left for the perspective P2.

Subsequently, if part / decoding of the perspectives P2 and / or P4 is performed in part or in whole, the part / decoding of neighboring perspectives in which the dependency is resolved can be started. At this time, the WPP processing direction of the neighbor perspective may be determined based on the WPP processing direction applied to the reference perspective. For example, in FIG. 23, when the encoding / decoding of the blocks of the first row and the second column of the perspective P4 is completed, the encoding / decoding of the blocks of the first row and the first column of the perspective P1 may be started. At this time, the negative / decoding of the perspective P1 may be performed in the order of the lower left to the upper right.

In addition, when the encoding / decoding of the blocks of the last row and the last second column of the perspective P2 is completed, the encoding / decoding of the blocks of the last row and the first column of the perspective P6 may be started. At this time, the negative / decoding of the perspective P6 may proceed in the order of the lower left to the upper right.

Subsequently, encoding / decoding may be started for neighboring perspectives in which the dependency on the perspective (s) in which encoding / decoding has been advanced so far has been resolved. At this time, the progress direction of the WPP may be determined according to a perspective in which dependency among the one or more reference perspectives is resolved more quickly. If the dependencies of two or more reference perspectives are resolved at the same time, the encoding / decoding of the current perspective can proceed simultaneously on several sides. In this case, the current perspective can be divided and processed.

Referring to FIG. 23, when the encoding / decoding of the blocks of the last second column and the last row of the perspective P2 is completed, the encoding / decoding of the blocks of the last row and the first column of the perspective P3 may be started. In addition, when the encoding / decoding of the blocks of the first row and the first column of the perspective P4 is completed, the encoding / decoding of the blocks of the first row and the last column of the perspective P3 may be started. That is, the encoding / decoding of the perspective P3 may proceed simultaneously on several sides with reference to the plurality of perspectives. In this case, the encoding / decoding order of the perspective P3 may be determined in consideration of at least one of the neighboring perspectives P1, P2, P4, and P6. At this time, the order of decoding / decoding of the perspective P3 may be determined in the direction in which the decoding / decoding can be completed as soon as possible.

The method described with reference to FIGS. 23 and 24 may be repeated until the encoding / decoding of all time points is completed.

In the example shown in FIG. 24, the negative / decoding of the perspective P4 proceeds from the upper left to the lower right on the developed view, and the negative / decoding of the perspective P2 proceeds from the lower right to the upper left. This is to better distribute the amount of data being processed in parallel for more efficient parallelism.

FIG. 25 is a diagram for describing a decoding / decoding procedure of two perspectives facing each other in a cube omnidirectional image. FIG.

The order of encoding / decoding of two perspectives at positions facing each other in the three-dimensional space may be set upside down in the up, down, left, and right directions as shown in FIG. Alternatively, as shown in (b) of FIG. 25, they may be set in the same direction from side to side.

FIG. 26 is a diagram for explaining a difference according to FIGS. 25A and 25B.

As shown in (a) of FIG. 25, when the sub / decoding of the perspectives P4 and P2 starts at the positions of up, down, left, and right symmetry, the CTU heat treatment sequence of the perspectives P2 and P1 is shown in FIG. Are opposite directions on the side where the perspectives P2 and P1 meet. The same applies to the side where the perspectives P6 and P4 meet. If two perspectives are reversed in the processing direction of each perspective on an adjacent side, information of neighboring perspectives cannot be utilized efficiently. That is, since the processing order with the neighbor perspective is reversed, a delay occurs until the encoding / decoding of the reference row of the neighbor perspective is completed.

As shown in (b) of FIG. 25, when negative / decoding of the perspectives P4 and P2 proceeds in the same direction from side to side, as shown in (c) of FIG. 26, the perspective P2 is formed at the side where the perspective P2 and P1 meet. The CTU heat treatment direction of and P1 becomes the same. The same applies to the side where the perspectives P6 and P4 meet. Therefore, there is no delay since the start of the encoding / decoding of the last row of the perspectives P1 and P6 coincides with the completion point of the two CTUs of the last row of the perspectives P4 and P2 to be used as reference regions. FIG. 27 is a diagram for describing encoding / decoding according to FIG. 25B.

As described above, if two CTUs are processed in each of the perspectives P2 and P4, encoding / decoding through neighboring perspectives through WPP is possible. Since the perspective P1 has continuity with the perspective P4, as shown in (b) of FIG. 27, when the sub- / decoding of the first CTU of the first row of the perspective P4 is completed, the dependency of the first CTU of the perspective P1 is resolved. Therefore, entropy context can be obtained from perspective P4 and used.

Similarly, in the perspective P6, the decoding / decoding may be started when the second CTU of the perspective P2 is completed / decoding. As shown in FIG. 27C, the WPP may proceed with respect to the perspective P6 in accordance with the direction in which the perspective P2 is processed. In FIGS. 27B and 27C, the solid arrows indicate the directions for taking and initializing the entropy context. The dashed arrows indicate the processing direction of the CTU in each row.

As shown in (b) and (c) of FIG. 27, entropy contexts can be derived from other perspectives. Reference of the entropy context may be performed via a context extractor, described below with reference to FIGS. 32 and 33.

FIG. 28 is a diagram for describing encoding / decoding of perspectives that may refer to two or more perspectives.

As shown in FIG. 27, in the case of perspectives P3 and P5, neighboring perspectives are both encoded / decoded. Therefore, by performing parallel processing by utilizing all of the neighbor perspectives, the encoding / decoding efficiency can be improved.

As shown in FIG. 28, perspectives P3 and P5 may utilize information of neighboring perspectives P1, P2, P4 and / or P6. The solid and dashed arrows used in FIG. 28 are used in the same meaning as the solid and dashed arrows used in FIG. 27. That is, the solid arrow indicates the direction of bringing entropy information (eg, entropy context, entropy context table, etc.). In addition, the dotted line arrow indicates the direction of row processing through WPP in each perspective. The number in the CTU block in each perspective represents the encoding / decoding process order.

As shown in FIG. 28, in the case of the perspective P3 or the perspective P5, the dependency may be resolved simultaneously in two or more directions according to the negative / decoding of neighboring perspectives. According to the sub / decoding sequence, perspectives P2 and P4 begin to decode / decode before perspectives P1 and P6. Therefore, the dependencies necessary for the encoding / decoding of the perspectives P3 and P5 can first be solved by the perspectives P2 and P4. Thus, perspective P3 and / or P5 may select one of perspectives P2 and P4 to determine the WPP processing direction. Alternatively, in order to utilize both perspectives, perspectives P3 and / or P5 may be divided into two regions, and the WPP processing direction according to the perspectives P2 and P4 may be determined for each of the divided regions.

For example, the encoding / decoding of the perspective P3 may be performed after the dependency with the perspective P2 is resolved. In this case, the CTU 8 of the perspective P3 may be encoded / decoded based on the two CTUs 1 and 3 of the perspective P2. Alternatively, when two CTUs of the perspective P2 and P4 are respectively encoded / decoded, the encoding / decoding of the perspective P3 may be started in both directions. In this case, the encoding / decoding of two CTUs of the perspective P3 in which the dependency is resolved by the encoding / decoding of the perspectives P2 and P4 can be started in parallel. By parallel decoding / decoding means that the perspective P3 is encoded / decoded independently of each other in both directions. Thus, parallel encoding / decoding may or may not be performed simultaneously.

FIG. 29 is a diagram for describing a method of dividing one perspective into two regions and encoding / decoding each region. FIG.

FIG. 29A shows one perspective divided into areas A and B, and FIG. 29B shows WPP processing after dividing the perspective into areas of the same size. 29C shows that the WPP process is performed after dividing the perspective into regions having different sizes. The meanings of the solid and dashed arrows in FIG. 29 are the same as those described with reference to FIGS. 27 and 28.

The region A and the region B may be represented by a first region and a second region. Each of the first region and the second region may be encoded / decoded based on the degree of progress of encoding / decoding of adjacent perspectives. For example, when the first region and the first perspective are adjacent to each other, the first region may be encoded / decoded from a block of the first region where the dependency is resolved by the encoding / decoding of the first perspective. The negative / decoding direction of the blocks included in the first region may be determined based on the negative / decoding direction with respect to the first perspective. For example, the negative / decoding directions for the first region and the first perspective may be directions perpendicular to each other.

The method of dividing one perspective into two regions is not limited to the embodiment described with reference to FIG. 29. For example, it may be divided into shapes or sizes not shown in FIG. 29. A method of dividing one perspective into two regions may be signaled through a bitstream or predefined in an encoder and a decoder.

Neighboring CTUs with continuity with perspectives P3 and / or P5 may be included in one row rather than columns in perspectives P2 and / or P6. As described above, in the case of WPP processing, since the decoding / decoding of the lower row can be started only after the encoding / decoding of the CTU 2 column of the upper row is completed, a delay per row occurs by 2 * t (CTU). Since the negative / decoding delay of the reference perspective occurs by 2 * t (CTU) per row, as shown in FIG. 28 (a), even for perspective P3, 2 * t (CTU) is not 1 * t (CTU) for each column. CTU) delay.

30 is a diagram for explaining a coding / decoding process procedure when referring to two or more perspectives.

If two or more perspectives can be referenced, encoding / decoding of the current perspective can be performed based on the degree of progression of the decoding / decoding of the two or more perspectives referenced. For example, even if the dependency is resolved by the encoding / decoding of the first perspective, after waiting until the dependency is resolved by the encoding / decoding of the second perspective, the encoding / decoding of the current perspective can be started. Compared to the case of referring to one perspective, referring to more than one perspective may cause a delay. However, since two or more blocks can be referenced, an improvement in coding efficiency can be expected.

As described above, a delay per column occurs by 2 * t (CTU) in the encoding / decoding process of the first row of the perspective P3. Therefore, as shown in Fig. 30A, even if the decoding / decoding of the lower left CTU 4 of the perspective P3 is completed, the CTU 6 of the next row cannot be directly decoded / decoded. There is a delay of 2 * t (CTU) per column, so if you start encoding / decoding each time the dependence of each CTU in perspective P3 is resolved, the last CTU of the first row in the processing order of perspective P3 is the 11th Can be decrypted.

In this case, as shown in (b) of FIG. 30, in consideration of the fact that the last CTU of the first row of the perspective P3 is encoded / decoded to the 11th, the start / decoding of the first CTU of the first row of the perspective P3 is started. The seventh time can be determined. As shown in FIG. 30 (b), even when the start / decoding start time of the first CTU of the perspective P3 is delayed, the start / decoding start time of the last CTU of the first row is not changed, and thus the parallel processing time of the entire image is not affected. You may not.

In addition, when processing the perspective P3 in the order shown in (b) of FIG. 30, there is an advantage that the neighboring CTUs that can be referred to are increased. That is, in encoding / decoding of the first CTU of the perspective P3, not only the pixel information of the CTU (5, hatched portion) of the second row of the perspective P6, but also the entropy context can be used.

In addition, as shown in FIG. 30 (c), if the encoding / decoding of the perspective P3 is further delayed by 1 * t (CTU), the CTU 11 (hatched area) of the neighbor perspective P1 can also be referred to. This is because the CTU processing times of the perspectives P1 and P2 differ only by 2 * t (CTU). Therefore, in the case of the perspective P3, as shown in the example shown in FIG. 30 (c), the start / decoding start time of the perspective P3 is determined based on the processing time of the right CTU (belonging to the neighbor perspective) of the last CTU of the first row. Can be.

(C) of FIG. 30 may have a slight delay more than that of FIG. 30 (b). However, since more regions can be referred to, the encoding / decoding efficiency can be improved.

In the present disclosure, information that can be referred to from adjacent blocks, CTUs, and / or perspectives is described as pixel information, entropy context, entropy context table, and the like. However, the present invention is not limited thereto, and reference may be made to all of the encoding / decoding information of the adjacent region. For example, the information that can be referred to is not only pixel information existing in an image, but also prediction modes used for spatial prediction, motion vectors used for temporal prediction, merge index, quantization parameter information, in-loop filter information, filter coefficient information, and the like. Contains all the information generated or used in the decoding process.

As described above, for the case of an octahedron or an icosahedron image other than a cube, the steps described in consideration of the cube may be repeatedly performed.

FIG. 31 is an exemplary diagram for describing an encoding / decoding procedure of a cube omnidirectional image. FIG.

When performing the encoding / decoding of the cube omnidirectional image in the order described with reference to FIGS. 23 to 30, encoding / decoding may be performed in the order shown in FIG. 31.

FIG. 31A is a diagram illustrating an encoding / decoding sequence on a developed view. The solid and dashed arrows used in FIG. 31A have the same meaning as the solid and dashed arrows used in FIG. 27 and the like.

(B) and (c) of FIG. 31 exemplarily show a procedure of encoding / decoding on a cube. Small blocks in FIGS. 31B and 31C denote coding unit blocks (eg, CTUs), and numbers indicate an order in which the corresponding blocks are processed. The shaded block refers to a block in which encoding / decoding has been completed, and the block not shaded refers to a block in which encoding / decoding has not yet started.

As described above, when performing encoding / decoding of the omnidirectional image according to the present disclosure, the WPP processing order may vary for each perspective. Alternatively, encoding / decoding may be started in several directions at the same time for one perspective. When the processing directions of the plurality of perspectives included in one omnidirectional image are different, separate encoding / decoding apparatuses are not considered to encode / decode all perspectives with the device of the same configuration, and to consider the direction of decoding / decoding of each perspective. There may be a problem that you must provide a separate.

32 is a diagram illustrating a configuration of an encoding device according to the present disclosure.

33 is a diagram illustrating a configuration of a decoding apparatus according to the present disclosure.

The encoding apparatus of FIG. 32 may include a transform and divider 3201, an image information converter 3202, a context information extractor 3203, and a plurality of encoders. The encoding apparatus of FIG. 32 may generate a bitstream by encoding an input image. The input image may be an omnidirectional image, and each encoder may encode one perspective of a plurality of perspectives constituting the omnidirectional image. Each of the encoders may include an encoding processor, a reconstruction picture buffer, and / or an entropy encoder, and may correspond to the encoding apparatus of FIG. 3.

The decoding apparatus of FIG. 33 may include a video information converter 3301, a context information extractor 3302, a transform and integrator 3303, and a plurality of decoders. The decoding apparatus of FIG. 33 may output an image by decoding the bitstream. The output image may be an omnidirectional image, and each decoder may decode one perspective of a plurality of perspectives constituting the omnidirectional image. Each of the decoders includes a decoding processor, an entropy decoder, and / or a reconstruction picture buffer, and may correspond to the decoding apparatus of FIG. 4.

If a separate sub / decoder is needed according to the sub / decoding direction of each perspective, each of the plurality of encoders and the plurality of decoders of FIGS. 32 and 33 has a different configuration, thus the complexity of the device configuration Will increase. Therefore, even if the processing direction of each perspective is different, an additional configuration is required to be processed by an encoder or decoder having the same configuration. To this end, the encoder according to the present disclosure may further include a transform and divider 3201 and / or an image information converter 3202. Also, the decoder according to the present disclosure may further include an image information converter 3301 and / or a transform and integrator 3302.

In order to process each perspective constituting the omnidirectional image in parallel, the transform and splitter 3201 of FIG. 32 may transmit image information of each perspective to one or more encoding processors. Before the image information is transmitted, the image information may be modified such that the processing direction of each perspective coincides with the predetermined processing direction. The predetermined processing direction may be, for example, a raster scan order, but is not limited thereto and may be predetermined or signaled as a processing direction commonly applied to all perspectives.

The transform and divider 3201 can operate as follows.

First, if the input image includes a plurality of perspectives in an integrated form, the input image is divided for each perspective (step 1). The omnidirectional image may be stored separately for each perspective or integrated and stored in a single exploded form. At this time, when stored in the form of a single development view, it is divided by perspective for parallel processing. Each perspective may be sent to a respective encoding processor.

Next, when the CTU processing order in the perspective is counterclockwise, a symmetric image is generated to be clockwise (step 2). 34 is an exemplary diagram for explaining a method of converting a processing order of a perspective. Each of (a), (b) and (c) of FIG. 34 shows an embodiment in which the processing order of the perspective P6 in which the CTU processing order is counterclockwise is converted to the clockwise direction.

In FIG. 34A, the symmetric perspective of stage 2 can be generated by symmetrically the perspective P6 of stage 1 from side to side. In (b) of FIG. 34, the symmetric perspective of stage 2 can be created by symmetry up and down perspective P6 of stage 1. FIG. In addition, in FIG. 34C, the symmetric perspective of stage 2 can be generated by diagonally symmetrical perspective P6 of stage 1. FIG. In each of (a), (b) and (c) of FIG. 34, it can be seen that the processing order of the CTU of the symmetric perspective of stage 2 is converted to the clockwise direction.

Finally, the image is rotated with respect to the center of the perspective so that the negative / decoding start position of the symmetric perspective of stage 2 is at the upper left (step 3). In the case of FIG. 34A, since the negative / decoding start position is located at the lower right end, the symmetric perspective of stage 2 can be rotated 180 degrees. In the case of Fig. 34B, since the negative / decoding start position is located at the upper left, there is no need to rotate the symmetric perspective of stage 2. In the case of (c) of FIG. 34, since the sub / decoding start position is located at the lower left end, the symmetric perspective of stage 2 can be rotated 90 degrees clockwise. The perspective of stage 3 can be generated by the rotation of the symmetric perspective of stage 2.

If the processing order of CTUs in the perspective of stage 1 is clockwise, the symmetry may not be performed, for example, only rotation may be performed. As described above, by symmetrical and / or rotating the perspective, the order of negative / decoding processing of the perspective can be matched in a predetermined order. The above example described with reference to FIG. 34 is an example of a method in which the encoding / decoding of each perspective can be started at the upper left. In addition, the symmetry and / or rotation can be expressed in a matrix. In addition, the symmetry and rotation may be expressed in the form of a matrix product may be performed in one operation.

Each embodiment described with reference to FIG. 34 is part of an embodiment for matching the CTU processing order in the perspective to a predetermined order. That is, the CTU processing order may be matched by applying a method other than the embodiments described with reference to FIG. 34. For example, unlike the embodiment described with reference to FIG. 34, the CTU processing order may be matched by first rotating the perspective and then symmetrically. That is, in FIG. 34A, the perspective of Stage 3 may be obtained by rotating the perspective of Stage 1 180 degrees and then symmetrically rotating the perspective from side to side.

In addition, a method of simultaneously applying rotation and symmetry may be considered. In addition to rotation and symmetry, the perspective may be converted using mapping information between the processing order of the perspective of the stage 1 and the processing order of the perspective of the stage 3. For example, the mapping information may include information for identifying a processing order of the perspective of stage 1 and transformation information about the perspective of stage 1. For example, the conversion information may be information for converting the perspective of the stage 1 into the perspective of the stage 3. For example, the transformation information may be information about a correspondence relationship between the CTU of stage 1 and the CTU of stage 3. The correspondence may be a position of the CTU of stage 1 and a position of the CTU of stage 3 corresponding thereto. The position of the CTU may be represented in an order according to a predetermined scan order. The predetermined scan order may be, for example, a raster scan order. Alternatively, the position of the CTU may be two-dimensional coordinates based on a predetermined corner of the perspective. The predetermined corner may be, for example, the upper left corner of the perspective.

In the example described with reference to FIG. 34, when the CTU processing order of the input perspective is different from the predetermined processing order, symmetry and / or rotation are performed to transform the perspective. However, it is also possible to modify the processing order of the sub / decoder without converting the input perspective. For example, when the input perspective is the same as the perspective of Fig. 34A, and the predetermined processing sequence is the raster scanning sequence, the sub / decoder performs the encoding / decoding in the counterclockwise direction starting from the lower left CTU of the input perspective. It can also be configured to. At this time, the above-described mapping information between the processing order of the perspective of the stage 1 and the processing order of the perspective of the stage 3 may be used.

As described above, when processing a block included in the perspective, it is possible to refer to the image information of the block included in the neighbor perspective. As described above, the image information may be a prediction mode used for spatial prediction, a motion vector used for temporal prediction, a merge index, quantization parameter information, in-loop filter information, filter coefficient information, etc., in addition to simply pixel information existing in the image. It can contain any information generated or used during the encoding / decoding process.

However, in the process of expressing an image on a three-dimensional space on a developed view, a difference occurs in the position and direction between perspectives. Therefore, in order to share image information that can be used for prediction between different perspectives, a process of converting image information according to the direction of development of each perspective is required. In addition, as described with reference to FIG. 34, it is necessary to convert image information by performing symmetrical and / or rotational transformation for matching the negative / decoding process order of the perspective from the upper left to the lower right.

The decoder may determine a method of converting image information to be referred to based on information about the developed view used by the encoder and / or information about the transformation applied to each perspective. The sub / decoder may promise information in advance about the exploded view, information about the location of the perspective on the exploded view, and / or information about the transformation applied to each perspective. Alternatively, all or part of the information may be signaled explicitly or implicitly through the bitstream. Alternatively, the decoder may derive some or all of the information based on other syntax or variables regarding encoding / decoding.

The above-described conversion of the image information may be performed by the image information converter 3202 of FIG. 32 and the image information converter 3301 of FIG. 33.

If the encoder / decoder can use one method promised to each other, the flag information is included in a video header such as a video parameter set (VPS) or a sequence parameter set (SPS). Signal whether or not to use.

If the encoder uses one of a plurality of development diagrams or if the processing order of the perspectives processed by the transform and distribution unit 3201 is variable, the information about the type of development scheme or the processing order of the perspectives used in encoding is explicitly expressed by the decoder. May be signaled. Alternatively, when the encoder / decoder shares a lookup table for the information, the index of the lookup table may be signaled.

The transform and integrator 3303 of FIG. 33 may reverse the process of the transform and divider 3201 of FIG. 32. More specifically, the following process can be performed.

First, when rotation is applied to the perspective image in the encoding apparatus, the image may be rotated in the opposite direction by the angle rotated in the encoding apparatus (step 1).

Whether rotation is applied and / or the rotation angle for the corresponding perspective image may be signaled through the bitstream. Whether the rotation is applied and / or the rotation angle can be directly negative / decoded. For example, a flag indicating whether rotation is applied and a syntax element regarding rotation angle may be encoded / decoded.

There may be a limited number of rotation angles depending on the shape of the perspective image. For example, when the perspective image is a quadrangle, the rotation angle may be one of 90 degrees, 180 degrees, and 270 degrees. Therefore, the syntax element regarding the rotation angle may be information indicating one of three cases.

Or using a 2-bit syntax element, for example, bit string “00” is “not rotated”, bit string “01” is rotated 90 degrees, bit string “10” is rotated 180 degrees, The bit string “11” may indicate “rotated 270 degrees”. The bit string and its corresponding meaning may be preset in the encoder / decoder and stored, for example, in the form of a table.

When the perspective image is n-square, there may be one case where rotation is not applied and n-1 cases where rotation is applied. The n-1 cases may refer to cases in which rotation angles are different. That is, when the perspective image is n-square, the information about the rotation transformation may be information indicating one of n cases. Thus, using log ₂ n bits of syntax elements, information about whether rotation is applied and the angle of rotation can be encoded / decoded.

Next, when symmetry is applied to the perspective image in the encoding apparatus, the symmetry image may be generated based on the center line used in the encoding apparatus (step 2).

Information about whether symmetry has been applied to the corresponding perspective image and / or centerline used for symmetry may be signaled through the bitstream. Information about whether the symmetry is applied and / or the center line may be directly encoded / decoded. For example, syntax elements for flags and centerlines indicating whether symmetry is applied may be encoded / decoded.

The number of centerlines available may be limited. For example, the decoder / decoder may promise in advance the number of available centerlines. Or the number of available centerlines may be signaled.

For example, when the center line is one of three types of vertical line, horizontal line, and diagonal line, the information about the center line may be information indicating one of three cases.

Or using a 2-bit syntax element, for example, bit string “00” is “not symmetrical”, bit string “01” is “symmetrical using a vertical line”, and bit string “10” is using a horizontal line. Symmetrical ”, bit string“ 11 ”may indicate“ symmetrical using diagonal ”. The bit string and its corresponding meaning may be preset in the encoder / decoder and stored, for example, in the form of a table.

Next, when generating one development view in which a plurality of perspectives are combined, perspective images that have passed through Step 1 and / or Step 2 may be integrated into one according to the development view.

Steps 1 and 2 may be performed by inversely applying an operation used by the transform and divider 3201 of the encoding apparatus. Step 3 may be performed when the user of the decoding apparatus requests one development view that combines a plurality of perspectives.

As described above, in order to share the pixel information that can be used for prediction between perspectives, the positional difference between the perspectives generated while expressing an image of a three-dimensional space on a development diagram and / or a process of encoding / decoding the perspectives in a predetermined order. In consideration of the transformation for matching, the pixel information can be corrected.

In addition, in order to smoothly refer to video information of adjacent perspectives used for prediction, it is necessary to modify video information of adjacent perspectives.

For example, in the case of a motion vector used for temporal prediction, a symmetrical and / or rotational transformation is performed, similarly to a correction taking into account a transformation for matching the positional difference between the perspectives on a developed view and / or the order of sub / decoding processes in a predetermined order. Can be modified.

However, modification of the spatial prediction mode may not be solved with this method. The reason for this is described later. In order to perform the intra prediction, for example, adjacent pixel information of the bottom left, left, top, and top right corners of the current block may be used. In addition, as the directional prediction mode, as shown in FIG. 35, 33 prediction modes may be used. Or, more directional prediction modes may be used to support finer directionality.

When the encoding / decoding of the image proceeds from the upper left to the lower right, pixel information exists only in a specific direction. Accordingly, the directional prediction mode that may be referred to may be a prediction mode of a lower left, a left end, an upper end, and / or an upper right end. In addition, unlike the motion vector, the mode number according to the direction of spatial prediction is stored. Thus, the mode number may not match the direction of the actual spatial prediction. Therefore, when the perspective is transformed into symmetry or rotation, the prediction information of the neighboring block cannot be used as it is.

36 is an exemplary diagram for describing spatial prediction of a current perspective with reference to a prediction mode of a neighbor perspective.

36 (a) shows the positions of block A of perspective P2 and block B of perspective P6 on the developed view of the cube omnidirectional image. 36B shows the positions of blocks A and B after correcting the positions of the perspective P2 and the perspective P6 in consideration of the positional difference on the developed view. FIG. 36C shows the positions of blocks A and B after correction for matching the negative / decoding process order of the perspective P6 from a predetermined direction, for example, from the upper left to the lower right.

The perspective P6 of FIG. 36C may be encoded in the order from the upper left to the lower right through the transform and divider 3201 of FIG. 32. Accordingly, block B may be predicted using the intra prediction mode illustrated in FIG. 35. However, the encoded pixel information of the block A of FIG. 36C is converted so that the block B can refer to it. However, since the image information other than the pixel information is before the conversion, the block B cannot use it as it is. Therefore, it is necessary to perform a conversion process corresponding to the conversion of the pixel information with respect to other image information. In this case, the conversion of the pixel information is as described with reference to FIG. 34, for example.

However, in the case of intra prediction, since the mode number is used for the directionality, after the conversion process, the mode number for the directionality may not be the same or may not exist.

If the directionality of the spatial prediction available for the current block (eg, block B of FIG. 36) and the reference block (eg, block A of FIG. 36) is the same, only the modification of the mode number changed by the transformation may be necessary. However, a mode corresponding to the prediction mode of the current block may not exist for the reference block.

37 is a diagram for explaining a difference between prediction modes of a current block and a reference block.

FIG. 37A illustrates the directional prediction modes available for the blocks A and B shown in FIG. 36C. As shown in (a) of FIG. 37, as the block A is transformed according to the conversion of the block B, it can be seen that there is no mode other than the mode number 26 for the block A that the block B can utilize for prediction. However, the prediction direction itself of block A may be useful for encoding / decoding of the prediction direction of block B. Therefore, if the mode number of the block A is interpreted to be used by the block B, the efficiency of prediction techniques such as Most Probable Mode (MPM) can be improved.

The interpretation of the directional prediction mode may be performed as follows. When the prediction direction of the reference block exists in the directional mode (eg, the directional mode described with reference to FIG. 35) according to a predetermined processing order (eg, sequential scanning order), the directional prediction mode of the reference block is the prediction direction of the reference block. Can be interpreted as a mode number assigned to. For example, in FIG. 37A, when the prediction direction of the reference block A is the upper right direction, the upper right direction is included in the directional mode shown in FIG. 35. Therefore, in this case, the directional prediction mode of the reference block A may be interpreted as '34'.

If the prediction direction of the reference block does not exist in the directional mode according to a predetermined processing order (for example, sequential scanning order), that is, when the prediction direction of the reference block is 180 degrees symmetric to the prediction direction of the reference block Can be interpreted as the mode number assigned to the direction. For example, in FIG. 37A, when the prediction direction of the reference block A is the right direction, the right direction is not included during the directional mode shown in FIG. 35. Therefore, in this case, the directional prediction mode of the reference block A may be interpreted as '10', which is a mode number assigned to the left direction that is 180 degrees symmetric to the right direction.

FIG. 37B shows an example in which the lower directionality (solid arrow) in FIG. 37A is interpreted as the upper directionality (dashed arrow). Mode 2 of the directional mode of FIG. 37B is included in the existing directional mode. Therefore, it may not be interpreted symmetrically by 180 degrees.

FIG. 37C shows an example in which the right directionality (solid line arrow) in FIG. 37A is interpreted as the left directionality (dashed arrow). Mode 34 of the directional mode of FIG. 37C is included in the existing directional mode. Therefore, it may not be interpreted symmetrically by 180 degrees.

38 is a diagram for explaining a method of analyzing a directional prediction mode of a reference block.

In operation S3810, image information of a reference block may be converted. Conversion of the image information of the reference block may be performed according to the current block. For example, the operation may be performed in consideration of the transformation of the current block or in consideration of the difference between the perspectives of the current block and the reference block.

In operation S3820, it may be determined whether the directional prediction mode of the uninterpreted reference block is available for the current block. For example, if the directional prediction mode of the uninterpreted reference block is included among the directional prediction modes available for the current block, it may be determined that the corresponding directional prediction mode of the reference block is available for the current block.

If the directional prediction mode of the reference block is not available for the current block (No in step S3820), in step S3830, a mode that is 180 degrees symmetric to the corresponding directional prediction mode of the reference block may be generated. Thereafter, in step S3840, the directional prediction mode of the reference block may be analyzed. For example, the directional prediction mode of the reference block may be interpreted as a mode of the current block having the same directionality as the directional prediction mode of the reference block generated in step S3830.

If the directional prediction mode of the reference block is available for the current block (Yes in step S3820), in step S3840, the directional prediction mode of the reference block is to be interpreted as the mode of the current block having the same directionality as that of the reference block. Can be.

In operation S3850, it may be determined whether all the directional prediction modes of the reference block have been interpreted. If there is an uninterpreted mode among the directional prediction modes of the reference block, the process moves to step S3820 and the process can be repeated.

Assuming the prediction mode shown in FIG. 35, the directional prediction mode of the reference block may be, for example, one of eight cases shown in FIG. 39, depending on which development of each perspective of the cube omnidirectional image is developed. Can be.

Of the eight cases shown in FIG. 39, except for the same case as the current block (eg, the case of FIG. 39A), it is necessary to interpret the directional mode of the reference block according to the current block. Analysis of the directional mode of the reference block may be performed as described with reference to FIG. 37. Alternatively, the directional mode of the reference block can be analyzed by referring to a lookup table (interpretation table) shared by the encoding / decoding device. When referring to the table, it is necessary to determine which of the eight cases of FIG. 39 corresponds to. This may be signaled via the bitstream. Or it may be derived based on the information on the developed view used, other encoding / decoding information.

Table 1 below is an example of the directional mode analysis table. Table 1 is a table of 33 orientations. However, the number of directionality is not limited to this.

Analysis mode (a) (b) (c) (d) (e) (f) (g) (h) 2 18 18 34 * * 34 2 3 19 17 33 19 17 3 33 4 20 16 32 20 16 4 32 5 21 15 31 21 15 5 31 6 22 14 30 22 14 6 30 7 23 13 29 23 13 7 29 8 24 12 28 24 12 8 28 9 25 11 27 25 11 9 27 10 26 10 26 26 10 10 26 11 27 9 25 27 9 11 25 12 28 8 24 28 8 12 24 13 29 7 23 29 7 13 23 14 30 6 22 30 6 14 22 15 31 5 21 31 5 15 21 16 32 4 20 32 4 16 20 17 33 3 19 33 3 17 19 18 2, 34 2, 34 18 2, 34 2, 34 18 18 19 3 33 17 3 33 19 17 20 4 32 16 4 32 20 16 21 5 31 15 5 31 21 15 22 6 30 14 6 30 22 14 23 7 29 13 7 29 23 13 24 8 28 12 8 28 24 12 25 9 27 11 9 27 25 11 26 10 26 10 10 26 26 10 27 11 25 9 11 25 27 9 28 12 24 8 12 24 28 8 29 13 23 7 13 23 29 7 30 14 22 6 14 22 30 6 31 15 21 5 15 21 31 5 32 16 20 4 16 20 32 4 33 17 19 3 17 19 33 3 34 * * 2 * 18 2 34

In addition, it is necessary to correct the image information in consideration of the difference in perspective during spatial prediction. 40 is a diagram for explaining a position of a reference sample used for spatial prediction of a cube omnidirectional image. The dotted box in FIG. 40 indicates the position of the reference sample used for spatial prediction of each CTU (blocks A, B and C). As shown in (a) of FIG. 40, when spatial prediction of a cube omnidirectional image is performed, there may not be a reference sample on the developed view. For example, in the case of the upper left CTU (block A) shown in FIG. 40A, the upper left reference sample X does not exist. In addition, in the case of the right CTU (block B) and the bottom CTU (block C), due to the difference between the perspectives, the correct reference sample may not be selected. As described above, when there is no reference sample and / or correction of pixel information due to a difference between perspectives, when performing spatial prediction, the sample values of the neighboring perspectives are used to determine the sample values that do not exist. The reference sample can be determined by deriving or transforming the neighbor perspective to fit the current perspective.

For example, the reference sample X of the block A of FIG. 40A can be generated by interpolating the image information of the perspectives P1 and P2. In addition, in the case of perspective P4 and / or P6 having different viewpoints from perspective P3, correct reference samples may be generated through viewpoint correction.

FIG. 41 is a view for explaining a method of parallel processing a cube omnidirectional image utilizing continuity on a developed view.

Two cube developments used in FIG. 41 are the same as in FIGS. 11A and 11E. In the example shown in FIG. 41, each perspective includes 5x5 CTUs, and the number within each CTU represents the WPP processing order. The processing of the encoding / decoding can be started from the hatched blocks a1 and a2 in Fig. 41A. In simple developments other than three dimensions, perspectives A and B may not refer to each other. Even when encoding / decoding is started simultaneously at the positions of blocks a1 and a2 in Fig. 41 (a), when the processing of the row to which the block a2 belongs reaches the column to which the block a1 belongs, the delay is caused by the dependency because the dependency is resolved. I never do that.

Similarly, even in the case of FIG. 41 (b), even when encoding / decoding is started at the positions of blocks b1 and b2 at the same time, the dependency is resolved when the processing of the row to which the block b2 belongs reaches the column to which the block b1 belongs, No delay is caused.

In the example shown in FIG. 41, the encoder will start encoding / decoding at two or more positions in the development view. Thus, for example, the encoder may use two entropy information (eg, an entropy context table) for blocks a1 and a2. The decoder needs to know the method used in the encoder to perform the decoding. Therefore, entropy information used in the encoder, information on an encoding start time point and / or an exploded view, and the like may be signaled through a bitstream. Or by referring to a lookup table. Or based on other coding parameters and / or internal variables.

Exemplary methods of the present disclosure are represented as a series of operations for clarity of description, but are not intended to limit the order in which the steps are performed, and each step may be performed simultaneously or in a different order as necessary. In order to implement the method according to the present disclosure, the illustrated step may further include other steps, may include other steps except some, or may include additional other steps except some.

The various embodiments of the present disclosure are not an exhaustive list of all possible combinations and are intended to describe representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For hardware implementations, one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), General Purpose It may be implemented by a general processor, a controller, a microcontroller, a microprocessor, and the like.

It is intended that the scope of the disclosure include software or machine-executable instructions (eg, an operating system, an application, firmware, a program, etc.) to cause an operation in accordance with various embodiments of the method to be executed on an apparatus or a computer, and such software or Instructions, and the like, including non-transitory computer-readable media that are stored and executable on a device or computer.

Claims (21)

In the encoding method of an image including a plurality of perspective images,
Identifying a processing order of an encoding target perspective image among the plurality of perspective images;
Determining whether the identified processing sequence is the same as a predetermined processing sequence; And
If the identified processing sequence is not the same as the predetermined processing sequence,
Converting the target image to be encoded,
The processing sequence is identified by the processing start position and the processing progress direction,
And said plurality of perspective images are N images obtained by projecting a 360 degree omnidirectional image onto an N-sided surface. The method of claim 1,
A processing start position of the predetermined processing sequence is an upper left position of the perspective image, and a processing progress direction of the predetermined processing sequence is clockwise; The method of claim 2,
And the transformation comprises at least one of a symmetric transformation and a rotation transformation for the encoding target perspective image. The method of claim 3,
And the symmetric transformation is performed when the processing progress direction of the identified processing sequence is different from the processing progress direction of the predetermined processing sequence. The method of claim 3,
And the symmetric transformation is one of symmetric transformation, vertical symmetry transformation, and diagonal symmetry transformation. The method of claim 3,
And the rotation conversion is performed when the processing start position of the identified processing sequence is different from the processing start position of the predetermined processing sequence. The method of claim 6,
And the rotation transformation is a transformation for rotating the encoding target perspective image such that the processing start position of the encoding target perspective image is the same as the processing start position of the predetermined processing sequence. The method of claim 1,
The converting of the target image to be encoded may include:
First converting the encoding target perspective image;
Determining whether a processing order of the first transformed encoding target perspective image is the same as the predetermined processing order; And
When the processing order of the first transformed encoding target perspective image is not the same as the predetermined processing order,
And a second transform of the first transformed encoded target perspective image. The method of claim 8,
And wherein the first transform is one of a symmetric transform and a rotation transform, and the second transform is the other. In the image encoding apparatus including a plurality of perspective images,
The encoding device includes a converter,
The converter,
Identifying a processing order of the encoding target perspective image among the plurality of perspective pictures, determining whether the identified processing order is the same as a predetermined processing order, and wherein the identified processing order is not the same as the predetermined processing order. In this case, the encoding target perspective image is configured to convert,
The processing sequence is identified by the processing start position and the processing progress direction,
And the plurality of perspective images are N images obtained by projecting a 360 degree omnidirectional image onto an N-sided surface. In the decoding method of an image including a plurality of perspective images,
Identifying a processing order of a decoding target perspective image among the plurality of perspective images;
Determining whether the identified processing sequence is the same as a predetermined processing sequence; And
If the identified processing sequence is not the same as the predetermined processing sequence,
Converting the perspective image to be decoded,
The processing sequence is identified by the processing start position and the processing progress direction,
And the plurality of perspective images are N images obtained by projecting a 360 degree omnidirectional image onto an N-sided surface. The method of claim 11,
And a processing start position of the predetermined processing sequence is an upper left position of the perspective image, and a processing progress direction of the predetermined processing sequence is clockwise. The method of claim 12,
The transformation method includes at least one of a symmetric transformation and a rotation transformation for the decoding target perspective image. The method of claim 13,
And the symmetric transformation is performed when the processing progress direction of the identified processing order is different from the processing progress direction of the predetermined processing order. The method of claim 13,
And the symmetric transformation is one of symmetric transformation, vertical symmetry transformation, and diagonal symmetry transformation. The method of claim 13,
And the rotation conversion is performed when the processing start position of the identified processing sequence is different from the processing start position of the predetermined processing sequence. The method of claim 16,
And the rotation transform is a transformation for rotating the decoding target perspective image so that the processing start position of the decoding target perspective image is the same as the processing start position of the predetermined processing sequence. The method of claim 11,
Converting the decoding target perspective image.
First converting the decoding target perspective image;
Determining whether a processing order of the first transformed decoding target perspective image is the same as the predetermined processing order; And
When the processing order of the first transformed decoding target perspective image is not the same as the predetermined processing order,
And secondly converting the first transformed decoding target perspective image. The method of claim 18,
The first transform is one of a symmetric transform and a rotation transform, and the second transform is the other. An apparatus for decoding an image including a plurality of perspective images,
The decoding device includes a converter,
The converter,
Identify a processing order of a decoding target perspective image among the plurality of perspective images, determine whether the identified processing sequence is the same as a predetermined processing sequence, and the identified processing sequence is not the same as the predetermined processing sequence In this case, the decoding target perspective image is configured to convert,
The processing sequence is identified by the processing start position and the processing progress direction,
And the plurality of perspective images are N images obtained by projecting a 360 degree omnidirectional image onto an N-sided surface. A computer-readable recording medium storing a bitstream generated by an encoding method of an image including a plurality of perspective images,
The encoding method of the video,
Identifying a processing order of an encoding target perspective image among the plurality of perspective images;
Determining whether the identified processing sequence is the same as a predetermined processing sequence; And
If the identified processing sequence is not the same as the predetermined processing sequence,
Converting the target image to be encoded,
The processing sequence is identified by the processing start position and the processing progress direction,
The plurality of perspective images are N images obtained by projecting a 360-degree omnidirectional image onto an N-sided surface.
A computer readable recording medium storing bitstreams.

Images