KR102537024B1 - A method for encoding/decoding a virtual reality video - Google Patents

Abstract

An image processing method according to an embodiment of the present invention includes processing encoding or decoding, including frame packing processing of the virtual reality image.

Description

Method and apparatus for encoding/decoding virtual reality video providing frame packing

The present invention relates to a video encoding/decoding method and apparatus. More specifically, the present invention relates to a method and apparatus for encoding/decoding a virtual reality image providing frame packing.

As digital image processing and computer graphics technology have recently developed, research on virtual reality (VR) technology that reproduces the real world and allows a realistic experience is being actively conducted.

In particular, recent VR systems such as HMD (Head Mounted Display) can not only provide a 3D stereoscopic image to both eyes of the user, but also track the viewpoint in all directions, providing realistic virtual reality that can be viewed 360 degrees. It is receiving a lot of attention in that it can provide reality (VR) video content.

However, since 360 VR content is composed of simultaneous omnidirectional multi-view image information in which temporal and binocular images are spatially and compositely synchronized, two large images synchronized with respect to the binocular space of all viewpoints must be used in the production and transmission of images. It is encoded, compressed, and transmitted. This increases the complexity and bandwidth burden, and in particular, in the decoding apparatus, decoding is performed even for an area that is not actually viewed outside the user's point of view, so that unnecessary processes are wasted.

Accordingly, there is a need for an encoding method that reduces the transmission data amount and complexity of an image and is efficient in terms of bandwidth and battery consumption of a decoding device.

The present invention is to solve the above problems, and to provide a method and apparatus for efficiently encoding/decoding virtual reality images such as 360-degree cameras or VR images using frame packing of virtual reality images. There is a purpose.

As a technical means for achieving the above technical problem, an image encoding method according to an embodiment of the present invention includes obtaining a virtual reality image; performing frame packing processing on the virtual reality image; and encoding the virtual reality image.

In addition, the method may be implemented in a computer-readable recording medium on which a program to be executed by a computer is recorded.

According to an embodiment of the present invention, by providing a frame packing process optimized for encoding and transmission of a virtual reality image, it is possible to efficiently reduce the transmission data amount, bandwidth, and complexity of an image.

1 is a block diagram schematically showing an entire system according to an embodiment of the present invention.
2 to 4 show video encoding and decoding based on spatial structure information according to an embodiment of the present invention.
5 to 9 illustrate frame packing according to an embodiment of the present invention.
10 to 11 show encoding and decoding apparatuses according to an embodiment of the present invention.

Hereinafter, embodiments of the present application will be described in detail so that those skilled in the art can easily practice them with reference to the accompanying drawings. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. And in order to clearly describe the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is said to be "connected" to another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element in between. do.

Throughout the present specification, when a member is said to be located on another member, this includes not only a case where a member is in contact with another member, but also a case where another member exists between the two members.

Throughout the present specification, when a part "includes" a certain component, it means that it may further include other components without excluding other components unless otherwise stated. As used throughout this specification, the terms "about," "substantially," and the like are used at or approximating that value when manufacturing and material tolerances inherent in the stated meaning are given, and do not convey the understanding of this application. Accurate or absolute figures are used to help prevent exploitation by unscrupulous infringers of the disclosed disclosure. The term "step of (doing)" or "step of" as used throughout the present specification does not mean "step for".

Throughout the present specification, the term of a combination thereof included in the expression of the Markush form means one or more mixtures or combinations selected from the group consisting of the components described in the expression of the Markush form, and includes the components It means including one or more selected from the group consisting of.

In an embodiment of the present invention, as an example of a method of encoding a virtual reality image, it is jointly standardized by Moving Picture Experts Group (MPEG) and Video Coding Experts Group (VCEG), which have the highest coding efficiency among video coding standards developed to date. Coding may be performed using High Efficiency Video Coding (HEVC) or a coding technology currently being standardized, but is not limited thereto.

Typically, an encoding device includes an encoding process and a decoding process, and a decoding device includes a decoding process. A decoding process of the decoding device is the same as that of the encoding device. Therefore, hereinafter, the encoding device will be mainly described.

1 shows the overall system structure according to an embodiment of the present invention.

Referring to FIG. 1 , the entire system according to an embodiment of the present invention includes a pre-processing device 10, an encoding device 100, a decoding device 200, and a post-processing device 20.

A system according to an embodiment of the present invention may process virtual reality image information according to an embodiment of the present invention. The virtual reality image is an image that provides an experience as if the user is actually there, and may be an image that is synchronized with the user's view and can express omnidirectional images, and may also be called a 360 video or a virtual reality video.

A pre-processing device 10 that obtains a synchronized video frame by pre-processing a plurality of images for each viewpoint through an operation such as merging or stitching, and an encoding device that encodes the synchronized video frame and outputs a bitstream ( 100), a decoding device 200 that receives the bitstream and decodes the synchronized video frame, and a post-processing device that outputs a synchronized image for each view to each display through post-processing of the video frame ( 20) may be configured.

Here, the input image may include individual images for each multi-viewpoint, and may include, for example, sub-image information of various viewpoints captured by one or more cameras in a time- and spatial-synchronized state. Accordingly, the preprocessing device 10 may acquire synchronized virtual reality image information by spatially merging or stitching the obtained multi-view sub image information according to time.

The encoding device 100 scans and predictively encodes the synchronized virtual reality image information to generate a bitstream, and the generated bitstream may be transmitted to the decoding device 200 . In particular, the encoding device 100 according to an embodiment of the present invention can extract spatial structure information from the synchronized image information and signal it to the decoding device 200.

Here, the spatial layout information may include basic information about attributes and arrangement of each subimage as one or more subimages are merged from the preprocessor 10 to form one video frame. . In addition, additional information about each sub-image and the relationship between sub-images may be further included, which will be described later.

Accordingly, spatial structure information according to an embodiment of the present invention may be delivered to the decoding apparatus 200. Also, the decoding apparatus 200 may determine a decoding target and a decoding order of a virtual reality image bitstream by referring to spatial structure information and user viewpoint information, which may lead to efficient decoding.

And, the decoded video frame is again separated into sub-images for each display through the post-processing device 20 and provided to a plurality of synchronized display systems such as HMD, so that the user can experience realistic virtual reality like virtual reality. video can be provided.

2 is a block diagram showing the configuration of a virtual reality video encoding apparatus according to an embodiment of the present invention.

Referring to FIG. 2 , an encoding device 100 according to an embodiment of the present invention includes a virtual reality image acquisition unit 110, a spatial structure information generator 120, a spatial structure information signaling unit 130, an image encoder and It includes a transmission processing unit 150.

The virtual reality image acquisition unit 110 obtains a virtual reality image using a virtual reality image acquisition means such as a 360-degree camera. The virtual reality image may include a plurality of sub-images synchronized in time and space, and may be received from the pre-processing device 10 or from a separate external input device.

The spatial structure information generation unit 120 divides the virtual reality image into time-unit video frames and extracts spatial structure information of the video frames. Spatial structure information may be determined according to the properties and arrangement state of each sub-image, or may be determined according to information acquired from the pre-processing device 10 .

And, the spatial structure information signaling unit 130 performs information processing for signaling the spatial structure information to the decoding apparatus 200 . For example, the spatial structure information signaling unit 130 may perform one or more processes for including in image data encoded by the image encoder, constructing a separate data format, or including in metadata of an encoded image. can

And, the image encoder encodes the virtual reality image according to the lapse of time. In addition, the image encoder may use the spatial structure information generated by the spatial structure information generation unit 120 as reference information to determine an image scanning order and a reference image.

Accordingly, the image encoder may perform encoding using High Efficiency Video Coding (HEVC) as described above, but may be improved in a more efficient manner for virtual reality images according to spatial structure information.

In addition, the transmission processing unit 150 performs one or more conversion and transmission processes for combining the encoded image data and the spatial structure information inserted from the spatial structure information signaling unit 130 and transmitting the combined image data to the decoding apparatus 200. can

3 and 4 are diagrams for explaining a signaling method of spatial structure information according to various embodiments of the present disclosure.

As described above, the sub-images of the input image may be arranged in various ways. Accordingly, spatial structure information may separately include a table index for signaling arrangement information. For example, as shown in FIG. 11, a virtual reality image is Equirectangular (ERP), Cubemap (CMP), Equal-area (EAP), Octahedron (OHP), Viewport generation using rectilinear projection, Icosahedron (ISP) according to the conversion method. ), Crasters Parabolic Projection for CPP-PSNR calculation, Truncated Square Pyramid (TSP), Segmented Sphere Projection (SSP), Adjusted Cubemap Projection (ACP), and Rotated Sphere Projection (RSP) layouts may be exemplified, and spatial structure information A table index shown in FIG. 12 corresponding to each layout may be inserted into .

More specifically, a 3D image of a coordinate system corresponding to 360 degrees may be projected as a 2D image according to each piece of spatial structure information.

ERP transforms a 360-degree image onto one face by projecting and converting the u/v coordinate system position corresponding to the sampling position of the 2D image and the longitude on the sphere corresponding to the u/v coordinate system position It may include latitude coordinate conversion processing. Accordingly, the spatial structure information may include an ERP index and single face information (for example, face index is set to 0).

CMP projects a 360-degree image onto six regular hexagonal faces, and each face index corresponding to PX, PY, PZ, NX, NY, NZ (P indicates positive, N indicates negative) Sub-images projected in f) may be arranged. For example, in the case of a CMP image, an image obtained by converting an ERP image into a 3x2 cubemap image may be included.

Accordingly, the spatial structure information may include the CMP index and each face index information corresponding to the sub-image. The post-processing device 20 may process the 2D positional information on the sub-image according to the plane index to calculate positional information corresponding to the 3D coordinate system, and inversely transform the result into a 3D 360-degree image.

ACP, like CMP, in projecting a 360-degree image onto six regular hexagonal faces, applies a function adjusted to 3-dimensional curvature in response to projection transformation into 2D and inverse transformation into 3D, respectively. , the processing function is different, but the spatial structure information used may include an ACP index and face index information for each sub-image. Therefore, the post-processing device 20 inversely transforms the 2D location information on the sub-image according to the face index according to the adjusted function to calculate the location information corresponding to the 3D coordinate system, and converts the 3D location information into a 360 degree image accordingly. can be printed out.

EAP is a transformation projected on one face, like ERP, and may include transformation of longitude and latitude coordinates on a sphere immediately corresponding to a sampling location of a two-dimensional image. Spatial structure information may include an EAP index and single face information.

OHP projects a 360 degree image onto 8 regular octagonal faces using 6 vertices. Sub-images projected using V0, V1, V2, V3, V3, V4, and V5) may be arranged in the transformed image.

Accordingly, the spatial structure information may include an OHP index, face index information corresponding to the sub-image, and one or more vertex index information matched to the face index information. In addition, sub-image arrangement of the converted image may be divided into a compact case and a non-compact case. Accordingly, the spatial construction information may further include compactness identification information. For example, face index and vertex index matching information and an inverse transformation process may be determined differently for a non-compact case and a compact case. For example, face index 4 may be matched with vertex indices V0, V5, and V1 when it is not compact, and other matching may be processed with V1, V0, and V5 when it is compact.

The post-processing device 20 inversely transforms the 2D location information on the sub-image according to the face index and vertex index to calculate vector information corresponding to the 3D coordinate system, and outputs the inverse transformation as a 3D 360-degree image. there is.

ISP projects a 360-degree image using 20 faces and 12 vertices, and sub-images according to each transformation can be arranged on the transformed image. Similar to OHP, the spatial structure information may include at least one of an ISP index, a face index, a vertex index, and compact identification information.

SSP divides the sphere of the 360-degree image into three segments, the North Pole, the Equator, and the South Pole, and processes them. , and the equator can use the same projection as ERP. Accordingly, the spatial structure information may include an SSP index and a face index corresponding to each of the equatorial, arctic and antarctic segments.

RSP may include a method of dividing a sphere of a 360-degree image into two equally sized sections and spreading the divided images on a 2D converted image and arranging them in two rows. In addition, the RSP may implement the above arrangement by using six surfaces with a 3X2 aspect ratio similar to that of the CMP. Accordingly, the transformed image may include the first segment image of the upper segment and the second segment image of the lower segment. Spatial structure information may include at least one of an RSP index, a segmentation image index, and a face index.

TSP may include a method of transforming and projecting a frame obtained by projecting a 360-degree image onto six cube planes corresponding to the planes of a truncated quadrangular pyramid. Accordingly, the sizes and shapes of the sub images corresponding to each side may be different. Spatial structure information may include at least one of TSP identification information and face index.

Viewport generation using rectilinear projection is obtained by converting a 360-degree image into a two-dimensional image projected with a viewing angle as the Z axis, and spatial structure information includes Viewport generation using rectilinear projection index information and a viewing port representing a viewpoint ( Viewport) information may be further included. Meanwhile, the spatial structure information may further include interpolation filter information to be applied in the image conversion. For example, the interpolation filter information may be different according to each projection transformation method, and may include at least one of a nearest neighbor filter, a bilinear filter, a bicubic filter, and a Lanczos filter.

Meanwhile, a transformation scheme and its index for evaluating processing performance of pre-processing transformation and post-processing inverse transformation may be separately defined. For example, performance evaluation may be used to determine a preprocessing method in the preprocessor 10, and in that method, a CPP method that measures PSNR by converting two different transformed images into a Crasters Parablic Projection (CPP) domain. This can be exemplified.

However, the table shown in FIG. 4 is arbitrarily arranged according to the input image, and may be changed according to encoding efficiency and content distribution in the market.

Accordingly, the decoding apparatus 200 may parse the separately signaled table index and use it for decoding processing.

In particular, in an embodiment of the present invention, each layout information may be usefully used for partial decoding of an image. That is, subimage arrangement information such as CUBIC LAYOUT can be used to distinguish independent subimages from dependent subimages, and accordingly, it can be used to determine an efficient coding/decoding scanning order or to perform partial decoding at a specific point in time.

Hereinafter, frame packing according to an embodiment of the present invention will be described with reference to FIGS. 5 to 9 .

Referring to FIG. 5, an efficient projection plane rearrangement technique of the ISP method, which is one of image format techniques for encoding 360 video, is proposed.

In order to encode a 360-degree video, projection into a 2D image is required. There are various projection methods such as ERP (Equirectangular Projection), and the non-active area and discontinuous boundaries are reduced through rearrangement of projection planes in the format.

ISP is a projection technique that transforms a spherical image by projecting it onto 20 planes, one of which is triangular. As shown in FIG. 5 , each projected surface may be aligned and expressed as a 2D image. An unaligned ISP can generally be expressed as in Figure 1(a). The discontinuity between the non-active area and the projection plane existing in a general ISP reduces the coding efficiency and deteriorates the subjective picture quality.

Accordingly, as shown in FIG. 5 , the inactive region may be removed and the discontinuity may be reduced by aligning the projection plane. This is called CISP (Compact ISP). In the process of arranging projection planes, some projection planes are divided into two pieces or placed upside down or upside down. If the two projection planes are simultaneously adjacent in the icosahedron and in the 2D projection image, no discontinuity occurs. However, a discontinuity occurs if the two projection planes are not adjacent in the icosahedron but are adjacent in the 2D projection image. By padding with a blank space between the two projection planes, coding can be simplified and visual artifacts appearing in the decoded image can be reduced. Blanks are filled with interpolation pixels using pixels closest to the boundary of the adjacent projection plane, and if there is no adjacent projection plane, pixels of the boundary are copied and filled. Currently, CISP is frame-packed with a total of 8 discontinuous boundaries, 4 horizontally and 4 diagonally. The discontinuous boundary generated in this way is the main cause of reducing the coding efficiency of CISP.

To solve the above problem, various frame packing methods have been proposed. As one of the methods, FIG. 5 presents a rearranged CISP in consideration of subjective picture quality.

However, the discontinuity generated in the existing CISP method reduces the coding efficiency in the process of encoding an image due to low correlation between neighboring pixels.

To solve this problem, the present invention can provide an efficient frame packing technique that can increase coding efficiency by generating diagonal discontinuities between triangles with high similarity, and can improve subjective picture quality by attaching equatorial regions into one.

More specifically, the frame packing technique proposed in the present invention causes a discontinuous boundary in a diagonal direction to occur in the same polar region with high similarity through realignment of projection planes. In addition, by arranging the triangles in the equatorial region containing the main information of the 360 image in a row and minimizing the discontinuity between the triangles in the equatorial region, subjective image quality can be improved.

6 is a diagram for explaining frame packing according to an embodiment of the present invention for implementing this.

More specifically, the present invention can be performed in the conversion step to the ISP format. As described above, CISP has 8 discontinuous surfaces, 4 horizontally and 4 diagonally, and frame packing proceeds. Encoding is performed. In addition, when a 360 image has a large motion, the directionality of adjacent projection planes may be different in the process of rearranging the projection plane. This causes a decrease in encoding efficiency because directions are not similar to each other in the image prediction step.

Therefore, the reduction in encoding efficiency caused by this may be reduced through the following method. The following processes may be exemplified as being processed by the format conversion process of the image format conversion unit 11 of the pre-processing unit 10.

According to the first embodiment, the format conversion process may include a process of arranging projection planes such that a diagonal discontinuity is generated between projection planes having high polar similarity.

According to the second embodiment, the format conversion process may include a process of arranging projection planes adjacent to each other to have similar directivity.

According to the third embodiment, the format conversion process may include a process of padding discontinuous areas by copying adjacent pixels between projection planes by N pixels.

According to the fourth embodiment, the format conversion process may include a process of padding discontinuous areas by performing N-pixel interpolation of adjacent pixels between projection planes.

More specifically, in step [D1] of FIG. 6, a 360 image format conversion step; At least one of various formats such as ERP, CMP, ACP, EAP, OHP, ISP, TSP, SSP, and RSP can be used to encode a 360-degree video. Among those formats, the one in which the projection plane of the ISP is aligned to remove the inactive area and the discontinuity between the projection planes is minimized is called CISP. Through rearrangement of the projection plane, the reduction in coding efficiency caused by the discontinuity plane in the existing CISP can be reduced.

Here, a step [D1-1] may be further included, which may include a step of rearranging the projection plane by using similarity of adjacent planes in the rearrangement of the projection plane.

FIG. 7 is an example of rearranging the projection plane of CISP, and FIG. 8 shows an example of CISP and a discontinuous plane configured through rearrangement of the projection plane.

The green line segment and the red line segment in [FIG. 7] and [FIG. 8] may mean discontinuity between projection planes.

Referring to FIGS. 7 and 8 , the image format conversion unit 11 may arrange the projection planes such that a diagonal discontinuity is generated between the projection planes having high polar similarity.

For example, [Fig. 7] can be frame-packed into [Fig. 8] through rearrangement of the projection plane. The rearrangement of the projection plane can be implemented so that a diagonal discontinuity is generated in the pole region where the inter-pixel correlation is high.

Also, the image format conversion unit 11 may arrange projection planes so that adjacent projection planes have similar directivity.

For example, when an image has great motion, the image format conversion unit 11 may make the projection planes have similar orientations through rearrangement of the projection planes.

For example, in [Fig. 7], projection planes 14 and 16 and projection planes 18 may have similar orientations.

Meanwhile, the [D1] may further include steps [D1-2]. This may include performing N-pixel padding on discontinuous boundaries existing on the rearranged projection planes.

In this case, the image format conversion unit 11 may copy and pad adjacent pixels of the projection plane.

For example, the image format conversion unit 11 may copy adjacent pixels of discontinuous boundaries between projection planes and perform the same in various sizes as many as N pixels.

Also, the image format conversion unit 11 may pad adjacent pixels of the projection plane using an interpolation method.

For example, the image format conversion unit 11 may perform discontinuous boundaries between projection planes in various ways as many as N pixels using various interpolation methods such as linear interpolation.

The proposed technique was implemented in HM16.16_360Lib4.0 and performance was verified according to CTC (Common Test Condition). The proposed method was evaluated using End to End (E2E) S-PSNR-NN, WS-PSNR, and Codec level (CL) S-PSNR-NN and WS-PSNR [6]. In terms of objective picture quality, BD-rate reductions of 1.0%, 1.0%, 1.55%, and 0.75% for E2E S-PSNR-NN, WS-PSNR, CL S-PSNR-NN, and WS-PSNR compared to existing CISP can be confirmed, respectively. could

(a) of FIG. 9 is an image display area of the existing CISP, and (b) shows an image display area of the proposed technique. In terms of subjective image quality, it was confirmed that the proposed method reduced visual artifacts in the equatorial region, which is the main source of 360 video information, compared to the existing CISP.

According to an embodiment of the present invention, it can be confirmed that frame packing by rearranging projection planes having high similarity to each other and frame packing is helpful in improving coding efficiency. Experimental results: 1.0%, 1.0%, 1.55%, 0.75% of BD- rate decreased. In addition, it was confirmed that visual artifacts in the equatorial region, which is the main source of 360 video information, were reduced in terms of subjective picture quality.

10 to 11 are diagrams for explaining encoding and decoding processes according to an embodiment of the present invention.

10 is a block diagram showing the configuration of a video encoding apparatus according to an embodiment of the present invention, which receives and processes each sub-image or entire frame of a virtual reality video as an input video signal according to an embodiment of the present invention. can

Referring to FIG. 10, a video encoding apparatus 100 according to the present invention includes a picture division unit 160, a transform unit, a quantization unit, a scanning unit, an entropy encoding unit, an intra prediction unit 169, and an inter prediction unit 170. , an inverse quantization unit, an inverse transform unit, a post-processing unit 171, a picture storage unit 172, a subtraction unit, and an addition unit 168.

The picture divider 160 analyzes the input video signal, divides the picture into coding units of a predetermined size for each Largest Coding Unit (LCU), determines a prediction mode, and determines the size of the prediction unit for each coding unit. decide

Then, the picture division unit 160 sends the prediction unit to be encoded to the intra prediction unit 169 or the inter prediction unit 170 according to the prediction mode (or prediction method). Also, the picture division unit 160 sends a prediction unit to be encoded to the subtraction unit.

A picture is composed of a plurality of slices, and a slice may be composed of a plurality of largest coding units (LCUs).

The LCU may be divided into a plurality of coding units (CUs), and an encoder may add information (flag) indicating whether division is performed to a bitstream. The decoder can recognize the location of the LCU using the address (LcuAddr).

A coding unit (CU) when division is not allowed is regarded as a prediction unit (PU), and the decoder can recognize the location of the PU using the PU index.

A prediction unit (PU) may be divided into a plurality of partitions. In addition, a prediction unit (PU) may be composed of a plurality of transform units (TUs).

In this case, the picture division unit 160 may send the image data to the subtraction unit in units of blocks (eg, PU units or TU units) of a predetermined size according to the determined coding mode.

CTB (Coding Tree Block) is used as a video coding unit, and at this time, CTB is defined in various square shapes. CTB is called a coding unit CU (Coding Unit).

A coding unit (CU) may have a form of a quad tree according to division. In addition, in the case of QTBT (Quadtree plus binary tree) division, the coding unit may have the form of a binary tree binary partitioned from the quad tree or terminal node, and the maximum size is 256X256 to 64x according to the standard of the encoder. It can consist of 64.

For example, when the maximum size is 64X64, the picture divider 160 sets the depth to 0 when the maximum coding unit is LCU (Largest Coding Unit), and until the depth becomes 3, that is, 8 × 8 size. Coding is performed by finding an optimal prediction unit recursively up to the coding unit (CU). Also, for a coding unit of a terminal node divided into, for example, QTBT, a prediction unit (PU) and a transform unit (TU) may have the same form as the divided coding unit or may have a further divided form.

A prediction unit that performs prediction is defined as a PU (Prediction Unit), and each coding unit (CU) is divided into a plurality of blocks and prediction is performed, and prediction is performed by dividing into square and rectangular shapes.

The conversion unit converts the original block of the input prediction unit and the residual block, which is a residual signal of the prediction block generated by the intra prediction unit 169 or the inter prediction unit 170. The residual block is composed of a coding unit or a prediction unit. A residual block composed of coding units or prediction units is divided into optimal transform units and transformed. Different transformation matrices may be determined according to prediction modes (intra or inter). Also, since the residual signal of intra prediction has a direction according to the intra prediction mode, a transform matrix may be adaptively determined according to the intra prediction mode.

A transformation unit can be transformed by two (horizontal, vertical) one-dimensional transformation matrices. For example, in the case of inter prediction, one predetermined transformation matrix is determined.

On the other hand, in the case of intra prediction, when the intra prediction mode is horizontal, since the probability that the residual block has a vertical direction increases, a DCT-based integer matrix is applied in the vertical direction and a DST-based integer matrix is applied in the horizontal direction. Alternatively, a KLT-based integer matrix is applied. When the intra prediction mode is vertical, a DST-based or KLT-based integer matrix is applied in the vertical direction, and a DCT-based integer matrix is applied in the horizontal direction.

In case of DC mode, DCT-based integer matrix is applied to both directions. Also, in the case of intra prediction, a transform matrix may be adaptively determined depending on the size of a transform unit.

The quantization unit determines a quantization step size for quantizing coefficients of the residual block transformed by the transform matrix. The quantization step size is determined for each coding unit (hereinafter, referred to as a quantization unit) equal to or larger than a predetermined size.

The predetermined size may be 8x8 or 16x16. Coefficients of the transform block are quantized using a quantization matrix determined according to the determined quantization step size and prediction mode.

The quantization unit uses a quantization step size of a quantization unit adjacent to the current quantization unit as a quantization step size predictor of the current quantization unit.

The quantization unit may generate a quantization step size predictor of the current quantization unit by using one or two effective quantization step sizes by searching in the order of the left quantization unit, the upper quantization unit, and the upper left quantization unit of the current quantization unit.

For example, the first effective quantization step size retrieved in the above order may be determined as a quantization step size predictor. In addition, an average value of two valid quantization step sizes retrieved in the above order may be determined as a quantization step size predictor, and if only one is valid, this may be determined as a quantization step size predictor.

When the quantization step size predictor is determined, a difference between the quantization step size of the current coding unit and the quantization step size predictor is transmitted to the entropy encoder.

Meanwhile, there is a possibility that all of the left coding unit, the above coding unit, and the top left coding unit of the current coding unit do not exist. On the other hand, a previously existing coding unit may exist in the coding order within the largest coding unit.

Therefore, in quantization units adjacent to the current coding unit and the maximum coding unit, a quantization step size of a quantization unit immediately preceding in coding order may be a candidate.

In this case, priority is given to 1) the left quantization unit of the current coding unit, 2) the upper quantization unit of the current coding unit, 3) the upper left quantization unit of the current coding unit, and 4) the order of the immediately preceding quantization unit in the coding order. can The order may be reversed, and the upper left quantization unit may be omitted.

The quantized transform block is provided to an inverse quantization unit and a scanning unit.

The scanning unit scans the coefficients of the quantized transform block and transforms them into one-dimensional quantization coefficients. Since the coefficient distribution of the transform block after quantization may depend on the intra prediction mode, the scanning method is determined according to the intra prediction mode.

In addition, the coefficient scanning method may be determined differently according to the size of the transform unit. The scan pattern may vary according to a directional intra prediction mode. The scanning order of quantization coefficients is scanned in the reverse direction.

When the quantized coefficients are divided into a plurality of subsets, the same scan pattern is applied to the quantized coefficients in each subset. A zigzag scan or a diagonal scan is applied to the scan pattern between the subsets. The scan pattern preferably scans from the main subset including DC to the remaining subsets in a forward direction, but the reverse direction is also possible.

In addition, a scan pattern between subsets may be set identically to a scan pattern of quantized coefficients within the subset. In this case, a scan pattern between subsets is determined according to an intra prediction mode. Meanwhile, the encoder transmits information indicating the position of the last non-zero quantization coefficient in the transform unit to the decoder.

Information indicating the position of the last non-zero quantization coefficient in each subset may also be transmitted to the decoder.

Inverse quantization 135 inverse quantizes the quantized quantization coefficients. The inverse transform unit restores the inverse quantized transform coefficient into a residual block in the spatial domain. The adder generates a reconstructed block by adding the residual block reconstructed by the inverse transform unit and the prediction block received from the intra predictor 169 or the inter predictor 170.

The post-processing unit 171 performs a deblocking filtering process to remove a blocking effect occurring in the reconstructed picture, an adaptive offset application process to supplement the difference value with the original video in pixel units, and a difference from the original video using a coding unit. Performs an adaptive loop filtering process to complement the value.

It is preferable to apply the deblocking filtering process to a boundary between a prediction unit and a transformation unit having a size greater than or equal to a predetermined size. The size may be 8x8. The deblocking filtering process includes determining a boundary to be filtered, determining a boundary filtering strength to be applied to the boundary, determining whether to apply a deblocking filter, and the deblocking filter. and selecting a filter to be applied to the boundary when it is determined to apply.

Whether or not the deblocking filter is applied is i) whether the boundary filtering strength is greater than 0 and ii) a value representing the degree of change in pixel values at the boundary of two blocks (P block, Q block) adjacent to the boundary to be filtered. It is determined by whether it is smaller than the first reference value determined by this quantization parameter.

It is preferable that the said filter is at least two or more. When the absolute value of the difference between two pixels located at the block boundary is greater than or equal to the second reference value, a filter that performs relatively weak filtering is selected.

The second reference value is determined by the quantization parameter and the boundary filtering strength.

The process of applying the adaptive offset is to reduce a distortion between a pixel in an image to which a deblocking filter is applied and an original pixel. It may be determined whether to perform the adaptive offset application process in units of pictures or slices.

A picture or slice may be divided into a plurality of offset regions, and an offset type may be determined for each offset region. Offset types may include a predetermined number (eg, 4) of edge offset types and two band offset types.

If the offset type is an edge offset type, an edge type to which each pixel belongs is determined, and an offset corresponding thereto is applied. The edge type is determined based on the distribution of values of two pixels adjacent to the current pixel.

The adaptive loop filtering process may perform filtering based on a value obtained by comparing a reconstructed image that has undergone a deblocking filtering process or an adaptive offset application process with an original video. Adaptive loop filtering may apply the determined ALF to all pixels included in a 4x4 or 8x8 block.

Whether to apply the adaptive loop filter may be determined for each coding unit. The size and coefficient of the loop filter to be applied may vary according to each coding unit. Information indicating whether the adaptive loop filter is applied for each coding unit may be included in each slice header.

* In the case of a color difference signal, whether to apply an adaptive loop filter may be determined on a picture-by-picture basis. Unlike the luminance, the shape of the loop filter may also have a rectangular shape.

Whether adaptive loop filtering is applied may be determined for each slice. Accordingly, information indicating whether adaptive loop filtering is applied to the current slice is included in the slice header or picture header.

If it indicates that adaptive loop filtering is applied to the current slice, the slice header or picture header additionally includes information indicating the horizontal and/or vertical filter length of the luminance component used in the adaptive loop filtering process.

A slice header or a picture header may include information indicating the number of filter sets. In this case, if the number of filter sets is 2 or more, filter coefficients may be coded using a prediction method. Accordingly, a slice header or a picture header may include information indicating whether filter coefficients are coded using a prediction method, and includes predicted filter coefficients when the prediction method is used.

Meanwhile, color difference components as well as luminance may be adaptively filtered. Accordingly, the slice header or the picture header may include information indicating whether each chrominance component is filtered. In this case, in order to reduce the number of bits, joint coding (ie, multiplex coding) of information indicating whether Cr and Cb are filtered may be performed.

At this time, in the case of color difference components, since it is most likely that both Cr and Cb are not filtered to reduce complexity, entropy encoding is performed by assigning the smallest index when both Cr and Cb are not filtered. .

And, when both Cr and Cb are filtered, entropy encoding is performed by assigning the largest index.

The picture storage unit 172 receives post-processed image data from the post-processing unit 171 and restores and stores the image in units of pictures. A picture may be an image in frame units or an image in field units. The picture storage unit 172 includes a buffer (not shown) capable of storing a plurality of pictures.

The inter prediction unit 170 performs motion estimation using at least one reference picture stored in the picture storage unit 172, and determines a reference picture index and a motion vector representing the reference picture.

Then, a prediction block corresponding to a prediction unit to be encoded is extracted and output from a reference picture used for motion estimation among a plurality of reference pictures stored in the picture storage unit 172 according to the determined reference picture index and motion vector. .

The intra prediction unit 169 performs intra prediction encoding using reconstructed pixel values within a picture including a current prediction unit.

The intra prediction unit 169 receives a current prediction unit to be predictively encoded and performs intra prediction by selecting one of a preset number of intra prediction modes according to the size of a current block.

The intra prediction unit 169 adaptively filters reference pixels to generate an intra prediction block. When reference pixels are not available, reference pixels may be generated using available reference pixels.

The entropy encoding unit entropy-codes the quantized coefficients quantized by the quantization unit, the intra prediction information received from the intra prediction unit 169, and the motion information received from the inter prediction unit 170.

Although not shown, the inter prediction encoding apparatus may include a motion information determining unit, a motion information encoding mode determining unit, a motion information encoding unit, a prediction block generator, a residual block generator, a residual block encoder, and a multiplexer. .

The motion information determining unit determines motion information of the current block. Motion information includes a reference picture index and a motion vector. The reference picture index indicates one of previously encoded and reconstructed pictures.

When the current block is unidirectional inter-prediction coded, it represents one of reference pictures belonging to list 0 (L0). On the other hand, if the current block is subjected to bidirectional predictive coding, it may include a reference picture index indicating one of the reference pictures of list 0 (L0) and a reference picture index indicating one of the reference pictures of list 1 (L1). .

In addition, when the current block is bi-directionally predicted-coded, it may include indices indicating one or two pictures among reference pictures of a composite list (LC) generated by combining list 0 and list 1.

* A motion vector represents the position of a prediction block within a picture indicated by each reference picture index. The motion vector may be a pixel unit (integer unit) or may be a sub-pixel unit.

For example, it may have a resolution of 1/2, 1/4, 1/8 or 1/16 pixels. When the motion vector is not an integer unit, a prediction block is generated from pixels of an integer unit.

The motion information encoding mode determining unit determines whether to encode the motion information of the current block in skip mode, merge mode, or AMVP mode.

The skip mode is applied when a skip candidate having the same motion information as that of the current block exists and the residual signal is 0. Also, the skip mode is applied when the current block has the same size as the coding unit. The current block can be viewed as a prediction unit.

The merge mode is applied when a merge candidate having the same motion information as that of the current block exists. The merge mode is applied when a residual signal exists when the size of the current block is different from that of the coding unit or when the size is the same as that of the coding unit. Merge candidates and skip candidates may be the same.

AMVP mode is applied when skip mode and merge mode do not apply. An AMVP candidate having a motion vector most similar to that of the current block is selected as an AMVP predictor.

The motion information encoding unit encodes the motion information according to the method determined by the motion information encoding mode determination unit. When the motion information encoding mode is a skip mode or a merge mode, a merge motion vector encoding process is performed. When the motion information encoding mode is AMVP, an AMVP encoding process is performed.

The prediction block generation unit generates a prediction block using motion information of the current block. When the motion vector is an integer unit, a block corresponding to a position indicated by the motion vector in the picture indicated by the reference picture index is copied to generate a prediction block of the current block.

However, when the motion vector is not an integer unit, pixels of a prediction block are generated from integer unit pixels in a picture indicated by a reference picture index.

In this case, in the case of a luminance pixel, a prediction pixel may be generated using an 8-tap interpolation filter. In the case of color difference pixels, prediction pixels may be generated using a 4-tap interpolation filter.

The residual block generator generates a residual block using a current block and a prediction block of the current block. When the size of the current block is 2Nx2N, a residual block is generated using the current block and a prediction block having a size of 2Nx2N corresponding to the current block.

However, when the size of the current block used for prediction is 2NxN or Nx2N, a prediction block for each of two 2NxN blocks constituting 2Nx2N is obtained, and then a final prediction block of size 2Nx2N is obtained using the two 2NxN prediction blocks. can create

Also, a 2Nx2N residual block may be generated using the 2Nx2N prediction block. In order to solve the discontinuity of the boundary of two prediction blocks of size 2NxN, pixels of the boundary may be overlapped and smoothed.

The residual block encoder divides the generated residual block into one or more transform units. Then, each transform unit is transform-encoded, quantized, and entropy-encoded. In this case, the size of the transform unit may be determined according to the quadtree method according to the size of the residual block.

The residual block encoder transforms the residual block generated by the inter prediction method using an integer-based transformation matrix. The transformation matrix is an integer-based DCT matrix.

The residual block encoder uses a quantization matrix to quantize coefficients of the residual block transformed by the transform matrix. The quantization matrix is determined by a quantization parameter.

The quantization parameter is determined for each coding unit having a predetermined size or larger. The predetermined size may be 8x8 or 16x16. Therefore, when the current coding unit is smaller than the predetermined size, only the quantization parameters of the first coding unit in the coding order among the plurality of coding units within the predetermined size are coded, and the quantization parameters of the remaining coding units are the same as the above parameters. No need to.

Coefficients of the transform block are quantized using a quantization matrix determined according to the determined quantization parameter and prediction mode.

A quantization parameter determined for each coding unit having a predetermined size or larger is predicted and encoded using a quantization parameter of a coding unit adjacent to the current coding unit. A quantization parameter predictor of the current coding unit may be generated using one or two effective quantization parameters by searching in the order of the left coding unit and the upper coding unit of the current coding unit.

For example, the first effective quantization parameter searched in the above order may be determined as a quantization parameter predictor. In addition, a first effective quantization parameter may be determined as a quantization parameter predictor by searching in the order of the left coding unit and the immediately preceding coding unit in coding order.

Coefficients of the quantized transform block are scanned to transform into one-dimensional quantization coefficients. A scanning method may be set differently according to an entropy encoding mode. For example, in the case of CABAC encoding, inter-prediction coded quantization coefficients may be scanned in a predetermined manner (zigzag or raster scan in a diagonal direction). On the other hand, if it is coded by CAVLC, it can be scanned in a different way from the above way.

For example, the scanning method may be determined according to zigzag in the case of inter and intra prediction mode in the case of intra. In addition, the coefficient scanning method may be determined differently according to the size of the transform unit.

The scan pattern may vary according to a directional intra prediction mode. The scanning order of quantization coefficients is scanned in the reverse direction.

The multiplexer multiplexes the motion information encoded by the motion information encoder and the residual signals encoded by the residual block encoder. The motion information may vary according to an encoding mode.

That is, in the case of skip or merge, only indexes representing predictors are included. However, in the case of AMVP, the reference picture index of the current block, the differential motion vector, and the AMVP index are included.

Hereinafter, an embodiment of the operation of the intra prediction unit 169 will be described in detail.

First, prediction mode information and prediction block size are received by the picture divider 160, and the prediction mode information indicates an intra mode. The size of the prediction block may be a square such as 64x64, 32x32, 16x16, 8x8, or 4x4, but is not limited thereto. That is, the size of the prediction block may be non-square rather than square.

Next, reference pixels are read from the picture storage unit 172 to determine the intra prediction mode of the prediction block.

It is determined whether a reference pixel is generated by examining whether the unavailable reference pixel exists. The reference pixels are used to determine the intra prediction mode of the current block.

When the current block is located on the upper boundary of the current picture, pixels adjacent to the upper side of the current block are not defined. Also, when the current block is located on the left boundary of the current picture, pixels adjacent to the left of the current block are not defined.

It is determined that these pixels are not usable pixels. Also, when the current block is located at the slice boundary and adjacent pixels to the upper or left side of the slice are not pixels to be encoded and reconstructed first, it is determined that they are not usable pixels.

As described above, when pixels adjacent to the left side or above the current block do not exist or pixels that have been previously encoded and reconstructed do not exist, the intra prediction mode of the current block may be determined using only available pixels.

However, reference pixels at unavailable locations may be generated using available reference pixels of the current block. For example, when the pixels of the upper block are not available, some or all of the pixels on the left side may be used to generate the upper side pixels, and vice versa.

That is, a reference pixel may be created by copying an available reference pixel at a nearest location in a predetermined direction from a reference pixel at an unavailable location. When there is no reference pixel available in a predetermined direction, a reference pixel may be created by copying an available reference pixel at the closest location in the opposite direction.

Meanwhile, even when there are pixels above or to the left of the current block, they may be determined as unavailable reference pixels according to the coding mode of the block to which the pixels belong.

For example, when a block to which a reference pixel adjacent to the upper side of the current block belongs is a block reconstructed by inter-encoding, the pixels may be determined as unavailable pixels.

In this case, a block adjacent to the current block is intra-coded, and usable reference pixels may be generated using pixels belonging to a reconstructed block. In this case, the encoder needs to transmit information indicating that usable reference pixels are determined according to the encoding mode to the decoder.

Next, an intra prediction mode of the current block is determined using the reference pixels. The number of intra prediction modes allowable for the current block may vary depending on the size of the block. For example, when the size of the current block is 8x8, 16x16, or 32x32, 34 intra prediction modes may exist, and when the size of the current block is 4x4, 17 intra prediction modes may exist.

The 34 or 17 intra prediction modes may include at least one non-directional mode and a plurality of directional modes.

One or more non-directional modes may be DC modes and/or planar modes. When the DC mode and the planar mode are included as non-directional modes, 35 intra prediction modes may exist regardless of the size of the current block.

In this case, two non-directional modes (DC mode and planar mode) and 33 directional modes may be included.

The planner mode generates a prediction block of the current block using at least one pixel value (or a predicted value of the pixel value, hereinafter referred to as a first reference value) located at the bottom-right of the current block and reference pixels. .

As described above, the configuration of the video decoding apparatus according to an embodiment of the present invention can be derived from the configuration of the video encoding apparatus described above. can do.

11 is a block diagram showing the configuration of a video decoding apparatus according to an embodiment of the present invention.

Referring to FIG. 11, the video decoding apparatus according to the present invention includes an entropy decoding unit 210, an inverse quantization/inverse transform unit 220, an adder 270, a deblocking filter 250, a picture storage unit 260, It includes an intra prediction unit 230, a motion compensated prediction unit 240, and an intra/inter conversion switch 280.

The entropy decoding unit 210 decodes the encoded bit stream transmitted from the video encoding device and separates the encoded bit stream into an intra prediction mode index, motion information, quantization coefficient sequence, and the like. The entropy decoding unit 210 supplies the decoded motion information to the motion compensation prediction unit 240 .

The entropy decoding unit 210 supplies the intra prediction mode index to the intra prediction unit 230 and the inverse quantization/inverse transformation unit 220 . Also, the entropy decoding unit 210 supplies the inverse quantization coefficient sequence to the inverse quantization/inverse transformation unit 220 .

The inverse quantization/inverse transformation unit 220 transforms the quantization coefficient sequence into a 2-dimensional array of inverse quantization coefficients. For the conversion, one of a plurality of scanning patterns is selected. One of a plurality of scanning patterns is selected based on at least one of the prediction mode of the current block (that is, one of intra prediction and inter prediction) and the intra prediction mode.

The intra prediction mode is received from an intra prediction unit or an entropy decoding unit.

The inverse quantization/inverse transformation unit 220 restores quantization coefficients by using a quantization matrix selected from among a plurality of quantization matrices for the inverse quantization coefficients of the two-dimensional array. Different quantization matrices are applied according to the size of a current block to be reconstructed, and a quantization matrix is selected for blocks of the same size based on at least one of a prediction mode and an intra prediction mode of the current block.

Then, a residual block is reconstructed by inverse transforming the reconstructed quantization coefficient.

The adder 270 restores an image block by adding the residual block reconstructed by the inverse quantization/inverse transformation unit 220 and the prediction block generated by the intra prediction unit 230 or the motion compensation prediction unit 240.

The deblocking filter 250 performs a deblocking filter process on the reconstructed image generated by the adder 270. Accordingly, deblocking artifacts caused by image loss due to the quantization process can be reduced.

The picture storage unit 260 is a frame memory that stores a locally decoded image on which the deblocking filter process is performed by the deblocking filter 250 .

The intra prediction unit 230 restores the intra prediction mode of the current block based on the intra prediction mode index received from the entropy decoding unit 210 . Then, a prediction block is generated according to the reconstructed intra prediction mode.

The motion compensation prediction unit 240 generates a prediction block for a current block from a picture stored in the picture storage unit 260 based on the motion vector information. When decimal precision motion compensation is applied, a prediction block is generated by applying the selected interpolation filter.

The intra/inter conversion switch 280 provides a prediction block generated by either the intra predictor 230 or the motion compensated predictor 240 to the adder 270 based on the encoding mode.

The current block is reconstructed using the predicted block of the current block reconstructed in this way and the residual block of the current block that has been decoded.

A video bitstream according to an embodiment of the present invention is a unit used to store coded data in one picture, and may include parameter sets (PS) and slice data.

Parameter sets (PS) are divided into a picture parameter set (hereinafter simply referred to as PPS) and a sequence parameter set (hereinafter simply referred to as SPS), which are data corresponding to the head of each picture. The PPS and SPS may include initialization information necessary for initializing each encoding, and may include spatial structure information (SPATIAL LAYOUT INFORMATION) according to an embodiment of the present invention.

The SPS is common reference information for decoding all pictures coded with a random access unit (RAU), and may include a profile, the maximum number of pictures usable for reference, and a picture size.

The PPS may include, for each picture coded by a random access unit (RAU), a type of variable length coding method, an initial value of a quantization step, and a plurality of reference pictures as reference information for decoding the picture.

Meanwhile, the slice header (SH) includes information on a corresponding slice when coding in slice units.

The method according to the present invention described above may be produced as a program to be executed on a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, and magnetic tape. , floppy disks, and optical data storage devices.

The computer-readable recording medium is distributed to computer systems connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the technical field to which the present invention belongs.

Claims (1)

In the image processing method of the image processing device,
Including frame packing processing of the virtual reality image, processing encoding or decoding of the virtual reality image,
The virtual reality image is framed in a CISP (Compact ISP) format having a certain number of horizontal and diagonal discontinuities so as to align projection planes of the virtual reality images converted to the ISP format, remove inactive regions, and reduce discontinuity between projection planes. Including packed virtual reality images,
The frame packing process,
In the virtual reality image converted to the CISP format, a format conversion process of rearranging a projection plane so that a diagonal discontinuity is generated in the virtual reality image frame-packed with triangles,
The format conversion process,
In the polar region of the virtual reality image converted to the CISP format, the triangles in the equatorial region are arranged in a line while rearranging such that a discontinuity in a diagonal direction occurs between the projection planes of triangles in the polar region where the inter-pixel correlation is higher than the reference value. a first process of rearranging the projection plane so that the discontinuity between the triangles of the equatorial portion is minimized;
In performing the first process, a second process of rearranging projection surfaces so that adjacent projection surfaces have similar directivity within a predetermined range;
In performing the second process, a third process of padding at least some of the discontinuous boundaries existing on the rearranged projection surfaces by copying adjacent pixels between projection planes by N pixels;
In performing the third process, a fourth process of padding at least some of the discontinuous boundaries existing on the rearranged projection surfaces by performing N-pixel interpolation of adjacent pixels between projection planes,
Including processing to reduce visual artifacts for subjective image quality due to frame packing of the virtual reality image
Image processing method.

Images