JP2021064979A - Encoding device, encoding method and program, and decoding device, decoding method and program - Google Patents

Abstract

To solve the problem in which: when inter-loop filtering is performed for pixels located over a discontinuous tile boundary, a deterioration in image quality may occur as a result of filtering processing; meanwhile, if the filtering processing is unconditionally disabled for the pixels located over the tile boundary, an inter-loop filter cannot be applied to a continuous tile boundary where originally image quality can be expected to be improved by inter-loop filter processing.SOLUTION: An encoding device for encoding an image having a plurality of tiles has: determination means that determines, for a plurality of boundaries formed by the plurality of tiles, whether or not to perform filter processing on pixels adjacent to the boundaries; and encoding means that, based on the determination made by the determination means, encodes control information indicating whether or not to perform the filter processing on the pixels adjacent to the boundaries for at least two of the plurality of boundaries.SELECTED DRAWING: Figure 1

Description

The present invention relates to a coding device, a coding method and a program, a decoding device, a decoding method and a program, and an image code data, and particularly to a filtering process at a block boundary.

As a coding method for compressed recording of moving images, a HEVC (High Efficiency Video Coding) coding method (hereinafter referred to as HEVC) is known. HEVC employs a method called Tile, which divides one frame into a plurality of rectangular regions to enable parallel processing of coding / decoding. By using tiles, it is possible to realize high speed by parallel processing of coding / decoding and to reduce the memory capacity of the image coding device / image decoding device.

Further, in HEVC, in-loop filter processing such as a deblocking filter and a sample adaptive offset is also adopted in order to improve the image quality of the encoded image. Such in-loop filtering can be applied to pixels that straddle tile boundaries, but if in-loop filtering is applied to pixels that straddle tile boundaries, coding and decoding are performed in parallel. It may interfere with the processing. Therefore, HEVC employs a loop_filter_acloss_tiles_enable_flag syntax element that can select whether or not to apply in-loop filtering to pixels that straddle the boundaries of tiles. When the syntax element is 1, the in-loop filter can be applied to the tile boundary, and when it is 0, the in-loop filter cannot be applied to the tile boundary. As a result, the syntax element can be set to 0 when the parallel mountability is emphasized, and the syntax element can be set to 1 when the image quality of the tile boundary is emphasized.

In recent years, with the development of VR (Virtual Reality) technology, a use case has been created in which a 360 ° image is captured by a plurality of cameras and the captured image is compressed and encoded. As a method of photographing a 360 ° image, there is a method of photographing images in each direction of up, down, left, right, front and back with six cameras and synthesizing them (Non-Patent Document 1). The images captured in this way are compressed and encoded by rearranging the six captured images and synthesizing them into one image. Regarding the rearrangement, the method of arranging the dice so as to unfold as shown in FIG. 9A, and the method of arranging the dice so as to expand the dice, and further making the image rectangular so that the area of the combined image is minimized as shown in FIGS. A method of rearranging is being studied (Non-Patent Document 2). In the method as shown in FIG. 9A, the boundaries of adjacent images such as left and front, front and right are always continuous, but useless areas are generated at the four corners of the combined image. On the other hand, in the methods shown in FIGS. 9 (b) to 9 (d), no wasted area is generated in the combined image, but continuous and discontinuous boundaries are formed in the boundaries of adjacent images. Mixed.

JVET Contribution JVET-C0021 Internet <http://phenix. int-every. fr / jvet / doc_end_user / documents / 3_Geneva / wg11 /> JVET Contribution JVET-D0022 Internet <http://phenix. int-every. fr / jvet / doc_end_user / documents / 4_Chengdu / wg11 />

It is considered effective from the viewpoint of coding efficiency to use an in-loop filter represented by the above-mentioned deblocking filter and sample adaptive offset. Further, as shown in FIGS. 9 (b) to 9 (d) described above, an image taken by a plurality of cameras and combined, that is, an image in which continuous boundaries and discontinuous boundaries are mixed in the boundary of the image is displayed. When encoding, it is natural to encode the image taken by each camera in association with the tile. However, in the existing method represented by HEVC, it is only possible to select whether or not to apply the in-loop filtering uniformly to all the tile boundaries in the image. That is, it is only possible to select whether or not to apply the in-loop filter without distinguishing between continuous tile boundaries and discontinuous tile boundaries. In that case, if the non-application of the in-loop filter is selected with priority given to the discontinuous tile boundary, the in-loop filter cannot be applied to the continuous tile boundary where the image quality can be originally expected to be improved by the in-loop filter processing. On the other hand, if you choose to apply the in-loop filter with priority given to continuous tile boundaries, the in-loop filter will be applied to the discontinuous tile boundaries, causing unnecessary image quality degradation around the discontinuous tile boundaries. It will cause it. Therefore, the present invention has been made to solve the above-mentioned problems, and an object of the present invention is to enable the in-loop filtering of tile boundaries to be applied adaptively in consideration of the continuity of tile boundaries. ..

In order to solve the above-mentioned problems, the coding apparatus of the present invention has the following configuration. That is, in a coding device for coding an image having a plurality of tiles, it is determined whether or not to filter pixels adjacent to the boundary with respect to the plurality of boundaries composed of the plurality of tiles. A determination means and a coding means that encodes control information indicating whether or not to filter pixels adjacent to a boundary based on the determination by the determination means with respect to at least two of the plurality of boundaries. It is characterized by having.

Further, the decoding device of the present invention has the following configuration. That is, in a decoding device for decoding an image having a plurality of tiles from a bit stream, a decoding means for decoding the image and a pixel adjacent to the boundary with respect to the plurality of boundaries composed of the plurality of tiles. Based on the generation means that generates control information indicating whether or not to perform filtering from the bitstream and the control information generated by the generation means, it is determined whether or not to perform filtering processing on pixels adjacent to at least a plurality of boundaries. It is characterized by having a determination means for filtering and a processing means for filtering the boundary determined to be filtered by the determination means.

It becomes possible to adaptively select whether or not to apply in-loop filtering for each tile boundary.

Block diagram showing the configuration of the image coding device Block diagram showing the configuration of the image decoding device Figures (a) to (b) show an example of the relationship between the continuity of tile boundaries and the target pixels for in-loop filtering. A flowchart showing an image coding process in an image coding apparatus A flowchart showing in-loop filtering in an image coding device and an image decoding device. (A)-(b) The figure which shows an example of the bit stream structure generated by an image coding apparatus and decoded by an image decoding apparatus. Flowchart showing decoding processing in image decoding apparatus A block diagram showing a computer hardware configuration example applicable to an image coding device and an image decoding device. (A)-(d) A diagram showing an arrangement example in which images taken by a plurality of cameras are rearranged into one image. (A)-(c) The figure which shows the syntax about the tile division information and the filter control information of the bit stream generated by the image coding apparatus and decoded by the image decoding apparatus.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings, based on its preferred embodiments. The configuration shown in the following embodiments is only an example, and the present invention is not limited to the illustrated configuration.

Hereinafter, embodiments of the coding side of the present invention will be described with reference to the drawings. In this embodiment, a case where the image shown in FIG. 9B is input and encoded will be described in particular.

FIG. 1 is a block diagram showing an image coding apparatus of this embodiment. Each block in FIG. 1 may be configured entirely as hardware, or some or all blocks may be configured by software.

In FIG. 1, the terminal 101 is a terminal for inputting image data.

The tile dividing unit 102 determines a tile dividing method for the input image and performs a process of dividing the input image.

The filter control information generation unit 103 generates and outputs filter control information, which is information about whether or not in-loop filter processing, which will be described later, is performed on the pixels at each tile boundary.

The prediction unit 104 performs intra-frame prediction, intra-frame prediction, inter-frame prediction, and the like on the image data in block units, and generates prediction image data. Further, the prediction error is calculated from the input image data and the predicted image data and output. In addition, information necessary for prediction, for example, information such as a prediction mode is also output together with a prediction error. Hereinafter, the information necessary for this prediction is referred to as prediction information.

The conversion / quantization unit 105 orthogonally transforms the prediction error in block units to obtain a conversion coefficient, and further performs quantization to obtain a quantization coefficient.

The inverse quantization / inverse conversion unit 106 inversely quantizes the quantization coefficient output from the conversion / quantization unit 105 to reproduce the conversion coefficient, and further performs inverse orthogonal transformation to reproduce the prediction error.

The frame memory 108 is a memory for storing the reproduced image data.

Based on the prediction information output from the prediction unit 104, the image reproduction unit 107 generates the prediction image data by appropriately referring to the frame memory 108, and reproduces the image from the generated prediction image data and the input prediction error. Generate and output data.

The in-loop filter unit 109 performs in-loop filter processing such as a deblocking filter and a sample adaptive offset on the reproduced image, and outputs the filtered image.

The coding unit 110 encodes the quantization coefficient output from the conversion / quantization unit 105 and the prediction information output from the prediction unit 104, generates code data, and outputs the code data.

The integrated coding unit 111 encodes the output from the tile dividing unit 102 and the filter control information generating unit 103 to generate header code data to be stored in the header. The generated header code data forms a bit stream and is output together with the code data output from the coding unit 110. The terminal 112 is a terminal that outputs the bit stream generated by the integrated coding unit 111 to the outside.

The image coding process in the image coding apparatus shown in FIG. 1 will be described below. In the present embodiment, the moving image data is input in frame units, but the still image data for one frame may be input.

The image data for one frame input from the terminal 101 is input to the tile dividing unit 102. The image data input at the terminal 101 is one frame of image data obtained by rearranging and synthesizing N images (N ≧ 3). The arrangement of N images (N ≧ 3) shall include the arrangement of rotated images. In the present embodiment, information indicating how many images are arranged and combined may be acquired.

The tile division unit 102 determines the tile division method of the input image, divides the input image data into tile units based on the determined division method, and outputs the tile division method to the prediction unit 104. That is,
Further, the determined tile division method is output as tile division information to the filter control information generation unit 103 and the integrated coding unit 111. The method for determining the tile division is not particularly limited, and the characteristics of the input image may be used, or the information indicating how many images are arranged and combined as described above is used. Alternatively, it may be determined by input from the user.
In the present embodiment, the input image data for one frame is divided along the boundary of the combined N images, and M (M ≧ 2) tiles are formed by N tiles (N ≧ 3). It will be described as assuming that the image data having the boundary of the tile is composed. For example, when coding an image as shown in FIG. 9B, each image of front, right, rear, bottom, left, and top is associated with an individual tile, that is, it is composed of six tiles. The image is encoded, and so on.

Next, the filter control information generation unit 103 determines whether or not to perform the in-loop filter processing for each tile boundary, and uses the information as the filter control information in the in-loop filter unit 109 and the integrated coding unit 111. Output. The method for determining the filter control information is not particularly limited, and the characteristics of the input image may be used, or the filter control information may be determined by input from the user. Further, the filter control information generation unit 103 may determine based on the information on the tile division state input from the outside or calculated internally and the information on the continuity of each tile boundary. For example, when coding an image as shown in FIG. 9B, the tile dividing portion 102 determines that an image composed of six tiles is encoded. In this case, the two vertical tile boundaries at the top of the image (between "front" and "right", between "right" and "rear") and the left part of the image1 Pixel values are continuous at one horizontal tile boundary (between "front" and "bottom"). On the other hand, the pixels are not continuous at the other four tile boundaries. In that case, the filter control information generation unit 103 performs in-loop filter processing on the three tile boundaries in which the pixels are continuous, and does not perform in-loop filter processing on the remaining four discontinuous tile boundaries. The indicated information is output as filter control information.

The integrated coding unit 111 encodes the tile division information and the filter control information, and generates the tile division information code and the filter control information code, respectively. Although the coding method is not particularly specified, Golomb coding, arithmetic coding, Huffman coding, or the like can be used.

The prediction unit 104 divides the image data for each tile input from the tile division unit 102 into a plurality of blocks, and the prediction process for each block is executed. As a result of the prediction processing, a prediction error is generated and input to the conversion / quantization unit 105. Further, the prediction unit 104 generates prediction information and outputs it to the coding unit 110 and the image reproduction unit 107.

The conversion / quantization unit 105 performs orthogonal transformation / quantization on the input prediction error to generate a quantization coefficient. The generated quantization coefficient is input to the coding unit 110 and the inverse quantization / inverse conversion unit 106.

The inverse quantization / inverse conversion unit 106 inversely quantizes the input quantization coefficient to reproduce the conversion coefficient, further performs inverse orthogonal conversion of the reproduced conversion coefficient to reproduce the prediction error, and causes the image reproduction unit 107 to reproduce the prediction error. Output.

The image reproduction unit 107 appropriately refers to the frame memory 108 based on the prediction information input from the prediction unit 104, and reproduces the predicted image. Then, the image data is reproduced from the reproduced predicted image and the reproduced predicted error input from the inverse quantization / inverse conversion unit 106, input to the frame memory 108, and stored.

The in-loop filter unit 109 reads the reproduced image from the frame memory 108, and performs in-loop filter processing such as a deblocking filter based on the block position to be filtered and the filter control information input from the filter control information 103. Then, the filtered image is input to the frame memory 108 again and stored again. This filtered image is used for inter-prediction and the like in the prediction unit 104.

The in-loop filter processing in this embodiment will be described with reference to FIG. One of the following filtering processes is performed based on the position of the block to be filtered and the filter control information. Here, only the filter processing at the tile boundary will be described, but the filter processing at the pixel positions other than the tile boundary shall be performed based on the existing image coding method. Further, in order to facilitate the explanation here, the block size is set to 4 × 4 blocks, but the present invention is not limited to this.

FIG. 3A is an example of a deblocking filter process straddling the boundary between two 4 × 4 blocks. Inside the outer frame 300, there are two blocks of 4 × 4 pixels, and filtering is executed for a total of 6 pixels, 3 pixels adjacent to the tile boundary of each block surrounded by the rounded rectangle 301. Shall be. The block on the left side shown in FIG. 3A belongs to tile 0, and the block on the right side belongs to tile 1. That is, the left and right blocks are blocks belonging to different tiles, and the boundary between the left and right blocks is also the boundary between tiles. It is assumed that this tile boundary is a tile boundary in which pixel values are continuous as seen in the upper part of FIG. 9B (for example, between "front" and "right"). In this case, since the filter control information generation unit 103 inputs the filter control information indicating that the in-loop processing is performed at the tile boundary where the pixels are continuous, all the 6 pixels indicated by the circles are filtered. It becomes a target, and filtering processing is performed on all 6 pixels.

On the other hand, FIG. 3B is different from FIG. 3A, and the boundary between the left and right tiles is a pixel value as seen in the lower part of FIG. 9B (for example, between “left” and “top”). Suppose that is a discontinuous tile boundary. In this case, since the filter control information generation unit 103 inputs the filter control information indicating that the in-loop processing is not performed at the tile boundary where the pixels are discontinuous, all the 6 pixels indicated by the x symbols are filtered. Not subject to processing. Therefore, the filtering process is not performed on these 6 pixels.

Although the deblocking filter has been mentioned in the present embodiment, the same control may be performed for other in-loop filter processing such as the adaptive loop filter and the sample adaptive offset. Further, in the present embodiment, the filter processing between the left and right blocks has been described, but the same processing is performed between the upper and lower blocks. Further, although an example in which 3 pixels of each block are filtered is shown, the number of pixels to be processed is not limited to this. For example, it is possible to perform asymmetric filtering processing in which the number of pixels to be processed differs in each block, such as 3 pixels for the left block and 2 pixels for the right block.

Returning to FIG. 1, the coding unit 110 entropy-encodes the quantization coefficient generated by the conversion / quantization unit 105 and the prediction information input from the prediction unit 104 in block units to generate code data. The method of entropy coding is not particularly specified, but Golomb coding, arithmetic coding, Huffman coding and the like can be used. The generated code data is output to the integrated coding unit 111.

In the integrated coding unit 111, a bit stream is formed by multiplexing the tile division information code, the filter control information code, the code data generated by the coding unit, and the like generated prior to the coding process. Finally, it is output from the terminal 112 to the outside.

FIG. 6A shows an example of a bit stream including encoded tile division information and filter control information. The tile division information is included as a tile division information code, and the filter control information is included as a filter control information code in any of the headers such as sequences and pictures. In the present embodiment, as shown in FIG. 6A, it is included in the header portion of the picture. FIG. 10A is a diagram showing an example of the syntax configuration of the header portion of the picture including these codes. tiles_enable_flag is a code indicating the presence or absence of tile division, and in the present embodiment, the case where the image shown in FIG. 9B is input and divided into 6 tiles is described, so this value is 1. Since entry_coding_sync_enable_flag is not related to this embodiment, the description thereof will be omitted. number_tile_columns_minus1 indicates the value obtained by subtracting 1 from the number of tiles in the horizontal direction. In this embodiment, there are three tiles in the horizontal direction, so this value is 2. On the other hand, number_tile_rows_minus1 indicates the value obtained by subtracting 1 from the number of tiles in the vertical direction. In this embodiment, since there are two tiles in the vertical direction, this value is 1. uniform_spacing_flag is a code indicating whether or not the sizes of the tiles (the number of pixels in the vertical and horizontal directions) match, and in this embodiment, the sizes of all the tiles match, so this value is 1. .. When this value is 0, that is, when all the tile sizes do not match, the volumes_wise_minus1 and low_height_minus1 indicating the horizontal and vertical sizes of the tiles are encoded, but the description thereof will be omitted in the present embodiment. .. loop_filter_aross_tiles_enabled_flag is a code indicating whether or not in-loop filtering of tile boundaries is possible, but in the present embodiment, in-loop filtering is possible at some boundaries, so this value is set to 1.

The following syntax is encoded only when the loop_filter_acloss_tiles_enabled_flag is 1. The loop_filter_accross_tiles_control_flag is a code indicating whether or not to individually encode the information indicating whether or not the in-loop filter processing is performed for each tile boundary. In this embodiment, it is necessary to individually set whether or not in-loop filter processing is possible, so this value is set to 1.

Furthermore, the following syntax is encoded only when the loop_filter_acloss_tiles_control_flag is 1. loop_filter_accross_bottom_tile_bound_enable_flag is a code indicating whether or not in-loop filtering is performed on the tile boundary below the tile. In the present embodiment, since there are three tiles having a tile boundary on the lower side (front, right, and back) in the upper row, this code is encoded three times. That is, loop_filter_accross_bottom_tile_bound_enable_flag is encoded for each boundary composed between the upper and lower tiles. Since the tile boundary below the "previous" tile is a continuous boundary, the first value of this is 1 indicating that in-loop filtering is to be performed. On the other hand, the tile boundaries below the "right" and "rear" tiles are discontinuous boundaries, so the second and third values are 0, which indicates no in-loop filtering. Become. Note that loop_filter_accross_bottom_tile_boundary_enable_flag is not encoded for the three tiles (bottom, left, top) in the lower row. Next, loop_filter_accross_right_tile_bound_enable_flag is a code indicating whether or not in-loop filtering is performed on the tile boundary on the right side of the tile. That is, loop_filter_accross_right_tile_bound_enable_flag is encoded for each boundary composed between the two left and right tiles. In the present embodiment, since there are four tiles (front, right, bottom, left) having a tile boundary on the right side, this code is encoded four times. Since the tile boundaries to the right of the "front" and "right" tiles are continuous boundaries, the first and second values are 1 indicating that in-loop filtering is to be performed. On the other hand, the tile boundaries to the right of the "bottom" and "left" tiles are discontinuous boundaries, so this third and fourth value is 0, which indicates no in-loop filtering. .. Note that loop_filter_acloss_right_tile_boundary_enable_flag is not encoded for the tiles (back, top) that exist in the rightmost column. That is, loop_filter_accross_bottom_tile_bound_enable_flag is encoded for the tile boundaries inside the image. Similarly, loop_filter_accross_right_tile_bound_enable_flag is encoded for the tile boundaries inside the image. Then, the boundaries of the tiles that form the outer circumference of the image are not encoded.

FIG. 4 is a flowchart showing a coding process in the image coding apparatus according to the present embodiment.

First, in step S401, the tile division unit 102 determines the tile division method of the input image, and divides the input image data into tile units based on the determined division method. Further, the determined tile division method is used as tile division information, and the tile division information is encoded by the integrated coding unit 111.

In step S402, the filter control information generation unit 103 determines whether or not to perform in-loop filter processing on the pixels at each tile boundary, uses that information as the filter control information, and the filter control information is also the integrated coding unit. It is encoded by 111.

In step S403, the prediction unit 104 cuts out the input image data for each tile into a plurality of blocks, performs intra-prediction or inter-prediction for each block, and generates prediction information and prediction image data. Further, the prediction error is calculated from the input image data and the predicted image data.

In step S404, the conversion / quantization unit 105 orthogonally transforms the prediction error calculated in step S403 to generate a conversion coefficient, and further performs quantization to generate a quantization coefficient.

In step S405, the inverse quantization / inverse conversion unit 106 reverse-quantizes / inverse-orthogonally transforms the quantization coefficient generated in step S404, and reproduces the prediction error.

In step S406, the image reproduction unit 107 reproduces the predicted image based on the prediction information generated in step S403. Further, the image data is reproduced from the reproduced predicted image and the predicted error generated in step S405.

In step S407, the coding unit 110 encodes the prediction information generated in step S403 and the quantization coefficient generated in step S404 to generate code data. It also generates a bitstream including other code data.

In step S408, the image coding apparatus determines whether or not the coding of all the blocks in the tile is completed, and if it is completed, proceeds to step S409, and if not, targets the next block. Then, the process returns to step S403.

In step S409, the image coding apparatus determines whether or not all the tiles in the frame have been coded, and if so, proceeds to step S410, and if not, targets the next tile. Then, the process returns to step S403.

In step S410, the in-loop filter unit 109 performs in-loop filter processing on the image data reproduced in step S406, generates a filtered image, and ends the processing.

FIG. 5 is a flowchart showing the details of the in-loop filter processing in step S410.

First, in step S501, the in-loop filter unit 109 determines whether or not the target pixel exists at the tile boundary from the position of the pixel targeted for the in-loop filter. If it is determined that it exists at the tile boundary, the process proceeds to S502, and if not, the process proceeds to S503. In S501, the meaning of being present at the tile boundary means that it is present at the tile boundary inside the image.

In step S502, the in-loop filter unit 109 determines whether or not the target pixel existing at the tile boundary is the target of filtering based on the filter control information generated in step S402. If it is determined that filtering is performed, the process proceeds to S503, and if not, the process proceeds to S504.

In step S503, the in-loop filter unit 109 performs an in-loop filter process on the target pixel.

In step S504, the in-loop filter unit 109 determines whether or not the in-loop filter processing of all the pixels is completed, and if it is completed, proceeds to step S505, and if not, targets the next pixel. Then, the process returns to step S501.

In step S505, the in-loop filter unit 109 determines whether or not all types of in-loop filter processing have been completed. If it is completed, the in-loop filter processing is terminated, and if not, the process returns to step S501 for the next type of in-loop filter processing.

For example, in HEVC, two in-loop filter processes such as a deblocking filter and a sample adaptive offset are defined, and switching between them is executed in this step. Specifically, first, a deblocking filter process for all pixels is performed, and then the process is switched to the sample adaptive offset process to return to step S501. When the sample adaptive offset processing is also completed, the in-loop filter processing is terminated. In the present embodiment, two in-loop filter processes such as a deblocking filter and a sample adaptive offset similar to HEVC are executed, but other in-loop filter processes (for example, an adaptive loop filter) may be executed. The order of in-loop filtering is not limited to these.

With the above configuration and operation, it is possible to control the applicability of the in-loop filter processing for each tile boundary, especially in step S410, so that the in-loop filter processing is applied to the tile boundary according to the continuity of the tile boundary. You can choose whether or not. As a result, the in-loop filter processing can be applied at the tile boundary where the pixels are continuous, and the in-loop filter processing cannot be applied at the tile boundary where the pixels are discontinuous, so that the image quality can be improved. it can.

In the present embodiment, as shown in FIG. 6A, the tile division information and the filter control information are encoded in the picture header portion, but the encoded position is not limited to this. As shown in FIG. 6B, it may be encoded at the sequence header portion of the image, or it may be encoded at another position.

In the present embodiment, a bitstream having a syntax structure as shown in FIG. 10A is encoded, but the bitstream structure is not limited to this. For example, it is possible to have a syntax configuration as shown in FIG. 10 (b). FIG. 10 (a) encodes the filter control information for each tile boundary, whereas FIG. 10 (b) shows a filter common to all horizontal tile boundaries and vertical tile boundaries. An example of encoding the control information is shown. loop_filter_accross_horizontal_tile_boundary_enable_flag is a flag indicating whether or not in-loop filtering is performed on all vertical tile boundaries in the image. That is, loop_filter_accross_horizontal_tile_boundary_enable_flag is horizontal control information common to a plurality of horizontal tile boundaries. Further, loop_filter_accross_vertical_tile_bound_enable_flag is a flag indicating whether or not to perform an in-loop filter on all horizontal tile boundaries in the image. That is, loop_filter_accross_vertical_tile_bound_enable_flag is vertical control information common to a plurality of vertical tile boundaries. For example, when the image shown in FIG. 9D is input, the former (... horizontal ... flag) is set to 1 and the latter (... vertical ... flag) is set to 0, so that the amount of code is smaller than that of the tile boundary. Filtering control can be realized. That is, in the case of an image in which a plurality of tiles are arranged in a certain direction (horizontal direction or vertical direction), the flag having the syntax configuration of FIG. 10B is an in-loop filter processing at the boundary of each tile. It is effective when the applicability is the same. Further, as a modification of FIGS. 10 (a) and 10 (b), the applicability of the in-loop filter processing may be set by using the flag of FIG. 10 (b) in one of the horizontal and vertical directions of the tile boundary. Good. Then, as shown in FIG. 3A, a flag may be assigned to each tile boundary in other directions to set whether or not the in-loop filter processing can be applied.

It is also possible to have a syntax configuration as shown in FIG. 10 (c). FIG. 10C is characterized in that loop_filter_acloss_tiles_control_idc is encoded as filter control information. loop_filter_accross_tiles_control_idc indicates an index corresponding to a specific tile arrangement method. For example, as filter control information, an index that associates "1" with FIG. 9 (b), "2" with FIG. 9 (c), "3" with FIG. 9 (d), and the like is encoded. In the syntax configuration of FIG. 10 (c), the arrangement of the six images combined to compose the one-frame image is defined in several formats, for example, FIGS. 9 (b) to 9 (d). It is effective when it is possible. In the case of the syntax configuration of FIG. 10 (c), the image decoding apparatus grasps whether or not the in-loop filter processing can be applied to the boundary of each tile in the arrangement of FIGS. 9 (b) to 9 (d) corresponding to each index. It is necessary to do it. As a result, it is possible to realize filtering control for the tile boundary with a smaller amount of code while supporting a plurality of tile arrangement methods.

Further, in the present embodiment, filtering for the boundary of the rectangular tile is controlled, but the control target is not limited to this. Filtering for slice boundaries that can take shapes other than rectangles can be controlled in the same way, and can be applied to new processing units other than tiles and slices.

Further, in the present embodiment, it is assumed that the tile has a rectangular shape based on the existing HEVC, but it can be applied even if the tile and other processing units take other shapes such as a triangle. is there.

FIG. 2 is a block diagram showing a configuration of an image decoding device according to an embodiment of the present invention. In this embodiment, the decoding process of the coded data generated by the image coding apparatus will be described. Each block in FIG. 2 may be configured entirely as hardware, or some or all blocks may be configured by software.

Terminal 201 is a terminal for inputting an encoded bit stream. The separation / decoding unit 202 separates the bitstream into information related to the decoding process and code data related to the coefficient, and further separates and decodes the code data existing in the header part of the bitstream. In the present embodiment, the tile division information and the filter control information are reproduced by the decoding process and output to the subsequent stage. That is, the separation / decoding unit 202 performs the reverse operation of the integrated coding unit 111 of FIG.

The decoding unit 203 decodes the code data output from the separation decoding unit 202 and reproduces the quantization coefficient and the prediction information.

Similar to the inverse quantization / inverse conversion unit 106 in FIG. 1, the inverse quantization / inverse conversion unit 204 inputs a quantization coefficient in block units, performs inverse quantization to obtain a conversion coefficient, and further inverse orthogonal conversion. To reproduce the prediction error.

Frame memory 207 A memory that stores images for at least one frame, and stores image data of the reproduced picture.

Similar to the image reproduction unit 107 of FIG. 1, the image reproduction unit 205 generates the prediction image data by appropriately referring to the frame memory 207 based on the input prediction information. Then, the reproduced image data is generated and output from the predicted image data and the predicted error reproduced by the inverse quantization / inverse conversion unit 204.

Similar to the in-loop filter unit 109 of FIG. 1, the in-loop filter unit 206 performs in-loop filter processing such as a deblocking filter on the reproduced image, and outputs the filtered image.

Terminal 208 outputs the filtered image data to the outside.

The image decoding process in the image decoding apparatus shown in FIG. 2 will be described below. In this embodiment, the bitstream generated by the image coding apparatus of FIG. 1 is decoded.

In FIG. 2, the bit stream input from the terminal 201 is input to the separation / decoding unit 202. The separation / decoding unit 202 separates the information related to the decoding process and the code data related to the coefficient from the bit stream, and further decodes the code data existing in the header unit of the bit stream. Specifically, the tile division information and the filter control information are reproduced by the decoding process. In the present embodiment, first, the code of the tile division information and the code of the filter control information are extracted from the picture header of the bitstream shown in FIG. 6A and decoded. In the following description, it is assumed that the picture header portion uses the syntax configuration shown in FIG. 10 (a). First, the tiles_enabled_flag code is decoded to obtain a value of 1, indicating that tile division is used. Since entry_coding_sync_enable_flag is not related to this embodiment, the description thereof will be omitted. Next, the number_tile_columns_minus1 code is decoded to obtain a value of 2, which indicates that three tiles are present in the horizontal direction. Further, the number_tile_rows_minus1 code is decoded to obtain a value of 1, which indicates that two tiles are present in the vertical direction. Subsequently, the uniform_spacing_flag code is decoded to obtain a value of 1 indicating that the sizes of the tiles (the number of pixels in the vertical and horizontal directions) match. Next, the loop_filter_aross_tiles_enabled_flag code is decoded to obtain a value of 1, indicating that in-loop filtering of tile boundaries is possible.

Since loop_filter_accross_tiles_enable_flag is 1, the syntax decoding is continued. The loop_filter_accross_tiles_control_flag code is decoded to obtain a value of 1, indicating that in-loop filtering is performed on each tile boundary.

Since loop_filter_accross_tiles_control_flag is 1, the syntax decoding is continued. The loop_filter_accross_bottom_tile_bound_enable_flag code is decoded to obtain information indicating whether or not in-loop filtering is performed on the tile boundary below each tile. In the present embodiment, the values 1, 0, and 0 are obtained as information for three tiles (front, right, and rear) having a tile boundary on the lower side, respectively. Next, the loop_filter_accross_right_tile_bound_enable_flag code is decoded to obtain information indicating whether or not the in-loop filter processing is performed on the tile boundary on the right side of each tile. In the present embodiment, a tile having a tile boundary on the right side obtains values of 1, 1, 0, and 0 for four tiles (front, right, bottom, and left) as information. The filter control information obtained in this way is output to the in-loop filter unit 206. Subsequently, the code data in block units of the picture data is reproduced and output to the decoding unit 203.

The decoding unit 203 decodes the code data and reproduces the quantization coefficient and the prediction information. The reproduced quantization coefficient is output to the inverse quantization / inverse conversion unit 204, and the reproduced prediction information is output to the image reproduction unit 205.

The inverse quantization / inverse conversion unit 204 performs inverse quantization on the input quantization coefficient to generate an orthogonal transformation coefficient, and further performs inverse orthogonal transformation to reproduce the prediction error. The reproduced prediction information is output to the image reproduction unit 205.

The image reproduction unit 205 appropriately refers to the frame memory 207 based on the prediction information input from the decoding unit 203, and reproduces the predicted image. The image data is reproduced from the predicted image and the prediction error input from the inverse quantization / inverse conversion unit 204, input to the frame memory 207, and stored. The stored image data is used as a reference when making a prediction.

Similar to 109 in FIG. 1, the in-loop filter unit 206 reads the reproduced image from the frame memory 207 and performs in-loop filter processing such as a deblocking filter based on the filter control information input from the separation / decoding unit 202. Then, the filtered image is input to the frame memory 207 again. Since the in-loop filter processing in the present embodiment is the same as the in-loop filter processing in the image coding apparatus of FIG. 1, detailed description thereof will be omitted.

The image filtered by the in-loop filter unit 206 is stored in the frame memory 207 again, and is finally output from the terminal 208 to the outside.

FIG. 7 is a flowchart showing an image decoding process in the image decoding apparatus of FIG.

First, in step S701, the separation / decoding unit 202 separates the bitstream into information related to the decoding process and code data related to the coefficient, decodes the code data of the header portion, and reproduces the tile division information and the filter control information.

In step S702, the decoding unit 203 decodes the code data separated in step S701 and reproduces the quantization coefficient and the prediction information.

In step S703, the inverse quantization / inverse conversion unit 204 performs inverse quantization on the quantization coefficient in block units to obtain a conversion coefficient, further performs inverse orthogonal transformation, and reproduces the prediction error.

In step S704, the image reproduction unit 205 reproduces the predicted image based on the prediction information generated in step S702. Further, the image data is reproduced from the reproduced predicted image and the predicted error generated in step S703.

In step S705, the image reproduction unit 205 or the control unit (not shown) of the image decoding device determines whether or not the decoding of all the blocks in the tile is completed. If it is completed, the process proceeds to step S706, and if not, the process returns to step S702 for the next block.

In step S706, the image reproduction unit 205 or the control unit (not shown) of the image decoding device determines whether or not the coding of all the tiles in the frame is completed. If it is completed, the process proceeds to step S707, and if not, the next tile is targeted and the process returns to step S702.

In step S707, the in-loop filter unit 206 performs in-loop filter processing on the image data reproduced in step S704, generates a filtered image, and ends the processing.

Since the flowchart showing the details of the in-loop filter processing is the same as FIG. 5 showing the processing in the image coding apparatus, the description thereof will be omitted.

With the above configuration and operation, it has become possible to select whether or not to apply the in-loop filter processing to the tile boundary according to the continuity of the tile boundary generated in the image coding apparatus of FIG. The bitstream can be decrypted.

In the present embodiment, as shown in FIG. 6A, the bit stream in which the tile division information and the filter control information are included in the picture header portion is decoded, but the coding position of the information is this. Not limited to. As shown in FIG. 6B, it may be encoded at the sequence header portion of the image, or it may be encoded at another position.

Although the present embodiment shows the decoding of a bitstream having a syntax configuration as shown in FIG. 10A, the configuration of the bitstream to be decoded is not limited to this. For example, a bitstream having a syntax configuration as shown in FIG. 10B may be decoded. FIG. 10B shows an example in which common filter control information is encoded for all horizontal tile boundaries and vertical tile boundaries. For example, when the image shown in FIG. 9D is encoded by the coding apparatus of the first embodiment and the bit stream generated is decoded, the former (... horizontal ... flag) is decoded and the value is 1, the latter. (... virtual ... flag) is decoded to obtain a value of 0. As a result, the filter control information that the in-loop filter processing is performed on all the horizontal tile boundaries and the in-loop filter processing is not performed on all the vertical tile boundaries is encoded with a smaller amount of code. The resulting bitstream can be decrypted.

Furthermore, it is also possible to decode a bit stream having a syntax configuration as shown in FIG. 10 (c). In FIG. 10 (c), the index indicated by loop_filter_acloss_tiles_control_idc can be associated with a particular tile placement method. For example, 1 is associated with FIG. 9 (b), 2 is associated with FIG. 9 (c), 3 is associated with FIG. 9 (d), and the like. As a result, it is possible to decode a bit stream that realizes filtering control for tile boundaries with a smaller amount of code while supporting a wider variety of tile arrangement methods.

Each of the processing units shown in FIGS. 1 and 2 has been described in the above embodiment as being configured by hardware. However, the processing performed by each processing unit shown in these figures may be configured by a computer program.

FIG. 8 is a block diagram showing a configuration example of computer hardware applicable to the image coding device and the image decoding device according to each of the above embodiments.

The CPU 801 controls the entire computer by using the computer programs and data stored in the RAM 802 and the ROM 803, and also executes the above-described processes as those performed by the image processing apparatus according to each of the above embodiments. That is, the CPU 801 functions as each processing unit shown in FIGS. 1 and 2.

The RAM 802 has an area for temporarily storing computer programs and data loaded from the external storage device 806, data acquired from the outside via the I / F (interface) 807, and the like. Further, the RAM 802 has a work area used by the CPU 801 to execute various processes. That is, the RAM 802 can be allocated as a frame memory, or can provide various other areas as appropriate.

The ROM 803 stores the setting data of the computer, the boot program, and the like. The operation unit 804 is composed of a keyboard, a mouse, and the like, and can be operated by a user of the computer to input various instructions to the CPU 801. The display unit 805 displays the processing result by the CPU 801. The display unit 805 is composed of, for example, a liquid crystal display.

The external storage device 806 is a large-capacity information storage device typified by a hard disk drive device. The external storage device 806 stores an OS (operating system) and a computer program for realizing the functions of the respective parts shown in FIGS. 1 and 2 in the CPU 801. Further, each image data as a processing target may be stored in the external storage device 806.

The computer programs and data stored in the external storage device 806 are appropriately loaded into the RAM 802 according to the control by the CPU 801 and become the processing target by the CPU 801. A network such as a LAN or the Internet, or other devices such as a projection device or a display device can be connected to the I / F807, and the computer acquires and sends various information via the I / F807. Can be done. 808 is a bus connecting the above-mentioned parts.

The processing of the flowchart shown in FIGS. 4, 5, and 7 described above reads the program stored in the ROM 803 into the RAM 802, and is executed by the CPU based on the program.

As described above, the present invention makes it possible to select whether or not to apply the in-loop filtering process for each tile boundary. Therefore, it is possible to improve the image quality of continuous tile boundaries in particular, and it is possible to further improve the coding efficiency.

(Other Examples)
The present invention supplies a system or device with a program that realizes one or more of the functions of the above-described embodiment via a network or a storage medium. It can also be realized by a process in which one or more processors in the computer of the system or device reads and executes a program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The present invention is used in a coding device / decoding device that encodes / decodes a still image / moving image. In particular, it can be applied to coding and decoding methods that use tile division and in-loop filtering.

101, 112, 201, 208 terminals 102 Tile division unit 103 Filter control information generation unit 104 Prediction unit 105 Conversion / quantization unit 106, 204 Inverse quantization / inverse conversion unit 107, 205 Image reproduction unit 108, 207 Frame memory 109, 206 In-loop filter section 110 Coding section 111 Integrated coding section 202 Separation decoding section 203 Decoding section

Claims (18)

In a coding device for coding an image having a plurality of tiles,
A determination means for determining whether or not to filter pixels adjacent to a plurality of boundaries composed of the plurality of tiles.
Having a coding means for encoding at least two of the plurality of boundaries, control information indicating whether or not to filter pixels adjacent to the boundary based on the determination by the determining means. A coding device characterized by. The coding apparatus according to claim 1, wherein the plurality of tiles have a rectangular shape and form at least two boundaries. The coding device according to claim 1, wherein the coding means encodes the control information for each boundary formed between two tiles. As control information, the coding means includes horizontal control information indicating whether or not to filter pixels adjacent to the horizontal boundary and vertical indicating whether or not to filter pixels adjacent to the vertical boundary. The coding apparatus according to claim 1, wherein the control information of the direction is encoded. The coding device according to claim 4, wherein the control information in the horizontal direction is control information common to a plurality of tile boundaries in the horizontal direction. The coding device according to claim 4, wherein the control information in the vertical direction is control information common to a plurality of tile boundaries in the vertical direction. The coding device according to claim 1, wherein the control information is an index corresponding to an arrangement of a plurality of images synthesized to form an image of one frame. In a decoding device for decoding an image having a plurality of tiles from a bit stream,
Decoding means to decode the image and
A generation means for generating control information from a bit stream indicating whether or not to filter pixels adjacent to the boundary with respect to a plurality of boundaries composed of the plurality of tiles.
A determination means for determining whether to filter pixels adjacent to at least a plurality of boundaries based on the control information generated by the generation means.
A decoding device comprising a processing means for filtering a boundary determined to be filtered by the determination means. The decoding device according to claim 8, wherein the plurality of tiles have a rectangular shape and form at least two boundaries. The decoding device according to claim 8, wherein the generation means generates the control information for each boundary formed between two tiles. As control information, the generation means has horizontal control information indicating whether or not to filter pixels adjacent to the horizontal boundary and vertical direction indicating whether or not to filter pixels adjacent to the vertical boundary. The decoding device according to claim 8, wherein the control information of the above is generated. The decoding device according to claim 11, wherein the control information in the horizontal direction is control information common to a plurality of tile boundaries in the horizontal direction. The decoding device according to claim 11, wherein the control information in the vertical direction is control information common to a plurality of tile boundaries in the vertical direction. The decoding device according to claim 8, wherein the control information is an index corresponding to an arrangement of a plurality of images synthesized to form an image of one frame. In a coding method for coding an image having multiple tiles,
A determination step for determining whether or not to filter pixels adjacent to the plurality of boundaries composed of the plurality of tiles, and a determination step.
Having a coding step that encodes control information indicating whether or not to filter pixels adjacent to a boundary based on the decision in the determination step for at least two of the plurality of boundaries. A coding method characterized by. In a decoding method for decoding an image having multiple tiles from a bitstream,
Decoding step to decode the image and
A generation step of generating control information from a bit stream indicating whether or not to filter pixels adjacent to the boundary with respect to the plurality of boundaries composed of the plurality of tiles, and a generation step.
A determination step for determining whether to filter pixels adjacent to at least a plurality of boundaries based on the control information generated in the generation step, and a determination step.
A decoding method comprising a processing step for filtering a boundary determined to be filtered in the determination step. A program for causing a computer to execute the processing of the coding apparatus according to any one of claims 1 to 7. A program for causing a computer to execute the processing of the decoding device according to any one of claims 8 to 14.

Images