JP2021182728A - Image encoding device and image decoding device, method, and program - Google Patents

Abstract

To reduce the amount of information in parameter information more than before.SOLUTION: An image encoding device that encodes an image composed of N (N>1) rectangles so that each rectangle can be decoded independently includes a segmentation unit for dividing an image into N rectangles and an encoding unit that encodes position and size information of the rectangles. Here, the encoding unit encodes the position information of second to N-first rectangles and does not encode position information of first and Nth rectangles.SELECTED DRAWING: Figure 15

Description

The present invention relates to a moving image coding technique.

An image pickup device represented by a digital video camera or the like is described by H. 264 (Non-Patent Document 1), H. et al. The moving image data obtained by imaging is encoded by using a compression coding technique such as 265 (Non-Patent Document 2). In this coding, the coded data obtained by coding one picture in units of blocks composed of a plurality of pixels and the parameters related to the reproduction of the blocks are stored in SPS (Sequence Parameter Set) and PPS (Picture Parameter Set). Generate a bitstream that is multiplexed with the data.

In recent years, as a successor to HEVC, activities to carry out international standardization of more efficient coding methods have been started. JVET (Joint Video Experts Team) was established between ISO / IEC and ITU-T, and is being standardized as a VVC (Versatile Video Coding) coding method (hereinafter referred to as VVC). In VVC, there are subpictures that are configured to contain one or more slices and form a rectangle. This sub-picture can be divided into one or more, and each sub-picture can be processed as independent code data.

ITU-T Rec. H.264 Edition 8.0 ITU-T Rec. H.265 Edition4.0 (V4)

However, the parameter information to be multiplexed has a redundant part, and there is room for improvement.

In order to solve this problem, for example, the image coding apparatus of the present invention has the following configuration. That is,
An image coding device that encodes an image composed of N (N> 1) rectangles so that each rectangle can be independently decoded.
A dividing means for dividing an image into N rectangles, and
It has a coding means for encoding information on the position and size of the rectangle.
The coding means encodes information about the positions of the second to N-1th rectangles and does not encode information about the positions of the first and Nth rectangles.

According to the present invention, the amount of parameter information can be reduced more than before.

The figure which shows the relationship between CTU, tile, and slice. The figure which shows the relationship between CTU, tile, and slice. The figure which shows the relationship between a CTU, a tile, and a subpicture. The figure which shows the syntax of SPS in an embodiment. The figure which shows the syntax of SPS in an embodiment. The figure which shows the syntax of SPS in an embodiment. The figure which shows the syntax of SPS in an embodiment. The figure which shows the syntax of SPS in an embodiment. The figure which shows the syntax of SPS in an embodiment. The figure which shows the syntax of PPS in an embodiment. The figure which shows the syntax of PPS in an embodiment. The figure which shows the syntax of PPS in an embodiment. The figure which shows the syntax of the slice header in an embodiment. The figure which shows the syntax of PPS in an embodiment. The figure which shows the syntax of PPS in an embodiment. The block block diagram of the image pickup apparatus (image coding apparatus) to which an embodiment applies. The block block diagram of the image decoding apparatus to which an embodiment applies. The block diagram which shows the structure of the image decoding apparatus in embodiment. The flowchart which shows the image coding processing in the image coding apparatus in embodiment. The flowchart which shows the image decoding processing in the image decoding apparatus in embodiment. The figure which shows the bit stream structure used in embodiment. The figure which shows an example of image division. The figure which shows the division of the image used in an embodiment. The block diagram which shows the structure of the image coding apparatus in embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicate explanations are omitted.

FIG. 7 is a diagram showing a block configuration of the image pickup apparatus 100 to which the embodiment is applied. The image pickup device 100 includes a CPU 101, a memory 102, a non-volatile memory 103, an operation unit 104, an image pickup lens 111, an image pickup unit 112, an image processing unit 113, a coding processing unit 114, a display control unit 115, a display unit 116, and a communication control unit. It has 117, a communication unit 118, a recording medium control unit 119, a recording medium 120, a detection unit 140, and an internal bus 130.

The CPU 101 controls the operation of each part of the image pickup apparatus 100 by executing a computer program stored in the non-volatile memory 103. The memory 102 is a rewritable volatile memory (RAM), and is a computer program that temporarily controls the operation of each part of the image pickup apparatus 100, information such as parameters related to the operation of each part, information received by the communication control unit 117, and the like. Remember. The memory 102 also functions as a work memory for temporarily storing images and information processed by the image pickup unit 112, the image processing unit 113, the coding processing unit 114, and the like.

The non-volatile memory 103 is a memory that can be electrically erased and recorded, and a storage medium such as an EEPROM or an SD memory card is used. The non-volatile memory 103 stores information such as a computer program that controls the operation of each part of the image pickup apparatus 100 and parameters related to the operation of each part. The computer program referred to here includes a program for executing various processes described later in the present embodiment.

The operation unit 104 provides a user interface for operating the image pickup apparatus 100. The operation unit 104 has a power button, a menu button, a shutter button, a mode switching button, and the like of the image pickup apparatus 100, and each button is composed of a switch, a touch panel, and the like. The CPU 101 controls the image pickup apparatus 100 according to a user's instruction input via the operation unit 104. The operation unit 104 may control the image pickup apparatus 100 in response to a remote control signal received from a remote controller (not shown) or a request notified from a mobile terminal (not shown) via the communication control unit 117.

The image pickup lens 111 is composed of a zoom lens, a lens group including a focus lens, a lens control unit, an aperture, and the like. The image pickup lens 111 has a lens control unit (not shown), and controls focus adjustment and aperture value (F value) by a control signal transmitted from the CPU 101. The image pickup unit 112 includes an image pickup element that converts an optical image of a subject into an electric signal. The image pickup device is an area image sensor composed of, for example, a CCD (charge-coupled device), a CMOS (complementary metal oxide semiconductor) device, or the like. The image pickup unit 112 takes an image at a frame rate of, for example, 30 FPS, and outputs the image obtained by this image pickup to the image processing unit 113 or the memory 102.

The image processing unit 113 performs resizing processing such as predetermined pixel interpolation and reduction with respect to the data output from the imaging unit 112 or the data read from the memory 102, image addition for matching the aspect ratio, and color conversion processing. And so on. Further, in the image processing unit 113, a predetermined calculation process is performed using the captured image data, and the CPU 101 performs exposure control and distance measurement control based on the obtained calculation result. As a result, AE (automatic exposure) processing, AWB (auto white balance) processing, and AF (autofocus) processing are performed.

Further, the image processing unit 113 aligns the image data, synthesizes a plurality of aligned images, and performs wide-angle composition to generate a wide-angle composite image. Details of wide-angle synthesis in this embodiment will be described later.

The coding processing unit 114 divides the input image data into a plurality of rectangular areas according to the settings from the CPU 101, and compresses and encodes the image in each area. The coding processing unit 114 in the embodiment shall perform compression processing according to, for example, the VVC method. However, the points to be described later are different from the VVC method proposed at this stage.

The display control unit 115 is a control unit for controlling the display unit 116. The display control unit 115 performs resizing processing, color conversion processing, and the like so that the image can be displayed on the display unit 116, and outputs the image signal to the display unit 116.

The display unit 116 is composed of a liquid crystal display, an organic EL, or the like, and displays an image based on the image data from the display control unit 115. A touch panel is provided on the front surface of the display unit 116, which also serves as an operation unit for receiving operations from the user.

The communication control unit 117 is controlled by the CPU 101, generates a modulation signal applied to a wireless communication standard predetermined by 802.11 or the like, and outputs the modulation signal to the communication unit 118. Further, the communication control unit 117 receives the modulation signal applied to the wireless communication standard from the communication unit 118 and decodes it, thereby converting the analog signal into a digital signal and notifying the CPU 101. Further, the communication control unit 117 has a register for setting communication, and by being controlled by the CPU 101, the transmission / reception sensitivity at the time of communication can be adjusted, and transmission / reception can be performed by a predetermined modulation method. The communication unit 118 is composed of an antenna that outputs a modulation signal sent from the communication control unit 117 to the outside or receives a modulation signal from the outside, an analog circuit, and the like.

The recording medium control unit 119 is a control unit for controlling the recording medium 120, and outputs a control signal for controlling the recording medium 120 in response to a request from the CPU 101. The recording medium 120 is composed of a removable or built-in non-volatile memory, a magnetic disk, or the like for recording image data captured and encoded. When the CPU 101 records on the recording medium 120, it is saved as file data in a format suitable for the file system of the recording medium.

The detection unit 140 includes a GPS sensor, a gyro sensor, an acceleration sensor, and the like.

The internal bus 130 is an internal bus for each processing unit to access the CPU 101 and the memory 102.

In the above configuration, the CPU 101 controls the coding processing unit 114, divides the picture into a plurality of rectangular areas, and causes the coding processing of each rectangular area to be performed. Further, the CPU 101 generates information (SPS and the like described later) for reproducing each rectangular area. Then, the CPU 101 multiplexes the coded data of the image generated by the coding processing unit 114 and the information such as SPS to generate a bit stream. Then, the CPU 101 controls the recording medium control unit 119 and records the moving image file on the recording medium 120.

Next, the coding process by the coding process unit 114 and the division of the rectangular area in the embodiment will be described. Under the control of the CPU 101, the coding processing unit 114 divides one picture (frame image) to be coded into one or more tile rows and one or more tile columns, and performs coding.

Here, one tile is an aggregate of blocks (CTU: Coding Tree Unit) covering a rectangular area composed of a plurality of pixels. Further, the slice in the following description means an area composed of a plurality of tiles or an area of one or more CTU rows in one tile.

1 and 2 are examples of the relationship between CTU, Tile, and Slice. In the example of FIG. 1, one picture is divided into tiles of 4 rows and 6 columns, and a total of 9 slices of 3 × 3 that collectively cover the tiles are defined. Further, in FIG. 2, one picture is divided into four tiles of 2 rows and 2 columns, the upper right tile is divided into two slices, and two vertically adjacent tiles on the left side form one slice. Then, the tile at the lower right becomes one slice, and an example in which a total of four slices are defined is shown.

Further, in the embodiment, a region composed of one or more slices collectively covering a rectangular region in a frame image and which can be independently decoded is referred to as a subpicture. FIG. 3 shows an example of the relationship between the CTU, tiles, and subpictures. The figure shows an example in which one picture is divided into 18 tiles in 3 rows and 6 columns, and a total of 24 slices and sub-pictures that collectively cover each tile are set.

It should be noted that the CPU 101 is responsible for setting how many pictures are divided into sub-pictures and what size they are divided into sub-pictures. This setting is determined based on, for example, which of the color patterns registered in advance is the closest to the color distribution derived from the image taken prior to recording the moving image.

In VVC, information such as parameters for realizing picture division using tiles, slices, and sub-pictures (information for reproduction for the decoding side) as shown in FIGS. 1 to 3 is encoded in the image. It is supposed to be stored in the header part (SPS: Sequence Parameter Set) of the sequence of the bitstream obtained in 1 and the header part (PPS: Picture Parameter Set) for the picture.

For example, the SPS stores sps_subpic_ctu_top_left_x and y-coordinates sps_subpic_ctu_top_left_y indicating the upper left end of each sub-picture. However, for example, the coordinates of the upper left of the last sub-picture located at the lower right of the picture can be calculated from the coordinates of each of the previous sub-pictures. That is, it can be said that the information indicating the position related to the last sub-picture is redundant.

Therefore, the CPU 101 of the present embodiment generates an SPS in which the parameter information related to the last sub-picture is omitted, and the amount of information of the SPS is reduced more than before.

4A-4F show the syntax of the SPS information generated in the embodiment. It should be noted that FIGS. 4A to 4F represent the syntax of SPS by making them continuous in this order.

"Sps_num_subpics_minus1" in the description of the lines of reference numerals 401 and 402 shown represents "the number of sub-pictures included in one picture-1". Then, by newly adding the condition "i <sps_num_subpics_minus1", when the number of sub-pictures included in one picture is N, the positions of N-2 sub-pictures, which is one less than that, {sps_subpic_ctu_top_left_x [i]. ], Sps_subpic_ctu_top_left_y [i]} and the size {sps_subpic_width_minus1 [i], sps_subpic_hight_minus1 [i]} are included in the SPS. In the present embodiment, the conditional expression is added to the lines of reference numerals 401 and 402, but the present invention is not limited thereto. For example, the conditional expression of the for statement in the line above 401 may be changed from i <= sps_num_subpics_minus1 to i <sps_num_subpics_minus1.

On the decoding side, in the process of decoding according to the SPS syntax shown in FIGS. 4A to 4F, the position and size of the last (Nth) sub-picture are automatically set from the decoding results of the previous sub-pictures. Since it can be obtained, the image of the sub-picture can be reproduced according to the obtained position, so that no problem occurs.

CtbSizeY, which is the number of vertical and horizontal pixels of the CTU, is represented by 1 << CtbSize (here, CtbSize = sps_log2_ctu_size_minus5 + 5). PicCtbNumH, which is the number of CTUs in the horizontal direction of the image, can be obtained by (sps_pic_width_max_in_luma_samples + CtbSizeY-1) >> CtbLog2SizeY. Similarly, PicCtbNumV, which is the number of CTUs in the vertical direction, can be obtained by (sps_pic_height_max_in_luma_samples + CtbSizeY-1) >> CtbLog2SizeY. The number of vertical and horizontal pixels of the basic block (CTU), CtbSizeY, is indicated by 1 << CtbSize (here, CtbLog2SizeY = sps_log2_ctu_size_minus5 + 5). Further, in the following description, the basic block will be described as being a CTU, but the basic block is not necessarily limited to that. Moreover, the size of the basic block and the CTU shown in each embodiment is an example. The size of the area such as tiles and slices shown in each embodiment is also an example.

sps_num_subpics_minus The derivation of the coordinates of the CTU on the upper left of the first (last) subpicture will be described. First, how to obtain the X coordinate will be described. First, the subpicture adjacent to the left of the last subpicture is obtained. Among the sub-pictures including the CTU at the lower end of the image, the sub-picture having the maximum X coordinate at the right end CTU of the sub-picture is the sub-picture adjacent to the left of the last sub-picture. First, the sub-picture for which the following equation holds is a sub-picture including the CTU at the lower end.
PicCtbNumV == sps_subpic_ctu_top_left_y [i] + sps_subpic_height_minus1 [i] + 1

Among all the sub-pictures satisfying the above conditions in the image, the sub-picture located on the rightmost side, that is, the sub-picture having the largest sps_subpic_ctu_top_left_x [i] is the sub-picture adjacent to the left side. Therefore, assuming that the sub-picture adjacent to the left of the last sub-picture is the Nth sub-picture, the X coordinate of the CTU on the upper left of the last sub-picture can be obtained by the following equation.
sps_subpic_ctu_top_left_x [N] + sps_subpic_width_minus1 [N] + 1

Next, how to obtain the Y coordinate of the CTU on the upper left of the last subpicture will be described. First, the sub-pictures adjacent to the last sub-picture are obtained. Among the sub-pictures including the CTU at the right end of the image, the sub-picture having the maximum Y coordinate of the CTU at the lower end of the sub-picture is the sub-picture adjacent to the last sub-picture. First, the sub-picture for which the following equation holds is a sub-picture including the CTU at the right end.
PicCtbNumH == sps_subpic_ctu_top_left_x [i] + sps_subpic_width_minus1 [i] + 1

Among all the sub-pictures satisfying the above conditions in the image, the sub-picture located at the bottom, that is, the sub-picture having the largest sps_subpic_ctu_top_left_y [i] is the sub-picture adjacent to the top. Therefore, assuming that the sub-picture adjacent to the last sub-picture is the M-th sub-picture, the Y coordinate of the CTU on the upper left of the last sub-picture can be obtained by the following equation.
sps_subpic_ctu_top_left_y [M] + sps_subpic_height_minus1 [M] + 1

Further, the SPS, PPS, and slice header in the conventional techniques employ a description that allows two or more sub-pictures to exist in one picture. In other words, so far, information that is meaningful only when two or more sub-pictures exist in one picture is included.

However, there are cases where only one sub-picture is included in one picture, and in such a case, the SPS, PPS, and slice header contain redundant information, which means that the amount of code is reduced. There is still room for improvement from the point of view.

Therefore, in the present embodiment, the lines of reference numerals 403 and 404 in FIGS. 4A to 4F showing the syntax of SPS are considered to be valid only when two or more subpictures are present in one picture {}. If the syntax in is encoded and there is only one subpicture in one picture, the syntax in {} is not encoded.

Further, the pps_no_pic_partition_flag present in the line above the line 501 in FIG. 5A is a flag indicating whether or not the image is divided into subpictures, tiles, and slices. When the syntax is 1, it indicates that the image is composed of a single subpicture, tile, or slice. The line indicated by reference numeral 501 in the syntax of PPS (Picture Parameter Set) shown in FIGS. 5A to 5C (in this order representing the syntax of PPS), and the condition "&& (pps_num_subpics_minus1" in the line indicated by reference numeral 503). > 0 ”is encoded as valid only when there are two or more subpictures in one picture, and the syntax in {} is encoded, and when there is only one subpicture in one picture. The syntax in {} is not encoded in.

Then, the condition "sps_num_subpix_minus1> 0" in the line indicated by the reference code 601 in the syntax of the slice header shown in FIGS. 6A to 6C (in this order, the syntax of the slice header is represented) is two or more in one picture. The syntax in {} is encoded as valid only if a subpicture is present. Then, when only one sub-picture exists in one picture, the syntax in {} is not encoded.

As a result of the above, the CPU 101 encodes the index number that identifies the sub-picture only when there are two or more sub-pictures in one picture. Then, when only one sub-picture exists in one picture, the CPU 101 does not generate the code of the index number of the sub-picture, and the code amount can be reduced by that amount.

As a result of the above, the amount of information in the SPS, PPS, and slice header can be reduced, especially when the number of sub-pictures contained in one picture is one.

In the above embodiment, an example of applying to an image pickup device as an image coding device has been described, but the device is limited to the image pickup device as long as it is a device that inputs or receives an image to be coded and encodes it. It's not something.

Further, in the embodiment, the image pickup apparatus has been described as having the configuration of FIG. 7, but the CPU 101 executes a program for some configurations of the figure, such as the image processing unit 113 and the coding processing unit 114. It doesn't matter if it is realized.

Next, the image coding apparatus described so far will be described in more detail with reference to FIGS. 10 and 12 to 15. In the following description, the image pickup process performed by the image pickup unit 112 and the like will be omitted, and further details of the process described mainly as the process in the coding process unit 114 will be described.

FIG. 15 is a block diagram showing an image coding apparatus of this embodiment. FIG. 15 is a functional block diagram showing the coding processing unit 114 and the like in more detail.

This device has an image division unit 802, a block division unit 803, a prediction unit 804, a conversion / quantization unit 805, an inverse quantization / inverse conversion unit 806, an image reproduction unit 807, a frame memory 808, an in-loop filter unit 809, and a code. It has a quantization unit 810 and an integrated coding unit 811.

The image dividing unit 802 divides the input image input via the input terminal 801 into one or a plurality of tile rows and one or a plurality of tile columns. A tile is a set of continuous basic blocks (CTUs: Coding Tree Units) that cover a rectangular area in an image. The image dividing unit 802 further divides the image into slices. A slice is composed of one or more tiles in an image, or is composed of one or more basic block lines in one tile. A slice is a basic unit of coding, and header information such as information indicating a slice type is added to each slice. The image dividing unit 802 further defines one or more sub-pictures of the input image. A subpicture is composed of a set of multiple slices (rectangular shape). FIG. 13 shows an example of dividing the input image into 4 tiles, 4 subpictures, and 11 slices. The upper left tile is divided into 1 slice, the lower left tile is divided into 2 slices, the upper right tile is divided into 5 slices, and the lower right tile is divided into 3 slices. The left subpicture is configured to contain 3 slices, the upper right subpicture is configured to contain 2 slices, the right center subpicture is configured to contain 3 slices, and the lower right slice is configured to contain 3 slices. ..

The block division unit 803 divides a line image composed of basic blocks output from the image division unit 802 into a plurality of basic blocks, and outputs an image in units of basic blocks to a subsequent stage.

The prediction unit 804 determines sub-block division for the image data in the basic block unit, performs intra-frame prediction such as intra-frame prediction and inter-frame prediction in the sub-block unit, and generates prediction image data. .. Here, the prediction unit 804 does not perform intra-prediction or motion vector prediction across sub-pictures. Further, the prediction unit 804 calculates and outputs a prediction error from the input image data and the predicted image data. Further, the prediction unit 804 also outputs information necessary for prediction, for example, information such as sub-block division, prediction mode, motion vector, and the like together with the prediction error. Hereinafter, the information necessary for this prediction is referred to as prediction information.

The conversion / quantization unit 805 orthogonally transforms the prediction error in subblock units to obtain a conversion coefficient, and further performs quantization to generate a quantization coefficient.

The inverse quantization / inverse conversion unit 806 inversely quantizes the quantization coefficient output from the conversion / quantization unit 805 to reproduce the conversion coefficient, and further performs inverse orthogonal conversion to reproduce the prediction error.

Based on the prediction information output from the prediction unit 804, the image reproduction unit 807 generates the prediction image data by appropriately referring to the frame memory 808, and generates the reproduction image data from the input prediction error. The image reproduction unit 807 re-stores the generated reproduced image data in the frame memory 807.

The in-loop filter unit 809 performs in-loop filter processing such as a deblocking filter and a sample adaptive offset on the reproduced image data stored in the frame memory 808, and re-stores the filtered image data in the frame memory 808. do.

The coding unit 810 encodes the quantization coefficient output from the conversion / quantization unit 804 and the prediction information output from the prediction unit 804, and generates and outputs code data.

The integrated coding unit 811 receives the division information from the image division unit 802 and generates the header code data. Further, the integrated coding unit 811 forms a bit stream together with the code data output from the coding unit 810, and outputs the formed bit stream via the output terminal 812.

The image coding operation in the image coding device according to the embodiment will be described below. In the present embodiment, the moving image data is input in frame units, but the still image data for one frame may be input. Further, in the present embodiment, for the sake of facilitation of explanation, only the intra-predictive coding process will be described, but the present invention is not limited to this and can be applied to the inter-predictive coding process. Further, in the present embodiment, for the sake of explanation, the block division unit 803 will be described as being divided into basic blocks of 64 × 64 pixels, but this is merely an example and is not limited thereto.

In the present embodiment, first, one frame of image data input from the terminal 801 is divided into tiles and slices in the image dividing unit 802 as shown in FIGS. 14A and 14B, and a sub-picture is defined. .. FIG. 14A is a diagram showing tile division and ID. In this embodiment, the image data is divided into nine tiles of 384 × 384 pixels. The tiles are assigned IDs in raster order from the upper left, the upper left tile ID is 0, and the lower right tile ID is 8.

FIG. 14B shows definitions of tiles and subpictures and an example of dividing slices. The tiles with tile ID 0 and tile ID 7 are further divided into two slices of 384 × 192 pixels. The tile with tile ID 2 is divided into two slices of 384 × 128 and 384 × 256 pixels. The tile with tile ID 3 is divided into three slices of 384 × 128. The other tiles consist of a single slice. IDs are assigned to each slice in order from the top in the tiles in raster order. The SID shown in FIG. 14B is the ID of the slice. The sub-pictures are sub-picture 0 including SID = 0 to 2, sub-picture 1 including SID = 3, sub-picture 2 including SID = 4, and sub-picture 3 including SID = 5-8 and 10-12, respectively. It is defined as a sub-picture 4 containing SID = 9 and 13. Information on the sizes of tiles, subpictures, and slices is sent to the integrated coding unit 811 as division information. Further, each slice is divided into basic block line images, which are image data for each basic block line, and sent to the block division unit 803.

The block division unit 803 divides the line image composed of the input basic blocks into a plurality of basic blocks, and outputs the image of each basic block unit to the prediction unit 804. The block division unit 803 of the present embodiment outputs an image in basic block units of 64 × 64 pixels.

The prediction unit 804 executes prediction processing on the image data of the basic block unit input from the block division unit 803. Specifically, the sub-block division that divides the basic block into smaller sub-blocks is determined, and the intra-prediction mode such as horizontal prediction or vertical prediction is determined for each sub-block.

The prediction unit 804 generates prediction image data from the determined intra prediction mode and encoded pixels, further generates a prediction error which is the difference between the input image data and the prediction image data, and converts / quantizes the prediction unit. Output to 805. Further, the prediction unit 804 supplies information such as sub-block division and intra prediction mode as prediction information to the coding unit 810 and the image reproduction unit 807.

The conversion / quantization unit 805 performs orthogonal transformation and quantization with respect to the input prediction error, and generates a quantization coefficient. The conversion / quantization unit 805 first performs orthogonal conversion processing corresponding to the size of the subblock to generate an orthogonal conversion coefficient. Next, the conversion / quantization unit 805 quantizes the orthogonal transformation coefficient and generates the quantization coefficient. Then, the conversion / quantization unit 805 outputs the generated quantization coefficient to the coding unit 810 and the inverse quantization / inverse conversion unit 806.

The inverse quantization / inverse conversion unit 806 inversely quantizes the input quantization coefficient and reproduces the conversion coefficient. The inverse quantization / inverse transformation unit 806 further performs inverse orthogonal transformation of the reproduced conversion coefficient to reproduce the prediction error. Then, the inverse quantization / inverse transformation unit 806 outputs the reproduced prediction error to the image reproduction unit 807.

The image reproduction unit 807 appropriately refers to the frame memory 808 based on the prediction information input from the prediction unit 804, and reproduces the predicted image. Then, the image reproduction unit 807 reproduces the image data from the reproduced predicted image and the prediction error input from the inverse quantization / inverse conversion unit 806, and stores the reproduced image in the frame memory 808.

The in-loop filter unit 809 reads the reproduced image from the frame memory 808 and performs in-loop filter processing such as a deblocking filter. Then, the in-loop filter unit 809 re-stores the filtered image in the frame memory 808.

The coding unit 810 entropy-codes the quantization coefficient generated by the conversion / quantization unit 805 and the prediction information input from the prediction unit 804 in block units to generate code data. The type of entropy coding is not particularly limited, but Golomb coding, arithmetic coding, Huffman coding, and the like can be used. The coding unit 810 outputs the generated code data to the integrated coding unit 811.

The integrated coding unit 811 receives the division information from the image division unit 802 and generates the coding data of the header. Then, the integrated coding unit 811 multiplexes the coded data of the header and the coded data received from the coding unit 810 to form a bit stream. Then, the integrated coding unit 811 outputs the formed bit stream to the outside via the output terminal 812.

FIG. 12 shows the format of the VVC-encoded data encoded by the image coding apparatus. In the coded data of FIG. 12, first, there is a sequence parameter set (SPS) which is header information including information related to the coding of the sequence. Further, a picture parameter set (PPS) which is header information including information related to picture coding, a slice header which is header information including information related to slice coding, and coding data of each slice. Followed by.

In the sequence parameter set (SPS), sps_pic_width_max_in_luma_samples and sps_pic_height_max_in_luma_samples are present as image size information. These represent the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the brightness of the image, respectively. In this embodiment, since the image of FIG. 14 is encoded, sps_pic_width_max_in_luma_samples is "1152" and sps_pic_height_max_in_luma_samples is "1152". Further, as basic block division information, sps_log2_ctu_size_minus5 indicating the size of the basic block exists. As described above, CtbSizeY, which is the number of vertical and horizontal pixels of the basic block, is indicated by 1 << CtbSize (here, CtbLog2SizeY = sps_log2_ctu_size_minus5 + 5). PicCtbNumH, which is the number of basic blocks in the horizontal direction of the image, can be obtained by (sps_pic_width_max_in_luma_samples + CtbSizeY-1) >> CtbLog2SizeY. Similarly, PicCtbNumV, which is the number of basic blocks in the vertical direction, can be obtained by (sps_pic_height_max_in_luma_samples + CtbSizeY-1) >> CtbLog2SizeY. In this embodiment, since the basic block has 64 × 64 pixels, the value of sps_log2_ctu_size_minus5 is 1.

Further, as the sub-picture definition information S (SPS), there is sps_subpic_info_present_flag indicating whether or not there is information regarding the definition of the sub-picture. When sps_subpic_info_present_flag is 1, it indicates that the integrated coding unit 811 encodes the information regarding the definition of the sub-picture, and when it is 0, it indicates that it is not encoded. When the sps_subpic_info_present_flag is 1, the integrated coding unit 811 encodes sps_num_subpics_minus1 representing the number of sub-pictures-1. Further, the integrated coding unit 811 encodes sps_subpic_ctu_top_left_x [i] (X coordinate) and sps_subpic_ctu_top_left_y [i] (Y coordinate) indicating the coordinates of the upper left basic block of the i-th subpicture. However, the integrated coding unit 811 does not encode the coordinate information of the 0th subpicture because it is always in the upper left of the image and it is determined that the coordinates are (0, 0). Further, the coordinates of the first (last & lower right) sub-pictures of sps_num_subpics_minus are also obtained as described later, and are not encoded. The integrated coding unit 811 also encodes sps_subpic_width_minus1 [i] and sps_subpic_height_minus1 [i], which indicate the number of basic blocks-1 in the horizontal and vertical directions of the i-th subpicture as well as the coordinates. The integrated coding unit 811 does not encode the first subpicture of sps_num_subpics_minus because the number of basic blocks in the horizontal and vertical directions is obvious.

Since five subpictures are defined in this embodiment, sps_subpic_info_present_flag is 1, and sps_num_subpics_minus1 is 4. Since the number of basic blocks of the 0th subpicture is 12 × 6, sps_subpic_width_minus1 [0] is 11, and sps_subpic_height_minus1 [0] is 5. Since the coordinates of the upper left basic block of the first subpicture are (12, 0), sps_subpic_ctu_top_left_x [1] is 12, and sps_subpic_ctu_top_left_y [1] is 0. Since the number of basic blocks of the first subpicture is 6 × 2, sps_subpic_width_minus1 [1] is 5, and sps_subpic_height_minus1 [1] is 1. Since the coordinates of the upper left basic block of the second subpicture are (12, 2), sps_subpic_ctu_top_left_x [2] is 12, and sps_subpic_ctu_top_left_y [2] is 2. Since the number of basic blocks of the second subpicture is 6 × 4, sps_subpic_width_minus1 [2] is 5, and sps_subpic_height_minus1 [2] is 3. Since the coordinates of the upper left basic block of the third subpicture are (0, 6), sps_subpic_ctu_top_left_x [3] is 0, and sps_subpic_ctu_top_left_y [3] is 6. Since the number of basic blocks of the third subpicture is 12 × 12, sps_subpic_width_minus1 [3] is 11, and sps_subpic_height_minus1 [3] is 11. The coordinates of the upper left basic block of the fourth subpicture are (12, 6), and the number of basic blocks is 6 × 12, but the integrated coding unit 811 does not encode this information.

sps_num_subpics_minus The derivation of the coordinates of the upper left basic block of the first (last) subpicture will be described. First, how to obtain the X coordinate will be described. First, the subpicture adjacent to the left of the last subpicture is obtained. Among the sub-pictures including the basic block at the lower end of the image, the sub-picture in which the right-most basic block of the sub-picture has the maximum X coordinate is the sub-picture adjacent to the left of the last sub-picture. First, the sub-picture for which the following formula holds is a sub-picture including the basic block at the lower end.
PicCtbNumV == sps_subpic_ctu_top_left_y [i] + sps_subpic_height_minus1 [i] + 1

Among all the sub-pictures satisfying the above conditions in the image, the sub-picture located on the rightmost side, that is, the sub-picture having the largest sps_subpic_ctu_top_left_x [i] is the sub-picture adjacent to the left side. Therefore, assuming that the subpicture adjacent to the left of the last subpicture is the Nth subpicture, the X coordinate of the upper left basic block of the last subpicture can be obtained by the following equation.
sps_subpic_ctu_top_left_x [N] + sps_subpic_width_minus1 [N] + 1

In the present embodiment, PicCtbNumV is 18, and it is subpicture 3 that satisfies sps_subpic_ctu_top_left_y [i] + sps_subpic_height_minus1 [i] +1 == 18. sps_subpic_ctu_top_left_x [3] + sps_subpic_width_minus1 [3] + 1 = 12, and the X coordinate of the upper left basic block of the last subpicture is 12.

Next, how to obtain the Y coordinate of the basic block on the upper left of the last subpicture will be described. First, the sub-pictures adjacent to the last sub-picture are obtained. Among the sub-pictures including the basic block at the right end of the image, the sub-picture in which the basic block at the lower end of the sub-picture has the maximum Y coordinate is the sub-picture adjacent to the last sub-picture. First, the sub-picture for which the following equation holds is a sub-picture including the rightmost basic block.
PicCtbNumH == sps_subpic_ctu_top_left_x [i] + sps_subpic_width_minus1 [i] + 1

Among all the sub-pictures satisfying the above conditions in the image, the sub-picture located at the bottom, that is, the sub-picture having the largest sps_subpic_ctu_top_left_y [i] is the sub-picture adjacent to the top. Therefore, assuming that the sub-picture adjacent to the last sub-picture is the M-th sub-picture, the Y coordinate of the upper left basic block of the last sub-picture can be obtained by the following equation.
sps_subpic_ctu_top_left_y [M] + sps_subpic_height_minus1 [M] + 1

In the present embodiment, PicCtbNumH is 18, and it is subpictures 1 and 2 that satisfy sps_subpic_ctu_top_left_x [i] + sps_subpic_width_minus1 [i] +1 == 18. Among them, subpicture 2 has the largest sps_subpic_ctu_top_left_y [i]. sps_subpic_ctu_top_left_y [2] + sps_subpic_height_minus1 [2] + 1 = 6, and the Y coordinate of the upper left basic block of the last subpicture is 6.

In this way, in the conventional method, the coordinate information of the upper left basic block of the last subpicture was encoded, but without using it, the coordinates of the upper left basic block of the last subpicture from the information of other subpictures. Can be asked. This makes it possible to reduce the amount of code in the bitstream.

Returning to the description of the coded data format of FIG. The integrated coding unit 811 then encodes the information regarding the ID of the sub-picture. Further, the integrated coding unit 811 encodes sps_subpic_id_len_minus1 indicating the bit length -1 of the sub-picture ID of the SPS. The integrated coding unit 811 then encodes sps_subpic_id_mapping_explicity_signaled_flag indicating whether or not to explicitly indicate the ID of the sub-picture. The integrated coding unit 811 encodes sps_subpic_id_mapping_present_flag indicating whether or not to indicate an ID to SPS only when sps_subpic_id_mapping_explicity_signed_flag is 1. Only when sps_subpic_id_mapping_present_flag is 1, the integrated coding unit 811 encodes sps_subpic_id [i] indicating the ID of the i-th subpicture. In the present embodiment, the integrated coding unit 811 encodes the information regarding the IDs of these sub-pictures only when sps_num_subpics_minus1 is 1 or more (only when the number of sub-pictures is 2 or more). Conventionally, if sps_subpic_info_present_flag is 1, encoding is performed even when the number of subpictures is 1, but in the present embodiment, when the number of subpictures is 1, the ID of the subpicture is set to 0, and the following regarding the ID The syntax is not encoded.
・ Sps_subpic_id_len_minus1
・ Sps_subpic_id_mapping_explicity_signaled_flag
・ Sps_subpic_id_mapping_present_flag
・ Sps_subpic_id

As a result, the amount of code can be reduced when the number of sub-pictures is 1. Further, in the present embodiment, when the number of sub-pictures is 1, the ID of the sub-pictures is set to 0, but the present invention is not limited to this. For example, it may be 1. Further, the integrated coding unit 811 does not have to always encode sps_subpic_id_mapping_explicity_signed_flag regardless of the number of subpictures.

Since there are five sub-pictures in this embodiment, the bit length can be represented by 3 bits. Therefore, sps_subpic_id_len_minus1 is obtained by subtracting 1 from 3 to become 2. In this embodiment, sps_subpic_id_mapping_explicity_signaled_flag is set to 0.

The integrated coding unit 811 encodes pps_no_pic_partition_flag indicating whether or not the image is divided in the picture parameter set. When pps_no_pic_partition_flag is 1, it means that the image is not divided into a plurality of tiles, slices, and subpictures, and is a single tile, slice, and subpicture, respectively. Only when pps_no_pic_partition_flag is 0, tile division information, sub-picture definition information P (PPS), and slice division information P (PPS) are included. As the sub-picture definition information P, the integrated coding unit 811 first encodes pps_subpic_id_mapping_present_flag indicating whether or not the ID of the sub-picture is explicitly indicated on the PPS. Only when pps_subpic_id_mapping_present_flag is 1, the integrated coding unit 811 encodes the following three types of syntax.
-Pps_num_subpics_minus1 indicating the number of sub-pictures-1
Pps_subpic_id_len_minus1 indicating the bit length -1 of the sub-picture ID.
-Pps_subpic_id [i] indicating the ID of the i-th subpicture.

In the conventional method, pps_subpic_id_mapping_present_flag is always encoded, but in this embodiment, it is encoded only when pps_no_pic_partition_flag is 0. This makes it possible to reduce the amount of code when the image is not divided.

In this embodiment, since the image is divided, pps_no_pic_partition_flag is 0. Further, pps_subpic_id_mapping_present_flag is set to 0.

Returning to the description of the format of FIG. The integrated coding unit 811 encodes pps_single_slice_per_subpic_flag only when pps_num_subpics_minus1 does not exist or is 1 or more as the slice division information P. pps_single_slice_per_subpic_flag indicates whether all subpictures in the image are encoded in a single slice. In this embodiment, since there is a subpicture containing a plurality of slices, pps_single_slice_per_subpic_flag is 0. However, pps_num_subpics_minus1 exists, and the integrated coding unit 811 encodes pps_single_slice_per_subpic_flag only when it is 1 or more. Therefore, when pps_num_subpics_minus1 exists and is 1 or more, the code amount can be reduced.

The slice header includes slice division information SH and the like, followed by the coded data of each slice. The integrated coding unit 811 encodes sh_subpic_id indicating the sub-picture ID of the slice only when the number of sub-pictures is 2 or more as the slice division information SH. Conventionally, if sps_subpic_info_present_flag is 1, encoding is performed even when the number of subpictures is 1, but in this embodiment, when the number of subpictures is 1, the ID of the subpicture is 0, so there is no need to encode. .. This makes it possible to reduce the amount of code.

In the case of this embodiment, the sh_subpic_id of the slice with SID = 0 to 2 is 0. The sh_subpic_id of the slice with SID = 3 is 1. The sh_subpic_id of the slice with SID = 4 is 2. The sh_subpic_id of slices with SID = 5-8 and SID = 10-12 is 3. The sh_subpic_id of slices with SID = 9 and 13 is 4.

FIG. 10 is a flowchart showing a coding process for a one-frame image in the image coding apparatus according to the embodiment.

First, in S1001, the image division unit 802 divides the image into tiles and slices as described above, defines subpictures, and sends the division information to the integrated coding unit 811. In S1009, the integrated coding unit 811 converts the division information into header information and then encodes it into a bit stream. Further, the image dividing unit 802 divides the image into basic block line images and sends the image to the block dividing unit 803.

In S1002, the block division unit 803 divides a line image composed of basic blocks into basic block units.

In S1003, the prediction unit 804 executes a prediction process on the image data in basic block units generated in S1002. Further, the prediction unit 804 generates prediction information such as sub-block division information and intra prediction mode, and prediction image data. Further, the prediction unit 804 calculates a prediction error from the input image data and the prediction image data.

In S1004, the conversion / quantization unit 805 orthogonally converts the prediction error calculated in S1003 to generate a conversion coefficient. Then, the conversion / quantization unit 805 quantizes the generated conversion coefficient to generate the quantization coefficient.

In S1005, the inverse quantization / inverse conversion unit 806 performs inverse quantization on the quantization coefficient generated in S1004 and reproduces the conversion coefficient. The inverse quantization / inverse transformation unit 806 further performs inverse orthogonal transformation with respect to the reproduced conversion coefficient, and reproduces the prediction error.

In S1006, the image reproduction unit 807 reproduces the predicted image based on the predicted information generated in S1003. The image reproduction unit 807 further reproduces the image data from the reproduced predicted image and the prediction error generated in S1005.

In S1007, the coding unit 810 encodes the prediction information generated in S1003 and the quantization coefficient generated in S1004 to generate code data.

In S1008, the image coding apparatus determines whether or not the coding of all the basic blocks in the slice is completed, and if it is completed, the process proceeds to S1009, and if not, the next basic block. The process is returned to S1002 in order to target.

In S1009, the integrated coding unit 811 generates and encodes the header information based on the division information sent from the image division unit 802.

Hereinafter, based on the information of these sub-pictures, the code data of the slices constituting the sub-picture are joined to create the code data of the sub-picture.

In S1010, the image coding apparatus determines whether or not the coding of all the basic blocks in the frame is completed, and if it is completed, the process proceeds to S1011, otherwise the next basic block. Is returned to S1002 in order to encode.

In S1011, the in-loop filter unit 809 performs in-loop filter processing on the image data reproduced in S1006, generates a filtered image, and ends the processing.

With the above configuration and operation, especially in S1009, when a plurality of subpictures are defined, the syntax related to the subpictures can be efficiently encoded by not encoding the upper left coordinates of the last subpicture. ..

In the present embodiment, the image is composed of 9 tiles, 5 subpictures, and 14 slices, but the image is not limited to this. The image may or may not be divided in any way. For example, an image may be composed of a single subpicture, tile, or slice without defining a plurality of subpictures and further dividing the image into a plurality of tiles or slices. In that case, even if sps_subpic_info_present_flag is 1, sps_num_subpics_minus1 becomes 0, and it is not necessary to encode the following sequence parameter set and slice header syntax.
・ Sps_subpic_id_len_minus1
・ Sps_subpic_id_mapping_explicity_signaled_flag
・ Sps_subpic_id_mapping_present_flag
・ Sps_subpic_id
・ Sh_subpic_id

Similarly, since the image is not divided, pps_no_pic_partition_flag is 1, and it is not necessary to encode the syntax of the following picture parameter set.
・ Pps_subpic_id_mapping_present_flag
・ Pps_num_subpics_minus1
・ Pps_subpic_id_len_minus1
・ Pps_subpic_id
As a result, unnecessary syntax is not encoded, so that the amount of code can be reduced as compared with the conventional case.

Next, 200 will be described for an image decoding device that acquires a bit stream generated by the image coding device (imaging device 100) described so far and decodes the bit stream. The image decoding device 200 basically performs the reverse operation of the image coding device (imaging device 100) described so far.

FIG. 8 is a block diagram showing the configuration of the image decoding device 200 according to the present embodiment. The image decoding device 200 in the present embodiment has an acquisition unit 218 for acquiring coded data (bitstream) encoded by the image coding device (imaging device 100) and a decoding processing unit for decoding the acquired bitstream. It has 214, a CPU 201, a memory 202, a non-volatile memory 203, and an internal bus 230.

As described above, the acquisition unit 218 acquires the coded data (bit stream) encoded by the image coding device (imaging device 100). The acquisition unit 218 can also acquire the coded data via the network, for example.

The CPU 201 controls the operation of each part of the image decoding device 200 by executing a computer program stored in the non-volatile memory 203. The memory 202 is a rewritable volatile memory (RAM), and is a computer program that temporarily controls the operation of each part of the image decoding device 200, information such as parameters related to the operation of each part, and data acquired by the acquisition unit 218. And so on. The memory 202 also functions as a work memory for temporarily storing images and information processed by each unit.

The non-volatile memory 203 is a memory that can be electrically erased and recorded, and a storage medium such as an EEPROM or an SD memory card is used. The non-volatile memory 203 stores information such as a computer program that controls the operation of each part of the image decoding device 200 and parameters related to the operation of each part. The computer program referred to here includes a program for executing various processes described later in the present embodiment.

The internal bus 230 is an internal bus for each processing unit to access the CPU 201 and the memory 202.

As described above, the bitstream acquired by the acquisition unit 218 is encoded by dividing the picture into a plurality of rectangular areas. The bit stream contains information for reproducing each rectangular area (SPS and the like described later). The decoding processing unit 214 separates the coded data of the image itself and the information such as SPS from the coded data, and decodes them. Then, the picture is reproduced based on the decoded data.

The decoding processing unit 214 shall perform decoding processing according to, for example, the VVC method. However, the points to be described later are different from the VVC method proposed at this stage.

Next, the division of the rectangular region in the picture decoded by the decoding processing unit 214 in the embodiment will be described. Under the control of the CPU 101, the decoding processing unit 214 decodes one picture (frame image) to be decoded for each one or more tile rows and one or more tile columns.

Here, one tile is an aggregate of blocks (CTU: Coding Tree Unit) covering a rectangular area composed of a plurality of pixels. Further, the slice in the following description means an area composed of a plurality of tiles or an area of one or more CTU rows in one tile.

1 and 2 are examples of the relationship between CTU, Tile, and Slice. In the example of FIG. 1, one picture is divided into tiles of 4 rows and 6 columns, and a total of 9 slices of 3 × 3 that collectively cover the tiles are defined. Further, in FIG. 2, one picture is divided into four tiles of 2 rows and 2 columns, the upper right tile is divided into two slices, and two vertically adjacent tiles on the left side form one slice. Then, the tile at the lower right becomes one slice, and an example in which a total of four slices are defined is shown.

Further, in the present embodiment, a region that is composed of one or more slices that collectively cover a rectangular region in a frame image and can be independently decoded is referred to as a subpicture. FIG. 3 shows an example of the relationship between the CTU, tiles, and subpictures. The figure shows an example in which one picture is divided into 18 tiles in 3 rows and 6 columns, and a total of 24 slices and sub-pictures that collectively cover each tile are set.

It is assumed that the setting of how many pictures are divided into sub-pictures and what size the pictures are divided into is performed in the coding apparatus 100. This setting is determined based on, for example, which of the color patterns registered in advance is the closest to the color distribution derived from the image taken prior to recording the moving image.

In VVC, information such as parameters for realizing picture division using tiles, slices, and sub-pictures (information for reproduction for the decoding side) as shown in FIGS. 1 to 3 is encoded in the image. It is supposed to be stored in the header part (SPS: Sequence Parameter Set) of the sequence of the bitstream obtained in 1 and the header part (PPS: Picture Parameter Set) for the picture. Information such as parameters stored in these header units is decoded by the decoding processing unit 214.

For example, the SPS stores sps_subpic_ctu_top_left_x and y-coordinates sps_subpic_ctu_top_left_y indicating the upper left end of each sub-picture. However, for example, the coordinates of the upper left of the last sub-picture located at the lower right of the picture can be calculated from the coordinates of each of the previous sub-pictures. That is, it can be said that the information indicating the position related to the last sub-picture is redundant.

Therefore, the decoding processing unit 214 of the present embodiment decodes the SPS in which the parameter information related to the last sub-picture is omitted. Therefore, the amount of information of the SPS is reduced more than before.

4A to 4F show the syntax of the SPS information generated by the image coding device (imaging device 100). It should be noted that FIGS. 4A to 4F represent the syntax of SPS by making them continuous in this order. The syntaxes shown in FIGS. 4A to 4F are decoded by the decoding processing unit 214 in the order shown in FIGS. 4A to 4F.

"Sps_num_sibics_minus1" in the description of the line of reference numerals 401 and 402 shown represents "the number of sub-pictures included in one picture-1". Then, by newly adding the condition "i <sps_num_sibpics_minus1", when the number of sub-pictures included in one picture is N, the position of N-1 sub-pictures, which is one less than that, {sps_subpic_ctu_top_left_x [i]. ], Sps_subpic_ctu_top_left_y [i]} and the size {sps_subpic_width_minus1 [i], sps_subpic_hight_minus1 [i]} are included in the SPS. In the present embodiment, the conditional expression is added to the lines of reference numerals 401 and 402, but the present invention is not limited thereto. For example, the conditional expression of the for statement in the line above 401 may be changed from i <= sps_num_subpics_minus1 to i <sps_num_subpics_minus1.

The decoding processing unit 214 obtains the position and size of the last sub-picture from the decoding results of the previous sub-pictures in the process of performing the decoding process according to the syntax of SPS shown in FIGS. 4A to 4F. Therefore, since the sub-picture image can be reproduced according to the obtained position, no problem occurs.

CtbSizeY, which is the number of vertical and horizontal pixels of the CTU, is represented by 1 << CtbSize (here, CtbSize = sps_log2_ctu_size_minus5 + 5). The decoding processing unit 214 obtains PicCtbNumH, which is the number of CTUs in the horizontal direction of the image, by (sps_pic_width_max_in_luma_samples + CtbSizeY-1) >> CtbLog2SizeY. Similarly, the decoding processing unit 214 obtains PicCtbNumV, which is the number of CTUs in the vertical direction, by (sps_pic_height_max_in_luma_samples + CtbSizeY-1) >> CtbLog2SizeY.

Next, the derivation of the coordinates of the CTU on the upper left of the first (last) subpicture of sps_num_subpics_minus will be described. First, how to obtain the X coordinate will be described. First, the decoding processing unit 214 obtains a subpicture adjacent to the left of the last subpicture. Among the sub-pictures including the CTU at the lower end of the image, it is determined that the sub-picture having the maximum X coordinate at the right end CTU of the sub-picture is the sub-picture adjacent to the left of the last sub-picture. The sub-picture for which the following equation holds is a sub-picture including the CTU at the lower end.
PicCtbNumV == sps_subpic_ctu_top_left_y [i] + sps_subpic_height_minus1 [i] + 1

The decoding processing unit 214 determines that, among all the sub-pictures satisfying the above conditions in the image, the sub-picture located on the rightmost side, that is, the sub-picture having the largest sps_subpic_ctu_top_left_x [i] is the sub-picture adjacent to the left side. do. Therefore, assuming that the subpicture adjacent to the left of the last subpicture is the Nth subpicture, the decoding processing unit 214 can obtain the X coordinate of the CTU on the upper left of the last subpicture by the following equation.
sps_subpic_ctu_top_left_x [N] + sps_subpic_width_minus1 [N] + 1

Next, how to obtain the Y coordinate of the CTU on the upper left of the last subpicture will be described. First, the decoding processing unit 214 obtains a subpicture adjacent to the last subpicture. The decoding processing unit 214 determines that among the sub-pictures including the CTU at the right end of the image, the sub-picture having the maximum Y coordinate of the CTU at the lower end of the sub-picture is a sub-picture adjacent to the last sub-picture. to decide. The sub-picture for which the following equation holds is a sub-picture including the CTU at the right end.
PicCtbNumH == sps_subpic_ctu_top_left_x [i] + sps_subpic_width_minus1 [i] + 1

The decoding processing unit 214 has the lowest sub-picture among all the sub-pictures satisfying the above conditions in the image, that is, the decoding processing unit 214 has the sub-picture having the largest sps_subpic_ctu_top_left_y [i] adjacent to the top. Judge that it is a sub-picture. Therefore, assuming that the sub-picture adjacent to the last sub-picture is the M-th sub-picture, the decoding processing unit 214 can obtain the Y coordinate of the CTU on the upper left of the last sub-picture by the following equation.
sps_subpic_ctu_top_left_y [M] + sps_subpic_height_minus1 [M] + 1

Further, the SPS, PPS, and slice header in the conventional techniques employ a description that allows two or more sub-pictures to exist in one picture. In other words, so far, information that is meaningful only when two or more sub-pictures exist in one picture is included.

However, there are cases where only one sub-picture is included in one picture, and in such a case, the SPS, PPS, and slice header contain redundant information, which means that the amount of code is reduced. There is still room for improvement from the point of view.

Therefore, in the present embodiment, the lines of reference numerals 403 and 404 in FIGS. 4A to 4F showing the syntax of SPS are considered to be valid only when two or more subpictures are present in one picture {}. If the syntax in is encoded and there is only one subpicture in one picture, the syntax in {} is not encoded.

Further, the pps_no_pic_partition_flag existing in the line above the line of reference numeral 501 in FIG. 5A is a flag indicating whether or not the image is divided into subpictures, tiles, and slices. When the syntax is 1, it indicates that the image is composed of a single subpicture, tile, or slice. The line indicated by reference numeral 501 in the syntax of PPS (Picture Parameter Set) shown in FIGS. 5A to 5C (in this order representing the syntax of PPS), and the condition "&& (pps_num_subpics_minus1" in the line indicated by reference numeral 503). The syntax in {} is encoded as "> 0" is valid only when there are two or more subpictures in one picture, and there is only one subpicture in one picture. In some cases, the syntax in {} is not encoded. The syntaxes shown in FIGS. 5A-5C are decoded by the decoding processing unit 214 in the order shown in FIGS. 5A-5C.

Then, the condition "sps_num_subpix_minus1> 0" in the line indicated by the reference code 601 in the syntax of the slice header shown in FIGS. 6A to 6C (in this order, the syntax of the slice header is represented) is two or more in one picture. The syntax in {} is encoded as valid only if a subpicture is present. When only one sub-picture exists in one picture, the syntax in {} is not encoded. The syntaxes shown in FIGS. 6A to 6C are decoded by the decoding processing unit 214 in the order shown in FIGS. 6A to 6C.

As a result of the above, the CPU 101 decodes the index number that identifies the sub-picture only when there are two or more sub-pictures in one picture. Then, when only one sub-picture exists in one picture, the CPU 101 does not decode the code of the index number of the sub-picture, and the code amount can be reduced by that amount.

As a result of the above, the amount of information in the SPS, PPS, and slice header can be reduced, especially when the number of sub-pictures contained in one picture is one. Further, since there is no redundant information, the processing speed of the image decoding apparatus 200 can be improved.

Next, the image decoding apparatus described so far will be described in more detail with reference to FIGS. 9 and 11 to 14. In the following description, further details of the processing described mainly as the processing in the decoding processing unit 214 will be described.

First, the image decoding apparatus will be described with reference to FIG. FIG. 9 is a functional block diagram showing the decoding processing unit 214 and the like in more detail.

The image decoding device (decoding processing unit 214) includes a separation decoding unit 902, a decoding unit 903, an inverse quantization / inverse conversion unit 904, an image reproduction unit 905, a frame memory 906, and an in-loop filter unit 907.

The separation / decoding unit 902 separates the bitstream input via the input terminal 901 into information related to the decoding process and code data related to the coefficient, and sends the information to the decoding unit 903. Further, the separation / decoding unit 902 decodes the code data existing in the header unit of the bit stream. The separation / decoding unit 902 of the present embodiment decodes header information related to image division / definition such as sub-pictures, tiles, slices, and basic block sizes, generates division information, and outputs the division information to the image reproduction unit 905. That is, the separation / decoding unit 902 performs the reverse operation of the integrated coding unit 811 of FIG.

The decoding unit 903 decodes the code data input from the separation decoding unit 902, and reproduces the quantization coefficient and the prediction information.

The inverse quantization / inverse conversion unit 904 inversely quantizes the quantization coefficient reproduced by the decoding unit 903 to obtain a conversion coefficient, and further performs inverse orthogonal conversion of the conversion coefficient to reproduce a prediction error.

The image reproduction unit 905 generates predicted image data by appropriately referring to the frame memory 906 based on the prediction information input from the separation / decoding unit 902. Then, the image reproduction unit 905 generates the reproduction image data from the predicted image data and the prediction error reproduced by the inverse quantization / inverse conversion unit 904. The image reproduction unit 905 identifies the positions of the sub-pictures, tiles, and slices in the frame based on the division information input from the separation / decoding unit 902, and outputs the generated reproduced image data to the frame memory 906.

Similar to the in-loop filter unit 809 in FIG. 15, the in-loop filter unit 907 performs in-loop filter processing such as a deblocking filter on the reproduced image, and outputs the filtered image.

The image data for one frame generated and stored in the frame memory 906 is output to the outside via the output terminal 908.

The image decoding operation in the image decoding device will be described below. In the present embodiment, the bitstream generated by the image coding apparatus described above is input in frame units, but a still image bitstream for one frame may be input. Further, in the present embodiment, only the intra-predictive decoding process will be described for the sake of simplicity, but the present embodiment is not limited to this and can be applied to the inter-predictive decoding process.

In FIG. 9, one frame of bitstream input from the input terminal 901 is input to the separation / decoding unit 902. The separation / decoding unit 902 separates the bitstream into code data related to information related to the decoding process and coefficients, and decodes the code data existing in the header unit of the bitstream. More specifically, the image size information, the basic block division information, the sub-picture definition information S, the sub-picture definition information P, the tile division information, the slice division information P, and the slice division information SH in FIG. 12 are decoded, and the division information is obtained. Generate and send to the image reproduction unit 905. Subsequently, the code data in the basic block unit of the picture data is reproduced and output to the decoding unit 903.

The decoding unit 903 decodes the code data and reproduces the quantization coefficient and the prediction information. The decoding unit 903 outputs the regenerated quantization coefficient to the inverse quantization / inverse conversion unit 904, and outputs the reproduced prediction information to the image reproduction unit 905.

The inverse quantization / inverse conversion unit 904 performs inverse quantization on the quantization coefficient input from the decoding unit 903 to generate an orthogonal conversion coefficient. Then, the inverse quantization / inverse transformation unit 904 performs the inverse orthogonal transformation on the generated orthogonal transformation coefficient and reproduces the prediction error. The inverse quantization / inverse transformation unit 904 outputs the reproduced prediction error to the image reproduction unit 905.

The image reproduction unit 905 appropriately refers to the frame memory 906 based on the prediction information input from the decoding unit 903, and reproduces the predicted image. The image reproduction unit 905 reproduces image data from this predicted image and the prediction error input from the inverse quantization / inverse conversion unit 904. Then, the image reproduction unit 905 uses the generated reproduction image data based on the division information input from the separation / decoding unit 902, for example, as shown in FIG. 5, in the shape of the tile, slice, and sub-picture, and the position in the frame. Is specified and then stored in the corresponding position in the frame memory 906. The image data stored in the frame memory 906 will be used as a reference at the time of prediction.

Similar to the in-loop filter unit 809 of FIG. 15, the in-loop filter unit 907 reads the reproduced image from the frame memory 906 and performs in-loop filter processing such as a deblocking filter. Then, the in-loop filter unit 907 stores the filtered image in the frame memory 906 again.

The reproduced image stored in the frame memory 906 is finally output to the outside from the output terminal 908.

FIG. 11 is a flowchart showing an image decoding process for one frame in the image decoding apparatus according to the embodiment.

First, in S1101, the separation / decoding unit 902 separates the bitstream into code data related to information related to the decoding process and coefficients, and decodes the code data of the header portion. The separation / decoding unit 902 decodes the sub-picture definition information S, the sub-picture definition information P, the tile division information, the slice division information P, the slice division information SH, etc. in FIG. 12, generates information for decoding, and reproduces the image. Send to section 905. The division of the image stored in the bit stream in the present embodiment is as shown in FIG.

First, from the values of sps_pic_width_max_in_luma_samples and sps_pic_height_max_in_luma_samples of the image size information, it is derived that the image is 1152 × 1152 pixels.

Next, since the value of sps_log2_ctu_size_minus5 of the basic block division information is 1, it can be seen that CtbLog2SizeY = 6. Then, the size of the basic block is derived from 1 << CtbLog2SizeY to be 64 × 64 pixels (CtbSizeY = 64). PicCtbNumH, which is the number of basic blocks in the horizontal direction of the image, is derived from (sps_pic_width_max_in_luma_samples + CtbSizeY-1) >> CtbLog2SizeY. Similarly, PicCtbNumV, which is the number of basic blocks in the vertical direction, is also derived as 18.

Next, the sub-picture definition information S is acquired. First, since sps_subpic_info_present_flag is 1, it can be seen that there is information regarding the definition of the sub-picture. Subsequently, the information of the number -1 of the sub-pictures represented by sps_num_subpics_minus1 is acquired. In the present embodiment, by acquiring the value 4 and adding 1 to this value, it can be seen that the number of sub-pictures is 5. Since sps_subpic_width_minus1 [0] is 11 and sps_subpic_height_minus1 [0] is 5, it can be seen that the number of horizontal basic blocks and the number of vertical basic blocks of subpicture 0 at the upper left corner of the image is 12. Since sps_subpic_ctu_top_left_x [1] is 12 and sps_subpic_ctu_top_left_y [1] is 0, it can be seen that the coordinates of the upper left basic block of subpicture 1 are (12, 0). Since sps_subpic_width_minus1 [1] is 5 and sps_subpic_height_minus1 [1] is 1, it can be seen that the number of basic blocks in the horizontal direction of subpicture 1 is 6 and the number of basic blocks in the vertical direction is 2. Since sps_subpic_ctu_top_left_x [2] is 12 and sps_subpic_ctu_top_left_y [2] is 2, it can be seen that the coordinates of the upper left basic block of subpicture 2 are (12, 2). Since sps_subpic_width_minus1 [2] is 5 and sps_subpic_height_minus1 [2] is 3, it can be seen that the number of basic blocks in the horizontal direction of subpicture 2 is 6 and the number of basic blocks in the vertical direction is 4. Since sps_subpic_ctu_top_left_x [3] is 0 and sps_subpic_ctu_top_left_y [3] is 6, it can be seen that the coordinates of the upper left basic block of the subpicture 3 are (0, 6). Since sps_subpic_width_minus1 [3] is 11 and sps_subpic_height_minus1 [3] is 11, it can be seen that the number of basic blocks in the horizontal direction of the sub-picture 3 is 12 and the number of basic blocks in the vertical direction is 12.

The coordinates of the upper left basic block of subpicture 4 (last subpicture) are derived. It is subpicture 3 that PicCtbNumV is 18, and sps_subpic_ctu_top_left_y [i] + sps_subpic_height_minus1 [i] + 1 == 18 is satisfied. sps_subpic_ctu_top_left_x [3] + sps_subpic_width_minus1 [3] + 1 = 12, and the X coordinate of the upper left basic block of the last subpicture is 12. It is subpictures 1 and 2 that have a PicCtbNumH of 18 and satisfy sps_subpic_ctu_top_left_x [i] + sps_subpic_width_minus1 [i] + 1 == 18. Among them, subpicture 2 has the largest sps_subpic_ctu_top_left_y [i]. sps_subpic_ctu_top_left_y [2] + sps_subpic_height_minus1 [2] + 1 = 6, and the Y coordinate of the upper left basic block of the last subpicture is 6. The number of basic blocks in the horizontal direction of the sub-picture 4 is 6 for PicCtbNumH-12, and the number of basic blocks in the vertical direction is 12 for PicCtbNumV-6.

In this way, in the conventional method, the coordinate information of the upper left basic block of the last subpicture was encoded, but without using it, the coordinates of the upper left basic block of the last subpicture from the information of other subpictures. Can be asked. This makes it possible to decode a bitstream with a reduced amount of code.

Next, since the value of sps_subpic_id_len_minus1 is 2, it can be seen that the bit length of the ID of the subpicture is 3. Further, in the present embodiment, since sps_subpic_id_mapping_explicity_signed_flag is 0, it can be seen that there is no explicit mapping of sub-picture IDs.

Conventionally, if sps_subpic_info_present_flag is 1, encoding is performed even when the number of sub-pictures is 1, but in the format of this embodiment, when the number of sub-pictures is 1, the ID of the sub-pictures is set to 0 to describe the following. Do not decrypt the syntax of.
・ Sps_subpic_id_len_minus1
・ Sps_subpic_id_mapping_explicity_signaled_flag
・ Sps_subpic_id_mapping_present_flag
・ Sps_subpic_id

As a result, when the number of sub-pictures is 1, it is possible to decode the bit stream in which the code amount is reduced. Further, in the present embodiment, when the number of sub-pictures is 1, the ID of the sub-pictures is set to 0, but the present invention is not limited to this. For example, it may be 1. Further, the separation / decoding unit 902 does not have to always decode sps_subpic_id_mapping_explicity_signed_flag regardless of the number of sub-pictures.

The separation decoding unit 902 then decodes pps_no_pic_partition_flag in the picture parameter set and obtains a value of 0. This shows that the image is not composed of a single subpicture, tile, or slice.

Since pps_no_pic_partition_flag is 0, the separation / decoding unit 902 subsequently decodes the tile division information, the sub-picture definition information P (PPS), and the slice division information P (PPS). In this embodiment, since the value of pps_subpic_id_mapping_present_flag is 0, the syntax regarding the ID of the sub-picture is not encoded in PPS.

In the conventional method, pps_subpic_id_mapping_present_flag is always encoded, but in the format of this embodiment, it is decoded only when pps_no_pic_partition_flag is 0. As a result, it is possible to decode a bit stream in which the amount of code is reduced when the image is not divided.

The separation / decoding unit 902 decodes pps_single_slice_per_subpic_flag as the slice division information P, and acquires a value of 0. From this, it can be seen that there may be a subpicture containing a plurality of slices. In the format of the present embodiment, since pps_num_subpics_minus1 exists and the separation / decoding unit 902 decodes the pps_single_slice_per_subpic_flag only when the number is 1 or more, the code amount can be reduced when the conditions are met.

Next, the separation / decoding unit 902 decodes sh_subpic_id as the slice division information SH in the slice header. In the present embodiment, since the slice with sh_subpic_id 0 is the slice with SID = 0 to 2, it can be seen that the slice constituting the sub-picture 0 is the slice with SID = 0 to 2. Since the slice with sh_subpic_id of 1 is the slice with SID = 3, it can be seen that the slice constituting the sub-picture 1 is the slice with SID = 3. Since the slice with sh_subpic_id 2 is the slice with SID = 4, it can be seen that the slice constituting the sub-picture 2 is the slice with SID = 4. Since the slice with sh_subpic_id 3 is a slice with SID = 5-8 and 10-12, it can be seen that it is a slice constituting the sub-picture 3. Since the slice with sh_subpic_id 4 is the slice with SID = 9 and 13, it can be seen that the slices constituting the sub-picture 4 are the slices with SID = 9 and 13.

Conventionally, if sps_subpic_info_present_flag is 1, encoding is performed even when the number of subpictures is 1, but in this embodiment, when the number of subpictures is 1, the ID of the subpicture is 0, so there is no need to decode. This makes it possible to decode a bit stream with a reduced amount of code.

The divided information obtained from the information necessary for decoding and derived from the separation / decoding unit 902 is sent to the image reproduction unit 905 and used to specify the position of the data processed by S1104 in the image.

In S1102, the decoding unit 903 decodes the code data separated in S1101 and reproduces the quantization coefficient and the prediction information.

In S1103, the inverse quantization / inverse conversion unit 904 performs inverse quantization on the quantization coefficient input from the decoding unit 903 to obtain a conversion coefficient, and further performs inverse orthogonal conversion on the conversion coefficient to predict an error. To play.

In S1104, the image reproduction unit 905 reproduces the prediction information and the prediction image generated in S1103. Further, the image reproduction unit 905 reproduces the image data from the reproduced predicted image and the prediction error generated in S1104. Then, the image reproduction unit 905 synthesizes the reproduced image data at an appropriate position in the image based on the division information generated in S1101.

In S1105, the image decoding device determines whether or not the decoding of all the basic blocks in the frame is completed, and if it is completed, the process proceeds to S1106, and if not, the decoding of the next basic block is performed. Therefore, the process is returned to S1102.

In S1106, the in-loop filter unit 907 performs in-loop filter processing on the image data reproduced in S1104, generates a filtered image, and ends the processing.

With the above configuration and operation, especially in S1101, when a plurality of subpictures are defined, the coordinates of the upper left of the last subpicture are calculated to decode the bitstream in which the syntax related to the subpicture is efficiently encoded. Can be done.

In the present embodiment, the image is composed of 9 tiles, 5 subpictures, and 14 slices, but the image is not limited to this. The image may or may not be divided in any way. For example, an image may be composed of a single subpicture, tile, or slice without defining a plurality of subpictures and further dividing the image into a plurality of tiles or slices. In that case, even if sps_subpic_info_present_flag is 1, sps_num_subpics_minus1 becomes 0, and it is not necessary to decode the following sequence parameter set and slice header syntax.
・ Sps_subpic_id_len_minus1
・ Sps_subpic_id_mapping_explicity_signaled_flag
・ Sps_subpic_id_mapping_present_flag
・ Sps_subpic_id
・ Sh_subpic_id

Similarly, since the image is not divided, pps_no_pic_partition_flag is 1, and it is not necessary to decode the syntax of the following picture parameter set.
・ Pps_subpic_id_mapping_present_flag
・ Pps_num_subpics_minus1
・ Pps_subpic_id_len_minus1
・ Pps_subpic_id
As described above, it is possible to decode a bit stream in which unnecessary syntax is not encoded and the amount of code is reduced as compared with the conventional case.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiment, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to publicize the scope of the invention.

100 ... image pickup device, 101 ... CPU 101, 102 ... memory, 103 ... non-volatile memory, 104 ... operation unit, 111 ... image pickup lens, 112 ... image pickup unit, 113 ... image processing unit, 114 ... coding processing unit, 115 ... display Control unit, 116 ... Display unit, 117 ... Communication control unit, 118 ... Communication unit, 119 ... Recording medium control unit, 120 ... Recording medium, 140 ... Detection unit, 130 ... Internal bus

Claims (5)

An image coding device that encodes an image composed of N (N> 1) rectangles so that each rectangle can be independently decoded.
A dividing means for dividing an image into N rectangles, and
It has a coding means for encoding information on the position and size of the rectangle.
The coding means is an image coding device that encodes information about the positions of the second to N-1st rectangles and does not encode information about the positions of the first and Nth rectangles. An image decoding device that decodes a bit stream encoded so that each rectangle can independently decode an image composed of N (N> 1) rectangles.
It has an acquisition means for acquiring the positions and sizes of the N rectangles in block units constituting the plurality of rectangles.
The acquisition means is characterized in that information regarding the positions of N-2 rectangles and the size of N-1 rectangles among the N rectangles is obtained by decoding the bitstream.
The acquisition means calculates information on the position and size of the last (Nth) rectangle among the N rectangles by using the information on the position and size of the N-1 rectangles. An image decoding device characterized by being acquired by. An image coding method for encoding an image composed of N (N> 1) rectangles so that each rectangle can be independently decoded.
A division step of dividing an image into N rectangles, and
It has a coding step for coding information on the position and size of the rectangle.
The coding step is an image coding method comprising encoding information regarding the positions of the second to N-1th rectangles and not encoding information regarding the positions of the first and Nth rectangles. An image decoding method for decoding an image composed of N (N> 1) rectangles by decoding a bit stream encoded so that each rectangle can be independently decoded.
It has an acquisition step of acquiring the positions and sizes of the N rectangles in block units constituting the plurality of rectangles.
The acquisition step is characterized in that the bit stream is decoded and acquired with information regarding the positions of N-2 rectangles and the size of N-1 rectangles among the N rectangles.
In the acquisition step, information on the position and size of the last (Nth) rectangle among the N rectangles is calculated using the information on the position and size of the N-1 rectangles. An image decoding method characterized by being acquired by. A program for causing the computer to execute the process of the method according to claim 3 or 4, by reading and executing the computer.

Images