JP6879401B2 - Image coding device, image coding method and image coding program, and image decoding device, image decoding method and image decoding program. - Google Patents

Description

The present invention relates to a technique of dividing an image into blocks and encoding and decoding the divided block units.

In image coding and decoding, an image is divided into blocks which are a set of a predetermined number of pixels, and coding and decoding are performed in block units. By appropriately dividing blocks, the coding efficiency of in-screen prediction (intra-prediction) and inter-screen prediction (inter-prediction) is improved. Further, in the intra prediction, the coding efficiency is improved by predicting the secondary signal from the decoded image of the primary signal.

Japanese Unexamined Patent Publication No. 2013-90015

However, when the secondary signal is predicted from the decoded image of the primary signal, the amount of processing increases and the dependency between the processing for the primary signal and the processing for the secondary signal occurs, which makes parallel processing difficult.

The present invention has been made in view of such a situation, and an object of the present invention is to provide a technique for improving coding efficiency by performing block division suitable for image coding and decoding.

In order to solve the above problems, the image coding device according to an embodiment of the present invention is an image coding device that divides an image into blocks and encodes the divided block units, and is a primary signal of the image. A primary signal block division unit that divides the image into a rectangle of a predetermined size to generate a primary signal block, a secondary signal block division unit that divides the secondary signal of the image into a rectangle of a predetermined size to generate a secondary signal block, and a primary unit. A primary signal prediction unit that predicts a signal and a secondary signal prediction unit that predicts a secondary signal are included, and the primary signal block and the secondary signal block are each independently divided, and the primary signal is a brightness signal and the secondary. The signal is a color difference signal, and the secondary signal prediction unit can perform inter-component prediction that predicts the secondary signal from the encoded primary signal, and when the size of the primary signal block is a predetermined size or more, the inter-components. Prohibit the use of forecasts.

Another aspect of the present invention is an image coding method. This method divides the image into blocks and
An image coding method that encodes in units of divided blocks, which is a primary signal block division step of dividing the primary signal of the image into rectangles of a predetermined size to generate a primary signal block, and a secondary signal of the image. The primary signal block includes a secondary signal block dividing step for generating a secondary signal block, a primary signal prediction step for predicting a primary signal, and a secondary signal prediction step for predicting a secondary signal. And the secondary signal block are independently divided, the primary signal is a brightness signal, the secondary signal is a color difference signal, and the secondary signal prediction step is a component that predicts a secondary signal from a coded primary signal. When the inter-component prediction is possible and the size of the primary signal block is a predetermined size or more, the use of the inter-component prediction is prohibited.

Yet another aspect of the present invention is an image decoding device. This device is an image decoding device that decodes an image in block units, and divides the primary signal of the image into rectangles of a predetermined size to generate a primary signal block, and a primary signal block dividing unit. Includes a secondary signal block division unit that divides the secondary signal of the image into rectangles of a predetermined size to generate a secondary signal block, a primary signal prediction unit that predicts the primary signal, and a secondary signal prediction unit that predicts the secondary signal. The primary signal block and the secondary signal block are independently divided, the primary signal is a brightness signal, the secondary signal is a color difference signal, and the secondary signal prediction unit is secondary from the decoded primary signal. Inter-component prediction for predicting a signal is possible, and when the size of the primary signal block is a predetermined size or more, the use of the inter-component prediction is prohibited.

Yet another aspect of the present invention is an image decoding method. This method is an image decoding method that decodes an image in units of divided blocks, and includes a primary signal block division step of dividing the primary signal of the image into rectangles of a predetermined size to generate a primary signal block. A secondary signal block division step of dividing the secondary signal of the image into a rectangle of a predetermined size to generate a secondary signal block, a primary signal prediction step of predicting the primary signal, and a secondary signal prediction step of predicting the secondary signal are included. The primary signal block and the secondary signal block are independently divided, the primary signal is a brightness signal, the secondary signal is a color difference signal, and the secondary signal prediction step is secondary from the decoded primary signal. Inter-component prediction for predicting a signal is possible, and when the size of the primary signal block is a predetermined size or more, the use of the inter-component prediction is prohibited.

It should be noted that any combination of the above components and the conversion of the expression of the present invention between methods, devices, systems, recording media, computer programs and the like are also effective as aspects of the present invention.

According to the present invention, block division suitable for image coding and decoding becomes possible, coding efficiency can be improved, and image coding and decoding with a small amount of processing can be provided.

It is a block diagram of the image coding apparatus which concerns on 1st Embodiment. It is a block diagram of the image decoding apparatus which concerns on 1st Embodiment. It is a flowchart explaining division into a tree block and division inside a tree block. It is a figure which shows the mode that the input image is divided into tree blocks. It is a figure explaining z-scan. It is the figure which divided the tree block into four horizontally and vertically. It is the figure which divided the tree block into two in the horizontal direction. It is the figure which divided the tree block into two in the vertical direction. It is a flowchart explaining the processing of each divided block when the tree block is divided into 4 in a horizontal direction and a vertical direction. It is a flowchart explaining the processing of each divided block when the tree block is divided into two in the horizontal direction. It is a figure which shows the state of the subdivision of the divided block when the division of a tree block is divided into two in the horizontal direction. It is a flowchart explaining the processing of each divided block when the tree block is divided into two in the vertical direction. It is a figure which shows the state of the subdivision of the divided block when the division of a tree block is divided into two in the vertical direction. It is a figure which shows the example of the syntax concerning the block division of the 1st Embodiment. It is a figure explaining the intra prediction. It is a figure explaining the inter-prediction. It is a figure explaining the color difference format. It is a figure explaining the luminance color difference intra prediction. It is a flowchart explaining the luminance color difference intra prediction. It is a figure explaining the case where the size of a color difference block is larger than the size of a luminance block. It is a figure explaining the case where the size of a color difference block is smaller than the size of a luminance block. It is a figure explaining the example of the syntax of the intra-color difference prediction mode. It is a figure explaining an example which limits the intra-color difference prediction by replacing a peripheral pixel. It is a figure explaining the luminance color difference intra prediction by the difference in the size of the luminance block.

In the embodiment of the present invention, the image is divided into rectangular blocks, and the divided blocks are encoded.
Provided is an image coding technique for decoding.

(First Embodiment)
The image coding device 100 and the image decoding device 200 according to the first embodiment of the present invention will be described.

FIG. 1 is a configuration diagram of an image coding device 100 according to the first embodiment. Here, FIG. 1 shows only the flow of data related to the image signal, and each component supplies the additional information other than the image signal such as the motion vector and the prediction mode to the coded bit string generation unit 105 and corresponds to the coding. Data is generated, but the flow of data related to additional information is not shown.

The block division unit 101 divides the image into coding target blocks that are coding processing units, and supplies the image signal in the coding target block to the residual signal generation unit 103. Further, the block division unit 101 supplies the image signal of the coded block to the prediction image generation unit 102 in order to evaluate the degree of coincidence of the prediction images.

The block division unit 101 recursively divides the image into rectangles of a predetermined size to generate a block to be encoded. The block division 101 divides the target block in the recursive division into four in the horizontal and vertical directions to generate four blocks, and the block division 101 divides the target block in the recursive division into two in the horizontal or vertical direction. Includes a two-part section that generates two blocks. The detailed operation of the block dividing unit 101 will be described later.

The prediction image generation unit 102 uses the decoded image signal supplied from the decoded image memory 108 to perform intra-picture prediction (intra-prediction) or inter-picture prediction (inter-prediction) based on the prediction mode.
To generate a predicted image signal. The image signal in the coded block supplied from the block division unit 101 is used for evaluation of intra-prediction and inter-prediction. In the intra prediction, the image signal of the coded block supplied from the block dividing unit 101 and the surroundings close to the coded block existing in the same picture as the coded block supplied from the decoded image memory 108. A predicted image signal is generated using the image signal of the encoded block of. In the inter-prediction, the image signal of the coded block supplied from the block dividing unit 101 is stored in the decoded image memory 108 located before or after the picture (encoded picture) including the coded block in the time series. The encoded picture is used as the reference picture.
A block matching degree evaluation such as block matching is performed between the coded picture and the reference picture, a motion vector indicating the amount of motion is obtained, motion compensation is performed from the reference picture based on this amount of motion, and a predicted image signal is generated. .. The prediction image generation unit 102 supplies the prediction image signal thus generated to the residual signal generation unit 103.

The residual signal generation unit 103 subtracts the encoded image signal and the predicted signal generated by the predicted image generation unit 102 to generate a residual signal, and supplies the residual signal to the orthogonal transformation / quantization unit 104.

The orthogonal transformation / quantization unit 104 orthogonally transforms / quantizes the residual signal supplied from the residual signal generation unit 103, and encodes the orthogonal transform / quantization residual signal into the coded bit string generation unit 105 and inverse quantization. -Supply to the inverse orthogonal transform unit 106.

The coded bit string generation unit 105 generates a coded bit string for the orthogonal transformation / quantized residual signal supplied from the orthogonal transformation / quantization unit 104. Further, the coded bit string generation unit 1
05 generates a corresponding coded bit string for additional information such as a motion vector, a prediction mode, and block division information.

The inverse quantization / inverse orthogonal transform unit 106 reverse-quantizes / inverse-orthogonal transforms the orthogonal transform / quantized residual signal supplied from the orthogonal transform / quantize unit 104, and performs inverse quantization / inverse orthogonal transform. The residual signal is supplied to the decoded image signal superimposing unit 107.

The decoded image signal superimposition unit 107 superimposes the predicted image signal generated by the predicted image generation unit 102 and the residual signal that has been inversely quantized and inversely orthogonally converted by the inverse quantization / anti-orthogonal conversion unit 106 to generate a decoded image. Generate and store in the decoded image memory 108. The decoded image may be stored in the decoded image memory 108 after being subjected to a filtering process for reducing block distortion or the like due to encoding.

FIG. 2 is a configuration diagram of the image decoding device 200 according to the first embodiment. Here, FIG. 2 shows only the flow of data related to the image signal, and the bit string decoding unit 201 supplies the additional information other than the image signal such as the motion vector and the prediction mode to each component and uses it for the corresponding processing. , The flow of data related to additional information is not shown.

The bit string decoding unit 201 decodes the supplied coded bit string and supplies the orthogonally converted / quantized residual signal to the block division unit 202.

The block division unit 202 determines the shape of the block to be decoded based on the decoded block division information, and orthogonally transforms the determined block to be decoded and the quantized residual signal is inversely quantized and inversely orthogonal. It is supplied to the conversion unit 203.

The block division unit 202 recursively divides the image into rectangles of a predetermined size based on the decoded block division information to generate a decoding target block. The block division unit 202 divides the target block in the recursive division into four in the horizontal and vertical directions to generate four blocks, and the block division unit 202 in the recursive division into two in the horizontal or vertical direction. Includes a two-part section that generates two blocks. The detailed operation of the block division unit 202 will be described later.

The inverse-orthogonal / inverse-orthogonal conversion unit 203 performs inverse-orthogonal conversion and inverse-quantization on the supplied orthogonal-transformed / quantized residual signal, and the inverse-orthogonal-converted / inverse-quantized residual signal. To get.

The predicted image generation unit 204 generates a predicted image signal from the decoded image signal supplied from the decoded image memory 206, and supplies the predicted image signal to the decoded image signal superimposing unit 205.

The decoded image signal superimposition unit 205 superimposes the predicted image signal generated by the predicted image generation unit 204 and the residual signal converted / inversely orthogonalized by the inverse orthogonal / inverse orthogonal transform unit 203. , The decoded image signal is generated, output, and stored in the decoded image memory 206.
The decoded image may be stored in the decoded image memory 206 after being subjected to a filtering process to reduce block distortion or the like due to encoding.

The operation of the block dividing unit 101 of the image coding apparatus 100 will be described in detail. FIG. 3 is a flowchart illustrating division into tree blocks and division inside the tree block.

First, the input image is divided into tree blocks of a predetermined size (S1000). For example, the tree block is 128 pixels x 128 pixels. However, the tree block is 128
The size and shape are not limited to 128 pixels, and any size and shape may be used as long as it is rectangular. Further, the size and shape of the tree block may be fixed values between the coding device and the decoding device, but the coding device determines the size and the shape and records the tree block in the coded bit stream for decoding. The device may be configured to use the recorded block size. FIG. 4 shows how the input image is divided into tree blocks. The tree blocks are encoded and decoded in raster scan order, left-to-right, top-to-bottom.

The inside of the tree block is further divided into rectangular blocks. The inside of the tree block is encoded / decoded in the order of z-scan. FIG. 5 shows the order of z-scans. In the z-scan, encoding and decoding are performed in the order of upper left, upper right, lower left, and lower right. The division inside the tree block can be divided into four and two, and the four division is divided in the horizontal direction and the vertical direction. 2
The division is horizontal or vertical. FIG. 6 shows the tree block horizontally and vertically 4
It is a divided figure. FIG. 7 is a diagram in which the tree block is divided into two in the horizontal direction. FIG. 8 is a diagram in which the tree block is vertically divided into two.

See FIG. 3 again. It is determined whether or not the inside of the tree block is divided into four in the horizontal and vertical directions (S1001).

When it is determined that the inside of the tree block is divided into four (S1001: Yes), the inside of the tree block is divided into four (S1002), and each process of the blocks divided into four in the horizontal and vertical directions is performed (S1003). The subdivision process of the four-divided block will be described later (FIG. 9).

When it is determined that the inside of the tree block is not divided into four (S1001: No), it is determined whether or not the inside of the tree block is divided into two (S1004).

When it is determined that the inside of the tree block is divided into two (S1004: Yes), it is determined whether or not the direction of dividing into two is the horizontal direction (S1005).

When it is determined that the direction of dividing into two is the horizontal direction (S1005: Yes), the inside of the tree block is divided into two in the horizontal direction (S1006), and each process of the block divided into two in the horizontal direction is performed (S1007). The subdivision process of the block divided into two in the horizontal direction will be described later (FIG. 10).

When it is determined that the direction of dividing into two is not the horizontal direction but the vertical direction (S1005: No), the inside of the tree block is divided into two in the vertical direction (S1008), and each process of the block divided into two in the vertical direction is performed (S1009). .. The subdivision process of the block divided into two in the horizontal direction will be described later (FIG. 11).

When it is determined that the inside of the tree block is not divided into two (S1004: No), the block division process is terminated without dividing the inside of the tree block into blocks (S1010).

Subsequently, the processing of each of the divided blocks when the tree block is divided into four in the horizontal direction and the vertical direction will be described with reference to the flowchart of FIG.

It is determined whether or not the inside of the block is divided into four again in the horizontal and vertical directions (S1101).
..

When it is determined that the inside of the block is divided into four again (S1101: Yes), the inside of the block is divided into four again (S1102), and each process of the block divided into four in the horizontal and vertical directions is performed (S1103).

When it is determined that the inside of the block is not divided into four again (S1101: No), it is determined whether or not the inside of the block is divided into two (S1104).

When it is determined that the inside of the block is divided into two (S1104: Yes), it is determined whether or not the direction of dividing into two is the horizontal direction (S1105).

When it is determined that the direction of dividing into two is the horizontal direction (S1105: Yes), the inside of the block is divided into two in the horizontal direction (S1106), and each process of the block divided into two in the horizontal direction is performed (S1).
107).

When it is determined that the direction of dividing into two is not the horizontal direction but the vertical direction (S1105: No), the inside of the block is divided into two in the vertical direction (S1108), and each process of the block divided into two in the vertical direction is performed (S1109).

When it is determined that the inside of the block is not divided into two (S1104: No), the block division process is terminated without dividing the inside of the block into blocks (S1110).

The process shown in the flowchart of FIG. 9 is executed for each block divided into four. The inside of the block divided into four is also encoded and decoded in the order of z-scan.

Subsequently, the processing of each of the divided blocks when the tree block is divided into two in the horizontal direction will be described with reference to the flowchart of FIG.

When the tree block is divided into two in the horizontal direction, each of the divided blocks first determines whether or not the inside of the block is divided into four in the horizontal and vertical directions (S1201).

When it is determined that the inside of the block is divided into four (S1201: Yes), the inside of the block is divided into four.
It is divided (S1202), and each process of the block divided into four in the horizontal and vertical directions is performed (S12).
03).

When it is determined that the inside of the block is not divided into four (S1201: No), it is determined whether or not the inside of the block is divided into two again (S1204).

When it is determined that the block is divided into two again (S1204: Yes), the inside of the block is divided in the vertical direction (S1205), and each process of the block divided into two in the vertical direction is performed (S1206).

When it is determined that the block is not divided into two again (S1204: No), the block division process is terminated without subdividing the inside of the block (S1207).

FIG. 11 shows a state of subdivision of the divided blocks when the division of the tree block is divided into two in the horizontal direction. Here, when the tree block, which is the parent block, is divided into two in the horizontal direction, the divided block is re-divided into two, allowing only the vertical division into two, and automatically dividing the divided block into two in the vertical direction. Further, when the tree block which is the parent block is divided into two, it is possible to completely prohibit the division into four in the child block. As a result, it is possible to prohibit the block from being divided in the same direction as the parent block, so that it is possible to prevent the block from being divided into a rectangular shape that is elongated in the horizontal direction, and it becomes easier to perform coding / decoding processing.

The process shown in the flowchart of FIG. 10 is executed for each block divided into two in the horizontal direction. The inside of the block divided into two is also encoded and decoded in the order of top and bottom.

Subsequently, the processing of each of the divided blocks when the tree block is divided into two in the vertical direction will be described with reference to the flowchart of FIG.

When the tree block is divided into two in the vertical direction, each of the divided blocks first determines whether or not the inside of the block is divided into four in the horizontal and vertical directions (S1301).

When it is determined that the inside of the block is divided into four (S1301: Yes), the inside of the block is divided into four.
It is divided (S1302), and each process of the block divided into four in the horizontal and vertical directions is performed (S13).
03).

When it is determined that the inside of the block is not divided into four (S1301: No), it is determined whether or not the inside of the block is divided into two again (S1304).

When it is determined that the block is divided into two again (S1304: Yes), the inside of the block is divided in the horizontal direction (S1305), and each process of the block divided into two in the horizontal direction is performed (S1306).

When it is determined that the block is not divided into two again (S1304: No), the block division process is terminated without subdividing the inside of the block (S1307).

FIG. 13 shows a state of subdivision of the divided blocks when the division of the tree block is divided into two in the vertical direction. Here, when the tree block, which is the parent block, is divided into two in the vertical direction, the divided block is re-divided into two, allowing only the horizontal division into two, and automatically dividing the divided block into two in the horizontal direction. Further, when the tree block which is the parent block is divided into two, it is possible to completely prohibit the division into four in the child block. As a result, it is possible to prohibit the block from being divided in the same direction as the parent block, so that it is possible to prevent the block from being divided into a rectangular shape that is elongated in the vertical direction, and it becomes easier to perform coding / decoding processing.

The process shown in the flowchart of FIG. 12 is executed for each block vertically divided into two. The inside of the block divided into two is also encoded and decoded in the order of left and right.

Although the subdivision of the divided block when the tree block is divided has been described, the parent block does not have to be the tree block. For example, a tree block (128x1)
When 28) is divided into four and the four-divided block (64x64) is further divided into four or two, the above processing is also applied to the subdivision of the subdivided block.

Next, the operation of the block division unit 202 of the image decoding device 200 will be described. The block is divided by the same processing procedure as the block division unit 101 of the image coding device 100, but the block division unit 101 of the image coding device 100 selects a block division pattern and outputs the selected block division information. On the other hand, the block division unit 202 of the image decoding device is
When dividing a block using the block division information decoded from the coded bit stream, or when decoding the block division information from the coded bit stream, subdivision in the same direction is prohibited. The difference is that information without options has a syntax structure that is not transmitted within the bitstream.

FIG. 14 shows an example of the syntax (coded bitstream syntax rule) relating to the block division of the first embodiment. The internal division of the tree block first sends and receives a flag (4_division_flag) as to whether or not to divide into four. When dividing into 4 (4_division_flag is 1)
Divides the inside of the tree block into four and ends the process. After that, the inside of the block divided into four is subdivided again by the syntax shown in FIG. When not dividing into 4 (4_division_f
When lag is 0), a flag (2_division_flag) for whether or not to divide into two is transmitted and received. When dividing into two (2_division_flag is 1), a flag indicating the direction of dividing into two (2_division_d)
irection) is sent and received. 2 When _division_direction is 1, it indicates a vertical division, and 2
When _division_direction is 0, it indicates horizontal division. After that, the inside of the block divided into two is subdivided again by the syntax shown in FIG. If it is not divided into two (2_division_flag is 0), the process ends without dividing the tree block.

Here, the process of subdividing the inside of the block divided into four or two will be described.
The syntax shown in FIG. 14 is also used for the process of subdividing the inside of the block, but it differs from the case where the tree block is divided in that there is a limitation in the division direction when the tree block is divided into two. That is, when the tree block is divided into two, and when the inside of the divided block is subdivided, it is prohibited to divide the tree block in the same direction as the dividing direction.
As a result, it is possible to prevent the divided blocks from becoming a more elongated rectangle, and to prevent an increase in the memory bandwidth required for intra-prediction and inter-prediction. Details on preventing the increase in memory bandwidth will be described later.

Further, it is of course possible to count the number of two divisions in the same direction and limit the division in the same direction when the predetermined number of times is exceeded. For example, two divisions in the same direction are permitted up to two times, but two divisions in the same direction are prohibited from the third time.

In FIG. 14, the four divisions are preferentially selected, and the information on whether or not to divide into four is used as the syntax for transmitting and receiving before the information on whether or not to divide into two. On the other hand, when the information on whether or not to divide into two is preferentially selected, the syntax may be such that the information on whether or not to divide into two is transmitted and received before the information on whether or not to divide into four. This is because the amount of code transmitted as a bit stream is smaller when an event that is more probabilistically likely to occur is transmitted and received first. That is, 4 divisions and 2 in advance
It may be a syntax for estimating which of the divisions is more likely to occur and sending and receiving the division information that is more likely to occur first. For example, by transmitting and receiving whether to prioritize 4-division or 2-division in the header information of the image, the encoding device adaptively determines the number of priority divisions having high coding efficiency, and the decoding device determines the number of priority divisions. It is also possible to divide the inside of the tree block by a syntax based on the selected priority division number.

In the image coding device 100 and the image decoding device 200, intra-prediction and inter-prediction are performed using the divided blocks. Both intra-prediction and inter-prediction involve copying pixels from memory.

15 (a) to 15 (d) show an example of intra-prediction. 15 (a) and 15 (Fig. 15 (a))
b) indicates the prediction direction and mode number of the intra prediction. As shown in FIGS. 15 (c) and 15 (d), the intra prediction is encoded / decoded by copying a pixel from an encoded / decoded pixel close to the coded / decoded target block. Generate a predicted image of the target block. In intra-prediction, since the prediction image generation and the encoding / decoding pixel generation are repeated in block units, the processing order is sequential in block units, and the smaller the block is divided, the heavier the overall processing load. Further, the longer the block shape is, the larger the pixel copy processing from the memory becomes. Further, since the orthogonal transformation of the residual signal is performed for coding / decoding, the more types of rectangular sizes are required, the more types of orthogonal transformation are required, and as a result, the circuit scale is increased. Therefore, when dividing the inside of the block into two,
By limiting the division into two in the same direction as the division method of the parent block, it is possible to prevent an increase in the memory bandwidth required for intra-prediction.

FIG. 16 shows an example of inter-prediction. Inter-prediction generates a predicted image of a block to be encoded / decoded by copying pixels in block units from pixels included in the encoded / decoded image. In inter-prediction, when copying pixels from a reference image in block units, it is often the case that the device needs to be acquired in the management unit of the memory including the necessary pixels. Therefore, the smaller the block is divided and the longer the block is formed into an elongated rectangle, the greater the overall processing load becomes. Further, when performing decimal-precision motion compensation using an interpolation filter on a reference image, it is necessary to copy pixels in which several pixels are added to the pixels included in the block, and the smaller the block size, the more. The relative ratio of several pixels to be added becomes large, and the load of the whole processing becomes large. Therefore, when the inside of the block is divided into two, it is possible to prevent an increase in the memory bandwidth required for inter-prediction by limiting the division into two in the same direction as the division direction of the parent block.

Next, the relationship between the luminance signal and the color difference signal in the intra prediction will be described. As a format between the luminance signal and the color difference signal, 4: 2: 0, 4: 2: 2, 4: 4: 4, and the like have been conventionally known. In the 4: 2: 0 format shown in FIG. 17 (a), the brightness signal samples two pixels in the horizontal / vertical direction, whereas the color difference signal samples one pixel in the horizontal / vertical direction. Since the human eye can perceive the luminance signal more sensitively than the luminance signal, the amount of information of the luminance signal is reduced compared to the luminance signal. In the 4: 2: 2 format shown in FIG. 17B, the luminance signal is sampled in two pixels in the horizontal direction, whereas the color difference signal is sampled in one pixel in the horizontal direction. In the vertical direction, the luminance signal is sampled in two pixels in the vertical direction, whereas the color difference signal is sampled in two pixels in the vertical direction. FIG. 17 (c)
The 4: 4: 4 format shown in (1) samples two pixels in the horizontal / vertical direction for the brightness signal, whereas it samples two pixels in the horizontal / vertical direction for the color difference signal.

The present embodiment will be described by taking the most widely used 4: 2: 0 format in image coding as an example. The block dividing units 101 and 202 include a luminance block dividing unit that divides the luminance signal of the image to generate a luminance block and a luminance block dividing unit that divides the luminance signal of the image to generate a luminance block, and intra-predicts the intra. In, the luminance signal and the color difference signal are divided into blocks independently. That is, in the intra prediction, the size of the luminance block and the size of the color difference block are determined independently. In intra-prediction, since each of the luminance signal and the color difference signal copies the pixel value from the peripheral pixels, the prediction efficiency is improved by dividing the luminance signal and the color difference signal into blocks independently. On the other hand, for inter-prediction, the luminance signal and the color difference signal are treated together and divided into blocks. That is, in the inter-prediction, the size of the luminance block and the size of the color difference block are the same. This is because it is not necessary to distinguish between luminance and color difference in motion compensation in inter-prediction.

The prediction image generation units 102 and 204 include a luminance signal prediction unit that predicts a luminance signal and a color difference signal prediction unit that predicts a color difference signal, and the luminance signal prediction unit is used to improve the prediction efficiency of the color difference signal in the intra prediction. , Luminance color difference intra-prediction that predicts a color difference signal from the encoded / decoded pixels of the luminance signal is performed. In the luminance color difference intra prediction, the primary signal is encoded / decoded before the secondary signal, and the secondary signal is predicted using the encoded / decoded primary signal. Here, since the amount of information of the luminance signal is larger than that of the color difference signal in the 4: 2: 0 format and the 4: 2: 2 format, the luminance signal is used as the primary signal and the luminance signal is used as the secondary signal. In the 4: 4: 4 format, the amount of information of the luminance signal and the color difference signal is the same, but it is usual that the luminance signal is used as the primary signal and the color difference signal is used as the secondary signal according to other formats.

FIG. 18 is a diagram for explaining the luminance color difference intra prediction, and FIG. 19 is a flowchart for explaining the luminance color difference intra prediction.

As shown in FIG. 18, the luminance color difference intra-prediction includes the encoded / decoded peripheral pixels 12a and 12b of the luminance block 10 and the encoded / decoded peripheral pixels 16a of the luminance block 14.
It is performed based on the degree of correlation with 16b. Since the luminance color difference intra prediction predicts a color difference signal, the peripheral pixels for which the degree of correlation is calculated are defined with reference to the peripheral pixels of the color difference block to be encoded / decoded. That is, the peripheral pixels of the luminance block located at the same position as the peripheral pixels determined for the color difference block are the targets of the calculation of the degree of correlation.

First, the degree of correlation between the peripheral pixels of the encoded / decoded luminance signal and the peripheral pixels of the color difference signal is calculated (S1901). Subsequently, the coded / decoded luminance signal of the coded / decoded target block is downsampled (S1902). Here, a plurality of filter types to be downsampled may be prepared so that the filter type can be selected. For example, filters having different intensities may be prepared and a plurality of filter types may be selected, or filters having different tap counts may be prepared and a plurality of filter types may be selected. The filter type may be automatically selected by using the degree of correlation between peripheral pixels, or the filter type may be encoded / decoded in a bit stream and transmitted. Further, unless the downsample filter of the luminance signal of the block to be encoded / decoded is determined by using the degree of correlation between the peripheral pixels, the processes of step S1901 and step S1902 are in no particular order, and step S1901
And step S1902 can be processed in parallel.

Finally, the color difference signal is predicted from the downsampled luminance signal based on the degree of correlation between the peripheral pixels (S1903). The downsample is halved in the horizontal / vertical direction in the case of 4: 2: 0 format. In the case of 4: 2: 2 format, it is halved in the horizontal direction and is not downsampled in the vertical direction. In the case of 4: 4: 4 format, do not downsample in both horizontal and vertical directions.

In the luminance color difference intra prediction, the color difference signal prediction processing can be started after the coding / decoding of the luminance signal of the block to be encoded / decoded is completed. Therefore, the timing at which the color difference block prediction process can be started depends on the size of the luminance block and the size of the color difference block.

20 (a) and 20 (b) are diagrams for explaining the luminance color difference intra prediction when the size of the luminance block is larger than the size of the luminance block. The number of pixels of the first to fourth luminance blocks 20a, 20b, 20c, and 20d shown in FIG. 20A is 16x16, and the number of pixels of the color difference block 20e shown in FIG. 20B is 16x16. ..

Here, the comparison of the sizes of the luminance block and the color difference block is not a comparison based on the number of pixels in the block, but a comparison based on the area considering the color difference format. That is, in the 4: 2: 0 format, the area occupied by the luminance block is 1/2 of the area occupied by the luminance block, so that the number of pixels of the luminance block is 16x16 and the number of pixels of the luminance block is 16x16. , The size of the brightness block is smaller. In the 4: 2: 0 format, when the number of pixels of the luminance block is 16x16 and the number of pixels of the color difference block is 8x8, the area occupied by both blocks is the same, and the luminance block and the color difference block have the same size. is there.

To compare the size of the luminance block and the color difference block by comparing the number of pixels in the block instead of comparing the area of the block, the number of pixels of the luminance block is determined by the ratio of the luminance signal to the luminance signal in the luminance format. It may be converted into the number of pixels of. In the case of the 4: 2: 0 format, the number of pixels of the luminance signal is twice the number of pixels of the color difference signal, so the number of vertical and horizontal pixels of the color difference block is doubled and converted into the number of vertical and horizontal pixels of the luminance block. For example, in the 4: 2: 0 format, when the size of the luminance block is 16x16 and the size of the luminance block is 16x16, the converted size of the luminance block converted to the number of pixels of the luminance block is 32x32, and the color difference block You can see that the size is larger.

Intra-prediction enables prediction processing of subsequent blocks after completing decoding processing in block units. That is, after the decoding of the first luminance block 20a is completed, the second luminance block 2
After the decoding of 0b becomes possible and the decoding of the second luminance block 20b is completed, the decoding of the third luminance block 20c becomes possible, and after the decoding of the third luminance block 20c is completed, the fourth luminance block 20d Decryption becomes possible.

When the size of the color difference block is larger than the size of the luminance block, the color difference block 20e
The peripheral pixels required for the prediction process are the four luminance blocks 20a for both the luminance pixel and the color difference pixel.
It exists before decoding 20b, 20c, 20d, and the four luminance blocks 20a, 20b,
It is possible to calculate the degree of correlation between the peripheral pixels of the luminance signal and the peripheral pixels of the color difference signal in step S1901 of FIG. 19 without waiting for the decoding of 20c and 20d.

Next, after the decoding of the first luminance block 20a is completed, the downsampling of the luminance signal in step S1902 of FIG. 19 is executed. The pixels of the color difference block 20e corresponding to the position of the first luminance block 20a can be predicted without waiting for the completion of decoding of the second luminance block 20b. Similarly
After the decoding of the second luminance block 20b is completed, the luminance signal is downsampled, and the pixels of the color difference block 20e corresponding to the position of the second luminance block 20b are predicted without waiting for the completion of decoding of the third luminance block 20c. .. Further, after the decoding of the third luminance block 20c is completed, the luminance signal is downsampled, and the pixels of the color difference block 20e corresponding to the position of the third luminance block 20c are generated without waiting for the completion of decoding of the fourth luminance block 20d. Predict. Finally, after the decoding of the fourth luminance block 20d is completed, the luminance signal is downsampled to execute the fourth luminance block 2
The pixel of the color difference block 20e corresponding to the position of 0d is predicted.

21 (a) and 21 (b) are diagrams for explaining the luminance color difference intra prediction when the size of the luminance block is smaller than the size of the luminance block. The number of pixels of the luminance block 21a shown in FIG. 21A is 16x16, and the number of pixels of the four-divided first to fourth color difference blocks 21b, 21c, 21d, 21e shown in FIG. 21B is 4x4. ..

In the 4: 2: 0 format, the area occupied by the luminance block is 1/2 of the area occupied by the color difference block. Therefore, when the number of pixels of the luminance block is 16x16 and the number of pixels of the luminance block is 4x4, the color difference The block size is smaller. To compare the size of the luminance block and the color difference block by comparing the number of pixels in the block instead of comparing the area of the block, in the case of 4: 2: 0 format, the number of vertical and horizontal pixels of the color difference block is doubled. Then, it is converted into the number of vertical and horizontal pixels of the luminance block. In the 4: 2: 0 format, when the size of the luminance block is 16x16 and the size of the color difference block is 4x4, the converted size of the luminance block converted to the number of pixels of the luminance block is 8x8, which is the size of the color difference block. You can see that it is smaller.

When the size of the color difference block is smaller than the size of the luminance block, the first color difference block 2
The peripheral pixels of 1b can be used before decoding the brightness block 21a for both the brightness pixel and the color difference pixel, but the peripheral pixels of the second to fourth color difference blocks 21c, 21d, and 21e are the brightness block 21.
It cannot be used until the decoding of a is completed. That is, if the decoding of the luminance block 21a is completed and the decoding of the first color difference block 21b is not completed, the second color difference block 21c is shown in FIG.
The degree of correlation between the peripheral pixels of the luminance signal and the peripheral pixels of the color difference signal in step S1901 cannot be calculated. Similarly, for the third color difference block 21d, if the decoding of the luminance block 21a is completed and the decoding of the first and second color difference blocks 21b and 21c is not completed, the third color difference block 2
For 1d, the degree of correlation between the peripheral pixels of the luminance signal and the peripheral pixels of the color difference signal cannot be calculated. Similarly, for the fourth color difference block 21e, the decoding of the luminance block 21a is completed, and the first
-If the decoding of the third color difference blocks 21b, 21c, 21d is not completed, the fourth color difference block 2
For 1e, the degree of correlation between the peripheral pixels of the luminance signal and the peripheral pixels of the color difference signal cannot be calculated.

In this way, when the size of the color difference block is smaller than the size of the luminance block, when the luminance color difference intra prediction is performed, there is a processing dependency between the luminance block and the luminance block and between the luminance blocks in the color difference block prediction processing. Not suitable for parallel processing. Therefore, when the size of the color difference block is smaller than the size of the luminance block, the luminance color difference intra prediction is limited. As a method of limiting the luminance color difference intra-prediction, (1) limit by syntax, (
2) Replace the intra color difference mode, (3) Replace the peripheral pixels, and so on.

FIG. 22 shows an example of the syntax of the intra color difference prediction mode. When the number of the color difference prediction mode is 0, the same intra prediction mode as the luminance prediction mode is used for the color difference prediction mode. For example, when the luminance prediction mode is the horizontal prediction mode, the color difference prediction mode is also the horizontal prediction mode. When the number of the color difference prediction mode is 1, the average value mode (DC mode) is used. In DC mode, intra-prediction is made based on the average value of peripheral pixels. When the number of the color difference prediction mode is 2, the luminance color difference intra prediction mode is used.

As a method of limiting the luminance color difference intra prediction, (1) when limiting by syntax, mode 2 is prohibited because it represents the luminance color difference intra prediction. That is, mode 2 is not transmitted, and the intra color difference mode is selected from mode 0 and mode 1.

As a method of limiting the luminance color difference intra prediction, (2) when replacing the intra color difference mode, when the color difference prediction mode number is specified as mode 2, the vertical prediction mode is used instead of the luminance color difference intra prediction mode. To do. However, the prediction mode to be replaced is not limited to the vertical prediction mode, and other prediction modes may be used. Further, it is more preferable that the brightness prediction mode of mode 0 and the mode to be replaced by mode 2 are the same so that the modes used do not overlap.

As a method of limiting the luminance color difference intra-prediction, (3) when replacing peripheral pixels, FIG.
The peripheral pixels for calculating the degree of correlation between the peripheral pixels of the luminance signal and the peripheral pixels of the color difference signal in step S1901 of 9 are replaced. The second to fourth color difference blocks 21c and 21d in FIG. 21
Also in 21e, the degree of correlation of peripheral pixels can be calculated without waiting for the decoding of the luminance block 21a.

An example of replacing peripheral pixels is shown in FIG. Normally, the pixels in the first color difference block 21b are used as the peripheral pixels on the left side of the second color difference block 21c, but in order to use the pixels in the first color difference block 21b, the luminance block 21a and the first color difference block 21b are used. You need to wait for the decryption to complete. Therefore, the region 21f that can be used without waiting for the completion of decoding of the luminance block 21a is used as the peripheral pixels of the second color difference block 21c. Similarly, for the third color difference block 21d, the region 21f that can be used without waiting for the completion of decoding of the luminance block 21a is used as the peripheral pixels of the third color difference block 21d. Similarly, for the fourth color difference block 21e,
The fourth color difference block 21 covers the area 21f that can be used without waiting for the decoding of the luminance block 21a to be completed.
It is used as a peripheral pixel of e.

As described above, in the first embodiment, when the luminance color difference intra prediction is performed, when the size of the luminance block is smaller than the size of the luminance block, the luminance block and the luminance block are limited by limiting the luminance color difference intra prediction. It becomes possible to relax the processing dependency between them. As a result, parallel processing of the luminance block and the color difference block becomes possible, and the amount of coding / decoding processing can be reduced.

(Second Embodiment)
A second embodiment of the present invention will be described. In the second embodiment, not the magnitude relationship between the luminance block and the color difference block, but the size of the luminance block and the size of the luminance block are evaluated independently to limit the luminance color difference intra-prediction. Unlike the first embodiment, the other configurations and operations are the same as those of the first embodiment.

First, the limitation based on the size of the luminance block will be described. 24 (a) to 24 (Fig. 24)
C) shows the luminance color difference intra prediction due to the difference in the size of the luminance block. FIG. 24 (a),
As shown in FIG. 24B, the first luminance block 24a corresponds to the position of the first luminance block 24e, the second luminance block 24b corresponds to the position of the second luminance block 24f, and the third luminance block 24c corresponds to the position of the second luminance block 24f. The fourth luminance block 24d corresponds to the position of the third color difference block 24g.
Corresponds to the position of the color difference block 24h. Further, the luminance block 24i in FIG. 24C corresponds to the positions of the first to fourth color difference blocks 24e, 24f, 24g, and 24h in FIG. 24B.

The dependency between the luminance block and the color difference block will be described. When the size of the luminance block is small as shown in FIG. 24A, when the decoding of the first luminance block 24a is completed, the first color difference block 24e can be decoded. When the decoding of the second luminance block 24b is completed, the second luminance block 24b is completed.
Decoding of the color difference block 24f becomes possible. When the decoding of the third luminance block 24c is completed,
Decoding of the third color difference block 24 g becomes possible. When the decoding of the fourth luminance block 24d is completed, the decoding of the fourth color difference block 24h becomes possible.

On the other hand, when the size of the luminance block is large as shown in FIG. 24 (c), the luminance block 24i
If the decoding of the first to fourth color difference blocks 24e, 24f, 24g, and 24h is not completed, all the decoding of the first to fourth color difference blocks 24e, 24f, 24g, and 24h becomes impossible.

When the division of the luminance block and the division of the luminance block are determined independently, if the absolute size of the luminance block is large, the size of the luminance block is likely to be smaller than the size of the luminance block. Therefore, when the absolute size of the luminance block is equal to or larger than a predetermined size, the luminance color difference intra prediction of the corresponding color difference block is limited.

Similarly, if the absolute size of the color difference block is small, it is more likely that the size of the color difference block will be smaller than the size of the luminance block. Therefore, when the absolute size of the color difference block is equal to or less than a predetermined size, the luminance color difference intra prediction of the color difference block is limited.

The method for limiting the luminance color difference intra-prediction is the same as that in the first embodiment.

As described above, in the second embodiment, the luminance color difference intra prediction is limited when the absolute size of the luminance block is larger than the threshold value or when the absolute size of the luminance block is smaller than the threshold value. This makes it possible to limit the luminance color difference intra-prediction and stochastically relax the processing dependency between the luminance blocks and the luminance blocks in anticipation of the case where the size of the luminance block is smaller than the size of the luminance block. become.

(Third Embodiment)
A third embodiment of the present invention will be described. In the third embodiment, the block division portion 101 is different from the first embodiment in that the color difference block is divided so that the size of the color difference block is not smaller than the size of the luminance block. And the operation is the same as that of the first embodiment. When dividing the color difference block, the block division unit 101 prohibits the division so that the size of the color difference block is smaller than the size of the luminance block. As a result, in the luminance color difference intra prediction, the size of the luminance block is prevented from becoming smaller than the size of the luminance block, and parallel processing of the luminance block and the luminance block is always possible.

The coded bit stream of the image output by the image coding apparatus of the above-described embodiment is
It has a specific data format so that it can be decoded according to the coding method used in the embodiment, and the image decoding device corresponding to the image coding device has a code of this specific data format. The converted bitstream can be decrypted.

When a wired or wireless network is used to exchange the coded bitstream between the image encoding device and the image decoding device, the coded bitstream is converted into a data format suitable for the transmission form of the communication path. May be transmitted. In that case, a transmission device that converts the coded bit stream output by the image coding device into coded data in a data format suitable for the transmission form of the communication path and transmits the coded data to the network, and a transmission device that receives the coded data from the network and receives the coded data. A receiving device that restores the coded bitstream and supplies it to the image decoding device is provided.

The transmission device includes a memory that buffers the coded bit stream output by the image coding device, a packet processing unit that packets the coded bit stream, and a transmission unit that transmits the packetized coded data via the network. And include. The receiving device generates a coded bit stream by packet processing the coded data and a receiver for receiving the packetized coded data via the network, a memory for buffering the received coded data, and a packet process for the coded data. Includes a packet processing unit provided to the image decoding device.

In addition, by adding a display unit that displays the image decoded by the image decoding device to the configuration,
It can also be used as a display device. In that case, the display unit reads the decoded image signal generated by the decoded image signal superimposing unit 205 and stored in the decoded image memory 206 and displays it on the screen.

It is also possible to use the image pickup device by adding the image pickup unit to the configuration and inputting the captured image into the image coding device. In that case, the imaging unit inputs the captured image signal to the block dividing unit 101.

The above processing related to encoding and decoding can be realized as a transmission, storage, and receiving device using hardware, and is stored in a ROM (read-only memory), a flash memory, or the like. It can also be realized by the existing firmware or software such as a computer. The firmware program and software program may be recorded on a recording medium readable by a computer or the like and provided, or provided from a server via a wired or wireless network, or provided as terrestrial or satellite digital broadcasting data broadcasting. Is also possible.

The present invention has been described above based on the embodiments. Embodiments are examples, and it will be understood by those skilled in the art that various modifications are possible for each of these components and combinations of each processing process, and that such modifications are also within the scope of the present invention. ..

100 image encoding device, 101 block division unit, 102 prediction image generation unit,
103 Residual signal generation unit, 104 Orthogonal conversion / quantization unit, 105 Coding bit string generation unit, 106 Inverse quantization / inverse orthogonal conversion unit, 107 Decoded image signal superimposition unit, 108
Decoded image memory, 200 image decoding device, 201 bit string decoding unit, 202 block division unit, 203 inverse quantization / inverse orthogonal conversion unit, 204 prediction image generation unit, 20
5 Decoded image signal superimposition unit, 206 Decoded image memory.

Claims (6)

An image coding device that divides an image into blocks and encodes the divided blocks.
A primary signal block dividing unit that divides the primary signal of the image into rectangles of a predetermined size to generate a primary signal block, and
A secondary signal block dividing unit that divides the secondary signal of the image into rectangles of a predetermined size to generate a secondary signal block, and
The primary signal predictor that predicts the primary signal,
Includes a secondary signal prediction unit that predicts secondary signals
In the case of intra-prediction , the primary signal block and the secondary signal block are divided independently, and in the case of inter-prediction, they are divided together.
The primary signal is a luminance signal, the secondary signal is a color difference signal, and
The secondary signal prediction unit can perform inter-component prediction that predicts a secondary signal from a coded primary signal, and when the size of the primary signal block is a predetermined size or more, the use of the inter-component prediction is prohibited.
An image encoding device characterized in that. An image coding method that divides an image into blocks and encodes the divided blocks.
A primary signal block division step of dividing the primary signal of the image into rectangles of a predetermined size to generate a primary signal block, and
A secondary signal block division step of dividing the secondary signal of the image into rectangles of a predetermined size to generate a secondary signal block, and
The primary signal prediction step for predicting the primary signal and
Includes a secondary signal prediction step that predicts the secondary signal
In the case of intra-prediction , the primary signal block and the secondary signal block are divided independently, and in the case of inter-prediction, they are divided together.
The primary signal is a luminance signal, the secondary signal is a color difference signal, and
The secondary signal prediction step enables inter-component prediction that predicts a secondary signal from a coded primary signal, and prohibits the use of the inter-component prediction when the size of the primary signal block is a predetermined size or more.
An image coding method characterized by the fact that. An image coding program that divides an image into blocks and encodes the divided blocks.
A primary signal block division step of dividing the primary signal of the image into rectangles of a predetermined size to generate a primary signal block, and
A secondary signal block division step of dividing the secondary signal of the image into rectangles of a predetermined size to generate a secondary signal block, and
The primary signal prediction step for predicting the primary signal and
Have the computer perform a secondary signal prediction step and a secondary signal prediction step to predict the secondary signal.
In the case of intra-prediction , the primary signal block and the secondary signal block are divided independently, and in the case of inter-prediction, they are divided together.
The primary signal is a luminance signal, the secondary signal is a color difference signal, and
The secondary signal prediction step enables inter-component prediction that predicts a secondary signal from a coded primary signal, and prohibits the use of the inter-component prediction when the size of the primary signal block is a predetermined size or more.
An image coding program characterized by this. An image decoding device that decodes an image in block units.
A primary signal block dividing unit that divides the primary signal of the image into rectangles of a predetermined size to generate a primary signal block, and
A secondary signal block dividing unit that divides the secondary signal of the image into rectangles of a predetermined size to generate a secondary signal block, and
The primary signal predictor that predicts the primary signal,
Includes a secondary signal prediction unit that predicts secondary signals
In the case of intra-prediction , the primary signal block and the secondary signal block are divided independently, and in the case of inter-prediction, they are divided together.
The primary signal is a luminance signal, the secondary signal is a color difference signal, and
The secondary signal prediction unit can perform inter-component prediction that predicts a secondary signal from the decoded primary signal, and when the size of the primary signal block is a predetermined size or more, the use of the inter-component prediction is prohibited.
An image decoding device characterized in that. This is an image decoding method that decodes an image in block units.
A primary signal block division step of dividing the primary signal of the image into rectangles of a predetermined size to generate a primary signal block, and
A secondary signal block division step of dividing the secondary signal of the image into rectangles of a predetermined size to generate a secondary signal block, and
The primary signal prediction step for predicting the primary signal and
Includes a secondary signal prediction step that predicts the secondary signal
In the case of intra-prediction , the primary signal block and the secondary signal block are divided independently, and in the case of inter-prediction, they are divided together.
The primary signal is a luminance signal, the secondary signal is a color difference signal, and
The secondary signal prediction step can perform inter-component prediction that predicts a secondary signal from a decoded primary signal, and when the size of the primary signal block is a predetermined size or more, the use of the inter-component prediction is prohibited.
An image decoding method characterized by that. An image decoding program that decodes an image in block units.
A primary signal block division step of dividing the primary signal of the image into rectangles of a predetermined size to generate a primary signal block, and
A secondary signal block division step of dividing the secondary signal of the image into rectangles of a predetermined size to generate a secondary signal block, and
The primary signal prediction step for predicting the primary signal and
Have the computer perform a secondary signal prediction step and a secondary signal prediction step to predict the secondary signal.
In the case of intra-prediction , the primary signal block and the secondary signal block are divided independently, and in the case of inter-prediction, they are divided together.
The primary signal is a luminance signal, the secondary signal is a color difference signal, and
The secondary signal prediction step can perform inter-component prediction that predicts a secondary signal from a decoded primary signal, and when the size of the primary signal block is a predetermined size or more, the use of the inter-component prediction is prohibited.
An image decoding program characterized by this.

Images