KR20120118463A - Image processing device, method, and program - Google Patents

Abstract

The present invention relates to an image processing apparatus and method and a program capable of improving the prediction accuracy in the B picture, especially near the end of the screen.
The motion compensator generates a predictive image in the reference region of the L0 reference picture by weighted prediction of the H.264 / AVC method, and in the reference region of the L0 reference picture, The prediction picture is generated using only the reference region of the L1 reference picture without using it. That is, in the L0 reference picture, as shown in the reference region of the L0 reference, the reference region is an outer dashed line square, but is not actually used for prediction other than the region inside the inner dashed line square. The present invention can be applied to, for example, an image encoding apparatus that encodes based on an H.264 / AVC scheme.

Description

Image processing apparatus and method, and program {IMAGE PROCESSING DEVICE, METHOD, AND PROGRAM}

TECHNICAL FIELD The present invention relates to an image processing apparatus, a method, and a program, and more particularly, to an image processing apparatus, a method, and a program, which make it possible to improve the prediction accuracy in the B picture, especially near the end of the screen.

As standard standards for compressing image information, there are H.264 and MPEG-4 Part 10 (Advanced Video Coding, hereinafter referred to as H.264 / AVC).

In H.264 / AVC, inter prediction that pays attention to correlation between frames or fields is performed. In the motion compensation process performed in this inter prediction, a predictive image (hereinafter referred to as an inter prediction image) by inter prediction is generated using a partial region in the referenceable image that is already stored.

For example, as shown in Fig. 1, when five frames of the already stored referenceable picture become a reference frame, a part of the inter prediction pictures of the frame to be inter predicted (the original frame) is any one of five. The reference frame pictures (hereinafter referred to as reference pictures) are configured with reference to. In addition, the position of a part of the reference picture which becomes a part of an inter prediction picture is determined by the motion vector detected based on the picture of a reference frame and an original frame.

More specifically, as shown in FIG. 2, when the face 11 in the reference frame moves downward from the original frame to the right side, and about one third of the lower part is covered, the upper left side is opposite to the lower right side. The motion vector is detected. Then, the part 12 of the face 11 which is not hidden in the original frame is referred to the part 13 of the face 11 in the reference frame at the position where the part 12 is moved by the motion indicated by the motion vector. It is composed.

In addition, in H.264 / AVC, as shown in FIG. 3, motion compensation can be performed at a block size of 16x16 pixels to 4x4 pixels. As a result, when a motion boundary occurs inside a macro block (e.g., 16 x 16 pixels), the block size can be divided smaller according to the boundary, so that accurate motion compensation can be performed.

In addition, in H.264 / AVC, it is conceivable to improve the resolution of the motion vector to a fractional precision such as 1/2 or quarter in the motion compensation process.

In such fractional precision motion compensation processing, a process of setting a pixel of a virtual fractional position called a sub pel and generating the sub pel (hereinafter referred to as interpolation) is performed between adjacent pixels. That is, in the fractional precision motion compensation process, since the minimum resolution of the motion vector becomes the pixel at the fractional position, interpolation for generating the pixel at the fractional position is performed.

Fig. 4 shows each pixel of the image in which the number of pixels in the vertical direction and the horizontal direction is increased four times by interpolation. In FIG. 4, a white square represents an integer pixel (Integer pel (Int. Pel)), and a hatched square represents a fractional pixel (Sub pel). In addition, the alphabet in a square has shown the pixel value of the pixel which the square represents.

The pixel values b, h, j, a, d, f and r of the fractional position pixel generated by interpolation are represented by the following equation (1).

[Equation 1]

b = (E-5F + 20G + 20H-5I + J) / 32

h = (A-5C + 20G + 20M-5R + T) / 32

j = (aa-5bb + 20b + 20s-5gg + hh) / 32

a = (G + b) / 2

d = (G + h) / 2

f = (b + j) / 2

r = (m + s) / 2

Also, pixel values aa, bb, s, gg and hh are the same as b, cc, dd, m, ee and ff are the same as h, c is the same as a, f, n, q are the same as d, e, p and g can be calculated | required similarly to r, respectively.

The above-described equation (1) is an equation employed in interpolation such as H.264 / AVC, and this equation is different depending on the difference in the standard, but the purpose of the equation is the same. This equation can be realized by a finite impulse response (FIR) filter having an even number of taps. For example, a 6-tap interpolation filter is used in H.264 / AVC.

In addition, in H.264 / AVC, when the reference region of the motion vector is outside the screen end (screen frame), the pixel value at the screen end is copied as shown in FIG.

In the reference picture shown in the example of FIG. 5, a dashed-dotted line shows a screen end (screen frame), and the area | region between a dashed-dotted line and the outer solid line shows the area extended by the copy of the screen end. That is, the reference picture is expanded by the copy of the screen end.

However, in H.264 / AVC, in particular, in the case of a B picture, as shown in FIG. 6, bidirectional prediction may be used. In Fig. 6, pictures are shown in display order, and reference pictures that have been encoded before and after the display order of the pictures to be encoded are listed. When the picture to be encoded is a B picture, for example, as shown in a target prediction block of the picture to be encoded, the motion vector of the L0 prediction in the forward direction is referred to by referring to two blocks of the reference picture in front and rear (bidirectional), It may have a motion vector of backward L1 prediction.

That is, L0 mainly has a display time earlier than a target prediction block, and L1 mainly has a display time later than a target prediction block. Such distinct reference pictures can be used separately for each encoding mode. As shown in Fig. 7, there are five types of coding modes: intra picture coding (intra prediction), L0 prediction, L1 prediction, pair prediction, and direct mode.

7 is a diagram illustrating a relationship between an encoding mode, a reference picture, and a motion vector. In FIG. 7, the reference picture indicates whether or not to use the reference picture as the reference picture in the encoding mode, and the motion vector indicates whether or not the encoding mode has motion vector information.

The intra picture encoding mode is a mode that predicts in the picture (that is, intra), and is an encoding mode in which neither the L0 reference picture nor the L1 reference picture is used, and neither the motion vector of the L0 prediction nor the motion vector information of the L1 prediction. The L0 prediction mode is a coding mode that performs prediction using only the L0 reference picture and has motion vector information of the L0 prediction. In the L1 prediction mode, the prediction is performed using only the L1 reference picture, and is an encoding mode that includes motion vector information of the L1 prediction.

In the pair prediction mode, prediction is performed using the L0 and L1 reference pictures, and is an encoding mode in which the motion vector information of the L0 and L1 prediction is included. In the direct mode, the prediction is performed using the L0 and L1 reference pictures, but the encoding mode does not have motion vector information. In other words, the direct mode is an encoding mode which does not have motion vector information but predicts and uses motion vector information of the target prediction block at this point in time from motion vector information of a block in which a reference picture is completed. In the direct mode, only one of the L0 or L1 reference pictures may be provided.

As described above, in the pair prediction mode and the direct mode, both L0 and L1 reference pictures may be used. When there are two reference pictures, the prediction signal of the pair prediction mode or the direct mode can be obtained by the weighted prediction shown in the following expression (2).

&Quot; (2) "

Y _{Bi - Pred} = W ₀ Y ₀ + W ₁ Y ₁ + D

Here, Y _{Bi - Pred} is an offset-weighted interpolation signal in pair prediction mode or direct mode, W ₀ and W ₁ are weighting coefficients for L0 and L1, respectively, and Y ₀ and Y ₁ are L0 and L1. Motion compensation prediction signal. These W ₀ , W ₁ , and D are included in the bit stream information explicitly, or those obtained by calculation implicitly on the decoding side are used.

If the coding deterioration of the reference picture is uncorrelated in two reference pictures of L0 and L1, the coding deterioration is suppressed by this weighted prediction. As a result, the residual signal, which is the difference between the prediction signal and the input signal, is reduced, and the bit amount of the residual signal is reduced, thereby improving the coding efficiency.

Regarding the direct mode, in Non Patent Literature 1, when the reference area includes the outside of the screen, a proposal is made to use the reference picture of only the other side without using the reference picture.

By the way, in the H.264 / AVC system, the macro block size is 16x16 pixels. However, setting the macroblock size to 16x16 pixels is not optimal for large screen frames such as Ultra High Definition (4000x2000 pixels), such as those targeted for the next generation coding method.

Therefore, in Non-Patent Document 2 and the like, it is also proposed to expand the macroblock size to a size such as 32x32 pixels.

Itani Yusuke, Yuichi Idehara, Sunichi Sekiguchi, Yoshihi Yamada (Mitsubishi Denki), "Work Review of Direct Mode Improvement Methods in Video Coding", 24th Symposium Document Image Symposium Hosted by Electronics and Telecommunications Department , Ohira, Izu City, Shizuoka Prefecture, October 7, 8, 9, 2009 "Video Coding Using Extended Block Sizes", VCEG-AD09, ITU-Telecommunications Standardization Sector STUDY GROUP Question 16-Contribution 123, Jan 2009

As described above, when direct mode or pair prediction is used, reference regions of the L0 reference picture and the L1 reference picture are used. Here, a case where either the reference area of the L0 reference or the reference area of the L1 reference may be outside of the screen may occur.

In the example of FIG. 8, from the left side, the L0 reference picture, the encoding target picture, and the L1 reference picture are shown in the order of passage of time. In each picture, the dashed-dotted line represents the screen end, and the area | region between a solid line and the dashed-dotted line shows the area extended by the copy of the screen end mentioned above in FIG.

In addition, the region enclosed by the broken line in each picture represents the reference region of the L0 reference in the L0 reference picture, the motion compensation region in the encoding target picture, and the reference region of the L1 reference in the L1 reference picture. In particular, the reference region of the L0 reference and the reference region of the L1 reference are shown in the lower part of FIG. 8.

In Fig. 8, the rhombic object P hatched in the picture to be encoded is in a state of moving from the upper left to the lower right, and in the L0 reference picture, part of the object P exceeds the end of the screen. An external example is shown.

As described above with reference to Fig. 5, when the reference area is outside the screen, it is determined to copy and use the pixel value at the end of the screen in the H.264 / AVC system. As a result, in the reference region of the L0 reference picture, since the pixel value at the screen end is copied, the shape is not a rhombus.

When the predictive image is generated by the weighted prediction of the reference regions of L0 and L1, as in the reference region of the L0 reference in FIG. 8, if the pixel values outside the screen differ from the actual, the difference between the predicted image and the original signal This is expected to increase. Of course, a large difference means that the bit amount of the residual signal increases, which may cause a decrease in the coding efficiency.

On the other hand, a method of reducing the block size of the motion compensation can be considered. However, dividing the block size into small blocks may increase the header information of the macro block and increase the overhead. When the quantization parameter QP is large (or at a low bit rate), in particular, since the header information of the macroblock is proportionally large as an overhead, a method of dividing the block size into a small size may also cause a decrease in coding efficiency.

In addition, since the direct mode does not require motion vector information, there is an effect of reducing the header information of the macroblock, and contributes to the improvement of the coding efficiency, especially at a low bit rate. However, as described above, when a predictive image is generated by weighted prediction of the reference regions of L0 and L1, the pixel value outside the screen is different from the actual one, and the difference between the predicted image and the original signal becomes large. The direct mode is hard to be selected, and there is a fear of a decrease in coding efficiency.

In contrast, in the above-described Non-Patent Document 1, in the direct mode, when the reference area includes the outside of the screen, the selection of the direct mode is increased by using the reference picture only on the other side without using the reference picture. It is proposed to.

However, in this proposal, since one reference picture is not used completely, weighted prediction is not performed, and improvement of prediction performance in weighted prediction cannot be expected very much. That is, in the proposal of Non-Patent Document 1, even when most of the reference area is inside the screen and only a part is outside the screen, the reference area is not used at all.

In addition, in the non-patent document 1, only improvement of the direct mode is proposed, and pair prediction is not mentioned.

The present invention has been made in view of such a situation, and it is possible to improve the prediction accuracy in the B picture, especially near the end of the screen.

In the image processing apparatus of one aspect of the present invention, in prediction using a plurality of different reference images referred to by an image to be processed, whether or not the pixel to which the block reference of the image is outside the screen is the plurality of reference images. Motion prediction compensation means for performing weighted prediction according to the present invention.

When the block reference destination of the picture is a pixel in the screen in the plurality of reference pictures, the motion prediction compensation means performs weighted prediction determined by the standard using those pixels, and the reference point of the block of the picture is determined. In the case where the pixel outside the screen is the one of the plurality of reference images, and the pixel inside the screen is the other reference image, the weighting prediction can be performed using those pixels.

The weight of the weighted prediction is greater in weight than pixels in the screen than in weights in pixels outside the screen.

The weight of the weighting prediction is 0 or 1.

Weight calculation means for calculating the weight of the said weighting prediction by the discontinuity between the pixels of the block vicinity of the said image may further be provided.

It may further comprise encoding means for encoding the information of the weight calculated by the weight calculating means.

Decoding means calculated by the discontinuity between the pixels in the vicinity of the block of the image, and further comprising: decoding means for decoding the information of the encoded weight, the motion prediction compensation means, when performing the weighted prediction, by the decoding means Decoded weight information can be used.

The prediction using the different plurality of reference pictures is at least one of pair prediction and direct mode prediction.

In the image processing method of one aspect of the present invention, in the prediction in which the motion prediction compensating means of the image processing apparatus uses a plurality of different reference images referred to by the image of the processing target, the reference of the block of the image is the plurality of reference points. And performing weighted prediction according to whether or not it is outside the screen in the reference picture.

The program of one aspect of the present invention, in prediction using a plurality of different reference images referred to by an image to be processed, weights according to whether a reference of a block of the image is outside the screen in the plurality of reference images. A computer is functioned as a motion prediction compensation means for performing prediction.

In one aspect of the present invention, in the prediction using a plurality of different reference pictures referred to by a picture to be processed, the weighted prediction according to whether the reference of the block of the picture is outside the screen in the plurality of reference pictures is Is done.

The above-described image processing apparatus may be an independent apparatus or may be an internal block constituting one image coding apparatus or image decoding apparatus.

According to the present invention, it is possible to improve the prediction accuracy in the B picture, especially near the end of the screen. As a result, the coding efficiency can be improved.

1 is a diagram for explaining conventional inter prediction.
2 is a diagram illustrating in detail the conventional inter prediction.
It is a figure explaining a block size.
4 is a diagram illustrating interpolation.
5 is a diagram for describing processing at the screen end.
6 is a diagram for explaining bidirectional prediction.
7 is a diagram illustrating a relationship between an encoding mode, a reference picture, and a motion vector.
8 is a diagram illustrating a conventional weighting prediction.
9 is a block diagram showing the configuration of an embodiment of a picture coding apparatus according to the present invention.
FIG. 10 is a diagram illustrating weighted prediction of the picture coding apparatus of FIG. 9.
11 is a block diagram illustrating an exemplary configuration of a motion compensation unit.
FIG. 12 is a flowchart for describing an encoding process of the image encoding device of FIG. 9.
FIG. 13 is a flowchart for explaining a prediction mode selection process of the picture coding apparatus of FIG. 9.
FIG. 14 is a flowchart for describing a compensation process for a B picture in the picture encoding apparatus of FIG. 9.
15 is a diagram illustrating a prediction block.
16 is a diagram illustrating a correspondence relationship between reference pixel positions and processing methods.
17 is a diagram for explaining the effect of the example of FIG. 14.
18 is a block diagram showing the configuration of an embodiment of an image decoding device to which the present invention is applied.
19 is a block diagram illustrating a configuration example of a motion compensation unit of FIG. 18.
20 is a flowchart for describing a decoding process of the image decoding device of FIG. 18.
21 is a diagram illustrating an example of an extended block size.
Fig. 22 is a block diagram showing an example of the hardware configuration of a computer.
Fig. 23 is a block diagram showing a main configuration example of a television receiver to which the present invention is applied.
24 is a block diagram showing a main configuration example of a mobile telephone to which the present invention is applied.
Fig. 25 is a block diagram showing a main configuration example of a hard disk recorder to which the present invention is applied.
Fig. 26 is a block diagram showing a main configuration example of a camera to which the present invention is applied.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Configuration Example of Image Coding Apparatus]

9 shows a configuration of an embodiment of an image coding apparatus as an image processing apparatus to which the present invention is applied.

The picture coding apparatus 51 compresses and inputs the inputted picture based on, for example, H.264 and MPEG-4 Part 10 (Advanced Video Coding) (hereinafter referred to as H.264 / AVC) schemes.

In the example of FIG. 9, the image encoding device 51 includes an A / D converter 61, a screen rearrangement buffer 62, an arithmetic unit 63, an orthogonal transform unit 64, a quantization unit 65, and a reversible. Encoding unit 66, accumulation buffer 67, inverse quantization unit 68, inverse orthogonal transform unit 69, computing unit 70, deblocking filter 71, frame memory 72, intra prediction unit 73 ), A motion predictor 74, a motion compensator 75, a predictive image selector 76, and a rate controller 77.

The A / D conversion unit 61 performs A / D conversion on the input image, outputs it to the screen rearrangement buffer 62, and stores it. The screen rearrangement buffer 62 rearranges the images of the stored frames in the display order in the order of the frames for encoding in accordance with Gop (Group of Picture).

The calculating part 63 is a prediction image from the intra prediction part 73 selected by the prediction image selection part 76, or the prediction image from the motion compensation part 75 from the image read out from the screen rearrangement buffer 62. Is subtracted and the difference information is output to the orthogonal transform unit 64. The orthogonal transform unit 64 performs orthogonal transform such as discrete cosine transform, Karunen rube transform, and the like on the difference information from the calculator 63, and outputs the transform coefficient. The quantization unit 65 quantizes the transform coefficients output from the orthogonal transform unit 64.

The quantized transform coefficients that are output from the quantization unit 65 are input to the reversible coding unit 66, where reversible coding such as variable length coding and arithmetic coding is performed and compressed.

The reversible coding unit 66 obtains information indicating intra prediction from the intra predicting unit 73, and obtains information indicating the inter prediction mode and the like from the motion compensating unit 75. Note that the information indicating intra prediction and the information indicating inter prediction are also referred to as intra prediction mode information and inter prediction mode information, respectively.

The reversible coding unit 66 encodes the quantized transform coefficients, encodes information indicating intra prediction, information indicating an inter prediction mode, and the like as part of the header information in the compressed image. The reversible encoding unit 66 supplies the encoded data to the accumulation buffer 67 and stores it.

For example, the reversible coding unit 66 performs reversible coding processing such as variable length coding or arithmetic coding. Examples of variable length coding include CAVLC (Context-Adaptive Variable Length Coding), which is determined by the H.264 / AVC method. Examples of the arithmetic coding include CABAC (Context-Adaptive Binary Arithmetic Coding).

The accumulation buffer 67 outputs the data supplied from the reversible coding unit 66 as a coded compressed image, for example, a recording device, a transmission path, or the like (not shown) at a later stage.

The quantized transform coefficients output from the quantization unit 65 are also input to the inverse quantization unit 68 and dequantized, and then inverse orthogonal transformation is performed by the inverse orthogonal transform unit 69. The inverse orthogonal transformed output is added to the predictive image supplied from the predictive image selecting unit 76 by the calculating unit 70 to form a locally decoded image.

The decoded image from the calculating unit 70 is output to the intra predicting unit 73 and the deblocking filter 71 as a reference image of the image to be encoded. The deblock filter 71 removes the block distortion of the decoded image, and then supplies it to the frame memory 72 for accumulation. The frame memory 72 outputs the accumulated reference picture to the motion predictor 74 and the motion compensator 75.

In the picture coding apparatus 51, for example, an I picture, a B picture, and a P picture from the screen rearrangement buffer 62 are intra-prediction (also referred to as intra processing). Supplied. Further, the B picture and the P picture read out from the screen rearrangement buffer 62 are supplied to the motion predicting unit 74 as an image for inter prediction (also referred to as inter processing).

The intra prediction unit 73 performs intra prediction processing of all intra prediction modes to be candidates based on the intra predicted image read out from the screen rearrangement buffer 62 and the reference image from the calculation unit 70, thereby predicting. Create an image.

At that time, the intra prediction unit 73 calculates a cost function value for all the intra prediction modes which are candidates, and selects the intra prediction mode in which the calculated cost function value gives the minimum value as the optimal intra prediction mode.

The intra prediction unit 73 supplies the predictive image generated in the optimal intra prediction mode and its cost function value to the predictive image selection unit 76. The intra predicting unit 73 supplies the reversible coding unit 66 with information indicating the optimal intra prediction mode when the predictive image generated in the optimal intra prediction mode is selected by the predictive image selecting unit 76. The reversible coding unit 66 encodes this information and makes it part of the header information in the compressed image.

The motion predicting unit 74 performs motion prediction of all inter prediction modes of the blocks as candidates based on the inter-processed picture and the reference picture from the frame memory 72 to generate motion vectors of the respective blocks. The motion compensator 74 outputs the generated motion vector information to the motion compensator 75.

In addition, when the predictive image of the target block of the optimal inter prediction mode is selected by the predictive image selection unit 76, the motion predictor 74 may include information indicating the optimal inter prediction mode (inter prediction mode information) and motion vector information. , The reference frame information, etc. are output to the reversible encoder 66.

The motion compensation unit 75 performs an interpolation filter on the reference image from the frame memory 72. The motion compensator 75 compensates for the reference image after the filter for all inter prediction mode blocks that are candidates by using the motion vector obtained from the motion vector from the motion predictor 74 or the motion vector of the neighboring block. The process is performed to generate a predictive image. At this time, the motion compensation unit 75, in the B picture, in the direct mode or the pair prediction mode, that is, in the prediction mode using a plurality of different reference pictures, the pixels to which the reference target of the target block is, A weighted prediction is performed according to whether or not the reference picture is outside the screen to generate a predicted picture.

For example, in the motion compensation unit 75, when the reference destination of the target block is outside the screen in one reference picture and inside the screen in the other reference picture, the weight of one reference picture is reduced, and the other Weighting prediction is performed in which the weight of the reference picture is increased.

This weight may be calculated by the motion compensation unit 75 or a fixed value may be used. In addition, when it calculates, it is supplied to the reversible coding part 66, added to the header of a compressed image, and transmitted to a decoding side.

In addition, the motion compensation unit 75 obtains the cost function value of the block to be processed for all the inter prediction modes which are candidates, and determines the optimal inter prediction mode having the smallest cost function value. The motion compensation unit 75 supplies the predictive image generated in the optimal inter prediction mode and its cost function value to the predictive image selection unit 76.

The predictive image selection unit 76 determines the optimal prediction mode from the optimal intra prediction mode and the optimal inter prediction mode based on each cost function value output from the intra prediction unit 73 or the motion compensation unit 75. . Then, the predictive image selection unit 76 selects the predictive image of the determined optimal prediction mode and supplies it to the calculating units 63 and 70. At this time, the predictive image selecting unit 76 supplies the selection information of the predictive image to the intra predicting unit 73 or the motion predicting unit 74, as indicated by the dotted line.

The rate control unit 77 controls the rate of the quantization operation of the quantization unit 65 so that overflow or underflow does not occur based on the compressed image accumulated in the accumulation buffer 67.

[Features of the motion compensation unit]

Next, with reference to FIG. 10, the characteristic of the motion compensator 75 is demonstrated.

In the pair prediction or direct mode in which the weighted prediction is performed using two reference pictures (pictures) in the motion compensation unit 75, if both reference pixels (pixels) of L0 and L1 are inside the screen, H.264 Weighted prediction of the / AVC method is performed. On the other hand, if one reference pixel (pixel) of L0 or L1 is outside the screen, and the other reference pixel is inside the screen, prediction is performed using only the reference pixels inside the screen.

In the example of FIG. 10, as in the example of FIG. 8, the L0 reference picture, the encoding target picture, and the L1 reference picture are shown in chronological order from the left side. In each picture, the dashed-dotted line represents the screen end, and the area | region between a solid line and the dashed-dotted line shows the area extended by the copy of the screen end mentioned above in FIG.

In addition, the region enclosed by the broken line in each picture represents the reference region of the L0 reference in the L0 reference picture, the motion compensation region in the encoding target picture, and the reference region of the L1 reference in the L1 reference picture. In particular, the reference region of the L0 reference and the reference region of the L1 reference are shown in the lower part of FIG. 10.

In FIG. 10, the rhombic object P hatched in the picture to be encoded is in a state of moving from the upper left to the lower right, and in the L0 reference picture, part of the object P exceeds the end of the screen. An external example is shown. That is, part of the reference area of the L0 reference picture is outside the screen, and all of the reference area of the L1 reference picture is inside the screen.

Therefore, the motion compensator 75 generates a predictive image in the reference region of the L0 reference picture by the weighted prediction of the H.264 / AVC method, and in the reference region of the L0 reference picture, For the external part of the screen, a prediction image is generated using only the reference region of the L1 reference picture without using it. That is, in the L0 reference picture, as shown in the reference region of the L0 reference, the reference region is an outer dashed line square, but is not actually used for prediction except for an area inside the inner dashed line square.

For example, among the reference areas of the L0 reference picture, a weighted prediction in which the weight of the reference area of the L0 reference picture is 0 and the weight of the reference area of the L1 reference picture is 1 is performed for the portion outside the screen. All. The weight may not be 0 or 1, and the weight of the outer portion of the screen in one reference region may be smaller than the weight of the inner portion of the screen in the other reference region. In this case, the weight may be fixed or an optimal weight may be calculated.

As a result, inaccurate information that was a copy of the pixel value inside the screen outside of the screen is not used, or the weight thereof is reduced, so that the prediction performance at the end of the screen can be improved.

[Configuration example of the motion compensation unit]

11 is a diagram illustrating a configuration example of a motion compensation unit.

The motion compensator 75 of FIG. 11 is composed of an interpolation filter 81, a compensation processor 82, a selector 83, a motion vector predictor 84, and a prediction mode determiner 85.

Reference frame (reference picture) information from the frame memory 72 is input to the interpolation filter 81. The interpolation filter 81 interpolates between the pixels of the reference frame, enlarges the image 4 times horizontally and outputs the same to the compensation processor 82.

The compensation processor 82 is composed of an L0 area selection unit 91, an L1 area selection unit 92, a calculation unit 93, a screen end determination unit 94, and a weight calculation unit 95. In the example of the compensation processing unit 82 in FIG. 11, an example in the case of a B picture is shown.

The reference frame information expanded from the interpolation filter 81 is input to the L0 area selection unit 91, the L1 area selection unit 92, and the screen end determination unit 94.

The L0 region selection unit 91 selects a corresponding L0 reference region from the enlarged L0 reference frame information according to the prediction mode information and the L0 motion vector information from the selection unit 83, and outputs it to the calculation unit 93. do. The output reference information is output to the prediction mode determination unit 85 as the L0 prediction information in the L0 prediction mode.

The L1 region selection unit 92 selects the corresponding L1 reference region from the enlarged L1 reference frame information according to the prediction mode information and the L1 motion vector information from the selection unit 83, and outputs it to the calculation unit 93. do. The output reference information is output to the prediction mode determiner 85 as L1 prediction information in the L1 prediction mode.

The calculating part 93 is comprised by the multiplier 93A, the multiplier 93B, and the adder 93C. The multiplier 93A multiplies the L0 reference area information from the L0 area selection unit 91 by the L0 weight information from the screen end determination unit 94 and outputs the result to the adder 93C. The multiplier 93B multiplies the L1 reference area information from the L1 area selection unit 92 by the L1 weight information from the screen end determination unit 94 and outputs the result to the adder 93C. The adder 93C adds the L0 reference region and the L1 reference region weighted from the L0 and L1 weight information, and outputs the weighted prediction information (Bi-pred prediction information) to the prediction mode determiner 85.

The enlarged reference frame information from the interpolation filter 81 and the motion vector information from the selection unit 83 are supplied to the screen end determination unit 94. The screen end determination unit 94 determines whether the L0 reference pixel or the L1 reference pixel is outside the screen based on those information, and supplies the multiplier 93A and the multiplier 93B according to the determination result. Output weighting coefficients. For example, if they are all inside or outside the screen, a weighting factor W = 0.5 is output. When either one is outside the screen and the other is inside the screen, a weighting coefficient smaller than at least the reference pixel inside the screen is given to at least the reference pixel outside the screen.

The weight calculator 95 calculates a weighting coefficient used when only one of the L0 reference pixel and the L1 reference pixel is outside the screen according to the characteristics of the input image, and supplies it to the screen end determination unit 94. do. The calculated weighting coefficient is also output to the reversible coding unit 66 in order to send it to the decoding side.

The selector 83 selects either the motion vector information searched by the motion predictor 74 or the motion vector information obtained by the motion vector predictor 84 according to the prediction mode, and selects the selected motion vector information. Is supplied to the screen end determination unit 94, the L0 area selection unit 91, and the L1 area selection unit 92.

The motion vector predicting unit 84 predicts the motion vector according to the mode in which the motion vector is not sent to the decoding side, such as the skip mode or the direct mode, and supplies it to the selection unit 83.

The motion vector prediction method is similar to the H.264 / AVC method, and the motion vector predictor 84 predicts spatial prediction and co-located blocks that predict median prediction from the motion vectors of neighboring blocks. Temporal prediction and the like that are predicted from the motion vector of the located block are performed according to the mode. The collocated block is a block of pictures (pictures located before or after) different from the picture of the target block, and is a block at a position corresponding to the target block.

In addition, in the example of FIG. 11, although the illustration is abbreviate | omitted, the motion vector information etc. of the surrounding block at the time of obtaining | requiring are acquired from the selection part 83. FIG.

[Explanation of Weighting Factor]

The weighting coefficient information supplied in accordance with the determination result by the screen end determining unit 94 and multiplied by the calculating unit 93 is the other reference pixel when one reference pixel of L0 and L1 is outside the screen. Multiply by. The value is taken from 0.5 to 1, and the weights multiplied by one pixel on the outside of the screen are added to each other.

Therefore, when the L0 weighting coefficient information is W _L0 , the L1 weighting coefficient information is W _L1 = 1-W _L0 . As a result, the calculation in the calculating part 93 of FIG. 11 becomes following Formula (3).

&Quot; (3) "

Y = W _L0 I _L0 + (1-W _L0 ) I _L1

Here, Y is a weighted prediction signal, I _L0 is an L0 reference pixel, and I _L1 is an L1 reference pixel.

In addition, this weighting coefficient can be calculated by the weight calculation part 95. The weight calculator 95 calculates a weight based on, for example, the strength and weakness of the correlation between pixels. When the correlation is weak for adjacent pixels in the screen, that is, when there are many gaps between adjacent pixel values, the weight information W approaches 1 because the pixel value copied from the pixel at the end of the screen has low reliability. When the correlation is strong, as in the H.264 / AVC method, since the pixel value obtained by copying the pixel at the end of the screen is reliable, the weight information W approaches 0.5.

The method of investigating the strength and weakness of the correlation between pixels is a method of calculating the average inside the screen of the absolute value of the difference between adjacent pixels, a method of calculating the magnitude of the dispersion of pixel values, a Fourier transform, or the like. Finding the size of and investigating.

As the simplest example, the outside of the screen may not be trusted and the weight W may be fixed to one. In this case, since the weight information does not need to be sent to the decoding side, it is not necessary to include it in the stream information.

In addition, since the weight outside the screen becomes 0, the multiplier 93A, the multiplier 93B, and the adder 93C of the calculation unit 93 are unnecessary, and can be replaced with a simpler selection circuit.

[Description of Encoding Process of Image Coding Device]

Next, with reference to the flowchart of FIG. 12, the encoding process of the image coding apparatus 51 of FIG. 9 is demonstrated.

In step S11, the A / D conversion unit 61 performs A / D conversion on the input image. In step S12, the screen rearrangement buffer 62 stores the image supplied from the A / D conversion unit 61 and rearranges the sequence from the display order of each picture to the encoding order.

In step S13, the calculating unit 63 calculates the difference between the rearranged image and the predictive image in step S12. The predicted image is supplied from the motion compensator 75 for inter prediction and from the intra predictor 73 for intra prediction to the computation unit 63 via the predictive image selection unit 76.

The difference data has a smaller data amount compared to the original image data. Therefore, the data amount can be compressed as compared with the case where the image is encoded as it is.

In step S14, the orthogonal transform unit 64 orthogonally transforms the difference information supplied from the calculation unit 63. Specifically, orthogonal transformations, such as a discrete cosine transform and a Karune Rube transform, are performed, and a transform coefficient is output. In step S15, the quantization unit 65 quantizes the transform coefficients. In this quantization, the rate is controlled as described in the processing of step S26 described later.

The difference information quantized as described above is locally decoded as follows. That is, in step S16, the inverse quantization unit 68 dequantizes the transform coefficient quantized by the quantization unit 65 to a characteristic corresponding to that of the quantization unit 65. In step S17, the inverse orthogonal transform unit 69 performs inverse orthogonal transform of the transform coefficients dequantized by the inverse quantization unit 68 into a characteristic corresponding to the characteristic of the orthogonal transform unit 64.

In step S18, the calculating unit 70 adds the predicted image input through the predictive image selecting unit 76 to the locally decoded difference information, and the image decoded locally (the image corresponding to the input to the calculating unit 63). ). The deblocking filter 71 filters the image output from the calculating part 70 in step S19. As a result, block distortion is eliminated. In step S20, the frame memory 72 stores the filtered image.

In step S21, the intra prediction unit 73 performs an intra prediction process. Specifically, the intra predicting unit 73 selects all intra candidates based on the intra predicted image read from the screen rearrangement buffer 62 and the image (unfiltered image) supplied from the calculating unit 70. Intra prediction processing in the prediction mode is performed to generate an intra prediction image.

The intra prediction unit 73 calculates a cost function value for all intra prediction modes which are candidates. The intra prediction unit 73 determines, as an optimal intra prediction mode, an intra prediction mode that gives a minimum value among the calculated cost function values. The intra prediction unit 73 supplies the intra prediction image generated in the optimal intra prediction mode and its cost function value to the prediction image selection unit 76.

When the image of the processing target supplied from the screen rearrangement buffer 62 is an interprocessed image, the referenced image is read from the frame memory 72, and the motion predicting unit 74 and the motion compensation are performed via the switch 73. It is supplied to the part 75.

In step S22, the motion predicting unit 74 and the motion compensating unit 75 perform a motion predicting and compensating process. Specifically, the motion predicting unit 74 performs motion prediction of all inter prediction modes of the blocks as candidates based on the inter-processed picture and the reference picture from the frame memory 72, and the motion vector of each block. Create The motion compensator 74 outputs the generated motion vector information to the motion compensator 75.

The motion compensation unit 75 performs an interpolation filter on the reference image from the frame memory 72. The motion compensator 75 uses the motion vector obtained from the motion vector from the motion predictor 74 or the motion vector of the surrounding blocks to compensate for the reference image after the filter for all the blocks of the inter prediction modes as candidates. Is performed to generate a predictive image.

In this case, the motion compensator 75 is a pixel in the reference picture of the target block when the B picture is in the direct mode or the pair prediction mode, that is, in the prediction mode using a plurality of different reference pictures. A weighted prediction is performed according to whether or not the reference picture is outside of the screen to generate a predicted picture. In addition, the compensation process in the case of this B picture is mentioned later with reference to FIG.

In addition, the motion compensation unit 75 obtains the cost function value of the block to be processed for all the inter prediction modes which are candidates, and determines the optimal inter prediction mode having the smallest cost function value. The motion compensation unit 75 supplies the predictive image generated in the optimal inter prediction mode and its cost function value to the predictive image selection unit 76.

In step S23, the predictive image selecting unit 76 selects one of the optimal intra prediction mode and the optimal inter prediction mode based on the respective cost function values output from the intra predicting unit 73 and the motion compensating unit 75, Determine the best prediction mode. The predictive image selection unit 76 then selects the predicted image of the determined optimal prediction mode and supplies it to the computation units 63 and 70. As described above, this predictive image is used for the calculation of steps S13 and S18.

In addition, the selection information of this predictive image is supplied to the intra predictor 73 or the motion predictor 74, as shown by the dotted line in FIG. When the predictive image of the best intra prediction mode is selected, the intra predicting unit 73 supplies the reversible coding unit 66 with information indicating the best intra prediction mode (that is, the intra prediction mode information).

When the predictive picture of the optimal inter prediction mode is selected, the motion predictor 74 outputs information indicating the optimal inter prediction mode, motion vector information, and reference frame information to the reversible coding unit 66. In addition, when the weight is calculated by the motion compensator 75, since information on which the inter prediction image is selected is also supplied to the motion compensator 75, the motion compensator 75 uses the calculated weighting coefficient information as a reversible encoder 66. )

In step S24, the reversible encoding unit 66 encodes the quantized transform coefficients output from the quantization unit 65. That is, the difference image is reversibly encoded such as variable length coding, arithmetic coding, and the like, and is compressed. At this time, the intra prediction mode information from the intra prediction unit 73, the optimum inter prediction mode from the motion compensation unit 75, the above-described respective information, and the like, which are input to the reversible coding unit 66 in step S23 described above, are also described. It is encoded and added to the header information.

For example, the information indicating the inter prediction mode is encoded for each macro block. Motion vector information and reference frame information are encoded for each target block. The information of the weighting coefficient may be for each frame or may be for each sequence (scene from start to end of imaging).

In step S25, the accumulation buffer 67 accumulates the difference image as a compressed image. The compressed image accumulated in the accumulation buffer 67 is appropriately read and transmitted to the decoding side via the transmission path.

In step S26, the rate control part 77 controls the rate of the quantization operation of the quantization part 65 so that an overflow or an underflow may not occur based on the compressed image accumulated in the accumulation buffer 67. FIG.

[Explanation of prediction mode selection processing]

In the image encoding device 51 of FIG. 9, when encoding the macroblock, it is necessary to determine an optimal mode among a plurality of prediction modes. A typical determination method is a multi-pass encoding method, wherein a motion vector, a reference picture, and a prediction mode are determined to minimize the cost (i.e., cost function value) using the following Equation 4 or Equation 5.

&Quot; (4) "

Cost = SATD + λ _Motion GenBit

[Equation 5]

Cost = SSD + λ _Mode GenBit

Here, SATD (Sum of Absolute Transformed Difference) is subjected to the Hadamard transform on the prediction error and is the sum of the absolute values. The sum of square difference (SSD) is a sum of squared errors, which is the sum of the squares of the prediction errors of each pixel. GenBit (Generated Bit) is the amount of bits generated when the macroblock is encoded in the candidate mode. λ _{M oti on} and λ _Mode are variables called Lagrange multipliers and are determined by the quantization parameters QP, I / P picture, and B picture.

The prediction mode selection processing of the picture coding apparatus 51 using the above-described equations (4) and (5) will be described with reference to FIG. In addition, this prediction mode selection process is a process which focuses on prediction mode selection in step S21-S23 of FIG.

In step S31, the intra predictor 73 and the motion compensator 75 (prediction mode determiner 85) calculate lambda from the quantization parameter QP and picture type, respectively. Although the arrow is not shown, the quantization parameter QP is supplied from the quantization unit 65.

In step S32, the intra prediction unit 73 determines the intra 4x4 mode so that the cost function value becomes small. There are nine types of prediction modes in the intra 4x4 mode, and among them, the one with the smallest cost function value is determined as the intra 4x4 mode.

In step S33, the intra prediction unit 73 determines the intra 16x16 mode so that the cost function value becomes small. There are four types of prediction modes in the intra 16x16 mode, and among them, the one with the smallest cost function value is determined as the intra 16x16 mode.

In step S34, the intra prediction unit 73 determines the mode having the smallest cost function value as the optimal intra mode among the intra 4 × 4 mode and the intra 16 × 16. The intra prediction unit 73 supplies the predictive image determined in the determined optimal intra mode and the cost function value to the predictive image selection unit 76.

The processes of the above steps S32 to S34 are the processes corresponding to the step S21 of FIG. 12.

In step S35, the motion predicting unit 74 and the motion compensating unit 75 calculate the motion vector and the reference picture for each of the following modes in the 8x8 macroblock subpartition shown in the lower part of FIG. Decide to be smaller. Each mode includes a direct mode for 8x8, 8x4, 4x8, 4x4, and B pictures.

In step S36, the motion predicting unit 74 and the motion compensating unit 75 determine whether the image being processed is the B picture, and if it is determined that the B picture is the B picture, the process proceeds to step S37. In step S37, the motion predicting unit 74 and the motion compensating unit 75 determine the motion vector and the reference picture so that the cost function value becomes smaller even for pair prediction.

If it is determined in step S36 that it is not a B picture, step S37 is skipped and the process proceeds to step S38.

In step S38, the motion predicting unit 74 and the motion compensating unit 75 determine, in the macroblock partition shown in the upper part of FIG. 3, the motion vector and the reference picture for each of the following modes so that the cost function value becomes small. . Each mode includes 16x16, 16x8, 8x16, direct mode and skip mode.

In step S39, the motion predicting unit 74 and the motion compensating unit 75 determine whether the image being processed is a B picture, and if it is determined that the B picture is a B picture, the process proceeds to step S40. In step S40, the motion predicting unit 74 and the motion compensating unit 75 determine the motion vector and the reference picture so that the cost function value becomes smaller even for pair prediction.

If it is determined in step S39 that the picture is not a B picture, step S40 is skipped and the process proceeds to step S41.

In step S41, the motion compensation unit 75 (prediction mode determination unit 85) determines, among the above-described macro block partitions and sub macro block partitions, the mode having the smallest cost function value as the optimal inter mode. The prediction mode determination unit 85 supplies the prediction image selection unit 76 with the prediction image obtained in the determined optimal inter mode and its cost function value.

The processes of the above steps S35 to S41 are the processes corresponding to the step S22 of FIG. 12.

In step S42, the predictive image selection unit 76 determines the mode having the smallest cost function value among the optimal intra mode and the optimal inter mode. The process of this step S42 is a process corresponding to step S23 of FIG.

As described above, the motion vector and the reference picture (in the case of the inter) and the prediction mode are determined. Here, for example, in the pair prediction and direct mode in the case of the B picture of step S37 or step S40 in FIG. 13, when the motion vector is determined, the predicted image compensated by the process of FIG. 14 described below is used.

14 is a flowchart for describing compensation processing in the case of a B picture. That is, FIG. 14 shows the process specialized in the B picture of the motion prediction / compensation process in step S22 of FIG. In the example of FIG. 14, for the sake of simplicity, a case where the weighting coefficient for the reference pixel outside the screen is 0 and the weighting coefficient for the reference pixel in the screen is 1 will be described.

In step S51, the selection unit 83 determines whether the mode to be processed is the direct mode or the pair prediction. When it is determined in step S51 that the direct mode and the pair prediction are not, the process proceeds to step S52.

In step S52, the compensation processing unit 82 predicts the block according to the mode (L0 prediction or L1 prediction).

That is, in the case of L0 prediction, the selection unit 83 sends the prediction mode information and the L0 motion vector information only to the L0 region selection unit 91. The L0 region selection unit 91 selects a corresponding L0 reference region from the enlarged L0 reference frame information according to the prediction mode (which indicates L0 prediction) information and the L0 motion vector information from the selection unit 83, The prediction mode determiner 85 outputs the result. The same applies to the case of L1.

If it is determined in step S51 that it is a direct mode or pair prediction, the process proceeds to step S53. In this case, the prediction mode information and the motion vector information from the selection unit 83 are supplied to the L0 area selection unit 91, the L1 area selection unit 92, and the screen end determination unit 94.

Correspondingly, the L0 region selection unit 91 corresponds to the prediction mode (which indicates the direct mode or the pair prediction) information from the selection unit 83 and the L0 reference frame information expanded according to the L0 motion vector information. The L0 reference area is selected and output to the calculation unit 93. The L1 region selection unit 92 selects the corresponding L1 reference region from the enlarged L1 reference frame information according to the prediction mode information and the L1 motion vector information from the selection unit 83, and outputs it to the calculation unit 93. do.

The screen end determination unit 94 then determines whether the reference pixel is outside the screen in steps S53 to S57 and S60 below. In the following description, the coordinates of the prediction pixel in the prediction block shown in FIG. 15 are referred to.

In FIG. 15, block_size_x represents the size of the prediction block in the x direction, and block_size_y represents the size of the prediction block in the y direction. In addition, i represents the x coordinate of the prediction pixel in the prediction block, and j represents the y coordinate of the prediction pixel in the prediction block.

In the case of FIG. 15, since the example of the prediction block is 4x4 pixels, (block_size_x, block_size_y) = (4, 4), 0≤i, j≤3. Therefore, it can be seen that the prediction pixel shown in FIG. 15 has coordinates of x = i = 2 and y = j = 0.

In step S53, the screen end determining unit 94 determines whether j starting from the value 0 is smaller than block_size_y, and terminates the process when it is determined that j is larger than block_size_y. On the other hand, when it is determined in step S53 that j is smaller than block_size_y, that is, while j is 0 to 3, the processing proceeds to step S54, and the processing thereafter is repeated.

In step S54, the screen end determining unit 94 determines whether i starting at a value of 0 is smaller than block_size_x, and if it is determined that i is larger than block_size_x, the process returns to step S53, and The process is repeated. In addition, when it is determined in step S54 that i is smaller than block_size_x, that is, while i is 0 to 3, the processing proceeds to step S55, and the processing thereafter is repeated.

In step S55, the screen end determination unit 94 obtains a reference pixel using the L0 motion vector information mvL0x and mvL0y and the L1 motion vector information mvL1x and mvL1y. That is, the y coordinate yL0, the x coordinate xL0 of the reference pixel of L0, and the y coordinate yL1, x coordinate xL1 of the reference pixel of L1 are calculated | required by following formula (6).

&Quot; (6) "

yL0 = mvL0y + j

xL0 = mvL0x + i

yL1 = mvL1y + j

xL1 = mvL1x + i

In step S56, the screen end determination unit 94 determines whether the y coordinate yL0 of the reference destination pixel of L0 is smaller than 0, or equal to or greater than the height of the screen frame (height: the size in the y direction of the screen), or the L0 reference. It is determined whether the x coordinate xL0 of the destination pixel is smaller than 0 or greater than the width of the screen frame (width: size in the x direction of the screen).

That is, in step S56, it is determined whether the following equation (7) is satisfied.

[Equation 7]

In step S56, when it is determined that equation (7) is satisfied, the process proceeds to step S57. In step S57, the screen end determining unit 94 determines whether the y coordinate yL1 of the reference pixel of L1 is smaller than 0, or is equal to or greater than the height of the screen frame (height: the size in the y direction of the screen), or the reference destination of L1. It is determined whether the x coordinate xL1 of the pixel is less than 0 or more than the width of the screen frame (the size of the x direction of the screen).

That is, in step S57, it is determined whether the following expression (8) is satisfied.

[Equation 8]

In step S57, when it is determined that equation (8) is satisfied, the process proceeds to step S58. In this case, since neither the L0 reference pixel nor the L1 reference pixel is a pixel outside the screen, the screen end determining unit 94 performs weighting coefficient information of weighted prediction based on the H.264 / AVC method with respect to the pixel. Is supplied to the calculating part 93. Correspondingly, in step S58, the calculating unit 93 performs weighting prediction on the pixel by the H.264 / AVC method.

In step S57, when it is determined that equation (8) is not satisfied, the process proceeds to step S59. In this case, since the L0 reference destination pixel is a pixel outside the screen, and the L1 reference destination pixel is a pixel inside the screen, the screen end determination unit 94 provides the L0 weighting coefficient information (0) and the L1 weighting coefficient with respect to the pixel. The information 1 is supplied to the calculating part 93. Correspondingly, in step S59, the calculating unit 93 performs prediction using only the L1 reference pixel for the pixel.

If it is determined in step S56 that equation (7) is not satisfied, the process proceeds to step S60. In step S60, the screen end determination unit 94 determines whether the y coordinate yL1 of the reference destination pixel of L1 is smaller than 0, or equal to or greater than the height of the screen frame (height: the size in the y direction of the screen), or the L1 reference. It is determined whether the x-coordinate xL1 of the destination pixel is smaller than 0 or more than the width of the screen frame (the size in the x direction of the screen).

That is, in step S60, it is determined whether or not the above expression (8) is satisfied. In step S60, when it is determined that equation (8) is satisfied, the process proceeds to step S61.

In this case, since the L1 reference pixel is a pixel outside the screen, and the L0 reference pixel is a pixel inside the screen, the screen end determination unit 94 provides the L0 weighting coefficient information (1) and the L1 weighting coefficient with respect to the pixel. The information (0) is supplied to the calculation unit 93. Correspondingly, in step S61, the calculation unit 93 performs prediction using only the L0 reference pixel for the pixel.

On the other hand, when it is determined in step S60 that Equation 8 is not satisfied, since all of the pixels are pixels in the screen, the process proceeds to step S58, and the pixels are weighted by the H.264 / AVC method. Prediction is made.

The weighted (Bi-pred) prediction information of the result of which the weighting prediction is performed by the calculating part 93 in step S58, S59, or S61 is output to the prediction mode determination part 85. FIG.

Incorporating the above processing, it is as shown in FIG. In the example of FIG. 16, the correspondence relationship between a reference pixel position and a processing method is shown.

In other words, when the position of the reference pixel in the L0 reference area and the position of the reference pixel in the L1 reference area are also inside the screen, that is, when " Yes " in step S57 of FIG. As a method, weighted prediction of the H.264 / AVC scheme is used.

If the position of the reference pixel in the L0 reference area is outside the screen and the position of the reference pixel in the L1 reference area is inside the screen, that is, "No" in step S57 of FIG. As the processing method, weighted prediction that weights the L1 reference pixel in the screen rather than the L0 reference pixel outside the screen is used. In addition, since the example shown in FIG. 14 is an example in which weighting coefficients are 0 and 1, the prediction using only L1 reference pixel is used.

If the position of the reference pixel in the L1 reference area is outside of the screen and the position of the reference pixel in the L0 reference area is inside the screen, that is, "yes" in step S60 of FIG. As a processing method, weighted prediction that weights the L0 reference pixel inside the screen rather than the L1 reference pixel outside the screen is used. In addition, since the example shown in FIG. 14 is an example in which weighting coefficients are 0 and 1, the prediction using only L0 reference pixel is used.

When the position of the reference pixel in the L0 reference area and the position of the reference pixel in the L1 reference area are also outside the screen, that is, when "No" in step S60 of FIG. 14, the processing method for the pixel is used. , Weighted prediction of H.264 / AVC method is used.

Next, with reference to FIG. 17, the effect in the case of the example of FIG. 14 is demonstrated.

In the example of FIG. 17, the inside of the screen of the L0 reference picture, the current picture, and the L1 reference picture is shown in order from the left. The broken line portion in the L0 reference picture shows the outside of the screen.

That is, the reference block of the L0 reference picture indicated by the motion vector MV (L0) found in the block of the current picture is composed of an outer portion (dashed line) and an inner portion (white line) of the current picture. The reference block of the L1 reference picture indicated by the motion vector MV (L1) found in the block is composed of a screen inner portion (white line portion).

Conventionally, in the H.264 / AVC system, both reference blocks have been used for weighted prediction of the block using the weighting coefficients w (L0) and w (L1), with or without an outside portion of the screen.

In contrast, in the present invention (particularly in the case of the example of FIG. 14), the out-of-screen portion in the L0 reference block is not used for weighted prediction of the block using the weighting coefficients w (L0) and w (L1). . Only the outer portion of the picture in the L0 reference block is used only for the pixels of the L1 reference block for weighted prediction of the block.

That is, since the pixels of the outside portion of the screen which are likely to be incorrect information are not used for prediction, the prediction accuracy is improved than the weight prediction of the H.264 / AVC method. Of course, it is not limited to the example of FIG. 14 in which the weighting coefficients are 0 and 1, and even when the weighting coefficient of the screen external portion is lower than the weighting coefficient of the screen internal portion, the prediction is more accurate than the weight prediction of the H.264 / AVC method. Precision is improved.

The encoded compressed image is transmitted via a predetermined transmission path and decoded by an image decoding device.

[Configuration example of image decoding device]

18 shows the configuration of an embodiment of an image decoding device as an image processing device to which the present invention is applied.

The image decoding device 101 includes an accumulation buffer 111, a reversible decoding unit 112, an inverse quantization unit 113, an inverse orthogonal transformation unit 114, an operation unit 115, a deblocking filter 116, and a screen rework. The array buffer 117, the D / A converter 118, the frame memory 119, the intra predictor 120, the motion compensator 121, and the switch 122 are configured.

The accumulation buffer 111 accumulates the compressed image that has been transferred. The reversible decoding unit 112 decodes the information encoded by the reversible coding unit 66 of FIG. 9 supplied from the accumulation buffer 111 in a manner corresponding to the coding method of the reversible coding unit 66. The inverse quantization unit 113 dequantizes the image decoded by the reversible decoding unit 112 in a manner corresponding to the quantization method of the quantization unit 65 in FIG. 9. The inverse orthogonal transform unit 114 inversely orthogonally transforms the output of the inverse quantization unit 113 in a manner corresponding to the orthogonal transform method of the orthogonal transform unit 64 in FIG. 9.

The inverse orthogonal transformed output is added to the predictive image supplied from the switch 122 by the calculation unit 115 and decoded. The deblocking filter 116 removes the block distortion of the decoded image, supplies it to the frame memory 119, accumulates it, and outputs it to the screen rearrangement buffer 117.

The screen rearrangement buffer 117 rearranges the images. That is, the order of the frames rearranged for the encoding order by the screen rearrangement buffer 62 of FIG. 9 is rearranged in the order of the original display. The D / A conversion unit 118 performs D / A conversion on the image supplied from the screen rearrangement buffer 117, outputs it to a display (not shown), and displays it.

The referenced image from the frame memory 119 is supplied to the motion compensation unit 121. The image before the deblocking filter from the calculator 115 is supplied to the intra predictor 120 as an image used for intra prediction.

The intra prediction unit 120 supplies information indicating the intra prediction mode obtained by decoding the header information from the reversible decoding unit 112. The intra prediction unit 120 generates a predictive image based on this information, and outputs the generated predictive image to the switch 122.

The motion compensation unit 121 supplies inter prediction mode information, motion vector information, reference frame information, and the like from the information obtained by decoding the header information from the reversible decoding unit 112. The inter prediction mode information is transmitted for each macro block. Motion vector information and reference frame information are transmitted for each target block. When the weighting coefficient is calculated by the image coding apparatus 51, the weighting coefficient is also sent for each frame or every sequence.

The motion compensator 121 compensates the reference image by using the supplied motion vector information or motion vector information obtained from surrounding blocks based on the inter prediction mode from the reversible decoder 112. A predictive image of the block is generated. In this case, like the motion prediction compensator 75 of FIG. 9, the motion compensator 121 is a direct mode or a pair prediction mode in the B picture, that is, a prediction mode using a plurality of different reference pictures. Then, weighted prediction is performed according to whether or not the pixel to which the target block is the target block is outside of the screen in those reference images to generate a predicted image. The generated predictive image is output to the calculating part 115 via the switch 122.

The switch 122 selects the predicted image generated by the motion compensator 121 or the intra predictor 120 and supplies it to the calculator 115.

[Configuration example of the motion compensation unit]

19 is a block diagram illustrating a detailed configuration example of the motion compensation unit 121.

In the example of FIG. 19, the motion compensator 121 is composed of an interpolation filter 131, a compensation processor 132, a selector 133, and a motion vector predictor 134.

Reference frame (reference picture) information from the frame memory 119 is input to the interpolation filter 131. Similar to the interpolation filter 81 of FIG. 11, the interpolation filter 131 interpolates between pixels of a reference frame, enlarges it four times in width and width, and outputs the same to the compensation processing unit 132.

The compensation processing unit 132 is configured by an L0 area selection unit 141, an L1 area selection unit 142, a calculation unit 143, and a screen end determination unit 144. In addition, in the compensation processing unit 132 in the example of FIG. 19, an example in the case of a B picture is shown.

The enlarged reference frame information from the interpolation filter 131 is input to the L0 area selection unit 141, the L1 area selection unit 142, and the screen end determination unit 144.

The L0 region selecting unit 141 selects a corresponding L0 reference region from the enlarged L0 reference frame information according to the prediction mode information and the L0 motion vector information from the selecting unit 133 and outputs the same to the calculating unit 143. do. The output reference information is output to the switch 122 as the L0 prediction information in the L0 prediction mode.

The L1 region selection unit 142 selects a corresponding L1 reference region from the enlarged L1 reference frame information according to the prediction mode information and the L1 motion vector information from the selection unit 133, and outputs the corresponding L1 reference region to the calculation unit 143. do. The output reference information is output to the switch 122 as L1 prediction information in the L1 prediction mode.

The calculating part 143 is comprised by the multiplier 143A, the multiplier 143B, and the adder 143C similarly to the calculating part 93 of FIG. The multiplier 143A multiplies the L0 reference area information from the L0 area selection unit 141 by the L0 weight information from the screen end determination unit 144 and outputs the result to the adder 143C. The multiplier 143B multiplies the L1 reference area information from the L1 area selection unit 142 by the L1 weight information from the screen end determination unit 144, and outputs the result to the adder 143C. The adder 143C adds the L0 reference region and L1 reference region weighted from the L0 and L1 weight information, and outputs the weighted prediction information (Bi-pred prediction information) to the switch 122.

The screen end determining unit 144 is supplied with inter prediction mode information from the reversible decoding unit 112, expanded reference frame information from the interpolation filter 131, and motion vector information from the selecting unit 133.

In the pair prediction or direct mode, the screen end determining unit 144 determines whether the L0 reference pixel or the L1 reference pixel is outside the screen based on the reference frame information and the motion vector information, and according to the determination result. The weighting coefficients supplied to the multiplier 143A and the multiplier 143B are output. For example, if they are all inside or outside the screen, a weighting factor W = 0.5 is output. At least, a weighting coefficient smaller than the reference pixel inside the screen is given to the reference pixel outside the screen.

In addition, when the weighting coefficient is calculated by the weight calculating part 95 of FIG. 11, since the weighting coefficient is also supplied from the reversible decoding part 112, the screen end determination part 144 calculates the weighting coefficient. According to the determination result, the weighting coefficients supplied to the multiplier 143A and the multiplier 143B are output.

The selection unit 133 is also supplied with inter prediction information from the reversible decoding unit 112 and, if there is motion vector information. The selecting unit 133 selects either the motion vector information from the reversible decoding unit 112 and the motion vector information obtained from the motion vector predicting unit 134 according to the prediction mode, and selects the selected motion vector information. The screen is supplied to the screen end determining unit 144, the L0 area selecting unit 141, and the L1 area selecting unit 142.

The motion vector predictor 134 predicts the motion vector according to the mode in which the motion vector is not sent to the decoding side, like the skip vector or the direct mode, similar to the motion vector predictor 84 in FIG. It supplies to 133. In addition, in the example of FIG. 19, although the illustration is abbreviate | omitted, the motion vector information etc. of the surrounding block at the time of obtaining | requiring are acquired from the selection part 133. As shown in FIG.

[Description of Decoding Process of Image Decoding Device]

Next, with reference to the flowchart of FIG. 20, the decoding process performed by the image decoding apparatus 101 is demonstrated.

In step S131, the accumulation buffer 111 accumulates the transferred image. In step S132, the reversible decoding unit 112 decodes the compressed image supplied from the accumulation buffer 111. That is, the I picture, P picture, and B picture coded by the reversible encoder 66 of FIG. 9 are decoded.

At this time, motion vector information, reference frame information, etc. are also decoded for each block. For each macro block, prediction mode information (information indicating an intra prediction mode or an inter prediction mode) is also decoded. In addition, when the weighting coefficient is calculated by the encoding side of Fig. 9, the information is also decoded.

In step S133, the inverse quantization unit 113 dequantizes the transform coefficient decoded by the reversible decoding unit 112 to a characteristic corresponding to that of the quantization unit 65 in FIG. 9. In step S134, the inverse orthogonal transform unit 114 inversely orthogonally transforms the transform coefficients inversely quantized by the inverse quantization unit 113 to a characteristic corresponding to that of the orthogonal transform unit 64 in FIG. As a result, the difference information corresponding to the input (output of the calculation unit 63) of the orthogonal transform unit 64 in FIG. 9 is decoded.

In step S135, the calculating part 115 adds the predictive image selected with the process of step S141 mentioned later, and inputs via the switch 122 with difference information. As a result, the original image is decoded. The deblocking filter 116 filters the image output from the calculating part 115 in step S136. As a result, block distortion is eliminated. In step S137, the frame memory 119 stores the filtered image.

In step S138, the reversible decoding unit 112, based on the reversible decoding result of the header portion of the compressed image, includes information indicating whether the compressed image is an inter predicted image, that is, the optimal inter prediction mode in the reversible decoding result. Determine whether there is.

When it is determined in step S138 that the compressed image is an inter predicted image, the reversible decoding unit 112 supplies the motion vector information, the reference frame information, the information indicating the optimal inter prediction mode, and the like to the motion compensation unit 121. When the weighting coefficient is decoded, it is also supplied to the motion compensation unit 121.

In step S139, the motion compensation unit 121 performs a motion compensation process. The motion compensator 121 compensates the reference picture by using the supplied motion vector information or motion vector information obtained from surrounding blocks based on the inter prediction mode from the reversible decoder 112. A predictive image of the block is generated.

In this case, similar to the motion prediction compensator 75 of FIG. 9, the motion compensator 121 is a direct mode or a pair prediction mode in a B picture, that is, a prediction mode using a plurality of different reference pictures. Then, weighted prediction is performed according to whether or not the pixel to which the target block is the target block is outside of the screen in those reference images to generate a predicted image. The generated predictive image is output to the calculating part 115 via the switch 122. In addition, since the compensation process in the case of this B picture is the same as the compensation process with reference to FIG. 14, the description is abbreviate | omitted.

On the other hand, when it is determined in step S138 that the compressed picture is not an inter predicted picture, that is, when the reversible decoding result includes information indicating the optimal intra prediction mode, the reversible decoding unit 112 selects the optimal intra prediction mode. The information indicated is supplied to the intra prediction unit 120.

In step S140, the intra prediction unit 120 performs an intra prediction process on the image from the frame memory 119 in the optimal intra prediction mode indicated by the information from the reversible decoding unit 112, and the intra prediction image. Create The intra prediction unit 120 then outputs the intra prediction image to the switch 122.

In step S141, the switch 122 selects the predictive image and outputs it to the calculating unit 115. That is, the predictive image generated by the intra predictor 120 or the predictive image generated by the motion compensator 121 is supplied. Therefore, the supplied predictive image is selected and output to the calculating part 115, and as mentioned above, it adds with the output of the inverse orthogonal transformation part 114 in step S135.

In step S142, the screen rearrangement buffer 117 rearranges. That is, the order of the frames rearranged for encoding by the screen rearrangement buffer 62 of the picture coding apparatus 51 is rearranged in the order of the original display.

In step S143, the D / A conversion unit 118 performs D / A conversion on the image from the screen rearrangement buffer 117. This image is output to a display (not shown), and an image is displayed.

As described above, in the image encoding apparatus 51 and the image decoding apparatus 101, either in the pair prediction and direct mode in which weighted prediction using a plurality of different reference pictures is performed, one of the L0 or L1 reference pixels is outside the screen. In reference to, weight prediction is performed in which the weight of the other pixel having higher reliability is increased than that of an external pixel which is likely to be incorrect information.

That is, in the case of this invention, in the case of the proposal of patent document 1, the pixel which exists in the inside of a screen is used among the blocks which are not used at all.

Therefore, according to the present invention, the prediction accuracy of inter coding in the B picture, especially near the end of the screen, is improved. As a result, the residual signal is reduced and the bit amount of the residual signal is reduced, thereby improving the coding efficiency.

This improvement is more effective for smaller screens such as portable terminals than when the screen is large. It is also more effective when the bitrate is low.

When the residual signal is reduced, the coefficient after the orthogonal transformation also becomes small, and many coefficients are expected to be zero after quantization. In the H.264 / AVC system, the number of consecutive zeros is included in the stream information. In general, it is possible to express the number of zeros with a much smaller amount of code than to replace a value other than zero with the determined code, so that many coefficients become zero by the present invention leads to a reduction in the amount of code bits. .

Further, according to the present invention, since the prediction accuracy of the direct mode is improved, the direct mode can be easily selected. Since the direct mode does not have the motion vector information, the header information by the motion vector information is reduced particularly near the edge of the screen.

That is, conventionally, even when the direct mode is selected when the reference area of the L0 or L1 reference picture is outside the screen, the above-described cost function value increases, so that the direct mode is difficult to be selected.

In order to avoid this, when small blocks are selected in pair prediction, the motion vector information of each block is increased, but the motion vector information is reduced by selecting a large block in the direct mode according to the present invention. In addition, since the bit string is determined so that the bit length is reduced for the large block, when the large block is easily selected by the present invention, the bit amount of the mode information is also reduced.

In addition, at low bit rates, since the quantization is performed using a large quantization parameter QP, the prediction accuracy directly affects the image quality. Therefore, when the prediction precision is improved, the image quality near the edge of the screen is improved.

In the above description, in the motion compensation in the pair prediction and direct mode, when either of the L0 or L1 reference pixels refers to the outside of the screen, the reliability is higher than that of the external pixels that are likely to be incorrect information. Although weight prediction with a larger weight for the other pixel is performed, in pair prediction, this weight prediction may be used during motion search. By applying the weighted prediction of the present invention at the time of motion search, the accuracy of the motion search is increased, so that the prediction accuracy can be further improved than at the time of motion compensation.

[Description of Application to Extended Macro Block Size]

It is a figure which shows the example of the block size proposed by the nonpatent literature 2. As shown in FIG. In Non-Patent Document 2, the macroblock size is extended to 32x32 pixels.

At the top of Fig. 21, macroblocks composed of 32x32 pixels divided into blocks (partitions) of 32x32 pixels, 32x16 pixels, 16x32 pixels, and 16x16 pixels are shown in order from the left. . In the middle of Fig. 21, blocks composed of 16x16 pixels divided into blocks of 16x16 pixels, 16x8 pixels, 8x16 pixels, and 8x8 pixels are shown in order from the left. In the lower part of FIG. 21, blocks of 8x8 pixels divided into blocks of 8x8 pixels, 8x4 pixels, 4x8 pixels, and 4x4 pixels are sequentially shown from the left side.

In other words, the macroblock of 32x32 pixels can be processed in blocks of 32x32 pixels, 32x16 pixels, 16x32 pixels, and 16x16 pixels shown at the upper end of FIG.

The block of 16 × 16 pixels appearing on the right side of the top is processed in blocks of 16 × 16 pixels, 16 × 8 pixels, 8 × 16 pixels, and 8 × 8 pixels appearing in the middle, similar to the H.264 / AVC method. It is possible.

Blocks of 8x8 pixels appearing on the right side of the interruption are processed in blocks of 8x8 pixels, 8x4 pixels, 4x8 pixels, and 4x4 pixels appearing at the bottom, similar to the H.264 / AVC method. It is possible.

By adopting such a hierarchical structure, in the proposal of Non-Patent Document 2, a larger block is defined as the enlarged set while maintaining compatibility with the H.264 / AVC system for blocks of 16x16 pixels or less.

The present invention can also be applied to the extended macroblock size proposed as described above.

In addition, in the above, although the H.264 / AVC system was used as a coding system based on this, this invention is not limited to this, The image coding apparatus / which uses the coding system / decoding system which performs another motion prediction and compensation process / It can also be applied to an image decoding device.

In addition, the present invention provides image information (bit stream) compressed by orthogonal transform such as discrete cosine transform and motion compensation, such as MPEG, H.26x, and the like, for example, by satellite broadcasting, cable television, Internet or mobile telephone. The present invention can be applied to a picture coding device and a picture decoding device used when receiving via a network medium. The present invention can also be applied to an image encoding device and an image decoding device used when processing on storage media such as optical, magnetic disks and flash memories. The present invention can also be applied to a motion prediction compensation device included in their picture coding apparatus, picture decoding apparatus and the like.

The series of processes described above may be executed by hardware or may be executed by software. When a series of processes are executed by software, the programs constituting the software are installed in the computer. The computer includes a computer built in dedicated hardware, a general-purpose personal computer capable of executing various functions by installing various programs, and the like.

[Configuration example of personal computer]

Fig. 22 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

In the computer, a CPU (Central Processing Unit) 251, a ROM (Read Only Memory) 252, and a RAM (Random Access Memory) 253 are connected to each other by a bus 254.

An input / output interface 255 is further connected to the bus 254. An input unit 256, an output unit 257, a storage unit 258, a communication unit 259, and a drive 260 are connected to the input / output interface 255.

The input unit 256 is composed of a keyboard, a mouse, a microphone, and the like. The output unit 257 is composed of a display, a speaker, and the like. The storage unit 258 includes a hard disk, a nonvolatile memory, or the like. The communication unit 259 consists of a network interface and the like. The drive 260 drives a removable medium 261 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 251 loads and executes a program stored in the storage unit 258 into the RAM 253 via the input / output interface 255 and the bus 254, for example. A series of processing is performed.

The program executed by the computer CPU251 can be recorded and provided to the removable media 261 as, for example, package media. In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

In the computer, the program can be installed in the storage unit 258 via the input / output interface 255 by attaching the removable media 261 to the drive 260. The program can be received by the communication unit 259 via a wired or wireless transmission medium and installed in the storage unit 258. In addition, the program can be installed in advance in the ROM 252 or the storage unit 258.

The program executed by the computer may be a program in which the processing is performed in time series in the order described in this specification, or may be a program in which the processing is performed at a necessary timing such as when the call is made in parallel or when a call is made.

Embodiment of this invention is not limited to embodiment mentioned above, A various change is possible in the range which does not deviate from the summary of this invention.

For example, the image coding device 51 and the image decoding device 101 described above can be applied to any electronic device. The example is demonstrated below.

[Configuration example of television receiver]

Fig. 23 is a block diagram showing a main configuration example of a television receiver using an image decoding device to which the present invention is applied.

The television receiver 300 shown in FIG. 23 includes a terrestrial tuner 313, a video decoder 315, a video signal processing circuit 318, a graphic generating circuit 319, a panel driving circuit 320, and a display panel 321. Has

The terrestrial tuner 313 receives and demodulates a broadcast wave signal of terrestrial analog broadcasting via an antenna, obtains a video signal, and supplies it to the video decoder 315. The video decoder 315 decodes the video signal supplied from the terrestrial tuner 313 and supplies the obtained digital component signal to the video signal processing circuit 318.

The video signal processing circuit 318 performs predetermined processing such as noise removal on the video data supplied from the video decoder 315, and supplies the obtained video data to the graphic generating circuit 319.

The graphic generating circuit 319 generates video data of a program displayed on the display panel 321, image data by processing based on an application supplied via a network, and generates the generated video data or image data on the panel. Supply to the drive circuit 320. The graphic generation circuit 319 also generates image data (graphics) for displaying a screen used by the user for selecting an item or the like, and superimposes the image data obtained by superimposing it on the image data of the program. Process such as supply to 320 is also appropriately performed.

The panel driving circuit 320 drives the display panel 321 based on the data supplied from the graphic generating circuit 319, and causes the display panel 321 to display an image of the program or the various screens described above.

The display panel 321 is formed of a liquid crystal display (LCD) or the like, and displays an image of a program or the like under the control of the panel driving circuit 320.

In addition, the television receiver 300 includes an audio A / D (Analog / Digital) conversion circuit 314, an audio signal processing circuit 322, an echo cancellation / audio synthesis circuit 323, an audio amplification circuit 324, and a speaker. Also has 325.

The terrestrial tuner 313 demodulates the received broadcast wave signal to acquire not only a video signal but also an audio signal. The terrestrial tuner 313 supplies the acquired audio signal to the audio A / D conversion circuit 314.

The audio A / D conversion circuit 314 performs A / D conversion processing on the audio signal supplied from the terrestrial tuner 313 and supplies the obtained digital audio signal to the audio signal processing circuit 322.

The speech signal processing circuit 322 performs predetermined processing such as noise removal on the speech data supplied from the speech A / D conversion circuit 314, and transmits the obtained speech data to the echo cancel / voice combining circuit 323. Supply.

The echo cancellation / voice combining circuit 323 supplies the voice data supplied from the voice signal processing circuit 322 to the voice amplifying circuit 324.

The audio amplifier 324 performs D / A conversion processing and amplification processing on the audio data supplied from the echo cancellation / audio synthesis circuit 323, adjusts the audio to a predetermined volume, and then adjusts the audio to the speaker 325. Output from

The television receiver 300 also has a digital tuner 316 and an MPEG decoder 317.

The digital tuner 316 receives and demodulates a broadcast wave signal of digital broadcasting (terrestrial digital broadcasting, BS (Broadcasting Satellite) / CS (Communications Satellite) digital broadcasting) via an antenna, and demodulates the moving picture experts. Group-Transport Stream) is obtained and supplied to the MPEG decoder 317.

The MPEG decoder 317 releases the scramble performed on the MPEG-TS supplied from the digital tuner 316 and extracts a stream containing data of a program to be played back (viewing target). The MPEG decoder 317 decodes the audio packet constituting the extracted stream, supplies the obtained audio data to the audio signal processing circuit 322, decodes the video packet constituting the stream, and outputs the video data obtained by the video signal. Supply to processing circuit 318. In addition, the MPEG decoder 317 supplies the EPG (Electronic Program Guide) data extracted from the MPEG-TS to the CPU 332 via a path not shown.

The television receiver 300 uses the above-described image decoding apparatus 101 as the MPEG decoder 317 which decodes the video packet in this way. Therefore, the MPEG decoder 317 can improve the prediction accuracy in the B picture, especially near the end of the screen, similarly to the case of the image decoding device 101. As a result, the coding efficiency can be improved.

As for the video data supplied from the MPEG decoder 317, similarly to the video data supplied from the video decoder 315, the video signal processing circuit 318 performs predetermined processing. The image data subjected to the predetermined processing is supplied by the graphic generating circuit 319 to the display panel 321 via the panel driving circuit 320 so that the generated image data and the like are superimposed appropriately. Is displayed.

The audio signal supplied from the MPEG decoder 317 is subjected to a predetermined process in the audio signal processing circuit 322 as in the case of the audio data supplied from the audio A / D conversion circuit 314. The audio data subjected to the predetermined processing is supplied to the audio amplifier circuit 324 via the echo cancellation / audio synthesis circuit 323 and subjected to D / A conversion processing or amplification processing. As a result, the sound adjusted to the predetermined volume is output from the speaker 325.

The television receiver 300 also has a microphone 326 and an A / D conversion circuit 327.

The A / D conversion circuit 327 receives a voice signal of a user introduced by the microphone 326 provided in the television receiver 300 as being for speech conversation. The A / D conversion circuit 327 performs A / D conversion processing on the received audio signal, and supplies the obtained digital audio data to the echo cancellation / audio synthesis circuit 323.

The echo cancellation / audio combining circuit 323 echoes the audio data of the user A when the audio data of the user (user A) of the television receiver 300 is supplied from the A / D conversion circuit 327. Cancel it. The echo cancellation / audio synthesis circuit 323 then outputs the audio data obtained by synthesizing with other audio data after the echo cancellation from the speaker 325 via the audio amplification circuit 324.

In addition, the television receiver 300 includes an audio codec 328, an internal bus 329, a synchronous dynamic random access memory (SDRAM) 330, a flash memory 331, a CPU 332, and a universal serial bus (USB). It also has an I / F 333 and a network I / F 334.

The A / D conversion circuit 327 receives a voice signal of a user introduced by the microphone 326 provided in the television receiver 300 as being for speech conversation. The A / D conversion circuit 327 performs A / D conversion processing on the received audio signal, and supplies the obtained digital audio data to the audio codec 328.

The voice codec 328 converts the voice data supplied from the A / D conversion circuit 327 into predetermined format data for transmission via the network, and passes through the internal bus 329 to the network I / F 334. To feed.

The network I / F 334 is connected to the network via a cable attached to the network terminal 335. The network I / F 334 transmits the voice data supplied from the voice codec 328 to, for example, another device connected to the network. In addition, the network I / F 334 receives, via the network terminal 335, voice data transmitted from another device connected via a network, for example, and receives the voice codec (via the internal bus 329). 328).

The voice codec 328 converts the voice data supplied from the network I / F 334 into predetermined format data, and supplies it to the echo cancel / voice synthesis circuit 323.

The echo cancellation / voice synthesis circuit 323 performs echo cancellation on the audio data supplied from the audio codec 328 and synthesizes the audio data obtained by synthesizing with other audio data via the audio amplification circuit 324. It outputs from the speaker 325.

The SDRAM 330 stores various data necessary for the CPU 332 to perform a process.

The flash memory 331 stores a program executed by the CPU 332. The program stored in the flash memory 331 is read by the CPU 332 at a predetermined timing such as when the television receiver 300 starts up. The flash memory 331 also stores EPG data acquired through digital broadcasting, data acquired from a predetermined server via a network, and the like.

For example, in the flash memory 331, MPEG-TS including content data acquired from a predetermined server via a network under the control of the CPU 332 is stored. The flash memory 331 supplies the MPEG-TS to the MPEG decoder 317 via the internal bus 329, for example, under the control of the CPU 332.

The MPEG decoder 317 processes the MPEG-TS as in the case of the MPEG-TS supplied from the digital tuner 316. Thus, the television receiver 300 can receive content data consisting of video, audio, and the like via a network, decode it using the MPEG decoder 317, display the video, or output audio. .

The television receiver 300 also includes a light receiver 337 that receives an infrared signal transmitted from the remote controller 351.

The light receiving unit 337 receives the infrared rays from the remote controller 351 and outputs a control code indicating the contents of the user operation obtained by demodulation to the CPU 332.

The CPU 332 executes the program stored in the flash memory 331 and controls the operation of the television receiver 300 as a whole in accordance with a control code supplied from the light receiving unit 337 or the like. Each part of the CPU 332 and the television receiver 300 is connected via a path (not shown).

The USB I / F 333 transmits and receives data between external devices of the television receiver 300 connected via a USB cable attached to the USB terminal 336. The network I / F 334 connects to the network via a cable attached to the network terminal 335, and also transmits and receives data other than voice data to various devices connected to the network.

The television receiver 300 can improve the coding efficiency by using the image decoding device 101 as the MPEG decoder 317. As a result, the television receiver 300 can obtain and display a higher precision decoded image from the broadcast wave signal received via the antenna or the content data acquired through the network.

[Configuration example of cellular phone]

24 is a block diagram showing an example of a main configuration of a mobile telephone using the picture coding apparatus and the picture decoding apparatus to which the present invention is applied.

The mobile phone 400 shown in FIG. 24 includes a main control unit 450, a power supply circuit unit 451, an operation input control unit 452, an image encoder 453, and a camera I / F unit configured to collectively control each unit. 454, LCD control unit 455, image decoder 456, multiple separation unit 457, recording / reproducing unit 462, modulation / demodulation circuit unit 458, and audio codec 459. These are connected to each other via the bus 460.

The mobile phone 400 also includes an operation key 419, a CCD (Charge Coupled Devices) camera 416, a liquid crystal display 418, a storage unit 423, a transmission / reception circuit unit 463, an antenna 414, a microphone. (Microphone) 421 and speaker 417.

The power supply circuit unit 451 activates the mobile phone 400 in an operable state by supplying electric power to each unit from the battery pack when the termination and power supply keys are turned on by a user's operation.

The mobile phone 400 transmits / receives a voice signal, sends an e-mail or image data in various modes such as a voice call mode and a data communication mode based on the control of the main controller 450 made of a CPU, a ROM, a RAM, and the like. Various operations such as transmission and reception, image shooting or data recording are performed.

For example, in the voice call mode, the cellular phone 400 converts the voice signal collected by the microphone (microphone) 421 into digital voice data by the voice codec 459, and modulates and demodulates the circuit 458. Spectrum spread processing is performed, and the transmission / reception circuit section 463 performs digital / analog conversion processing and frequency conversion processing. The cellular phone 400 transmits the transmission signal obtained by the conversion process to a base station (not shown) via the antenna 414. The transmission signal (audio signal) transmitted to the base station is supplied to the cellular phone of the call partner via the public telephone line network.

For example, in the voice call mode, the cellular phone 400 amplifies the received signal received by the antenna 414 in the transmission / reception circuit section 463, further performs a frequency conversion process and an analog / digital conversion process to perform a modulation / demodulation circuit section. In 458, the spectrum despreading process is performed, and the speech codec 459 converts the analog speech signal. The cellular phone 400 outputs the analog audio signal obtained by the conversion from the speaker 417.

For example, when sending an e-mail in the data communication mode, the cellular phone 400 receives the text data of the e-mail input by the operation of the operation key 419 in the operation input control unit 452. . The cellular phone 400 processes the text data in the main controller 450 and displays the text data on the liquid crystal display 418 as an image via the LCD control unit 455.

In addition, the mobile phone 400 generates the electronic mail data in the main control unit 450 based on the text data received by the operation input control unit 452, a user instruction, or the like. The cellular phone 400 performs spread spectrum processing on the electronic demodulation circuit section 458, and performs digital / analog conversion processing and frequency conversion processing on the transmission / reception circuit section 463. The cellular phone 400 transmits the transmission signal obtained by the conversion process to a base station (not shown) via the antenna 414. The transmission signal (e-mail) transmitted to the base station is supplied to a predetermined destination via a network, a mail server, or the like.

For example, when receiving an e-mail in the data communication mode, the cellular phone 400 receives, amplifies, and frequency-converts the signal transmitted from the base station through the antenna 414 in the transmission / reception circuit section 463. Processing and analog / digital conversion processing. The mobile phone 400 despreads the received signal by the demodulation circuit unit 458 to restore original electronic mail data. The cellular phone 400 displays the restored e-mail data on the liquid crystal display 418 via the LCD control unit 455.

The mobile phone 400 can also record (remember) the received e-mail data in the storage unit 423 via the recording / reproducing unit 462.

This storage unit 423 is any rewritable storage medium. The storage unit 423 may be, for example, a semiconductor memory such as a RAM or a built-in flash memory, or may be a hard disk, or may be a removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, a USB memory or a memory card. Of course, other than these may be used.

For example, when transmitting image data in a data communication mode, the cellular phone 400 generates image data by the CCD camera 416 by imaging. The CCD camera 416 has an optical device such as a lens or an aperture (throttling valve) and a CCD as a photoelectric conversion element. The CCD camera 416 captures a subject, converts the received light intensity into an electrical signal, and generates image data of the subject image. . The image data is converted into coded image data by compression encoding the image data through a camera I / F unit 454 by, for example, a predetermined coding scheme such as MPEG2 or MPEG4.

The mobile telephone 400 uses the above-described image coding apparatus 51 as the image encoder 453 which performs such processing. Therefore, the image encoder 453 can improve the prediction accuracy in the B picture, especially near the end of the screen, similarly to the case of the image encoding apparatus 51. As a result, the coding efficiency can be improved.

At this time, the cellular phone 400 analog-to-digital-converts and encodes the sound collected by the microphone (microphone) 421 during imaging with the CCD camera 416 at the same time.

The cellular phone 400 multiplexes the encoded image data supplied from the image encoder 453 and the digital audio data supplied from the audio codec 459 in a predetermined manner in the multiplexer 457. The cellular phone 400 performs spread spectrum processing on the resultant multiplexed data in the demodulation circuit section 458, and performs digital / analog conversion processing and frequency conversion processing on the transmission / reception circuit section 463. The cellular phone 400 transmits the transmission signal obtained by the conversion process to a base station (not shown) via the antenna 414. The transmission signal (image data) transmitted to the base station is supplied to the communication counterpart via a network or the like.

When the image data is not transmitted, the cellular phone 400 passes the image data generated by the CCD camera 416 through the LCD control unit 455 without passing through the image encoder 453. You can also mark

For example, in the data communication mode, when receiving data of a moving picture file linked to a simple home page or the like, the cellular phone 400 transmits and receives a signal transmitted from a base station via the antenna 414 to the transmission / reception circuit unit 463. Receive, amplify, and perform frequency conversion and analog / digital conversion. The mobile phone 400 despreads the received signal by the demodulation circuit section 458 to restore the original multiplexed data. In the multiplexing section 457, the cellular phone 400 separates the multiplexed data and divides the multiplexed data into encoded image data and audio data.

The cellular phone 400 decodes the encoded image data by a decoding method corresponding to a predetermined coding method such as MPEG2 or MPEG4 in the image decoder 456 to generate the reproduced moving image data, and the LCD control unit 455. It displays on the liquid crystal display 418 via the. Thereby, for example, moving picture data included in the moving picture file linked to the simplified homepage is displayed on the liquid crystal display 418.

The mobile telephone 400 uses the above-described image decoding apparatus 101 as the image decoder 456 which performs such a process. Therefore, similarly to the case of the image decoding apparatus 101, the image decoder 456 can improve the prediction accuracy in the B picture, especially near the end of the screen. As a result, the coding efficiency can be improved.

At this time, the cellular phone 400 simultaneously converts digital voice data into an analog voice signal by the voice codec 459 and outputs it from the speaker 417. As a result, for example, audio data contained in a moving image file linked to a simple homepage is reproduced.

In addition, similarly to the case of the e-mail, the mobile telephone 400 can also record (remember) the data linked to the received simple homepage or the like in the storage unit 423 via the recording / reproducing unit 462. .

In addition, the mobile phone 400 can acquire the information recorded in the two-dimensional code by analyzing the two-dimensional code picked up by the main control unit 450 and obtained by the CCD camera 416.

In addition, the cellular phone 400 can communicate with external devices by infrared rays in the infrared communication unit 481.

The mobile telephone 400 uses the image coding device 51 as the image encoder 453, thereby improving the prediction accuracy. As a result, the cellular phone 400 can provide encoded data (image data) having good encoding efficiency to other devices.

In addition, the mobile phone 400 uses the image decoding device 101 as the image decoder 456, thereby improving the prediction accuracy. As a result, the mobile phone 400 can obtain and display a higher precision decoded image, for example, from a moving image file linked to a simple homepage.

In addition, in the above, although the mobile telephone 400 demonstrated using the CCD camera 416, even if it is made to use the image sensor (CMOS image sensor) using CMOS (Complementary Metal Oxide Semiconductor) instead of this CCD camera 416, good. Also in this case, the cellular phone 400 can capture the subject and generate image data of the subject image, similarly to the case where the CCD camera 416 is used.

In addition, although it demonstrated above as the mobile telephone 400, it is the same as this mobile telephone 400, for example, PDA (Personal Digital Assistants), smart phone, UMPC (Ultra Mobile Personal Computer), netbook, notebook type personal computer. As long as the device has an imaging function or a communication function, the image coding device 51 and the image decoding device 101 can be applied as in the case of the mobile telephone 400.

[Configuration example of the hard disk recorder]

Fig. 25 is a block diagram showing a main configuration example of a hard disk recorder using the picture coding apparatus and the picture decoding apparatus to which the present invention is applied.

The hard disk recorder (HDD recorder) 500 shown in FIG. 25 includes audio data and video data of a broadcast program included in a broadcast wave signal (television signal) transmitted from a satellite, a terrestrial antenna, or the like received by a tuner. Is stored in a built-in hard disk, and the stored data is provided to the user at the timing according to the user's instruction.

The hard disk recorder 500 can extract audio data and video data from a broadcast wave signal, for example, decode them suitably, and store them in a built-in hard disk. In addition, the hard disk recorder 500 can acquire audio data and video data from another apparatus, for example via a network, decode them suitably, and can store them in the internal hard disk.

The hard disk recorder 500 decodes, for example, audio data and video data recorded on an internal hard disk, and supplies the decoded data to the monitor 560 so as to display the image on the screen of the monitor 560. In addition, the hard disk recorder 500 can output the sound from the speaker of the monitor 560.

The hard disk recorder 500, for example, decodes audio data and video data extracted from a broadcast wave signal acquired through a tuner, or audio data or video data obtained from another device via a network, and supplies the same to the monitor 560. The image is displayed on the screen of the monitor 560. In addition, the hard disk recorder 500 can output the sound from the speaker of the monitor 560.

Of course, other operations are also possible.

As shown in FIG. 25, the hard disk recorder 500 includes a receiver 521, a demodulator 522, a demultiplexer 523, an audio decoder 524, a video decoder 525, and a recorder controller 526. Have The hard disk recorder 500 further includes an EPG data memory 527, a program memory 528, a work memory 529, a display converter 530, an OSD (On Screen Display) controller 531, and a display controller 532. And a recording / reproducing section 533, a D / A converter 534, and a communication section 535. FIG.

The display converter 530 also has a video encoder 541. The recording and playback section 533 has an encoder 551 and a decoder 552.

The receiving unit 521 receives an infrared signal from a remote controller (not shown), converts it into an electric signal, and outputs it to the recorder control unit 526. The recorder control unit 526 is configured by, for example, a microprocessor, and executes various processes in accordance with a program stored in the program memory 528. At this time, the recorder control unit 526 uses the work memory 529 as necessary.

The communication unit 535 is connected to a network and performs communication processing with other devices via the network. For example, the communication unit 535 is controlled by the recorder control unit 526, communicates with a tuner (not shown), and mainly outputs a channel selection control signal to the tuner.

The demodulator 522 demodulates the signal supplied from the tuner and outputs it to the demultiplexer 523. The demultiplexer 523 separates the data supplied from the demodulator 522 into audio data, video data, and EPG data, and outputs them to the audio decoder 524, the video decoder 525, or the recorder control unit 526, respectively. do.

The audio decoder 524 decodes the input audio data by, for example, the MPEG method, and outputs it to the recording / playback section 533. The video decoder 525 decodes the input video data by, for example, the MPEG method, and outputs it to the display converter 530. The recorder control unit 526 supplies the input EPG data to the EPG data memory 527 and stores it.

The display converter 530 encodes the video data supplied from the video decoder 525 or the recorder control unit 526 into video data of the National Television Standards Committee (NTSC) system, for example, by the video encoder 541. The data is output to the recording and playback section 533. In addition, the display converter 530 converts the screen size of the video data supplied from the video decoder 525 or the recorder control unit 526 into a size corresponding to the size of the monitor 560. The display converter 530 converts the video data whose screen size is converted into NTSC video data by the video encoder 541, converts the video data into an analog signal, and outputs the analog signal to the display control unit 532.

The display control unit 532 superimposes the OSD signal output by the OSD (On Screen Display) control unit 531 on the video signal input from the display converter 530 under the control of the recorder control unit 526 and monitors 560. ) Is displayed on the display.

The audio data output from the audio decoder 524 is converted into an analog signal by the D / A converter 534 and supplied to the monitor 560. The monitor 560 outputs from the speaker incorporating this audio signal.

The recording and playback section 533 has a hard disk as a storage medium for recording video data, audio data, and the like.

The recording / reproducing unit 533 encodes audio data supplied from the audio decoder 524, for example, by the encoder 551 in the MPEG method. The recording / reproducing unit 533 also encodes the video data supplied from the video encoder 541 of the display converter 530 by the encoder 551 in the MPEG method. The recording / reproducing unit 533 synthesizes the encoded data of the audio data and the encoded data of the video data by a multiplexer. The recording / reproducing section 533 performs channel coding on the synthesized data, and amplifies the synthesized data into the hard disk via the recording head.

The recording / reproducing section 533 reproduces, amplifies, and decodes the data recorded on the hard disk via the reproduction head, and separates the audio data and the video data by the demultiplexer. The recording and playback section 533 decodes the audio data and the video data by the decoder 552 by the MPEG method. The recording / reproducing unit 533 converts the decoded audio data to D / A and outputs the decoded audio data to the speaker of the monitor 560. The recording / reproducing unit 533 also D / A-converts the decoded video data and outputs the decoded video data to the display of the monitor 560.

The recorder control unit 526 reads the latest EPG data from the EPG data memory 527 based on a user instruction indicated by the infrared signal received from the remote controller received via the receiving unit 521, and the OSD control unit 531. To feed. The OSD control unit 531 generates image data corresponding to the input EPG data and outputs it to the display control unit 532. The display control unit 532 outputs and displays the video data input from the OSD control unit 531 to the display of the monitor 560. As a result, the EPG (Electronic Program Guide) is displayed on the display of the monitor 560.

In addition, the hard disk recorder 500 can acquire various data, such as video data, audio data, or EPG data, supplied from another apparatus via a network such as the Internet.

The communication unit 535 is controlled by the recorder control unit 526 and acquires coded data such as video data, audio data, and EPG data transmitted from another device via a network, and supplies it to the recorder control unit 526. The recorder control unit 526 supplies, for example, the encoded data of the acquired video data or audio data to the recording / reproducing unit 533, and stores the encoded data in the hard disk. At this time, the recorder control unit 526 and the recording / reproducing unit 533 may perform reencoding or the like as necessary.

The recorder control unit 526 decodes the encoded data of the obtained video data or audio data and supplies the obtained video data to the display converter 530. Similar to the video data supplied from the video decoder 525, the display converter 530 processes the video data supplied from the recorder control unit 526, and supplies the same to the monitor 560 via the display control unit 532. Display an image.

In addition, in accordance with this image display, the recorder control unit 526 may supply the decoded audio data to the monitor 560 via the D / A converter 534 and output the sound from the speaker.

The recorder control unit 526 decodes the encoded data of the acquired EPG data, and supplies the decoded EPG data to the EPG data memory 527.

The hard disk recorder 500 described above uses the image decoding device 101 as a decoder built in the video decoder 525, the decoder 552, and the recorder control unit 526. Therefore, the decoders built in the video decoder 525, the decoder 552, and the recorder control unit 526, as in the case of the image decoding apparatus 101, can improve the prediction accuracy in the B picture, especially near the end of the screen. Can be. As a result, the coding efficiency can be improved.

Therefore, the hard disk recorder 500 can generate a highly accurate predictive image. As a result, the hard disk recorder 500 obtains, for example, encoded data of video data received via a tuner, encoded data of video data read from the hard disk of the recording / reproducing unit 533, or obtained via a network. From the encoded data of the video data, a higher precision decoded image can be obtained and displayed on the monitor 560.

In addition, the hard disk recorder 500 uses the image coding apparatus 51 as the encoder 551. Accordingly, the encoder 551 can improve the prediction accuracy in the B picture, especially near the end of the screen, similarly to the case of the picture coding apparatus 51. As a result, the coding efficiency can be improved.

Therefore, the hard disk recorder 500 can improve the coding efficiency of the coded data recorded on a hard disk, for example. As a result, the hard disk recorder 500 can use the storage area of the hard disk more efficiently at a higher speed.

In addition, although the hard disk recorder 500 which records video data and audio data on the hard disk was demonstrated above, of course, what kind of recording medium may be sufficient. For example, even in a recorder to which a recording medium other than a hard disk is applied, such as a flash memory, an optical disk, or a video tape, the image encoding apparatus 51 and the image decoding apparatus 101 are the same as in the case of the hard disk recorder 500 described above. ) Can be applied.

[Configuration example of the camera]

Fig. 26 is a block diagram showing a main configuration example of a camera using the image decoding device and image coding device to which the present invention is applied.

The camera 600 shown in FIG. 26 captures an image of a subject and displays an image of the subject on the LCD 616 or records it on the recording medium 633 as image data.

The lens block 611 causes light (ie, an image of a subject) to enter the CCD / CMOS 612. The CCD / CMOS 612 is an image sensor using a CCD or a CMOS. The CCD / CMOS 612 converts the intensity of the received light into an electric signal and supplies it to the camera signal processing unit 613.

The camera signal processing unit 613 converts the electric signal supplied from the CCD / CMOS 612 into a color difference signal of Y, Cr, and Cb and supplies it to the image signal processing unit 614. The image signal processing unit 614 performs predetermined image processing on the image signal supplied from the camera signal processing unit 613 under the control of the controller 621, or transmits the image signal to the encoder 641, for example. It is encoded by the MPEG method. The image signal processing unit 614 supplies the coded data generated by encoding the image signal to the decoder 615. The image signal processing unit 614 also acquires display data generated by the on-screen display (OSD) 620 and supplies it to the decoder 615.

In the above process, the camera signal processing unit 613 appropriately uses a DRAM (Dynamic Random Access Memory) 618 connected via the bus 617, and if necessary, the image data or the encoded data in which the image data is encoded. Etc. are held in the DRAM 618.

The decoder 615 decodes the encoded data supplied from the image signal processing unit 614 and supplies the obtained image data (decoded image data) to the LCD 616. The decoder 615 also supplies the display data supplied from the image signal processing unit 614 to the LCD 616. The LCD 616 appropriately synthesizes the image of the decoded image data supplied from the decoder 615 and the image of the display data, and displays the synthesized image.

The on-screen display 620, under the control of the controller 621, outputs display data such as a menu screen or icon consisting of symbols, characters, or figures to the image signal processing unit 614 via the bus 617.

The controller 621 executes various processes based on the signal indicating the contents commanded by the user using the operation unit 622, and passes the image signal processing unit 614 and the DRAM 618 via the bus 617. ), An external interface 619, an on-screen display 620, a media drive 623, and the like. The FLASH ROM 624 stores programs, data, and the like necessary for the controller 621 to execute various processes.

For example, the controller 621 encodes the image data stored in the DRAM 618 instead of the image signal processing unit 614 or the decoder 615, or encodes the encoded data stored in the DRAM 618. It can be decoded. At this time, the controller 621 may perform encoding / decoding processing by the same method as that of the encoding / decoding method of the image signal processing unit 614 or the decoder 615, and the image signal processing unit 614 and the decoder 615 may be used. The encoding / decoding process may be performed by a method that does not support.

For example, when the start of image printing is instructed from the operation unit 622, the controller 621 reads image data from the DRAM 618 and sends the image data to the external interface 619 via the bus 617. It is supplied to the printer 634 to be connected and printed.

For example, when image recording is instructed from the operation unit 622, the controller 621 reads the encoded data from the DRAM 618 and mounts it to the media drive 623 via the bus 617. The recording medium 633 is supplied and stored.

The recording medium 633 is any readable removable medium, such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory. The recording medium 633 may, of course, also be a removable medium, and may be a tape device, a disk, or a memory card. Of course, a contactless IC card etc. may be sufficient.

In addition, the media drive 623 and the recording medium 633 may be integrated to be configured by a non-reversible storage medium such as an internal hard disk drive or a solid state drive (SSD).

The external interface 619 is configured with, for example, a USB input / output terminal, and is connected to the printer 634 when printing an image. In addition, a drive 631 is connected to the external interface 619 as necessary, and a removable medium 632 such as a magnetic disk, an optical disk, or a magneto-optical disk is appropriately mounted so that a computer program read therefrom is provided. If necessary, it is installed in the FLASH ROM 624.

In addition, the external interface 619 has a network interface connected to a predetermined network such as a LAN or the Internet. The controller 621 can read the encoded data from the DRAM 618 according to an instruction from the operation unit 622, for example, and can supply the encoded data from the external interface 619 to other devices connected via a network. In addition, the controller 621 acquires the encoded data and the image data supplied from the other device via the network via the external interface 619 and maintains it in the DRAM 618 or the image signal processing unit 614. You can also supply.

The camera 600 described above uses the image decoding device 101 as the decoder 615. Therefore, the decoder 615 can improve the prediction accuracy in the B picture, especially near the end of the screen, similarly to the case of the image decoding device 101. As a result, the coding efficiency can be improved.

Therefore, the camera 600 can generate the high precision prediction image. As a result, the camera 600 acquires, for example, the image data generated by the CCD / CMOS 612, the encoded data of the video data read out from the DRAM 618 or the recording medium 633, or via a network. From the encoded data of the video data, a higher precision decoded image can be obtained and displayed on the LCD 616.

In addition, the camera 600 uses the image coding apparatus 51 as the encoder 641. Accordingly, the encoder 641 can improve the prediction accuracy in the B picture, especially near the end of the screen, similarly to the case of the picture coding apparatus 51. As a result, the coding efficiency can be improved.

Therefore, the camera 600 can improve the coding efficiency of the coded data recorded on a hard disk, for example. As a result, the camera 600 can use the storage area of the DRAM 618 and the recording medium 633 more efficiently at a higher speed.

In addition, the decoding method of the image decoding device 101 may be applied to the decoding process performed by the controller 621. Similarly, the encoding method of the image encoding device 51 may be applied to the encoding process performed by the controller 621.

The image data picked up by the camera 600 may be a moving image or a still image.

Of course, the image coding apparatus 51 and the image decoding apparatus 101 can be applied also to apparatuses and systems other than the apparatus mentioned above.

51: picture coding apparatus
66: reversible encoder
75: motion prediction / compensation unit
81: interpolation filter
82: compensation processing unit
83: selection
84: motion vector predictor
85: prediction mode determiner
91: L0 area selection
92 L1 area selection
93: calculator
93A, 93B: Multipliers
93C: adder
94: screen end determination unit
95: weight calculation unit
101: image decoding device
112: reversible decoder
121: motion compensation unit
131: interpolation filter
132: compensation processing unit
133:
134: motion vector predictor
141: L0 area selection unit
142: L1 region selection section
143: calculator
143A, 143B: Multiplier
143C: adder
144: screen end determination unit

Claims (10)

In the prediction using a plurality of different reference pictures referred to by the image to be processed, motion prediction compensation means for performing weighted prediction according to whether or not the pixel of the block reference destination of the picture is outside the screen in the plurality of reference pictures. An image processing apparatus having a. The method of claim 1,
The motion prediction compensation means performs weighted prediction determined by the standard using those pixels when the block reference destinations of the pictures are pixels within a screen in the plurality of reference pictures,
When the reference point of the block of the picture is a pixel outside the screen in one of the plurality of reference pictures, and the pixel inside the screen in the other reference picture, the weighting prediction is performed using those pixels. An image processing apparatus. The image processing apparatus according to claim 2, wherein a weight of the weighted prediction is larger than a weight of a pixel outside the screen, and a weight of a pixel inside the screen is larger. The image processing apparatus according to claim 3, wherein the weight of the weighted prediction is 0 or 1. 4. The image processing apparatus according to claim 3, further comprising weight calculation means for calculating a weight of the weighted prediction by discontinuity between pixels near a block of the image. The image processing apparatus according to claim 5, further comprising encoding means for encoding the information of the weight calculated by the weight calculating means. The method of claim 3,
Decoding means calculated by the discontinuity between the pixels in the vicinity of the block of the image, the decoding means for decoding the information of the encoded weight,
And the motion prediction compensation means uses the information of the weight decoded by the decoding means when performing the weighted prediction. The image processing apparatus according to claim 2, wherein the prediction using the plurality of different reference pictures is at least one of pair prediction and direct mode prediction. The motion prediction compensation means of the image processing apparatus,
In the prediction using a plurality of different reference pictures referred to by the image to be processed, the image includes a step of performing weighted prediction according to whether the reference of the block of the picture is outside the screen in the plurality of reference pictures. Treatment method. In the prediction using a plurality of different reference pictures referred to by an image to be processed, as motion prediction compensation means for performing weighted prediction according to whether a reference of a block of the picture is outside the screen in the plurality of reference pictures, Program to operate the computer.

Images