KR100742762B1 - Detector for predicting of movement - Google Patents

Abstract

The present invention provides a motion prediction information detecting apparatus capable of determining any motion information and weight information with high precision.

Thus, in the present invention, the first weight information calculating section 1 determines the provisional weight information from the reference image and the processing target image. The motion information calculation unit 2 determines the motion information from the reference image weight-compensated with the estimated weight information and the processing target image. The encoding mode determiner 5 selects the encoding mode from the precision of the processing target image compensated with the weight information determined by the first weight information calculator 1 and the motion information determined by the motion information calculator 3. Decide The second weight information calculating section 6 determines weight information with high precision from the reference picture compensated with the motion information determined by the motion information calculating section, the processing target image, and the encoding mode.

Description

Motion prediction information detection device {DETECTOR FOR PREDICTING OF MOVEMENT}

1 is a flowchart showing a flow of processing in the present invention;

FIG. 2 is a diagram showing a relationship between a process target image and a reference image in each process of FIG. 1;

3 is a flowchart showing an outline of weight information calculation processing according to a fade type;

4 is an explanatory diagram showing an encoding mode in a process target image;

5 is an explanatory diagram showing a motion vector erroneously detected by a fade;

6 is a block diagram showing a video encoding apparatus to which the motion prediction information detecting apparatus according to the present invention is applied.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion prediction information detecting apparatus, and in particular, by applying motion compensation and weight compensation to a reference image, the accuracy of the reference image with respect to the processing target image (encoding target image) is improved, and thus the processing target image and the reference image The present invention relates to a motion prediction information detecting device for determining motion information and weight information for minimizing errors.

As an encoding method for encoding moving picture signals that are continuously input, there is an inter-frame prediction encoding method. In the inter-frame predictive coding method, motion information is used to perform motion compensation in which the prediction efficiency of temporal correlation is increased.

In motion compensation, a local motion compensation method on the premise that one small area (pixel block) in the encoding target image is moved in parallel is widely used. This method has no change in illuminance and is effective when the movement of the subject is parallel movement.

On the other hand, a weight compensation method has been proposed for an image in which the luminance value changes in time such as sudden illuminance change, fade or dissolve that cannot be coped with the above motion compensation.

For example, Patent Documents 1 and 2 below propose a method of performing motion compensation by selecting weight information obtained by either linear external insertion or linear internal insertion of two reference images. In addition, the present inventor first proposed a method of determining weighted motion prediction information from sum, square sum, and multiplication of pixel values of a processing target image and a reference image in the following Patent Document 3 (prior application).

[Patent Document 1]

Japanese Patent Application Laid-Open No. 2003-284075

[Patent Document 2]

Japanese Patent Application Laid-Open No. 2004-7379

[Patent Document 3]

Japanese application 2004-21328 (prior application)

In the methods proposed in Patent Documents 1 and 2 above, only two kinds of weight information are obtained by weighting the linear external insertion or linear internal insertion of two reference images, for example, (0.5, 0.5) or (-1, 2). There is a problem that no optimal weight information can be obtained. Moreover, although the method of the said patent document 3 can obtain optimal weight information, there exists a subject that the encoding efficiency is not limited.

In general, weight compensation is not used singly, but together with motion compensation using motion vectors representing parallel movement. At this time, the optimal solution obtained separately for the weight compensation and the motion compensation is not necessarily the optimal solution for the whole. Also, if the order of weight compensation and motion compensation is different, the individual optimality for weight compensation and motion compensation is also different. That is, even if any compensation of the motion compensation and the weight compensation is optimized individually, the optimal solution of the whole cannot be derived.

For example, in the case of an image involving a fade out of a scene in which a scene involving clear movement by a moving object or a camera walk is performed, weight compensation is applied before motion compensation to obtain optimal weight information using a conventional method. In this case, even in a moving object region that can cope with parallel movement sufficiently, weight information is calculated by multiplying irrelevant pixel values. Therefore, even if an optimal solution is obtained as weight compensation, application of subsequent motion compensation becomes difficult, so that the accuracy of approximation is not optimal as a whole.

On the contrary, if motion compensation is applied before weight compensation, the conventional method may detect motion information that does not exist due to the effect of fade. For example, since the luminance value itself decreases in fade out, a random motion vector occurs in a flat block.

In the case of an image in which the pixel value is smoothly changed, such as an image with a gradation shown in Fig. 5, for example, since a bright area is changed to the same pixel value as the target block by fading out to black, The motion vector to the bright area is misdetected. That is, in an image with gradation, when there is no change in the image content and only the pixel value is changed, it is determined that the pixel block A of the reference image has moved to the pixel block A 'of the processing target image and the motion vector ( C) is incorrectly detected.

In addition, although there are some pixel blocks in which an encoding is used as an intra prediction mode in the image, it is preferable to derive weight information by including the pixel values as the least-squares processing targets for the pixel blocks employing the intra prediction mode. Unnecessarily increases the processing load. In addition, a pixel value having a large error, such as the intra prediction mode, has a problem of deriving wrong weight information in the least square method sensitive to the deviation value.

SUMMARY OF THE INVENTION An object of the present invention is to provide a motion prediction information detecting apparatus capable of solving the above problems and determining arbitrary motion information and weight information with high precision.

MEANS TO SOLVE THE PROBLEM In order to solve the said subject, this invention improves the precision of the reference image with respect to a process target image by applying weighted motion compensation to a reference image, and the motion information and weight information which minimize the error of a process target image and a reference image. A motion prediction information detecting apparatus for determining a weight, comprising: first weight information calculating means for determining tentative weight information from a reference image and a processing target image, a reference image compensated with weight information determined by the first weight information calculating means, and Motion information calculating means for determining motion information from the processing target image, and encoding from the precision of the processing target image compensated with the weight information determined by the first weight information calculating means and the motion information determined by the motion information calculating means Encoding mode determining means for determining a mode, and the motion information And a second weight information calculating means for determining weight information from the reference image compensated with the motion information determined by the calculating means, the processing target image, and the encoding mode.

Hereinafter, the present invention will be described with reference to the drawings. Fig. 1 is a flowchart showing the flow of processes in the present invention, and each part which executes these processes can be configured either in hardware or in software. 2 is a diagram illustrating a relationship between a process target image (frame) and a reference image (frame) in each part.

The processing in the present invention can be broadly divided into weight information calculation and weight compensation processing by the first weight information calculation unit 1 and the weight compensation unit 2, the motion information calculation unit 3, and the motion compensation unit 4. Motion information calculation / movement compensation processing by < RTI ID = 0.0 >), < / RTI > encoding mode determination processing by the encoding mode determination section 5, and weight information calculation / weight compensation processing by the second weight information calculation section 6 and the weight compensation section 7 Is made of. Hereinafter, the process in each part is demonstrated one by one.

Here, the input information is a pixel value of the reference image and a pixel value of the processing target image (encoding target image), and the output information is a pixel value of the motion compensated and weight-compensated reference image, or motion information and weight information.

1. First weight information calculator 1 and weight compensator 2

The first weight information calculating section 1 estimates the provisional weight information from the input processing target image and the reference image. However, in the beginning, since the motion compensation is applied, the comparison in the pixel at the position where the processing target image and the reference image are the same in the moving object region has no meaning. In addition, even in the presence of a camera work, the pixel unit ratio cannot be used at all as weight information.

Therefore, in order to exclude the influence of motion, for example, the pixel value average is calculated as the feature amount of the entire frame, and the weight information is calculated using a calculation method according to the type of the fade. That is, the weight information is calculated from the pixel value average which is the characteristic amount of the whole frame so that the error of the weight compensation becomes small even without the motion compensation. At this stage, the optimal solution for weight compensation cannot be derived.

The weight compensation unit 2 performs weight compensation using the weight information tentatively calculated by the above.

Fig. 2 shows the image to be processed and the first and second reference images before and after the time. FIG. 2 (a) is an image as it is input, and FIG. 2 (b) shows a state in which the first and second reference images are weight-compensated with the weight information calculated by the first weight information calculator 1. have. By the processing here, the pixel value average of each of the first and second reference images is close to the pixel value average of the processing target image. In the figure, a circle indicates the position of the moving object in each frame. In this case, since the motion compensation has not been performed yet, the positional correspondence of the moving object cannot be taken. However, since the pixel value average is calculated as the feature amount of the entire frame, weight information capable of accurate weight compensation can be determined.

As the weight information calculation method according to the type of fade, for example, the invention of Japanese Patent Application No. 2004-121017 proposed by the present inventor can be used. The outline of this weight information calculation method is demonstrated using the flowchart of FIG.

First, the pixel value of the reference image and the pixel value of the processing target image are given to the fade discrimination processing (S31). In the fade discrimination processing (S31), first, it is determined whether there is a fade, and when there is a fade, it is again determined whether the fade is a black fade or a back fade.

Whether or not a fade exists is characterized by temporal change between frames of sum of pixel values, for example, by examining the continuity of temporal changes such as whether it simply decreases or simply increases, or in one frame. This can be determined by examining the continuity of temporal changes such as the average value, mode value, or representative value of the pixel. If no fade exists, that is, non-fade, the calculation process of the weight information is not performed.

Whether the fade is a black fade or a back fade can be determined based on whether the rate of change of the pixel value from the reference image to the processing target image is proportional or inversely proportional to the pixel value. If the fade type of the black fade or the back fade can be determined, the fade or fade out can be determined by a comparison between the front-rear relation of the frame and the sum of the pixel values.

In the case of black fade, the first weight information calculation process S32 is executed, and in the case of the back fade, the second weight information calculation process S33 is executed.

In the first weight information calculation process S32, the offset coefficient w ₂ which is a part of the weight information is set to a fixed value 0, and only the weight coefficient w ₁ is calculated. At this time, the weight coefficient w ₁ which minimizes the error e between the pixel value average and the weight-compensated pixel value average in the image to be processed is obtained as the optimal weight coefficient.

In the second weight information calculation process S33, it is necessary to determine not only the weight coefficient w ₁ but also the offset coefficient w ₂ , so that the increase or decrease of the pixel value average is not the pixel value average itself, but the difference of the pixel value average. Use proportional to (change width). In this case, the weight coefficient w ₁ can be obtained by the ratio of the complement of the pixel value average of the processing target image and the complement of the pixel value average of the reference image. The offset coefficient w ₂ can be obtained depending on the weight coefficient w ₁ .

2. Motion information calculating unit 3 and motion compensating unit 4

The motion information calculator 3 performs motion search using a reference image weighted by the provisional weight information calculated by the first weight information calculator 1 to calculate motion information. In the reference image used here, since the effect of the fade is canceled to some extent in the weight compensator 2, the motion information used for motion compensation can be calculated with precision. Any method can be used for the motion search here.

2 (c) and 2 (d) show a state of motion search and a state in which the first and second reference images are motion compensated by the motion compensator 4, respectively. By the processing in the motion information calculating section 3 and the motion compensating section 4, the positional correspondence of the moving object (○ mark) in the first reference picture and the second reference picture can be taken.

3. Encoding Mode Determination Unit (5)

The encoding mode determiner 5 determines the encoding mode using the reference image compensated by the weight compensator 2 and the motion compensator 4. That is, in order to determine whether it can be sufficiently approximated by the weight compensation by the weight compensator 2 and the motion compensation by the motion compensator 4, a reference method in which the error is the minimum among the plurality of reference methods is selected. Select the coding mode that minimizes the prediction error. As an error in this determination, a square error between a reference image and an image to be processed can be used.

The reference method includes a method of referring to the first reference picture, a method of referring to the second reference picture, a method of referring to the first and second reference pictures, and a method of not referring to any of the reference pictures.

A set of pixel blocks having an encoding mode that refers to the first reference picture is referred to as F, and a set having an encoding mode that refers to the second reference picture is referred to as B, and a coding mode at which the first reference picture and the second reference picture are referred to. A set having C and a set having a coding mode that does not use a reference picture are called D. In addition, since the weight information does not affect the set D, it is not necessary to consider the calculation of the weight information for the pixel block of the set D.

4 exemplarily shows an encoding mode in a process target image. Here, the sets F, B, C, and D are shown as being distributed in a single mass, but in reality, they are distributed in blobs in units of pixel blocks.

4. The second weight information calculator 6 and the weight compensator 7

The second weight information calculating section 6 inputs a reference image compensated using only the motion information without using the provisional weight information so as to exclude only the influence by the motion. In addition, the reference method by the encoding mode obtained by the process target image and the encoding mode determination part 5 is input.

As the weight information, one set of the weight coefficient and the offset coefficient is calculated for each reference image. The weight compensation may be the case of a reference method using only one reference image or a reference method using a plurality of reference images. However, the latter may be saved by using the former weight information to save the amount of code necessary for the weight information, Can be common with motion compensation.

However, the rotation using the weight information does not necessarily mean that the optimal weight information in the case of the reference method using only one reference image becomes the optimal weight information even in the case of the reference method using a plurality of reference images. Both of these reference methods require evaluation by an evaluation function that is optimal weight information.

Here, the pixel values of the processing target image consisting of n pixels and the pixel values of the first and second reference images that are motion compensated by the motion information are p _i , p _i ', and p _i "(1 ≦ i ≦ n), respectively. In addition, i indicates the order of the pixels when the pixels in one frame are arranged in the scanning order.

If pixel values for each set of coding modes F, B, and C are used, weight information (w ₀ , w ₂ ) for the first reference picture and weight information (w ₁ , w ₃ ) for the second reference picture are used. The square error (e) with respect to the pixel of the image to be processed by compensation can be expressed by the following equation (1).

The first item on the right side of equation (1) is the square error in set F, the second item is the square error in set B, the third item is the square error in set C, and the addition of those square errors is It is the total squared error (e). Equation (1) is an evaluation function of the optimal weight information.

The distribution of the sets F, B, C, and D is already determined by the encoding mode determiner 5. It is assumed that n _f , n _b , n _c , and n _d pixels (n _f + n _b + n _c + n _d = n) exist in each set.

Here, since w ₀ , w ₁ , w ₂ , and w ₃ , which minimize the square error (e), become optimal weight information, the condition that becomes 0 by partial derivative of the squared error (e) is determined to determine the weight information. Obtain First, the partial derivative of the squared error (e) is derived from Equation (2). However, the sum of pixels belonging to the set X is represented by the following equation (3) using a Gaussian symbol.

When the weight information is expressed in a vector, the condition for the partial derivatives of the equation (1) represented by the equation (2) becomes zero is expressed by equation (5).

However, the matrix A and the vector b have the product sum of the pixels for each set of coding modes F, B, and C as elements, and are defined by equations (6) and (7), respectively. The matrix A is a symmetric matrix composed of the sum, the sum of squares, and the product sum of the pixel values of the reference images for each encoding mode, and b is a vector composed of the sum or multiplication including the pixel values of the processing target image for each encoding mode.

By substituting the pixel values into the matrix A and the vector b and solving Equation (5) mathematically, the vector w of the weight information can be obtained. (5) is a linear equation that can be transformed into 0 = Aw-b. Since matrix A is a symmetric matrix, it can be solved by backward substitution by Cholesky decomposition without obtaining an inverse matrix.

The second weight information calculating section 6 calculates the weight information with high precision, and the weight compensating section 7 performs the weight compensation using this weight information, so as to have high precision as shown in FIG. A weight compensated first and second reference picture are generated. If necessary, motion information can be obtained from the motion information calculating section 3, and weight information can be obtained from the second weight information calculating section 6.

In the above, the processing in the first weight information calculating unit 1 and the weight compensating unit 2 and the processing in the motion information calculating unit 3 and the motion compensating unit 4 are performed only once, respectively, and the encoding mode determination unit ( 5) As described above as proceeding the processing in the second weight information calculating section 6 and the weight compensating section 6, after performing the processing in the motion compensating section 4, the first weight information calculating section 1 Return to the processing, repeat the weight compensation and motion compensation thereon, and then proceed the processing to the encoding mode determination section 5, the second weight information calculating section 6, and the weight compensation section 7. It may be. In this way, motion compensation and weight compensation with higher precision are possible.

The reference method is not limited to the method of using one frame before and after the processing target image as a reference image, and may be a method of using any other frame or any combination of those frames as a reference image.

6 is a block diagram showing a video encoding apparatus to which the motion prediction information detecting apparatus according to the present invention is applied. I is an intra coded frame that does not perform prediction between frames. I, a forward predictive coded frame that performs prediction using a past reference picture, P, and a bidirectional predictive coded frame that performs prediction using a past and future reference picture, B. , Successive moving images, for example, I (1), B (2), B (3), P (4), B (5), B (6), P (7),... In order to be encoded by the present invention, the apparatus has a sequence of I (1), P (4), B (2), B (3), P (7), B (5), B (6),... The input image a replaced by is input.

As a result, P (4) can predict I (1) as a reference picture, and B (2) and B (3) can predict I (1) and P (4) as a reference picture. . In addition, P (7) enables prediction of P (4) as a reference picture, and B (5) and B (6) enables prediction of P (4) and P (7) as a reference picture. .

The input image a is input to the forward predictor 61, the backward predictor 62, the bidirectional predictor 63, the weighted motion compensator 66, and the prediction selector 64. The forward predicting unit 61 outputs a prediction error between the input image a and the weighted motion compensated past reference image. In addition, the backward predictor 62 outputs a prediction error between the input image a and the weighted motion compensated future reference image, and the bidirectional predictor 63 compensates the input image a and the weighted motion. The prediction error between the past and future reference pictures is output.

The weighted compensated reference image input to the forward predictor 61, the reverse predictor 62, and the bidirectional predictor 63 is weighted compensated by the weighted motion compensator 66 for the image accumulated in the frame memory 65. It is obtained by. The weighted motion compensator 66 includes a motion prediction information detection apparatus according to the present invention.

The prediction selector 64 is controlled according to the encoding mode from the selection controller 67, i.e., whether the frame is I, P or B. In I, the input selection a is selected, and in P, the forward predictor 61 is selected. ), And in B, according to the encoding mode determined for each macroblock, the input image a, the prediction error from the forward predictor 61, the prediction error from the backward predictor 62, or The prediction error from the bidirectional predictor 63 is selected. In order to enable selection in each macroblock in the case of B, the selection controller 67 inputs an encoding mode determined by the motion prediction information detecting apparatus according to the present invention. Since the prediction selection unit 64 makes a selection after the encoding mode is determined, the output is delayed here.

The DCT (orthogonal transformation) unit 68 converts the input image a or the prediction error selected by the prediction selecting unit 64 into DCT coefficients in units of 8x8 pixels, for example, and the quantization unit 69. ) Quantizes the DCT coefficients according to the quantization parameters from the quantization controller 70. The output of the quantizer 69 is variable length coded by the variable length coding (VLC) unit 71, multiplexed by the multiplexer 72 together with the weight information and the motion vector information, and then transmitted through the buffer 73. do.

The outputs for I and P output from the quantization unit 69 are inversely quantized by the inverse quantization unit 74, reproduced by a prediction error signal in the inverse DCT unit 75, and weighted by the adder 76. It is added with the reference image from the compensator 66. The local decoded image obtained by this addition is stored in the frame memory 65. In this way, local decoded images for I and P are stored in the frame memory 65 and updated.

The weighted motion compensator 66 calculates motion information and weight information from the local decoded image and the input image a stored in the frame memory 55 and simultaneously generates a reference image to generate the reference image. The backward predictor 62, the bidirectional predictor 63, and the adder 76 are output.

According to the present invention, after weight compensation of a reference picture with the temporarily determined weight information, motion information is determined from the reference picture and the processing target image, and again weight information is obtained from the reference image and the processing target image motion-compensated with the motion information. Since the reference image is composed of a high precision reference image, optimal motion information and weight information can be determined.

In the weight information calculation, a combination of a processing method having both resistance to motion and high speed and a method of determining weight information with high precision by selecting pixels to be processed are performed. And weight information. This makes it possible to realize an improvement in the compression ratio when compressing an image with temporal pixel change such as fading.

Further, if the motion compensation and the weight compensation are repeatedly applied alternately, motion information and weight information closer to the optimum solution can be obtained. Considering the iterative operation at this time, it is preferable that the processing load per one is lower than the height of the precision of the motion prediction information.

Also, the coding mode is determined, and the coding mode selects pixels used for calculating weight information and unused pixels so as to refer to the minimum required pixels, thereby reducing the amount of computation in determining final weight information. In addition, high-precision weight information can be derived.

Claims (10)

By applying the weighted motion compensation to the reference image, the motion prediction information detection apparatus for determining the motion information and the weight information for minimizing the error between the processing target image and the reference image by improving the accuracy of the reference image for the processing target image In First weight information calculating means for determining provisional weight information from the reference image and the processing target image; Motion information calculating means for determining motion information from the reference image and the processing target image compensated with the weight information determined by the first weight information calculating means; Encoding mode determination means for determining an encoding mode from the precision of the processing target image compensated with the weight information determined by the first weight information calculating means and the motion information determined by the motion information calculating means; And second weight information calculating means for determining weight information from the reference image compensated with the motion information determined by the motion information calculating means, the processing target image, and the encoding mode. The method of claim 1, And the first and second weight information calculating means and the motion information calculating means are repeated. The method of claim 1, And said first weight information calculating means is configured to calculate weight information from the feature amounts of the entire frame. delete The method of claim 1, And the encoding mode determining means determines the optimal encoding mode according to the magnitude of the error of the processing target image compensated with the motion information and the weight information. The method of claim 1, And the second weight information calculating means selects pixels used for calculating weight information and pixels not used in the encoding mode. The method of claim 1, And said second weight information calculating means performs calculation processing in accordance with an encoding mode. The method of claim 1, And the second weight information calculating means determines weight information such that the sum of the errors set for each encoding mode is minimized. The method of claim 1, And said second weight information calculating means determines weight information by solving a first equation. The method of claim 9, The first equation includes a symmetric matrix consisting of the sum, the sum of squares, or the product sum of the pixel values of the reference frame for each encoding mode, and a vector consisting of the sum or product sum including the pixel values of the encoding target frame for each encoding mode. Motion prediction information detection device.

Images