JP4763241B2 - Motion prediction information detection device - Google Patents

Description

  The present invention relates to a motion prediction information detection device, and more particularly to a motion prediction information detection device that determines weight information that minimizes an error of an approximate image with respect to a processing target image in weighted motion compensation.

  One of the encoding methods for encoding continuously input moving image signals is an inter-frame predictive encoding method. In the inter-frame predictive coding method, motion prediction information is used to perform motion compensation with improved temporal correlation prediction efficiency.

  In a fade scene where the brightness changes over time and a cross-fade (dissolve) scene where two scenes are mixed, conventional simple motion compensation cannot perform efficient compression, so several measures are proposed. Has been.

  For example, Patent Documents 1 and 2 describe that correction is performed using a frame average luminance difference, and Patent Documents 3, 4, and 5 adaptively allocate bit allocation, the encoder itself, and an error evaluation function for motion search, respectively. In order to prevent deterioration of encoding efficiency in a fade scene by changing to, it has been proposed to devise a coding process in a fade section. Patent Document 6 proposes to suppress the generation of a random motion vector by forcibly setting the motion vector to 0 in a scene determined to be a fade.

The MPEG-4 AVC / H.264 (ISO / IEC 14496-10) international standard for video compression has decided to adopt a weighted motion compensation method in addition to the conventional motion compensation method. It has been shown that the attached motion compensation method is effective for compression of fade scenes. However, since MPEG-4 AVC / H.264 (ISO / IEC 14496-10) does not stipulate a method for determining a weighting factor, Patent Document 7 restricts the weighting factor to 0, 0.5, 1, and 2, The weighting factor is determined based on the transition of the frame average difference.
JP-A-6-46412 JP-A-8-65684 Japanese Patent Laid-Open No. 10-336641 JP 2001-510964 A Japanese Patent Laid-Open No. 2003-87795 JP 2002-51341 A JP 2003-284075 A

  Since the technique of Patent Document 1 requires additional information, there is a problem that it is not compatible with the existing compression method, and the technique of Patent Document 2 performs processing in units of macroblocks (MB), which increases the amount of calculation. There is a problem of doing. In the method using the frame average luminance difference, the global frame average luminance difference is not correctly applied to the local change, and may not function correctly. For example, when the pixel value deviates from the range that can be taken near the start point or end point of the fade, an effect different from the change due to the fade appears in the frame average luminance difference when rounded to the minimum value or the maximum value.

  The techniques of Patent Documents 3, 4, and 5 are intended for simple motion compensation based on only translation, and this naturally has a limit in terms of compression rate. Further, Patent Document 6 aims only to prevent useless information due to an incorrect motion vector, and this does not necessarily improve the compression efficiency.

  Furthermore, since the technique of Patent Document 7 limits the weighting factor that can originally set an arbitrary value, it cannot cope with a non-linear fade that changes exponentially or logarithmically. In addition, since the method for determining the optimum weight coefficient depends on the fade detection based on the transition of the frame average luminance, it has the same problem as the method using the frame average luminance difference.

  An object of the present invention is to solve the above-described problems and provide a motion prediction information detection apparatus capable of determining weight information of an arbitrary weighted motion prediction method at high speed and with high accuracy.

In order to solve the above-described problem, the present invention is used to weight an approximate image that has been motion-compensated from an arbitrary number of reference images of one or more, and improves the accuracy of the approximate image to improve the error from the processing target image. In the motion prediction information detection apparatus that determines weight information that minimizes the number of reference images, an arbitrary number of reference images referred to by the processing target image is given for each partial region, and each reference image in a weighted compensation scheme according to the number A derivation formula deciding means for deriving a derivation formula necessary to simultaneously determine the weight information on the image, and a combination of the pixel value of the processing target image and the pixel value of the reference image according to the derivation formula derived by the derivation formula decision means It includes a motion prediction information deriving means for determining the weight information for each partial area from the calculation result of the number of elements of the weighted compensation method and the weight information is arbitrarily given, the derived formula determining means, sheets A plurality of derivation formulas necessary for determining the weight information of the weighted compensation scheme according to the above are derived in advance, and the motion prediction information derivation means appropriately calculates the plurality of derivation formulas derived in advance by the derivation formula determination means. There is a basic feature in that it is configured to select and determine weight information for each partial region .

  According to the present invention, a derivation formula corresponding to the weighted motion compensation method is derived, and weight information is determined according to the derivation formula. Therefore, the reference image for weighted motion compensation is obtained from the pixel values of the reference image and the processing target image. The optimum weight information can be determined according to an arbitrary number of reference images without depending on the number. Further, when determining the weight information, the calculation amount can be greatly reduced by devising the calculation process, and the processing load can be reduced.

First, the principle of the present invention will be described. In the process of compressing each frame of the moving image, the processing target image is defined as _Pc . Further, when the pixels constituting the processing target image _Pc are arranged in the scanning order, the pixel value of the i-th pixel is expressed by p (i; c). At this time, the pixel values of the pixels of any of the reference image P _{c + s} and approximate image P _^'c reconstructed using the conventional motion prediction information u _s away s frame from the processed image P _c is, p (i + u _s ; c + s).

On the other hand, assuming that j = 1, 2,..., M is a parameter and the pixel value converted by the weight information w _j is p ″ (i; c), the pixel value of the reconstructed image subjected to weighted motion compensation Is expressed by equation (1) using a weight function f (), which defines a weighted motion compensation scheme.

p "(i; c) = f (p (i + u s; c + s), w j) (1)

In the case of using a plurality of reference images, yet another reference from the target image P _c distant r frame images P _{c + r} of the pixel values p; if (i c + r) and the use of the motion prediction information u _r, weight The pixel value of the reconstructed image that has undergone additional motion compensation is expressed by equation (2) using a weight function g ().

p "(i; c) = g (p (i + u s; c + s), p (i + u r; c + r), w j) (2)

Determining the weight information includes weight information w ₁ , w ₂ ,... For matching the pixel value p ″ (i; c) obtained by the above method with the pixel value p (i; c) of the original image. , W _m .

Therefore, as a measure for measuring the similarity between the pixel value p (i; c) of the processing target image P _c and the weighted motion compensated pixel value p ″ (i; c), the sum of square errors of both pixel values e Is defined by the equation (3), and w _j that minimizes e is the optimum weight information.

In order for the weight information w _j to minimize the sum of squared errors e, equations (4) obtained by partial differentiation of e with respect to the individual weight information w _j must be zero.

_{Here, p oi = p (i;} c), p 1i = f (p (i + u s; c + s), w j), p gi = g (p (i + u s; c + s), p (i + u r; c + r) , w _j ), the expression (4) is expressed by the expression (5) or the expression (6).

In general, the weight functions f () and g () are linear with respect to the unknown weight information w _j , and there are m constraint equations for m unknowns w _j , so the weight information w _j is algebraic. Can be derived.

(5) or (6) is weight prediction information in which the sum obtained by multiplying the term obtained by partial differentiation of the weighting function by weighting information and the error component term by the approximated image obtained by weighted motion compensation is zero. This is a derivation formula for w _j .

The expression (5) or the expression (6) can be separated into, for example, a term that is multiplied by unknown weight information w _j and a term that is not multiplied. Here, the sum of the individual pixel values assigned to the product of the derivative of the weight function f () and the weight function f () itself, and the product of the derivative of the weight function g () and the weight function g () itself is Is a matrix with <A>, and the sum of the individual function values assigned to the product of the derivative of the weight function f () and the pixel to be processed or the product of the weight function g () and the pixel to be processed Is defined as <b>, the weight information <w _j > can be obtained by <A> ⁻¹ <b>. Note that lowercase letters with <> represent vectors, and uppercase letters with <> represent matrices.

  Next, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a flowchart showing a processing procedure in the motion prediction information detecting apparatus according to the present invention based on the above principle.

The processing procedure of this example is a step (S1) of calculating a matrix <A> made up of a product of weight function and weight function, and a vector <b> made up of the product of weight function derivative and pixel value. It comprises a step (S2) for calculating and a step (S3) for calculating the weight information <w _j >.

  In S1, the sum of the individual pixel values assigned to the product of the derivative of the weight function f () and the weight function f () itself, or the product of the weight function g () and the weight function g () itself As a matrix <A>.

  In S2, a vector <b> having, as elements, the product of the derivative of the weighting function f () and the pixel to be processed, or the sum of the individual pixel values assigned to the product of the derivative of the weighting function g () and the pixel to be processed. Is calculated. Note that the pixels to be processed in S1 and S2 do not have to be all pixels of the processing target image, and may be a part thereof. The weighting function and its parameters are arbitrary, and the matrix <A> and vector <b> for a plurality of weighting functions and parameters can be calculated in advance so that they can be selected as appropriate.

In S3, <A> ⁻¹ <b> is calculated from the matrix <A> and the vector <b> calculated in S1 and S2, and weight information <w _j > is obtained.

  Next, a procedure for determining weight information using a reference image will be specifically described. In the following, a case where there are one reference image and a case where there are two reference images will be described. However, the number of reference images is arbitrary, and even in that case, the same procedure can be performed.

First, a case where there is one reference image will be described. When a pixel value subjected to motion compensation in an arbitrary reference image P _{c + s} of the processing target image P _c is abbreviated by Expression (7), a pixel value p ″ (i; c) by weighted motion compensation uses a weight function f (). The weight coefficient w ₁ and the offset coefficient w ₂ are defined here as weight information.

p _1i = p (i + u _s ; c + s) (7)

p "(i; c) = f (p _1i , w _j )
= W ₁ p _1i + w ₂ (1 ≦ i ≦ n) (8)

At this time, the sum e of square errors between the pixel value p0i of the processing target image Pc and the pixel value obtained by weighted motion compensation is expressed by Expression (9). n represents the number of the processing target pixel, a processing target pixel constitutes a part of the processing target image.

The partial differential term based on the weight information w _j of the weighting function f () is expressed by equation (10), and the partial differential based on the unknown variable (w ₁ , w ₂ ) of the sum of squared errors e is expressed by equation (11). However, in the coefficient of equation (11) [
] Is a symbol of the sum of Gauss. For example, [p ₀ ] indicates n sums of pixel values p _0i as represented by the equation (12).

Expression (11) is a simultaneous linear equation with coefficients as the sum of pixel values of the processing target image, the sum and square sum of the pixel values of the reference image, the product sum of the pixel values of both images, and the number of processing target pixels. Thereby, the weight information w _j can be obtained.

When Expression (11) is expressed in the form of a vector and a matrix with <w> = (w ₁ , w ₂ ) ^t , Expression (13) is obtained.

_{<A 1><w> - <} b 1> = <0> (13)

Here, the matrix <A _1> is given by equation (14), the vector <b _1> is given by equation (15).

As is clear from the equations (14) and (15), the individual pixel values p _0i in the processing target image P _c, the individual pixel values p _1i obtained by motion compensation from the reference image, the product of both, and the processing target image from the number of pixels n, it can be determined each element of the matrix <A _1> and vector _{<b 1>.} Vector of weight information using one reference picture _{<w> = (w 1,} w 2) t can be obtained by calculating the ^{_{<A 1> -1 <b 1>}} .

Further, <A _1> without calculating the inverse matrix _<A ^{1> -1} since it is symmetric matrix, decomposing the <A _1> in the lower and upper triangular matrices by Cholesky (Cholesky) decomposition, < The weight information vector <w> can also be obtained by performing stepwise substitution of the lower triangular matrix and stepwise substitution of the upper triangular matrix for b ₁ >.

Lower triangular matrix _{<L 1>} can be calculated by _{<A 1> = <L 1>} <L 1> t, <A 1><w> - <b 1> = deforming the equation (16) from <0> Therefore, first, <L ₁ > ^t <w> is obtained by backward substitution, and then <w> is obtained.

<L ₁ >(<L ₁ > ^t <w>) = <b ₁ > (16)

Moreover, since the equation (17) is obtained by solving the equation (11) or the equation (13) algebraically, the weight information w _j can also be obtained by this.

Next, a case where there are two reference images will be described. When pixel values p _1i and p _2i motion-compensated in arbitrary reference images P _{c + s} and P _{c + r} of the processing target image P _c are abbreviated by equation (18), pixel values p ″ (i; c by weighted motion compensation) ) Is given by equation (19) using a weight function g (), where weighting factors w ₁ and w ₂ and an offset factor w ₃ are defined as weighting information.

p _1i = p (i + u _s ; c + s)
_{p 2i = p (i + u} r; c + r) (18)

p "(i; c) = g (p _1i , p _2i , w _j )
= W ₁ p _1i + w ₂ p _2i + w ₃ (1 ≦ i ≦ n) (19)

At this time, the sum e of the square errors between the pixel value p _0i of the processing target image P _{c and} the pixel value by weighted motion compensation is expressed by Expression (20). n represents the number of pixels to be processed, and the pixel to be processed is composed of all the pixels of the processing target image or a part thereof.

The partial differential term based on the weight information w _j of the weight function g () is expressed by Equation (21). Therefore, the partial differentiation of the sum of squared errors e by unknown variables (w ₁ , w ₂ , w ₃ ) is given by equation (22). However, [] in the coefficient of equation (22) is a symbol of the sum of Gauss, and for example, [p ₀ ] represents n sums of pixel values p _0i as represented by equation (23). Show.

Equation (22) is a simultaneous linear equation with the coefficient of the sum of pixel values of the processing target image, the sum and square sum of the pixel values of the reference image, the product sum of the pixel values of the two images, and the number of processing target pixels. With this, the weight information w _j can be obtained.

<W> = (w ₁ , w ₂ , w ₃ ) When Expression (22) is expressed in the form of a vector and a matrix as ^t , Expression (24) is obtained.

_{<A 2><w> - <} b 2> = <0> (24)

Here, the matrix <A _2> is given by equation (25), the vector <b _2> is given by equation (26).

As is clear from the equations (14) and (15), the individual pixel values p _0i in the processing target image P _c, the individual pixel values p _1i and p _2i obtained by motion compensation from the reference image, from the pixel number n of the product and the processing target image, it is possible to determine the elements of the matrix <A _2> and vector <b _2>. Vector of weight information using two reference images _{<w> = (w 1,} w 2, w 3) t can be obtained by calculating the ^{_{<A 2> -1 <b 2>}} .

Further, <A _2> without calculating the inverse matrix _<A ^{2> -1} since it is symmetric matrix, is decomposed into a lower triangular matrix and an upper triangular matrix <A _2> by Cholesky decomposition, _<b 2> On the other hand, the weight information vector <w> can also be obtained by performing stepwise substitution of the lower triangular matrix and stepwise substitution of the upper triangular matrix.

Lower triangular matrix _{<L 2>} can be calculated by _{<A 2> = <L 2>} <L 2> t, <A 2><w> - <b 2> = deforms the formula (27) from <0> First, <L ₂ > ^t <w> is obtained by backward substitution, and then <w> is obtained.

<L ₂ >(<L ₂ > ^t <w>) = <b ₂ > (27)

Moreover, since the equation (28) is obtained by solving the equation (22) or the equation (24) algebraically, the weight information w _j can also be obtained.

  According to the present invention, it is possible to determine optimum weight information according to an arbitrary number of reference images, without depending on the number of reference images for weighted motion compensation, from the pixel values of the reference image and the processing target image. By using the weight information determined in this way, MPEG-4 AVC / H.264 can achieve more efficient compression efficiency than conventional motion compensation. In addition, in the fade and dissolve scenes, the weight information determined by the present invention improves the compression efficiency compared to the conventional motion compensation. Therefore, the present invention examines the place where the compression efficiency has been improved and performs the fade and dissolve scenes. It can also be used for detection purposes.

It is a flowchart which shows the process sequence in the motion prediction information detection apparatus which concerns on this invention.

Explanation of symbols

S1... Matrix <A> calculation step, S2... Vector <b> calculation step, S3... Weight information <w _j > calculation step

Claims (8)

Motion prediction is used to weight a motion-compensated approximate image from an arbitrary number of reference images of one or more, and determines weight information that improves the accuracy of the approximate image and minimizes an error from the processing target image. In the information detection device,
An arbitrary number of reference images to be referred to by the processing target image is given for each partial area, and a derivation formula necessary for simultaneously determining weight information for each reference image in the weighted compensation method according to the number is derived. A derivation determining means;
A motion prediction information deriving unit that determines weight information for each partial region from a calculation result by a combination of a pixel value of the processing target image and a pixel value of the reference image according to the deriving equation derived by the deriving equation determining unit ;
The weighted compensation method and the number of elements of weight information are arbitrarily given,
The derivation formula determining means derives in advance a plurality of derivation formulas necessary for determining weight information of the weighted compensation method according to the number of sheets, and the motion prediction information deriving means is derived in advance by the derivation formula determining means. A motion prediction information detection apparatus configured to determine weight information for each partial region by appropriately selecting a plurality of derived equations . 2. The motion according to claim 1, wherein the derivation formula determining means uses a square error between the pixel value of the processing target image and the pixel value obtained by weighted motion compensation as the accuracy evaluation criterion of the weight information. Prediction information detection device. The derivation formula determining means includes a plurality of derivations in which a sum obtained by multiplying a term obtained by partial differentiation of the weighting function in the weighted compensation scheme by weight information and an error component term based on the approximate image obtained by weighted motion compensation becomes zero. The motion prediction information detection apparatus according to claim 2 , wherein an equation is derived. The derivation formula determining means includes a sum of pixel values of the processing target image, a sum and square sum of pixel values of the reference image, a product sum of pixel values of two images of the processing target image and the reference image, and the processing target image. The motion prediction information detection apparatus according to claim 2 , wherein a derivation formula comprising simultaneous linear equations with the number of pixels as a coefficient is determined. The motion prediction information detection apparatus according to claim 4 , wherein the prediction information deriving unit includes a deriving unit that determines weight information by solving a deriving equation configured as a coefficient of simultaneous linear equations. The derivation formula determination means derives a derivation formula from a weight function in the weighted motion compensation scheme, and the motion prediction information derivation means uses the derivation formula coefficients derived by the derivation formula determination means based on actual pixel values. The motion prediction information detection apparatus according to claim 1, wherein weight information is determined by calculating and solving the derivation formula. The motion prediction information deriving means multiplies weight information by multiplying an inverse matrix of a coefficient matrix whose elements are coefficients applied to weight information in the derivation formula as a solution of the derivation formula and a coefficient vector consisting of other coefficients. The motion prediction information detection apparatus according to claim 6 , wherein: In place of the motion prediction information deriving means of claim 7 , the coefficient matrix is decomposed into a lower triangle matrix and an upper triangle matrix based on Cholesky decomposition, and backward substitution of the lower triangle matrix and backward substitution of the upper triangle matrix are performed on the coefficient vector. A motion prediction information detection apparatus characterized in that weight information is determined by performing stepwise processing by omitting an inverse matrix calculation process of a coefficient matrix.

Images