JPH08242454A - Method for detecting global motion parameter - Google Patents

Abstract

PURPOSE: To reduce the required processing amount without much deteriorating estimate accuracy in the method that panning and zooming parameters are estimated with the least squares method by using a motion vector MV in the unit of blocks. CONSTITUTION: When the reliability of estimating detection of a motion vector MV from each reference position block of a coding object frame 2 is low, a block with high reliability of estimating detection of the MV is sought from object positions around the reference position instead. An MV detection circuit 11 detects the MV based on image data in each reference area corresponding to each block to be selected and a coding reference frame 1 and it is used to estimate panning parameters e, f and a zooming parameter (m) by using the MV by means of the least squares method. A substitute block is sought as to blocks where a difference between the expanded MV memory 15 and the detection MV memory 12 to estimate a final parameter.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to detection of a global motion parameter of a moving image, and more particularly to a parameter representing pan (horizontal and vertical translation) and zoom (enlargement / reduction) of global motion. Regarding the detection of.

The present invention relates to communication, storage of digital moving images,
It can be used in fields such as broadcasting where data amount compression is required by motion-compensated inter-frame prediction of moving images, and is particularly suitable for use in moving image coding systems for small digital video cameras, etc. where reduction of processing amount is important. Is.

[0003]

2. Description of the Related Art Generally, a motion compensation interframe prediction method is used for moving image data compression (MPEG, H.261).
etc). In this motion-compensated inter-frame prediction method, the screen is divided into multiple blocks, the translation vector is detected between each of the decoded image and the image of another frame, and the motion vector information is shifted together with the motion vector information. The difference is encoded by predicting from the position image.

By the way, when the whole screen is moved by a camera operation such as pan (translation in horizontal and vertical directions) and zoom (enlargement or reduction), motion vectors are generated in most blocks in the screen, and the motion is generated. The amount of vector information data also increases. Here, the amount of transmission data of motion vector information can be reduced by extracting parameters for panning and zooming, that is, parallel movement of the screen and enlargement / reduction, and transmitting only these parameters. Also, in a zoom scene, it is essentially impossible to compensate for accurate motion by parallel movement in block units, but if zoom parameters are accurately extracted, it is possible to expand motion vectors in pixel units using zoom parameters. By this, accurate compensation is also possible.

Several methods for extracting pan and zoom parameters from image information have been proposed. One of them is a method in which an image is divided into a plurality of blocks, the motion vector of each block is obtained, and then the pan and zoom parameters are estimated by least-squares estimation using all of these motion vectors. 3-191688). This method
Since it is directly related to the present invention, it will be briefly described below.

A motion vector for each pixel when the entire screen moves can be expressed by six parameters a to f as in the following expression (1).

[0007]

[Equation 1]

However, (x, y) is a pixel position (x coordinate, y coordinate), and (Vx, Vy) is a pixel motion vector (horizontal component, vertical component). here,

[0009]

[Equation 2]

In this equation (2), p1 = (a + d) / 2 p2 = (-b + c) / 2 p3 = (ad) / 2 p4 = (b + c) / 2, where p1 is the zoom parameter and p2 Is the rotation parameter,
p3 is the vertical / horizontal distortion parameter, p4 is the diagonal distortion parameter, e
And f are pan parameters in the horizontal and vertical directions.

Here, among a to f, the zoom parameter m =
Consider a model of the following expression (3) using only (a + d / 2) and pan parameters e and f. Therefore, b = c =
0 and p2 = p3 = P4 = 0.

[0012]

(Equation 3)

Then, it is considered that the motion vector of each block detected in advance is applied to the equation (3) and each parameter of m, e, and f is estimated by the least square error method. FIG.
3 is a simplified C language source list showing this estimation algorithm.

In FIG. 13, (Y_WIDTH, Y_HEIGHT)
Is the horizontal and vertical size of the screen in pixels, ME_B
LK_SZ is the size of one side of the square block, and these are specified in advance. (MB_X_SIZE, MB_Y_SIZE) is the horizontal and vertical size of the screen in block units. (X, y)
Is the x, y coordinate of the position of the central pixel of the block, (vx, v
y) are vertical and horizontal components of the motion vector of the block center pixel, which are obtained in advance for the representative block and given as a three-dimensional array mv. zoom
_param (= gmc_params [D]) corresponds to m in equation (3), and g
mc_params [E] corresponds to e in equation (3), gmc_params [F]
Corresponds to f in the equation (3), and these are the ones to be obtained here.

The outline of the processing shown here is as follows. The first two lines are preprocessor control statements for defining the screen size in block units. The next two lines are parts for initializing various variables to 0. F of the next nested structure
The or statement portion increments the block vertical direction number blno_y by 1 from 0 (moving the block row from top to bottom), and the block horizontal direction number bloc_x.
Is incremented by 1 from 0 (while selecting blocks in a block row in order from left to right), the position (x, y) of the central pixel of each block is obtained, and the vertical direction component of the motion vector at that pixel position is calculated. The horizontal component is read from the array mv, substituted into the variables vy and vx, and the operations of incrementing and adding various variables are repeated, and the processing is performed up to the block at the lower right of the screen. Array variable gmc in the next two lines
_params [B] and gmc_parms [C] are cleared to 0, and double precision arithmetic is performed in the next 7 lines to obtain the array variable gmc_params [A] = gmc_params [D]. Then, in the next three lines, three target parameters are obtained.

Since the blocks are arranged symmetrically with respect to the screen center point, when motion vectors of all blocks are used for parameter calculation, sum_ in the estimation formula shown in FIG. 13 is used.
x and sum_y become 0, and the estimation formula becomes simpler. Japanese Patent Laid-Open No. 3-191688 describes the above method in this case in a simpler form.

[0017]

[Problems to be Solved by the Invention] Japanese Patent Publication No. 3-19
The parameter estimation method of No. 1688 is based on the premise that motion vectors are obtained for all divided blocks on the screen and all of them are used for global motion parameter estimation. This is because, in normal moving image coding, it is usual that a motion vector in a block unit is already obtained by some method for motion compensation prediction in a block unit.

However, the block-by-block motion detection processing is
Since it occupies most of the processing amount of the entire encoding processing, performing motion detection for all blocks as in the conventional method is a heavy burden on a moving image encoding system having a simple configuration in terms of processing amount. There was a problem.

Solving such a problem is the main object of the present invention. That is, an object of the present invention is to provide a means for obtaining a global motion parameter of a moving image with sufficient accuracy with a smaller processing amount, thereby realizing a moving image coding system having a simple structure for global motion compensation. It is possible.

[0020]

In order to achieve the above object, the invention according to claim 1 does not use the motion vectors of all the blocks in the screen for estimating the pan and zoom parameters, but follows a certain rule. It is characterized in that only the motion vectors of some selected blocks are used for estimating the pan and zoom parameters. A main feature of the invention described in each of claims 2 to 10 relates to selection of a block used for estimation of pan and zoom parameters.

That is, according to the second aspect of the present invention, some blocks are selected as a representative block every predetermined number of blocks in the horizontal and vertical directions from a plurality of blocks in the screen, and only the blocks selected as the representative block are selected. , The motion vector is detected and used for estimating the pan and zoom parameters.

According to the third aspect of the present invention, like the second aspect of the invention, some blocks are selected as a representative block in the horizontal and vertical directions from a plurality of blocks in the screen at a fixed number of blocks at a time. Among the selected blocks, blocks that are estimated to have low reliability of motion vector detection are excluded from the representative block, and the motion vector of only the remaining representative block is detected, and the motion vector is detected as a pan and zoom parameter. It is characterized by being used for estimating.

According to a fourth aspect of the present invention, similarly to the third aspect of the present invention, among the blocks selected as the representative block at every fixed number of blocks, the block estimated to have low reliability of motion vector detection. Is excluded from the representative block, but at a predetermined position around the excluded block, if there is a block estimated to have high reliability of motion vector detection, that block is set as the representative block, and The motion vector of only the block that is the representative block is detected and used for estimating the pan and zoom parameters.

According to the fifth aspect of the invention, a part of the blocks is selected as a representative block from a plurality of blocks on the screen according to a certain rule, and a motion vector is detected for the selected block. The parameters of pan and zoom are estimated by the least square error method using the motion vector. From the estimated pan and zoom parameters, expand the motion vector of the block selected as the representative block, and in the block selected as the representative block, the difference between the expanded motion vector and the detected motion vector is A block exceeding a predetermined threshold is excluded from the representative block. Then, the final pan and zoom parameters are estimated by the least-squares error method using only the motion vector detected for the block as the representative block remaining without being excluded.

According to a sixth aspect of the present invention, a part of blocks is selected as a representative block in the horizontal and vertical directions from a plurality of blocks in the screen at a fixed number of blocks, and among the blocks selected as the representative block, A block estimated to have low reliability of motion vector detection is excluded from the representative block, and a block estimated to have high reliability of motion vector detection is placed at a predetermined position around the excluded block. When it exists, the block is set as a representative block. At this stage, the motion vector is detected for the block that is the representative block, and the pan and zoom parameters are estimated by the least square error method using the obtained motion vector. From the estimated pan and zoom parameters, the motion vector of the block that is the representative block is expanded, and the difference between the expanded motion vector and the detected motion vector in the block that is the representative block is predetermined. Blocks that exceed the threshold are excluded from the representative block. At a predetermined position around the excluded block, when there is a block that is estimated to have high reliability of motion vector detection, the motion vector of the block that is estimated to have high reliability is detected, When the difference between the detected motion vector and the motion vector corresponding to the block estimated to have high reliability, which is developed by the previously estimated parameter, is less than or equal to a predetermined threshold, the reliability is A block estimated to be high is set as a representative block. Then, the final pan and zoom parameters are estimated by the least-squares error method using only the motion vector detected for the block that is the representative block at this stage.

In the invention according to claim 3, 4 or 6, it is necessary to estimate the reliability of detection of the motion vector of the block. The inventions according to claims 7, 8, 9 and 10 relate to the estimation of the reliability of the motion vector detection, and for the estimation of the reliability of the motion vector detection,
The magnitude of the spread of the gradient vector of the luminance signal in the block is used. The feature of the invention according to claim 8 is that, when the target moving image is an interlaced image, the vertical direction component of the gradient vector is obtained when the size of the spread of the distribution of the gradient vector of the luminance signal in the block is obtained. About 1
It is to calculate the slope every line. The feature of the invention according to claim 9 is that, as an evaluation value of the spread of the distribution of the gradient vector of the luminance signal in the block, the average direction of the gradient vectors of all the pixels in the block, and the direction orthogonal to this direction, To use the smallest rectangular area orthogonal to the axes of these directions that covers the gradient vector distribution. The feature of the invention according to claim 10 is that, as the evaluation value of the spread of the distribution of the gradient vector of the luminance signal in the block, the smallest rectangular shape orthogonal to the horizontal and vertical axes covering the distribution of the gradient vector is used. The smaller area of the area and the area of the smallest rectangle orthogonal to the axis obtained by rotating the horizontal and vertical axes covering the distribution of the gradient vector by 45 degrees is used.

[0027]

According to the invention described in each of claims 1 to 10,
Since the pan and zoom parameters can be estimated only by detecting the motion vectors of some selected blocks in the screen in advance, or by only detecting them after the block selection or during the block selection process, the motion vector detection The processing amount can be reduced.

According to the second aspect of the present invention, the processing amount for selecting the block used for parameter estimation can be reduced, and the detection of the motion vector other than the selected block is not required at all. Can be significantly reduced. For block selection, for example, N is displayed vertically on the screen.
It is possible to adopt a method in which the block at the center position of the divided area is selected by equally dividing the image into M pieces in the vertical direction. An example thereof is shown in FIG. In FIG. 14, the vertical and horizontal lines are dividing lines of the screen, and each one block is shown by a rectangular grid. The solid black grid means a block at the center position (hereinafter referred to as a basic selection position) of the divided area. According to the invention of claim 2, the block at the basic selection position is used for parameter estimation. Is selected as the representative block.

According to the third aspect of the invention, the reliability of the motion vector detection is determined before the motion vector of the block selected as the representative block is actually detected in the same manner as the second aspect of the invention. Estimate and detect the motion vector only for blocks excluding the blocks with low estimated reliability. In this way, since the motion vector of the estimated block with low reliability is not used for parameter estimation, the parameter detection accuracy is improved.

According to the invention described in each of claims 7 to 10, the extent of the spread of the gradient vector distribution of the luminance signal in the block is used for the estimation of the reliability of the motion vector detection. The spread of the gradient vector distribution of the luminance signal in the block is, for example, as shown in FIG. In FIG. 15, the brightness gradient of each pixel in the block is plotted on the xy plane, and each plotted point is connected to the origin by a straight line.

The magnitude of the spread of the gradient vector distribution of the luminance signal in such a block, the reliability of the motion vector detection (the degree of reliability of the motion vector obtained when the motion vector detection is actually performed), and There is a qualitative relationship as described below.

If the luminance change in the block is flat,
Although the reliability of the motion vector detection of the block is low, the spread of the gradient vector distribution becomes small. Even in the case of a block that includes only linear edge components, the reliability of motion vector detection is low because the component of the motion vector in this edge direction cannot be specified, but since the gradient vectors are aligned in the same direction, the spread of the distribution is Get smaller. In addition, even in the case of regular texturer, the reliability of motion vector detection is low,
This low reliability is not reflected in the size of the spread of the gradient vector distribution of the luminance signal in the block. This is the same when there is no corresponding portion on the reference pixel as when a portion hidden in the background comes out.

Because of this relationship, the reliability of motion vector detection can be estimated according to the invention described in each of claims 7 to 10.

In the case of an interlaced image, the moving part is 1
An image with a time difference of 1/60 seconds appears for each line in a comb shape. Therefore, when calculating the gradient using the vertically adjacent pixels when obtaining the vertical component of the gradient vector of the luminance signal, it is possible to obtain There is an inconvenience that the spread of the distribution of the obtained gradient vector is likely to be large, and as a result, the reliability of motion vector detection is estimated to be high. According to the invention described in claim 8, since only field images of the same parity are used, such inconvenience is eliminated.

According to the ninth and tenth aspects of the invention, the extent of the spread of the gradient vector distribution of the luminance signal in the block is roughly evaluated. That is, in the invention according to claim 9, for example, as shown in FIG. 16, with respect to the average direction (direction of axis 1001) of the gradient vectors of all pixels in the block and the direction (direction of axis 1002) orthogonal thereto, Each axis 10 covering the gradient vector distribution
The area of the smallest rectangle 1003 orthogonal to 01 and 1002 is used as the evaluation value of the spread of the distribution of the gradient vector of the luminance signal in the block. The rectangle 10
03 is symmetrical with respect to the origin. Also, since the average direction of the gradient vector (1001) and the direction obtained by rotating it by 180 degrees are essentially the same, for example, the left half of the gradient vector (x <0) in the xy plane is An average vector can be taken after rotating 180 degrees and aligning with the right half of the xy plane.

In the invention described in claim 10, for example, FIG.
As shown in, the smallest rectangle 10 that covers the gradient vector distribution and is orthogonal to the x and y axes (horizontal axis, vertical axis)
05 and the area of the smallest rectangle 1008 that covers the gradient vector distribution and is orthogonal to the axes 1006 and 1007 with the x and y axes rotated by 45 degrees, whichever is smaller (the area of rectangle 1005 in FIG. 17). , It is used as a table value of the spread of the distribution of the gradient vector of the luminance signal in the block. The rectangles 1005 and 1008 are symmetrical with respect to the origin. This evaluation method is a simplification of the evaluation method of the invention described in claim 9, and evaluates the edge components only in the horizontal and vertical directions and the 45-degree diagonal direction. According to the invention described in claim 4, the block estimated to have low reliability of motion vector detection is excluded from the blocks selected as the representative block in the same manner as in the invention described in claim 3, but its surroundings are excluded. When a block with high reliability of motion vector detection is found at a predetermined position of, it is used as a representative block instead of the deleted block. For example, as shown in FIG. 14, candidate selection position blocks (shown by a meshed grid) are set around the basic selection position block, and when the basic selection position block is excluded from the representative block, If there is a surrounding candidate selected position block that is estimated to have high reliability in motion vector detection, that block is used as the representative block instead of the deleted representative block. According to the invention described in claim 4, it is possible to prevent the number of blocks used for parameter estimation from becoming excessively small and maintain the accuracy of parameter estimation.

Depending on the nature of the image, the reliability of motion vector detection is high, and the block in which the correct motion vector is actually detected may be the part that is moving regardless of the global motion. . It is desirable not to use the motion vector of such a part having a motion different from the global motion of the background for parameter estimation. However, such an undesired block can be determined only after actually detecting the motion vector.

According to the fifth or sixth aspect of the invention, in order to exclude such undesired blocks from the representative block, the motion vector is detected by the conventional technique such as the block matching method for the block selected as the representative block. , The parameters of pan and zoom are estimated by the least squares error method using the motion vector. Then, this parameter is expanded to obtain a motion vector, and a block in which the difference between the motion vector and the actually detected motion vector exceeds a predetermined threshold value is determined as the above-mentioned undesirable block and excluded from the representative blocks. According to the invention of claim 6, an operation of finding an alternative block suitable for the representative block is performed for the excluded block by the same method. Thus, the final estimation of the parameter is performed by the least square error method using the motion vector detected for the block finally determined as the representative block.

As described above, by excluding the block of the motion portion unrelated to the global motion from the representative block, it is possible to estimate the parameter with higher accuracy. The motion vector detected for the block that has been once excluded as the representative block is wasted, and the detection process is wasted. However, for other blocks that have not been excluded, it is only necessary to detect the motion vector once, and the obtained motion vector is used for the final parameter estimation. There is no.

Further, according to the comparison between the invention according to claim 5 and the invention according to claim 6, since the former block does not replenish the alternative block of the block excluded from the representative block, it is more advantageous than the latter in terms of throughput. Is. However, since the motion vector of the block excluded from the representative block is not used, the accuracy of parameter estimation may be inferior to the latter.

[0041]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Some embodiments of the present invention applied to MPEG encoding will be described below in detail with reference to the accompanying drawings. First, an example of a processing system according to each embodiment will be described with reference to FIG. The MPEG encoding means may be the same as the conventional one, which is not shown in FIG.

In FIG. 1, reference frame image memory 1
Is a memory for storing the image data of the reference frame, and the encoding target frame image memory 2 is a memory for storing the image data of the encoding target frame. The selected block position address generation circuit 3 is a circuit that generates a selected block position address, and the reference area image read circuit 4 is a circuit that reads the image data of the reference area from the reference frame image memory 1 according to the selected block position address. The image reading circuit 5 is a circuit for reading the image data of the selected block from the encoding target frame image memory 2 in accordance with the selected block position address. The read reference area image data is temporarily stored in the reference area image memory 6, and the read image data of the selected block is temporarily stored in the selected block image memory 7.

The evaluator 8 is a circuit for obtaining an evaluation value of the extent of spread of the gradient vector of the luminance signal in the same block from the image data of the selected block in the memory 7. The threshold value judging unit 9 calculates the evaluation value and the threshold value memory (#
1) The threshold value stored in 10 is compared, and when the evaluation value ≦ the threshold value, the reject signal R1 is output to the selected block position address generation circuit 3.

The MV detection circuit 11 is a circuit for reading image data from the memories 6 and 7 and detecting a motion vector (MV) in block units by a block matching method or the like. The detection MV memory 12 is a memory for storing the MV obtained by the MV detection circuit 11. The global motion parameter estimation circuit 13 reads the motion parameters from the memory 12 and uses the least squares error method to pan (horizontal and vertical directions) parameters e and f and a zoom parameter m.
Is a circuit for estimating.

The MV expansion circuit 14 is a circuit for obtaining a motion vector by expanding the estimated parameter, and the motion vector obtained here is the expansion MV memory 15
Is temporarily stored in. The MV difference determination circuit 16 expands M
Motion vector read from V memory 15 and detected MV
The difference from the motion vector read from the memory 12 is compared with the threshold value stored in the threshold value memory (# 2) 17, and when the motion vector difference> the threshold value, the reject block address R2 for the detected MV memory 12 is output. This reject block address R2 can also be supplied to the selected block position address generation circuit 3 via the switch 18.

<Embodiment 1> FIG. 2 is a flow chart showing an outline of the processing flow of this embodiment. The processing content of each step will be described with reference to the block diagram of FIG.

Step 100 (basic selection of MV detection block): In the selected block position address generation circuit 3, the number of horizontal and vertical divisions of the screen is input, and the block of the basic selection position shown by the solid black grid in FIG. Generates the address of. Here, the block size is 16 × 16 pixels, the screen size is 704 × 480 pixels, and the screen area is divided into 6 equal parts in the horizontal direction and 4 equal parts in the vertical direction. However, the area division is performed excluding the one block line around the screen so that the division is performed for each integer number of blocks.

The selected block image reading circuit 5 sequentially reads the image data of the block at the basic selected position from the encoding target frame image memory 2 and stores it in the selected block image memory 7. At the same time, the image data of the read reference area (usually selected as ± N pixels around the block address) corresponding to the read basic selection position block is read from the reference frame image memory 1 by the reference area image reading circuit 6. It is output and stored in the reference area image memory 6.

As can be understood from the above description, if the block selection in this step corresponds to the description of the claims, the basic selection position defined by the constant number of blocks in the horizontal and vertical directions is selected. A certain block is selected as the representative block.

Step 110 (MV detection block primary selection): This step and the previous step are repeatedly executed for all basic selection positions (this repetition is shown in FIG. 2).
Not specified).

The evaluator 8 uses the image data of the basic selection position block stored in the memory 7 to obtain an evaluation value of the spread of the gradient vector of the luminance signal in the block. The threshold determination unit 9 performs comparison determination between the evaluation value and the threshold stored in the threshold memory (# 1) 10. When the evaluation value is larger than the threshold value, the reliability of the motion vector detection of the block at the basic selection position is estimated to be high. Therefore, the basic selection position is set as the MV detection block primary selection position, and the threshold value judgment unit 9 determines Is the reject signal R1
Is not output.

If the evaluation value is less than or equal to the threshold value, the reliability of the motion vector detection of this basic selection position block is estimated to be low, so this basic selection position is not regarded as the MV detection block primary selection position. The reject signal R1 is output from the threshold value determiner 9.

When the reject signal R1 is output, the selected block position address generation circuit 3 selects one of a plurality of candidate positions around the basic selected position according to the input candidate position pattern, The address of the block at the candidate position is generated, and the image data of the block at the candidate position is read from the encoding target frame image memory 2 by the selected block image reading circuit 3 and stored in the selected block image memory 7. At the same time, the image data of the reference area corresponding to this block is also read from the reference frame image memory 1 to the reference area image memory 6.

With respect to this block, the evaluator 8 obtains an evaluation value of the magnitude of the spread of the gradient vector of the luminance signal, and the threshold judgment device 9 makes a threshold judgment for this evaluation value. If the evaluation value> threshold value, the block at this candidate position is estimated to have high reliability in motion vector detection, so this candidate position is set as the MV detection block primary selection position. However, if the evaluation value ≦ threshold value, the reject signal R1 is output again from the threshold value determiner 9. The selected block position address generation circuit 3 selects a candidate position next to the previously selected candidate position according to the candidate position pattern and generates the address of the block at that position. The image data of this block is read to the selected block image memory 7, and the same table value determination is performed. The image data of the corresponding reference area is also read at the same time.

If the reject signal R1 is not output,
The candidate position is set as the MV detection block primary selection position, but when the reject signal R1 is output, the image data of the block at the next candidate position is read and the evaluation value is determined. When the process proceeds to the last candidate position and the reject signal R1 is output for that, the MV detection block primary selection position is not obtained for the basic selection position.

FIG. 3 is a flow chart showing the simplified contents of the MV detection block primary selection processing described so far.

It should be noted that the required number of candidate positions may be set around the basic selected position, but in the present embodiment, eight candidate positions shown by a hatched grid in FIG. 14 are set. If the number of candidate positions is increased, the number of cases in which the primary selected position of the MV detection block is not found instead of the basic selected position is reduced, which is advantageous in terms of parameter estimation accuracy. However, on the other hand, the time for the MV detection block primary selection processing may increase.

Here, the processing contents of this step are as follows, corresponding to the description of the claims. The reliability of the motion vector detection of the block selected in the previous step is estimated by the evaluation value of the spread of the distribution of the gradient vector of the luminance signal, and when it is low, the block is excluded from the representative block and its surroundings. A block in which the reliability of the motion vector is estimated to be high is searched for from the blocks at the predetermined positions, and if found, the block is replaced with the excluded block and set as the representative block.

Now, the evaluator 8 obtains an evaluation value of the spread of the distribution of the gradient vector of the luminance signal. An example of an algorithm for obtaining the evaluation value will be described below.

First, it is necessary to find the gradient of the luminance signal at each pixel in the block. The algorithm can be described in C language as shown in FIG. 4 for a non-interlaced image. Here, ME_BLK_SIZE is the size of the block, and since the block has 16 × 16 pixels in the present embodiment, ME_BLK_SIZE = 16 is defined in the preprocessor control statement in the first line of the C language source list in FIG. MBImg is an array of luminance values of each pixel in the block. i and j are horizontal and vertical relative positions of pixels in the block.
In the nested for statement of FIG. 4, i and j are ME_B
Gradient grad [i] [j] [HORZ] in the x (horizontal) direction and gradient grad [i] in the y (vertical) direction at each pixel by incrementing from 0 to 1 until reaching LK_SZ-1. ]
[j] [VERT] is required.

When the target image is an interlaced image, the difference in luminance value is taken every other line for the gradient in the y (vertical) direction, and it is obtained by halving the difference. Therefore, the algorithm in this case can be expressed as shown in FIG. 5 by describing it in C language. The end condition of the first for sentence is ME_BLK because the difference in luminance value is taken every other line.
_SZ-2 has been changed, and the arithmetic expression of grad [i] [j] [VERT] has also been changed.

By using the gradient of each pixel thus obtained, the table value of the spread of the distribution of the gradient vector of the luminance signal in the block (the estimated reliability of the detection of the motion vector of the block) is obtained. An example of an algorithm therefor is shown in FIGS. 6 and 7 as a simplified C language source list. * reliable is the target evaluation value. abs () is a function for finding an absolute value, and MIN () is a function for finding a maximum value.

In the algorithm shown in FIG. 6, a value of 1/4 of the area of the rectangle corresponding to the rectangle 1003 shown in FIG. 16 is obtained as the table value (corresponding to claim 9). In the algorithm shown in FIG. 7, the rectangles 1005 and 1 shown in FIG.
A value of 1/4 of the smaller area of the rectangle corresponding to 008 is obtained as the table value (corresponding to claim 10).

Step 120 (Global motion parameter primary estimation): The address of the block of the MV detection block primary selection position determined in the previous step (the block which has been the representative block up to this stage) is the selected block position address generation circuit. 3, the image data of the reference area corresponding to each of the blocks is sequentially read from the encoding target frame image memory 4 and the reference frame image memory 1 by the selected block image reading circuit 5 and the reference area image reading circuit 4, respectively. It is output and input to the MV detection circuit 11 via the selected block image memory 7 and the reference area image memory 6. The MV detection circuit 11 detects the motion vector of the block from the input block and the image data of the reference area by a block matching method or the like. The detected motion vector and block address are stored in the detected MV memory 12.

When the motion vectors have been obtained for all the blocks of the MV detection block primary selection positions, the global motion parameter estimation circuit 13 uses the motion vectors (representative MV) and the pan parameters e and f by the least square error method. And the zoom parameter m is estimated.

FIG. 8 is a simplified C language source list showing this estimation algorithm. In FIG. 8, mv is the array of representative MVs, and Y_WIDTH and Y_HEIGHT are the horizontal and vertical sizes of the screen in pixel units. ME_BLK_SZ is the size of one side of the block, and is defined as 16 in the control statement on the first line here. MB_X_SIZE and MB_Y_SIZE are the horizontal and vertical sizes of the screen in block units, and are defined by the control statements on the second and third lines. DEF_MV_SPLT_X is the number of horizontal divisions of the screen (the number of representative MVs in the horizontal direction), DEF_MV_SPLT_
Y is the number of vertical divisions of the screen (the number of MVs in the vertical direction), which are defined as 6 and 4 in the control statements on the 4th and 5th lines, respectively. (X, y) is the position of the central pixel of the block, (vx,
vy) is the vertical direction of the motion vector of the block center pixel,
This is the horizontal component. gmc_params [E], gmc_params [F]
Is the desired horizontal and vertical pan parameters ((3) above).
E, f) of the formula, and zoom_params (= gmc_params [D])
Is the target zoom parameter (m in equation (3)).

The outline of this process is as follows. After initializing various variables, in the nested for statement,
While incrementing the vertical direction number blno_y and horizontal direction number bloc_x of the block, the position (x, y) of the center pixel of each block is obtained, and the vertical direction component and horizontal direction component of the motion vector at that pixel position are read from the array mv. (Actually read from the memory 15)
The operation of substituting for x and incrementing and adding various variables is repeated. After this, the parameters are calculated.

Step 130 (MV expansion primary): In the MV expansion circuit 14, the parameters e, f, m obtained in the previous step are substituted into the above equation (3) to determine the primary selection position of each MV detection block. Expand the motion vector of a block.
The expanded motion vector is stored in the expanded MV memory 15.

Step 140 (second selection of MV detection block): In the MV difference determination circuit 16, the expanded MV memory 1 for the block of each MV detection block primary selection position
The difference between the parameter expanded motion vector in 5 and the detected motion vector in the detection MV memory 12 (for example,
The sum of the absolute values of the differences between the x and y components of the motion vector) is compared with the threshold value stored in the threshold value memory (# 2) 17.
If the motion vector difference <threshold value, it is determined that the block should be included in the block of the MV detection block secondary selection position, and the MV difference determination circuit 16 does nothing. That is,
The detected motion vector of the block is stored in the detected MV memory 12 as valid.

However, if the motion vector difference ≧ threshold value, it is determined that the block should be excluded from the block of the MV detection block secondary selection position. Therefore, the MV difference determination circuit 16
Sends the address of the block (reject block) to the detection MV memory 12, and deletes the detected motion vector of the block from the detection MV memory 12. At this time, it is possible to select a mode in which a substitute block of the reject block is searched or a mode in which it is not searched.

When the mode in which the alternative block is not searched is selected, the switch 18 in FIG. 1 is in the open state as shown in the figure, the reject block address is not given to the selected block position address generation circuit 3, and special processing is performed. Is not done.

On the other hand, when the mode for searching for an alternative block is selected, the switch 18 in FIG. 1 is closed and the reject block address is also sent to the selected block position address generation circuit 3 to perform the processing described below. Done.

The selected block position address generation circuit 3 generates a substitute block position address corresponding to the reject block. When the reject block is the block at the basic selection position, the alternative block position is sequentially selected from the first candidate position according to the candidate position pattern. If the reject block is a block at a certain candidate position, alternative blocks are sequentially selected from the next candidate position according to the candidate position pattern. For this alternative block, the reliability of motion vector detection is evaluated in the same manner as in step 110. When it is determined that the reliability is low, the reject signal R1 is output from the threshold value determiner 9. Therefore, the selected block position address generation circuit 3 generates the address of the next alternative block, and the same evaluation is performed for that block. When reading the alternative block, the corresponding reference area is also read at the same time.

In this way, when a substitute block estimated to have high reliability in motion vector detection is found, the motion vector of this substitute block is detected by the MV detection circuit 11, and the detected motion vector is detected by the MV memory 12. To store. Then, the MV difference determination circuit 16 determines the difference between the detected motion vector and the developed motion vector of the reject block in the developed MV memory 15. If the motion vector difference <threshold value, the substitute block is set as the MV detection block secondary selection position instead of the reject block.

However, when the motion vector difference ≧ threshold value again, the address of the substitute block is sent to the detection MV memory 12 as the reject block address and the corresponding detection motion vector is deleted, and the selected block position address is also deleted. It is also sent to the generation circuit 3, and the process of searching for a substitute block from the next candidate position is continued. If a valid alternative block is not found even after reaching the last candidate position, the block used for final parameter estimation will not be selected in the divided area corresponding to the reject block.

FIG. 9 is a flow chart showing a simplified content of the above-mentioned MV detection block secondary selection processing in the mode for searching for an alternative block.

Although there is a possibility that the mode for searching for the alternative block will increase the processing amount as compared with the mode for not searching, the final parameter estimation accuracy can be improved because it is difficult to generate a divided area without a representative block. .

Step 150 (Secondary estimation of global motion parameters): This is the final processing step of the present invention.
In the global motion parameter estimation circuit 13, the detection M
Using the MV detection block secondary selection position stored in the V memory 12 and the block detection motion vector at that position, the pan parameters e and f and the zoom parameter m of the global motion are estimated by the same processing as step 120. To do. This is the final parameter estimation result.

Step 160 (MV expansion (secondary)), step 170 (MPEG encoding): The MV expansion circuit 14 uses the parameters e, f, and m estimated in the previous step, and all of them are performed according to the above equation (3). The motion vector of the block is expanded and stored in the expanded MV memory 15 (step 1
60). Using this expansion motion vector, MPEG coding is performed by a known coding means (not shown) (step 170).

<Embodiment 2> FIG. 10 is a flow chart showing the outline of the processing flow of this embodiment. Since the present embodiment is a simplification of the processing of the first embodiment, the processing content will be described focusing on the differences from the first embodiment.

Steps 400, 410 and 420 are the same processing steps as steps 100, 110 and 120 of FIG. 2, respectively. Steps 430 and 440 are shown in FIG.
This is the same processing step as steps 160 and 170, but step 430 uses the pan and zoom parameters estimated in step 420 as the final estimation result.

In this embodiment, steps 130 and 14 in FIG.
Since the processing steps corresponding to 0 and 150 are omitted (the operation related to the MV difference determination circuit 16 is not performed), there is an advantage that the processing amount is reduced as compared with the first embodiment. However, as compared with the above-mentioned embodiment, the risk that the block of the motion part different from the global motion of the background is finally selected as the representative block for parameter estimation increases, which is disadvantageous in terms of parameter estimation accuracy. . Therefore, this embodiment does not require the accuracy of parameter estimation as in the first embodiment, but is suitable when a further reduction in the processing amount is desired.

<Embodiment 3> This embodiment is a further simplification of Embodiment 2, and FIG. 11 is a flow chart showing the outline of the processing flow of this embodiment.

In FIG. 11, step 500 is the same processing step as step 100 in FIG. Step 5
Reference numeral 10 is a processing step corresponding to step 110 in FIG. 2, in which the reliability of motion vector detection of the block at the basic selection position is evaluated using the evaluation value of the spread of the gradient vector of the luminance signal in the block. The block determined to have low reliability is excluded from the representative block for parameter estimation, but unlike step 110, the operation of searching for a block that is a substitute for the excluded block is not performed (the threshold decision unit 9 rejects the reject signal R1. Is output, the selected block position address generation circuit 3 does not generate the address of the block at the candidate position, and the excluded block is simply ignored). Step 52 of FIG.
0, 530, and 540 are steps 120, 160, and
These are the same processing steps as 170.

In the present embodiment, since the alternative block of the block excluded in step 510 is not searched, there is a possibility that the number of blocks used for parameter estimation is reduced and the parameter estimation accuracy is deteriorated as compared with the second embodiment. However, the throughput is reduced. Therefore, the present embodiment is suitable for an application that prioritizes reduction of the processing amount.

<Embodiment 4> This embodiment is a further simplification of Embodiment 3, and FIG. 12 is a flow chart showing the outline of the processing flow of this embodiment.

In FIG. 12, step 600 and step 610 are the same processing steps as step 100 and step 120 of FIG. 2, respectively. Step 620
And step 630 are the same processing steps as step 160 and step 170 of FIG. 2, respectively.

In this embodiment, since the block at the basic selection position is directly used as the representative block for parameter estimation, there is a high risk that the motion parameter of the block with low reliability of motion vector detection will be used for parameter estimation. , It may not be possible to expect such high parameter estimation accuracy, but the amount of processing is extremely small. Evaluator 8, threshold value determiner 9,
The MV difference determination circuit 16 and its peripheral hardware are unnecessary. Therefore, it can be said that the present embodiment is suitable for an application in which reduction of the processing amount, acceleration of the processing, and reduction of the hardware are given the highest priority.

Although some embodiments of the present invention have been described above, it will be apparent to those skilled in the art that modifications and combinations thereof are possible. Further, it is natural that the parameter estimation method of the present invention can be applied to purposes other than moving picture coding.

[0090]

As described in detail above, claim 1 is as follows.
The inventions described in the respective items (1) to (10) can achieve a reduction in the amount of processing for motion vector detection, which has been a bottleneck in the global motion parameter estimation processing of moving images, without significantly lowering the parameter estimation accuracy. The present invention greatly contributes to the realization of a simple moving image coding system that compensates only global motion, such as a small digital video camera in which the reduction of the processing amount is important.

According to the second aspect of the present invention, the processing amount for selecting the block used for parameter estimation is small, and it is not necessary to detect the motion vector other than the selected block. Therefore, the processing amount is greatly reduced. And the processing speed can be increased.

According to the third aspect of the invention, the reliability of the motion vector detection is estimated before the motion vector is detected, and the global motion parameter is estimated using only the blocks estimated to have high reliability. Since the processing is performed, it is possible to estimate the parameter with high accuracy while reducing the processing amount for detecting the motion vector.

According to the fourth aspect of the present invention, it is possible to prevent the number of blocks used for parameter estimation from becoming too small and maintain the accuracy of parameter estimation.

According to the fifth or sixth aspect of the present invention, it is possible to prevent the deterioration of the parameter estimation accuracy due to the use of the motion vector of the block of the portion having a motion different from the global motion of the background for parameter estimation. Further, according to the invention as set forth in claim 6, it is possible to avoid an excessive number of blocks used for parameter estimation by searching for a substitute block of a block of such a portion, so that it is possible to estimate the parameter with higher accuracy. Is possible.

According to the invention described in each of claims 7 to 10, the motion vector of the block is actually detected by the size of the spread of the distribution of the gradient vector of the luminance signal in the block or its evaluation value. Without doing so, it is possible to easily evaluate the reliability of motion vector detection and efficiently select a block suitable for use in parameter estimation.

[Brief description of drawings]

FIG. 1 is a block diagram showing an example of a processing system for carrying out the present invention.

FIG. 2 is a schematic flowchart showing the flow of processing of the first embodiment.

FIG. 3 is a flowchart showing a simplified processing content of step 110 in FIG.

FIG. 4 is a diagram showing a C language source list describing an example of an algorithm for defining a gradient of a luminance signal at each pixel in a block.

FIG. 5 is a diagram showing a C language source list describing an example of an algorithm for defining a gradient of a luminance signal at each pixel in a block in the case of an interless image.

FIG. 6 is a diagram showing a C language source list describing an example of an algorithm for obtaining an evaluation value of a spread magnitude of a gradient vector distribution of a luminance signal of a block.

FIG. 7 is a diagram showing a C language source list describing another example of the algorithm for obtaining the evaluation value of the spread of the distribution of the gradient vector of the luminance signal of the block.

8 is a diagram showing a C language source list describing an example of a global motion parameter estimation algorithm in step 120 in FIG.

FIG. 9 is a flowchart showing a simplified processing content of step 140 in FIG.

FIG. 10 is a schematic flowchart showing the flow of processing of the second embodiment.

FIG. 11 is a schematic flowchart showing the flow of processing of the third embodiment.

FIG. 12 is a schematic flowchart showing the flow of processing of the fourth embodiment.

FIG. 13 is a diagram showing a C language source list describing an algorithm for estimating a global motion parameter by the least square error method.

FIG. 14 is a diagram showing an example of screen area division, a basic selection position, and a candidate selection position related to selection of a representative block for parameter estimation.

FIG. 15 is a diagram showing an example of the spread of the distribution of the gradient vector of the luminance signal in the block.

FIG. 16 is a diagram for explaining an example of a method of evaluating the extent of spread of the gradient vector distribution of the luminance signal in a block.

FIG. 17 is a diagram for explaining another example of the evaluation method of the extent of the spread of the gradient vector distribution of the luminance signal in the block.

[Explanation of symbols]

 1 Reference Frame Image Memory 2 Encoding Target Frame Image Memory 3 Selected Block Position Address Generation Circuit 4 Reference Area Image Reading Circuit 5 Selected Block Image Reading Circuit 6 Reference Area Image Memory 7 Selected Block Image Memory 8 Evaluator 9 Threshold Judgment Device 10 Threshold Memory (# 1) 11 MV detection circuit 12 Detection MV memory 13 Global motion parameter estimation circuit 14 MV expansion circuit 15 Expansion MV memory 16 MV difference determination circuit 17 Threshold memory (# 2)

Claims (10)

[Claims] 1. A global motion parameter detection method in which a screen of each frame of a moving image is divided into a plurality of blocks, and parameters representing pan and zoom in global motion are estimated by a least square error method from motion vectors in block units. , A global motion parameter detection method characterized in that only motion vectors of some blocks selected from a plurality of blocks in a screen by a certain rule are used for estimation of pan and zoom parameters. 2. A global motion parameter detection method for dividing a screen of each frame of a moving image into a plurality of blocks, and estimating parameters representing pan and zoom in global motion from a motion vector in block units by a least square error method. , Select some blocks as a representative block every certain number of blocks in the horizontal and vertical directions from multiple blocks in the screen, and only for the block selected as the representative block,
A global motion parameter detection method characterized by detecting the motion vector and using the pan and zoom parameters for estimation. 3. A global motion parameter detection method for dividing a screen of each frame of a moving image into a plurality of blocks and estimating parameters representing pan and zoom in global motion from a motion vector of each block by a least square error method. , Select some blocks as a representative block in the horizontal and vertical directions from a plurality of blocks in the screen at a constant number of blocks, and if the reliability of motion vector detection is low among the blocks selected as the representative block, Global motion parameter detection characterized by excluding the estimated block from the representative block, detecting the motion vector only for the representative block that remains unexcluded, and using it for estimating the pan and zoom parameters Method. 4. A global motion parameter detection method for dividing a screen of each frame of a moving image into a plurality of blocks, and estimating parameters representing pan and zoom in global motion from a motion vector of each block by a least square error method. , Select some blocks in the horizontal and vertical direction from a plurality of blocks in the screen at regular intervals as a representative block, and if the reliability of motion vector detection is low among the blocks selected as the representative block, Exclude the estimated block from the representative block, and if there is a block estimated to have high reliability of motion vector detection at a predetermined position around the block excluded from the representative block, remove that block. As a representative block, only for the block that was finally made a representative block,
A method for detecting a global motion parameter, characterized by detecting the motion vector and using it for estimating pan and zoom parameters. 5. A global motion parameter detection method, wherein a screen of each frame of a moving image is divided into a plurality of blocks, and parameters representing pan and zoom in global motion are estimated from a motion vector of each block by a least square error method. , Select some blocks from the multiple blocks on the screen as a representative block according to a certain rule, detect the motion vector for the block selected as the representative block, and use the obtained motion vector to calculate the least squares error. Estimate the pan and zoom parameters with, expand the motion vector of the block selected as the representative block from the estimated pan and zoom parameters, and expand the motion vector in the block selected as the representative block. The difference between the detected motion vector and Block is excluded from the representative block, and the final pan and zoom parameters are estimated by the least-squares error method using only the motion vector detected for the block as the representative block that remains without being excluded. Method for detecting global motion parameters. 6. A global motion parameter detection method, wherein a screen of each frame of a moving image is divided into a plurality of blocks, and a parameter representing pan and zoom in global motion is estimated from a motion vector of each block by a least square error method. , (A) select a certain block as a representative block every certain number of blocks in the horizontal and vertical directions from a plurality of blocks in the screen, and (b) among the blocks selected as the representative block,
A block estimated to have low reliability of motion vector detection is excluded from the representative block, and a block estimated to have high reliability of motion vector detection is placed at a predetermined position around the excluded block. If it exists, the block is set as a representative block, and (c) the motion vector is detected for the block that is set as the representative block at this stage, and the obtained motion vector is used to perform pan and zoom parameters by the least squares error method. And (e) expand the motion vector of the block that is the representative block from the estimated pan and zoom parameters, and detect the expanded motion vector and the detected motion in the block that is the representative block. Blocks that differ from the vector by more than a predetermined threshold are excluded from the representative block, and the surrounding blocks of the excluded blocks are excluded. When there is a block that is estimated to have a high reliability of motion vector detection at the determined position, the motion vector of the block that is estimated to have a high reliability is detected. When the difference between the motion vector developed in (c) corresponding to the block estimated to be high is less than or equal to a predetermined threshold value, the block estimated to have high reliability is set as a representative block, and (f) A global motion parameter detection method characterized in that the final pan and zoom parameters are estimated by the least-squares error method using only the motion vector detected for the block that is the representative block at this stage. 7. The global motion parameter detecting method according to claim 3, 4 or 6, wherein the magnitude of the spread of the gradient vector distribution of the luminance signal in the block is estimated in order to estimate the reliability of the motion vector detection of the block. A method for detecting global motion parameters, characterized in that 8. The global motion parameter detection method according to claim 7, wherein when the target moving image is an interlaced image, the extent of the spread of the gradient vector distribution of the luminance signal in the block is calculated. A global motion parameter detection method characterized in that the gradient is calculated every other line in the vertical direction component of the gradient vector. 9. The global motion parameter detection method according to claim 7, wherein an average value of the gradient vectors of all pixels in the block is used as an evaluation value of the spread of the distribution of the gradient vector of the luminance signal in the block, A method for detecting a global motion parameter, characterized by using the smallest rectangular area orthogonal to the axis of these directions that covers the distribution of gradient vectors in the direction orthogonal to this. 10. The global motion parameter detection method according to claim 7, wherein horizontal and vertical axes that cover the gradient vector distribution are used as evaluation values of the extent of spread of the gradient vector distribution of the luminance signal in the block. It is characterized by using the smallest rectangular area that is orthogonal to the minimum rectangular area and the minimum rectangular area that is orthogonal to the axis that rotates the horizontal and vertical axes by 45 degrees that covers the gradient vector distribution. Global motion parameter detection method.