JP2014007658A - Motion vector derivation device, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To derive a motion vector for minimizing image quality deterioration, with high precision, from among a plurality of motion vectors, even if a plurality of motion vector candidates exist for some block.SOLUTION: A motion vector derivation unit 14 derives a motion vector between corresponding blocks in two image frames. A candidate vector generation unit 20 generates a plurality of candidate vectors for a calculation object block of motion vector. A block texture extraction unit 22 extracts a block texture in an image frame where a candidate vector is generated. A motion vector selection unit 26 selects, as the motion vector of an object block, a candidate vector generated from a block texture having a highest degree of matching of local spatial structure of intensity change, between corresponding blocks out of a plurality of block textures thus extracted.

Description

  The present invention relates to a technique for obtaining a motion vector between consecutive frames in a moving image.

  When a thin display device using a liquid crystal panel has a large screen, blurring or blurring due to an afterimage becomes noticeable. In order to cope with this, it is effective to increase the frame rate of a moving image reproduced by a thin display device. In a general frame rate conversion technique, a motion vector is derived in units of blocks of a predetermined size between a frame at a certain point in time and a frame immediately before or after in time. Then, an intermediate frame between both frames is created by motion compensation using the derived motion vector. In order to realize a smooth moving image after frame rate conversion, it is important to accurately obtain a motion vector.

  In general, motion vectors are derived by performing block matching between blocks in both frames, searching for a pair of blocks that minimize the absolute value error or square error of the luminance value, and a vector connecting these pairs. Is adopted as a motion vector. However, in practice, the absolute value error or the square error of the luminance values in a plurality of block pairs often has the same magnitude. In such a case, if a motion vector is determined simply because the error is minimal, an inappropriate motion vector may be selected from the viewpoint of continuity between frames.

  There is also known a technique for selecting a more accurate motion vector by using the correlation of motion vectors between adjacent blocks, instead of determining a motion vector based only on block matching. For example, motion vector detection that performs median filtering on the motion vector of the reference block based on whether or not the position of the reference block in the current frame of the input video signal is a boundary region between a video region with a small amount of motion and an adjacent thing An apparatus has been proposed (see, for example, Patent Document 1).

JP-A-2005-033788

  In the above-described motion vector detection device that performs median filtering, a medium motion having a medium size is selected between a plurality of motion vectors of a reference block, and the other motion vectors are not taken into account, so that a correct motion vector may be rejected. .

  The present invention has been made in view of such a situation, and the object thereof is as a result of block matching between two temporally continuous frames, even when a plurality of motion vector candidates exist for a certain target block. It is an object of the present invention to provide a technique for accurately deriving a motion vector that minimizes image quality degradation from the plurality of motion vectors.

  A motion vector deriving device according to an aspect of the present invention is a device for deriving a motion vector between corresponding blocks in two temporally continuous image frames, and a motion vector of a target block which is a motion vector calculation target. A candidate vector generating unit that generates a plurality of candidate vectors that are candidates, a block texture extracting unit that extracts a block in an image frame in which the candidate vector is generated as a block texture, and a block texture extracting unit Motion vector selection for selecting a candidate vector generated from a block texture having the highest matching degree of local spatial structure of luminance change among corresponding blocks among a plurality of block textures as a motion vector of the target block A section.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, even when a plurality of motion vector candidates exist for a certain target block as a result of block matching between two temporally continuous frames, image quality degradation is minimized among the plurality of motion vectors. Can be derived with high accuracy.

It is a block diagram which shows schematic structure of the frame rate conversion apparatus which concerns on one Embodiment of this invention. It is a figure explaining the method to produce an interpolated frame. It is a figure which shows the case where an object cross | intersects in an interpolated frame. It is a block diagram which shows the detailed structure of a motion vector derivation | leading-out part. It is a figure which shows a block texture. It is a flowchart of motion vector derivation in this embodiment. It is a figure which shows the function for expressing the local structure of a luminance value change. It is a figure which shows the pixel used for the correlation determination in case a block size is 8x8.

  FIG. 1 is a block diagram showing a schematic configuration of a frame rate conversion apparatus 10 according to an embodiment of the present invention. This configuration can be realized in terms of hardware by a CPU, memory, or other LSI of an arbitrary computer, and in terms of software, it is realized by a program loaded in the memory. Draw functional blocks. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The frame rate conversion device 10 is a device that converts the frame rate of an input moving image and outputs it. In the present embodiment, double-speed conversion in which the frame rate of an input moving image is doubled and output will be described.

  The frame selection unit 12 extracts two temporally continuous frames from the moving image. Of these two frames, the temporally subsequent one is called the “current frame”, and the temporally previous one is called the “delayed frame”. The frame selection unit 12 continuously extracts two frames from the moving image while shifting the frames one frame at a predetermined timing, and sequentially outputs them to the motion vector deriving unit 14.

  The motion vector deriving unit 14 divides the input current frame and delay frame into blocks each having a predetermined size (for example, a 16 × 16 pixel macroblock). Subsequently, the motion vector deriving unit 14 performs block matching between the current frame and the delayed frame, and derives one motion vector for each block. The derived motion vector of each block is supplied to the interpolated frame creation unit 16.

  The interpolated frame creating unit 16 performs known motion compensation using the motion vector supplied from the motion vector deriving unit 14 and creates an interpolated frame intermediate between the current frame and the delayed frame.

  FIG. 2 is a diagram for explaining a method of creating an interpolated frame. In the figure, F1 is the current frame, F2 is a delay frame, and it is assumed that an interpolated frame F3 at an intermediate point between the two frames F1 and F2 is created. The moving image output from the frame rate conversion apparatus 10 is played back in the order of frames F1, F3, and F2.

  The motion vector deriving unit 14 divides the interpolated frame F3 into blocks of a predetermined size. Consider a straight line that passes through the coordinates (x, y) of the upper left corner of a block (block B3) in the interpolated frame, and this straight line passes through the upper left corner on the current frame F1 and the delayed frame F2, respectively. Perform block matching between B2. This block matching is performed over the entire current frame F1 and delayed frame F2, and the sum of absolute values of luminance differences is calculated for each block in order to evaluate the similarity between the blocks. In the example of FIG. 2, the block B1 whose upper left corner coordinates are (x + i, y + j) on the current frame F1, and the block B2 whose upper left corner coordinates are (x-i, y-j) on the delayed frame F2. When the block matching is executed in the above, the motion vector connecting the two is represented by (i, j).

  When it is determined that the motion vector (i, j) is optimal, the interpolated frame creation unit 16 changes the texture of the block B2 in the delay frame F2 (or the block B1 in the current frame F1) to the block B3 in the interpolated frame F3. Put in. By repeating such processing for all blocks in the interpolated frame F3, the interpolated frame F3 can be created.

  Since block matching and creation of an interpolated frame are well known to those skilled in the art, further detailed description is omitted.

  When there are a plurality of moving objects in the moving image, the objects may cross each other at the position where the interpolated frame F3 is created. As shown in FIG. 3, when the object C1 moves from the upper left to the lower right in the frame and the object C2 moves from the lower left to the upper right in the frame, the two objects intersect near the center of the interpolated frame F3. It will be. Even if the motion vectors of the blocks including the objects C1 and C2 are respectively obtained, it is not possible to obtain information about which object should be arranged on the near side in the interpolated frame F3.

  Therefore, in this embodiment, when it is considered that an object crossing has occurred in the interpolated frame, a motion vector can be selected so that an optimal object is arranged on the near side.

  FIG. 4 is a block diagram showing a detailed configuration of the motion vector deriving unit 14. The motion vector derivation unit 14 includes a candidate vector generation unit 20, a block texture extraction unit 22, a luminance difference determination unit 24 between adjacent pixels, a motion vector selection unit 26, a correlation determination unit 28, and a luminance difference calculation unit 30.

The candidate vector generation unit 20 receives the current frame and the delayed frame, and executes the block matching described with reference to FIG. 2 for each block in the interpolated frame. Then, the sum of absolute values of luminance differences (or the sum of squares of luminance differences) is calculated for each combination of blocks of the current frame and delayed frames for which matching has been performed. A combination of blocks in which the sum is sufficiently small is selected, and each motion vector is obtained. “Selecting a sufficiently small block combination” means, for example, selecting a block combination whose sum is equal to or less than a predetermined threshold value. Alternatively, sorting is performed in ascending order of the sum, and when (A _i −A _i−1 ) / (A _{i + 1} ) is a predetermined value (for example, 1/100) or less, all block combinations up to A _i are selected. Say. Here, A _i represents the i-th value from the top. The motion vector obtained for the selected combination of blocks is referred to as a “candidate vector”.

  In the present embodiment, it is assumed that there are two or more candidate vectors generated by the candidate vector generation unit 20. In other cases, when the minimum value is extremely small, for example, when the minimum value of the sum of luminance differences in a certain block combination is 1/100 of the second smallest value, the candidate vector is not generated. In addition, it is sufficient to directly obtain a motion vector from a combination of blocks having the smallest sum of luminance differences.

  The block texture extraction unit 22 extracts a plurality of blocks in the delay frame corresponding to the candidate vector generated by the candidate vector generation unit 20. Hereinafter, these are referred to as “extracted block texture”. FIG. 5 shows the extracted block texture. It is assumed that the texture has a size of p × p pixels.

The luminance difference calculation unit 30 calculates the accumulated value Z of luminance differences between adjacent pixels for each of the extracted block textures as shown in the following (Equation 1).
Here, Y (i, j) represents the luminance value of the pixel (i, j).

   On the other hand, the luminance difference determining unit 24 between adjacent pixels calculates the luminance difference with the surrounding pixels for each pixel of the extracted block texture, and roughly represents the local spatial structure of the luminance value change. The correlation determination unit 28 calculates the correlation between the magnitudes of the luminance for the two corresponding blocks.

  The motion vector selection unit 26 compares the luminance difference accumulated values Z of the plurality of extracted block textures calculated by the luminance difference calculation unit 30. If the maximum value of the luminance difference accumulated value Z is larger than the remaining one by a predetermined value (for example, twice or more), the motion vector selection unit 26 determines the candidate obtained for the extracted block texture having the largest luminance difference accumulated value Z. The vector is selected as the motion vector of the target block. When the maximum value of the luminance difference accumulated value Z is less than a predetermined value than the remaining one, the motion vector selection unit 26 obtains the extracted block texture having the largest or relatively large correlation calculated by the correlation determination unit 28. The obtained candidate vector is selected as the motion vector of the target block. That is, the candidate vector obtained for the extracted block texture having the largest or relatively large local spatial structure matching degree of luminance change between corresponding blocks of the current frame and the delayed frame is used as the motion vector of the target block. Choose as.

  The reason for the above is based on the following knowledge. That is, the motion vector of the extracted block texture having a large accumulated luminance difference value Z is selected because the object on the foreground side is higher in definition than the object on the far side in a front light environment, and includes more high frequency components. This is because the probability is considered high. Moreover, since the block texture containing many high frequency components has a large amount of information, there is a high possibility that highly accurate motion vector estimation is performed.

  Extraction block texture motion vector with high correlation of local structure of brightness value change is selected because the local structure of brightness value change is correct without being affected by subtle changes such as lighting conditions and shooting conditions. This is because there is a high possibility that In this regard, general block matching for calculating the sum of absolute values of luminance differences between blocks (or the sum of squares of luminance differences) is affected by subtle changes such as illumination conditions and photographing conditions. For example, if the lighting conditions are different between the current frame and the delayed frame, the sum of absolute values of luminance differences (or the sum of squares of luminance differences) can be large even if the local spatial structure is almost the same. There is sex. That is, the matching degree may be determined to be lower than actual. In this regard, if the matching degree is determined based on the correlation of the local structure of the luminance value change, it is less likely to be affected by the difference in illumination state and photographing conditions between frames.

  In this manner, the motion vector of the target block can be selected in consideration of the object context in the interpolated frame. In addition, it is possible to perform highly accurate correlation determination that is not easily affected by illumination conditions or imaging conditions.

  FIG. 6 is a flowchart of motion vector derivation in this embodiment. First, the candidate vector generation unit 20 performs block matching between blocks in the current frame and the delay frame to generate a plurality of candidate vectors (S10). The block texture extraction unit 22 extracts the textures of the blocks in the delay frame from among a plurality of combinations of blocks selected by the candidate vector generation unit 20 (S12). The luminance difference calculation unit 30 calculates a cumulative value Z of luminance differences between adjacent pixels for each of the extracted block textures (S14). The luminance difference determining unit 24 between adjacent pixels compares the luminance values of the neighboring pixels with each of the constituent pixels of the extracted block texture (S16). The correlation determination unit 28 determines the correlation of the above-described magnitude relationship for each of the extracted block textures (S18).

  The motion vector selection unit 26 compares the luminance difference accumulated values Z of the plurality of extracted block textures calculated by the luminance difference calculation unit 30 (S20). When the maximum value of the luminance difference accumulated value Z is larger than the remaining one by a predetermined value (for example, twice or more) (Y in S22), the motion vector selection unit 26 extracts the extracted block texture having the largest luminance difference accumulated value Z. The candidate vector obtained for is selected as the motion vector of the target block (S24). When the maximum value of the luminance difference accumulated value Z is less than a predetermined value than the remaining one (N in S22), the motion vector selection unit 26 obtains the extracted block texture having the maximum correlation calculated by the correlation determination unit 28. The obtained candidate vector is selected as the motion vector of the target block (S26).

  Next, processing in the adjacent pixel luminance difference determination unit 24 will be described in detail. In order to schematically represent the local spatial structure of the luminance change, a function str [m] as shown in FIG. 7 is introduced. This expresses the magnitude relationship for the surrounding 8 pixels of the luminance value Y [i, j], m = 0 for the upper left, m = 1 for the upper, m = 2 for the upper right, m = 3 for the left, m = The size relationship is expressed as 4 for right, m = 5 for lower left, m = 6 for lower left, and m = 7 for lower left. th is a threshold value for reducing the influence of noise or the like, and is set to, for example, about 5 to 10 in the case of 8-bit processing. Basically, 0 is output when the pixel value of the central pixel is smaller, 1 when the pixel value is equal, and 2 when it is larger.

Specific processing in the correlation determination unit 28 is realized by expanding the str [m], which is a local spatial structure expression for one pixel, into a block texture. For convenience of calculation with adjacent pixels, str [m] cannot be calculated at the outermost periphery of the block. As shown in FIG. 8, when the block size is 8 × 8, calculation is performed for 6 × 6 = 36 pixels with pixel numbers. When the pixel number is (n = 0, 1,... 35), the str [m] applied to the pixel n of any block A is described as strblkA [n] [m] ( m = 0,1, ... 7). Similarly, what is applied to an arbitrary block B is described as strblkB [n] [m]. The correlation between the local structures of both block textures can be described as (Equation 2) below.
However, f (0) = 1, f (1) = 0, and f (2) =-1. This means that, in the magnitude relationship of the pixel values of the corresponding adjacent pixels of the corresponding pixels of block A and block B, the matching combination counts up, and the contradictory combination such as increase and decrease Is counted down, and combinations such as increase or decrease and no change are not counted. Therefore, it shows that the correlation of the local structure of the block A and the block B is so high that the value of the function of said (Formula 2) is large.

  As described above, according to the present embodiment, when a plurality of motion vector candidates exist for a certain block in the interpolated frame as a result of block matching between two temporally continuous frames, An optimal motion vector can be selected from the motion vectors. That is, by determining the local spatial structure matching degree of the luminance change between the blocks, it is possible to realize highly accurate block matching that is not easily affected by the illumination state or the photographing condition. In addition, by selecting a motion vector of a block texture that contains a lot of high-frequency components, a motion vector corresponding to an object located on the near side can be selected even when an object crossing occurs at the location of the target block. Accordingly, the possibility that an erroneous motion vector is used in the interpolated frame is reduced, and an accurate moving image can be obtained even after frame rate conversion.

  The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to the combination of each component and each processing process, and such modifications are also within the scope of the present invention. .

  In the embodiment, the luminance difference determining unit 24 between adjacent pixels determines the magnitude relationship between the luminance value of the target pixel of the block texture and the luminance values of eight pixels adjacent to the periphery of the target pixel. In this regard, the pixels for determining the magnitude relationship are not limited to the eight pixels adjacent to the surroundings. For example, 16 pixels on the outer periphery of the 8 pixels may be added and compared with a total of 24 pixels. Although the accuracy increases as the number of pixels for determining the magnitude relationship increases, the amount of calculation increases. The designer can determine the pixel for determining the magnitude relationship in consideration of this trade-off relationship.

  In the embodiment, the case where the interpolated frame at the intermediate point between the current frame and the delay frame is generated has been described. However, the current frame and the delay frame are divided into two equal parts and three equal parts (or more) by the same method. It will be apparent to those skilled in the art that a plurality of interpolated frames can be created.

  In the embodiment, it has been described that the present invention is applied to the derivation of a motion vector when an interpolated frame is generated at a change in the frame rate. However, the method of the present invention is also applied to the derivation of a motion vector at the time of video encoding. can do. By using the motion vector derived by using the method of the present invention for moving picture coding, the amount of code may be increased as compared with a normal method, but the image quality can be improved.

  Furthermore, it is applicable not only to the above-described interpolation of time-series images but also to generation of an intermediate virtual viewpoint image among a plurality of viewpoint images.

  DESCRIPTION OF SYMBOLS 10 Frame rate conversion apparatus, 12 Frame selection part, 14 Motion vector derivation part, 16 Interpolated frame production part, 20 Candidate vector generation part, 22 Block texture extraction part, 24 Brightness difference determination part between adjacent pixels, 26 Motion vector selection part 28 correlation determination unit, 30 luminance difference calculation unit.

Claims (6)

An apparatus for deriving a motion vector between corresponding blocks in two temporally consecutive image frames,
A candidate vector generation unit that generates a plurality of candidate vectors that are motion vector candidates of a target block that is a motion vector calculation target;
A block texture extraction unit for extracting a block in the image frame in which the candidate vector is generated as a block texture;
Among the plurality of block textures extracted by the block texture extraction unit, candidate vectors generated from block textures having the highest matching degree of local spatial structure of luminance change between corresponding blocks are selected as the target block. A motion vector selection unit that selects the motion vector of
A motion vector deriving device comprising: For each of the plurality of block textures extracted by the block texture extraction unit, an inter-pixel luminance difference determination unit that determines the magnitude of the luminance of the pixels constituting the block texture and the luminance of surrounding pixels,
A correlation determining unit that evaluates the correlation between corresponding blocks based on the magnitude matching and mismatching of the luminance,
The motion vector deriving device according to claim 1, wherein the motion vector selection unit selects a candidate vector generated from a block texture having a maximum correlation value as a motion vector of the target block.   The motion vector derivation according to claim 2, wherein the inter-pixel luminance difference determination unit determines the magnitudes of the luminance of pixels constituting the block texture and the luminance of eight pixels adjacent to the block texture. apparatus. For each of the plurality of block textures extracted by the block texture extraction unit, further comprising a luminance difference calculation unit for calculating a cumulative value of luminance differences between adjacent pixels of the pixels constituting the block texture,
The motion vector selection unit selects a candidate vector generated from the block texture from which the maximum value of the cumulative value is obtained when the maximum value of the cumulative value is a predetermined multiple or more larger than the remaining cumulative value. A candidate vector generated from the block texture from which the maximum value of the correlation is obtained, when selected as a motion vector of the target block, and the maximum value of the cumulative value is less than a predetermined multiple of the remaining cumulative value 4. The motion vector deriving device according to claim 2, wherein the motion vector deriving device is selected. A method of deriving a motion vector between corresponding blocks in two temporally consecutive image frames,
Generating a plurality of candidate vectors that are motion vector candidates of a target block that is a target of motion vector calculation;
Extracting a block in the image frame in which the candidate vector is generated as a block texture;
A candidate vector generated from a block texture having the highest matching degree of the local spatial structure of luminance change between corresponding blocks among the plurality of extracted block textures is selected as a motion vector of the target block. ,
A motion vector derivation method characterized by the above. A program for deriving a motion vector between corresponding blocks in two temporally consecutive image frames,
A process of generating a plurality of candidate vectors that are motion vector candidates of a target block that is a motion vector calculation target;
A process of extracting a block in the image frame in which the candidate vector is generated as a block texture;
A candidate vector generated from a block texture having the highest matching degree of the local spatial structure of luminance change between corresponding blocks among the plurality of extracted block textures is selected as a motion vector of the target block. Processing,
A motion vector derivation program characterized by having a computer realize the above.