JP2013030838A - Motion vector derivation device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To derive a motion vector that minimizes deterioration of image quality accurately from among a plurality of motion vectors, even when there are a plurality of candidates of motion vector in a certain block.SOLUTION: A motion vector derivation unit 14 derives a motion vector between corresponding blocks in two image frames continuous temporally. A candidate vector generation unit 20 generates a plurality of candidate vectors which are the candidates of motion vector of an object block for which the motion vector is calculated. A reference vector generation unit 22 generates a reference vector becoming a reference for determining the motion vector of an object block, by using a motion vector determined in a plurality of blocks located around the object block. An inner product calculation unit 24 calculates the inner product of each of the plurality of candidate vectors and the reference vector. A motion vector selection unit 26 selects a vector candidate having a maximum inner product calculated by the inner product calculation unit 24 as the motion vector of the object block.

Description

  The present invention relates to a technique for obtaining a motion vector between consecutive frames in a moving image.

  When a thin display device using a liquid crystal panel has a large screen, blurring or blurring due to an afterimage becomes noticeable. In order to cope with this, it is effective to increase the frame rate of a moving image reproduced by a thin display device. In a general frame rate conversion technique, a motion vector is derived in blocks of a predetermined size between a frame at a certain point in time and a frame immediately before or after in time, and the derived motion vector is used. An intermediate frame between both frames is created by motion compensation. In order to realize a smooth moving image after frame rate conversion, it is important to accurately obtain a motion vector.

  In general, motion vectors are derived by performing block matching between blocks in both frames, searching for a pair of blocks that minimize the absolute value error or square error of the luminance value, and a vector connecting these pairs. Is adopted as a motion vector. However, in practice, the absolute value error or the square error of the luminance values in a plurality of block pairs often has the same magnitude. In such a case, if a motion vector is determined simply because the error is minimal, an inappropriate motion vector may be selected from the viewpoint of continuity between frames.

  There is also known a technique for selecting a more accurate motion vector by using the correlation of motion vectors between adjacent blocks, instead of determining a motion vector based only on block matching. For example, Patent Document 1 discloses a motion vector of a reference block based on whether or not the position of a reference block of a current frame of an input video signal is a boundary region between things adjacent to a video region with a small amount of motion. A motion vector detection device for median filtering is disclosed.

JP 2010-166153 A

  In the technique described in Patent Document 1, since the medium size among the plurality of motion vectors of the reference block is selected and the other is not considered, there is a possibility that the correct motion vector is rejected.

  The present invention has been made in view of such a situation, and the object of the present invention is to determine whether a plurality of motion vector candidates exist for a block as a result of block matching between two temporally continuous frames. It is an object of the present invention to provide a technique for easily and accurately deriving a motion vector that minimizes image quality degradation from a plurality of motion vectors.

  One aspect of the present invention is a motion vector detection apparatus that derives a motion vector between corresponding blocks in two temporally continuous image frames. The apparatus includes a candidate vector generation unit that generates a plurality of candidate vectors that are motion vector candidates of a target block that is a target of motion vector calculation, and motion vectors that have been determined in a plurality of blocks that are located around the target block. A reference vector generation unit that generates a reference vector serving as a reference for determining a motion vector of the target block; an inner product calculation unit that calculates an inner product of each of the plurality of candidate vectors and the reference vector; A motion vector selection unit that selects a candidate vector having the maximum inner product calculated by the inner product calculation unit as a motion vector of the target block.

  According to this aspect, even when a plurality of motion vector candidates exist for a certain target block as a result of block matching between two temporally continuous frames, image quality degradation is minimized among the plurality of motion vectors. It is possible to easily and accurately derive the motion vector to be performed.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, as a result of block matching between two temporally continuous frames, even when a plurality of motion vector candidates exist for a certain block, image quality degradation is minimized among the plurality of motion vectors. The motion vector to be performed can be derived easily and accurately.

It is a block diagram which shows schematic structure of the frame rate conversion apparatus which concerns on one Embodiment of this invention. It is a figure explaining the method to produce an interpolated frame. It is a block diagram which shows the detailed structure of a motion vector derivation | leading-out part. It is a figure explaining the production | generation method of a reference | standard vector. It is a figure explaining selection of a candidate vector. It is a flowchart of motion vector derivation in this embodiment.

  FIG. 1 is a block diagram showing a schematic configuration of a frame rate conversion apparatus 10 according to an embodiment of the present invention. This configuration can be realized in terms of hardware by a CPU, memory, or other LSI of an arbitrary computer, and in terms of software, it is realized by a program loaded in the memory. Draw functional blocks. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The frame rate conversion device 10 is a device that converts the frame rate of an input moving image and outputs it. In the present embodiment, double-speed conversion in which the frame rate of an input moving image is doubled and output will be described.

  The frame selection unit 12 extracts two temporally continuous frames from the moving image. Of these two frames, the temporally subsequent one is called the “current frame”, and the temporally previous one is called the “delayed frame”. The frame selection unit 12 continuously extracts two frames from the moving image while shifting the frames one frame at a predetermined timing, and sequentially outputs them to the motion vector deriving unit 14.

  The motion vector deriving unit 14 divides the input current frame and delay frame into blocks each having a predetermined size (for example, a 16 × 16 pixel macroblock). Subsequently, the motion vector deriving unit 14 performs well-known block matching between the current frame and the delayed frame, and derives one motion vector for each block. The derived motion vector of each block is supplied to the interpolated frame creation unit 16.

  The interpolated frame creating unit 16 performs known motion compensation using the motion vector supplied from the motion vector deriving unit 14 and creates an interpolated frame intermediate between the current frame and the delayed frame.

  FIG. 2 is a diagram for explaining a method of creating an interpolated frame. In the figure, F1 is the current frame, F2 is a delay frame, and it is considered to create an interpolated frame 3 at an intermediate point between the two frames F1 and F2. The moving image output from the frame rate conversion apparatus 10 is reproduced in the order of frames F2, F3, and F1.

  The motion vector deriving unit 14 divides the interpolated frame F3 into blocks of a predetermined size. Consider a straight line that passes through the coordinates (x, y) of the upper left corner of a block (block B3) in the interpolated frame, and this straight line passes through the upper left corner on the current frame F1 and the delayed frame F2, respectively. Perform block matching between B2. This block matching is performed over the entire current frame F1 and delayed frame F2, and the sum of absolute values of luminance differences is calculated for each block in order to evaluate the similarity between the blocks. In the example of FIG. 2, the block B1 whose upper left corner coordinates are (x + i, y + j) on the current frame F1, and the block B2 whose upper left corner coordinates are (x-i, y-j) on the delayed frame F2. When the block matching is executed in the above, the motion vector connecting the two is represented by (i, j).

  When it is determined that the motion vector (i, j) is optimal, the interpolated frame creation unit 16 changes the texture of the block B2 in the delay frame F2 (or the block B1 in the current frame F1) to the block B3 in the interpolated frame F3. Put in. By repeating such processing for all blocks in the interpolated frame F3, the interpolated frame F3 can be created.

  Since block matching and creation of an interpolated frame are well known to those skilled in the art, further detailed description is omitted.

  FIG. 3 is a block diagram showing a detailed configuration of the motion vector deriving unit 14. The motion vector derivation unit 14 includes a candidate vector generation unit 20, a reference vector generation unit 22, an inner product calculation unit 24, and a motion vector selection unit 26.

  The candidate vector generation unit 20 receives the current frame and the delayed frame, and executes the block matching described with reference to FIG. 2 for each block in the interpolated frame. Then, the sum of absolute values of luminance differences (or the sum of squares of luminance differences) is calculated for each combination of blocks of the current frame and delayed frames for which matching has been performed, and a predetermined number (for example, 3 The combination of blocks) is selected and the respective motion vectors are obtained. These predetermined number of motion vectors are called “candidate vectors”. The candidate vector is supplied to the inner product calculation unit 24.

  The candidate vector generation unit 20 may select a plurality of candidate vectors only when there is no significant difference (for example, about 50) between the sums of absolute values calculated for each combination of blocks. When the minimum value is extremely small, as in the case where the minimum value of the sum of brightness differences in a combination of blocks is 1/100 of the second smallest value, the sum of brightness differences is not generated without generating a candidate vector. It is sufficient to obtain the motion vector directly from the combination of blocks with the smallest. Below, the case where a some candidate vector is produced | generated is demonstrated.

  The reference vector generation unit 22 uses a predetermined number of motion vectors already derived in another block existing around a block (referred to as a “target block”) that is a motion vector calculation target in the interpolated frame. Get only (for example, four).

  The reference vector generation unit 22 calculates a reference vector having the average of the obtained motion vectors, that is, the average value of the horizontal and vertical components of each motion vector as its own horizontal and vertical components. This reference vector is based on the premise that the motion vector of the target block and the motion vectors of the blocks located around the target block should be close in direction. Instead of simply taking the average, a weighted average corresponding to the distance from the target block may be calculated. The reference vector is supplied to the inner product calculation unit 24.

  The inner product calculation unit 24 calculates the inner product of each of the plurality of candidate vectors supplied from the candidate vector generation unit 20 and the reference vector supplied from the reference vector generation unit 22. The closer the candidate vector is in the direction to the reference vector, the larger the inner product.

  The motion vector selection unit 26 selects the candidate vector having the maximum inner product calculated by the inner product calculation unit 24 as the motion vector of the target block. The selected motion vector is supplied to the interpolated frame creation unit 16.

  Hereinafter, the operation of the motion vector deriving unit 14 will be described with reference to FIGS. 4 and 5.

  FIG. 4 is a diagram for explaining a reference vector generation method. It is assumed that the hatched block in the figure is a target block that is a motion vector calculation target. Further, the arrow from the left to the right in the drawing represents the scan direction when the motion vector is calculated by the raster scan method from the upper left to the lower right of the interpolated frame.

  As described above, the reference vector generation unit 22 acquires a predetermined number (for example, four) of motion vectors already derived in the blocks existing around the target block. The surrounding blocks are, for example, four blocks (indicated by bold lines in FIG. 4) located at the upper left, upper, upper right, and left of the target block. The motion vectors V1 to V4 in these blocks are indicated by arrows in FIG. A reference vector Vn that is an average value of these four vectors V1 to V4 is shown at the left end of FIG.

  The reference vector generation unit 22 may further use motion vectors in a wider range of blocks (for example, a range indicated by a dotted frame in FIG. 4). In this case, instead of calculating a simple average of motion vectors in all blocks, a weighted average corresponding to the distance from the target block may be calculated. For example, a weight vector of 1 may be given to the motion vector in the range of the thick line frame and an average may be taken after giving a weight of 0.5 to the motion vector in the range of the dotted line frame.

  FIG. 5 is a diagram for explaining selection of candidate vectors. It is assumed that the candidate vector generation unit 20 generates three candidate vectors indicated by Va, Vb, and Vc. The inner product calculation unit 24 calculates an inner product of the reference vector Vn and each of the candidate vectors Va, Vb, and Vc. The inner product has the maximum value in the same upper right candidate vector Va as the reference vector, which is selected as the motion vector. In this way, with a very simple calculation, a motion vector similar to the motion vector in the surrounding blocks (that is, a motion vector that minimizes image quality degradation from the viewpoint of image correlation) can be selected for the target block. .

  FIG. 6 is a flowchart of motion vector derivation in this embodiment. First, the candidate vector generation unit 20 performs block matching between blocks in the current frame and the delay frame to generate a plurality of candidate vectors (S10). The reference vector generation unit 22 generates a reference vector using the motion vectors determined in a plurality of blocks located around the block for which the motion vector is to be calculated (S12). The inner product calculation unit 24 calculates the inner product of each of the plurality of candidate vectors and the reference vector (S14). The motion vector selection unit 26 selects the candidate vector having the maximum inner product as the motion vector of the target block (S16).

  As described above, according to the present embodiment, when a plurality of motion vector candidates exist for a certain target block as a result of block matching between two temporally continuous frames, It becomes possible to easily select a motion vector similar to the motion vector obtained in the blocks around the target block. This increases the possibility that an accurate motion vector can be selected. Therefore, the possibility that an apparently inappropriate motion vector is selected is reduced, and as a result, a smooth moving image can be obtained after frame rate conversion.

  The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  Instead of calculating the average of a plurality of vectors as a reference vector, the reference vector generation unit 22 may select a medium (median) having a medium size among the plurality of vectors as a reference vector. Compared to the case where the average is used as a reference vector, a median can eliminate a protruding vector.

  In the embodiment, the case where the interpolated frame at the intermediate point between the current frame and the delay frame is generated has been described. However, the current frame and the delay frame are divided into two equal parts and three equal parts (or more) by the same method. It will be apparent to those skilled in the art that a plurality of interpolated frames can be created.

  In the embodiment, it has been described that the present invention is applied to the derivation of a motion vector when an interpolated frame is created at a change in the frame rate. However, the method of the present invention is also used to derive a motion vector at the time of video encoding. Can be applied. By using the motion vector derived by using the method of the present invention for moving picture coding, the amount of code may be increased as compared with a normal method, but the image quality can be improved.

  10 frame rate conversion device, 12 frame selection unit, 14 motion vector derivation unit, 16 interpolated frame creation unit, 20 candidate vector generation unit, 22 reference vector generation unit, 24 inner product calculation unit, 26 motion vector selection unit

JP 2005-033788 A

Claims (4)

An apparatus for deriving a motion vector between corresponding blocks in two temporally consecutive image frames,
A candidate vector generation unit that generates a plurality of candidate vectors that are motion vector candidates of a target block that is a motion vector calculation target;
A reference vector generation unit that generates a reference vector serving as a reference for determining a motion vector of the target block using motion vectors determined in a plurality of blocks located around the target block;
An inner product calculation unit for calculating an inner product of each of the plurality of candidate vectors and the reference vector;
A motion vector selection unit that selects a candidate vector having a maximum inner product calculated by the inner product calculation unit as a motion vector of the target block;
A motion vector deriving device comprising:   The motion vector derivation device according to claim 1, wherein the reference vector generation unit generates an average of motion vectors determined in the plurality of blocks as the reference vector.   The motion vector deriving device according to claim 1, wherein the reference vector generation unit uses, as the reference vector, a motion vector having a medium size among motion vectors determined in the plurality of blocks. A method of deriving a motion vector between corresponding blocks in two temporally consecutive image frames,
Generating a plurality of candidate vectors that are motion vector candidates of a target block that is a target of motion vector calculation;
A reference vector serving as a reference for determining a motion vector of the target block is generated using motion vectors determined in a plurality of blocks located around the target block.
Calculating an inner product of each of the plurality of candidate vectors and the reference vector;
A motion vector derivation method comprising: selecting a candidate vector having a maximum calculated inner product as a motion vector of the target block.

Images