JP2011191972A - Device and method for measurement of motion vector - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately detect a motion vector, even if objects varying in moving speed are included in a frame. <P>SOLUTION: The motion vector in a target pixel position on a target frame Frt is measured based on the target frame Frt in a plurality of consecutive frames and a past frame Frt-i separated by a predetermined frame interval from the target frame. In this process, an interval parameter setting part 4 sets an interval parameter representing an interval between the target frame Frt and the past frame Frt-i in measurement of the motion vector for each area or each pixel on the target frame Frt. Specifically, the interval parameter is set so that the frame interval is narrower in the area/pixel corresponding to an object moving at a higher speed. A motion vector measurement part 5 measures the motion vector according to the interval parameter. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a motion vector measuring apparatus and method for measuring a motion vector representing the motion of an object in a moving image, and a program for causing a computer to execute the motion vector measuring method.

  As a means for measuring an image motion vector, a block matching method based on a correlation calculation is conventionally known. The block matching method divides a target frame to be subjected to motion vector measurement into a plurality of appropriately sized blocks (for example, 8 × 8 pixels), calculates a pixel difference from a past frame in this block unit, This is a method of searching for a pixel position in a past frame in which the sum of absolute values of the differences is minimized. In the block matching method, a block shift between frames indicates a motion vector of a pixel position at the center of the block in the target frame.

  In addition, when measuring a motion vector, there is a method of creating a low-resolution frame with a reduced number of pixels from the original resolution frame, and measuring the motion vector by the block matching method using the created low-resolution frame. It has been proposed (see Patent Document 1). According to the technique of Patent Document 1, since the amount of calculation for measuring a motion vector can be reduced by using a low-resolution frame, the motion vector can be measured at high speed.

  In addition, various methods have been proposed for improving the accuracy when measuring a motion vector or reducing the amount of calculation (see Patent Documents 2 to 4). The technique described in Patent Document 2 drops the frame between the target frame used for measuring the motion vector and the past frame by the set number of frames, and the block size according to the number of frames dropped. Is a method of measuring a motion vector at high speed without reducing accuracy. The method described in Patent Document 3 is a method for obtaining a motion vector for each area of a moving object when there are a plurality of moving objects in the frame. Further, the method described in Patent Document 4 is a method of expanding a motion vector search range because the amount of movement of an object increases as the distance between the target frame and the past frame increases.

Japanese Patent No. 3617671 JP-A-10-13836 JP-A-8-251474 JP-A-7-67116

  However, since the method described in Patent Document 1 uses a high-level frame for detecting a motion vector, it cannot detect a small motion of an object included in the frame. In addition, in order to detect a small motion, it is necessary to detect a motion vector again using a low-level (high resolution) frame, so that the amount of calculation cannot be reduced. Further, in the method described in Patent Document 2, when an object having a different moving speed is included in a frame, if the frame interval is large, a motion vector of an object having a low moving speed can be accurately detected, but the moving speed is high. Since the object may move out of the frame, the motion vector may not be detected. In addition, the method described in Patent Document 3 can obtain a motion vector for each area of a moving object, but cannot obtain a motion vector with high accuracy because processing according to the moving speed of the object is not performed. . Furthermore, since the method described in Patent Document 4 expands the search range of the motion vector according to the amount of movement of the object, although the motion vector can be detected with high accuracy, objects with different moving speeds are included in the frame. In case you can not cope.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to accurately detect a motion vector even when an object having a different moving speed is included in a frame.

The motion vector measuring apparatus according to the present invention calculates a motion vector at a target pixel position on the target frame based on the target frame in a plurality of frames included in the moving image and a past frame separated from the target frame by a predetermined frame interval. In the motion vector measuring device to measure,
Interval parameter setting means for setting an interval parameter representing a frame interval between the target frame and the past frame when measuring the motion vector for each region or pixel on the target frame;
And a motion vector measuring means for measuring the motion vector according to the interval parameter.

The motion vector measurement device according to the present invention further includes a hierarchizing means for hierarchizing the target frame and the past frame,
The interval parameter changing unit may be a unit that sets the interval parameter for each region or each pixel on the target frame having a layer other than the lowest layer, that is, a resolution other than the original resolution.

  The motion vector measurement device according to the present invention further includes a storage unit that stores the interval parameter, and the interval parameter setting unit includes the target frame and the past frame as the measured motion vector is smaller. It is also possible to update the stored interval parameter so that the interval between is increased.

  The motion vector measurement apparatus according to the present invention further includes a storage unit that stores the interval parameter, and the interval parameter setting unit includes a plurality of pixel positions at the target pixel position and a plurality of pixel positions in the vicinity of the target pixel position. The stored interval parameter may be updated based on the motion vector.

  “Updating the stored interval parameter based on the motion vector of the target pixel position and a plurality of pixel positions in the vicinity of the target pixel position” means not only the motion vector of the target pixel position but also the vicinity of the target pixel position. This means that the interval parameter is changed in consideration of the motion vector. For example, if there is no variation between the motion vector at the target pixel position and the motion vector near the target pixel position, the interval parameter obtained by measuring the motion vector is used to measure the motion vector for the next target frame, and the motion vector varies. If there is a frame, the frame interval represented by the interval parameter, or if the interval parameter also represents the number of frames in the past frame used to measure the motion vector, change the number of frames to measure the motion vector more accurately. You can do it.

  In the motion vector measuring device according to the present invention, the interval parameter setting means may be a means for setting the interval parameter representing the number of frames of the past frame used for measuring the motion vector together with the frame interval. .

  In this case, the motion vector measuring means is means for measuring the motion vector in order from the past frame at a frame interval close to the target frame or from the past frame at a frame interval away from the target frame. When the motion vector having a value other than 0 is measured, the motion vector measurement using the past frame after that may be stopped.

  In the motion vector measuring device according to the present invention, the motion vector measuring means may be configured to use the motion vector at the target pixel position based on the plurality of motion vectors at the target pixel position and a plurality of pixel positions near the target pixel position. It is good also as a means to determine.

The motion vector measurement method according to the present invention calculates a motion vector at a target pixel position on the target frame based on a target frame in a plurality of frames included in the moving image and a past frame separated from the target frame by a predetermined frame interval. In the motion vector measurement method to measure,
For each region or pixel on the target frame, set an interval parameter representing a frame interval between the target frame and the past frame when measuring the motion vector,
The motion vector is measured according to the interval parameter.

  The motion vector measuring method according to the present invention may be provided as a program for causing a computer to execute the method.

  Of the objects included in the moving image, it is difficult to detect the motion vector accurately when the frame interval between the target frame and the past frame is small, but the accuracy increases as the frame interval increases. Can detect motion vectors well. Conversely, an object with a high moving speed may move out of the frame if the frame interval between the target frame and the past frame is large. However, if the frame interval is small, the motion vector can be accurately detected. .

  The present invention sets an interval parameter representing a frame interval between a target frame and a past frame when measuring a motion vector for each region or pixel on the target frame or the past frame, and measures the motion vector according to the interval parameter. It is what you do. Therefore, an appropriate frame interval can be set according to the moving speed of the object in the frame, and as a result, the motion vector can be measured with high accuracy regardless of the moving speed of the object.

  In addition, the target frame and the past frame are hierarchized, and by setting an interval parameter for each region or pixel on a layer other than the lowest layer, that is, a low-resolution target frame, the amount of calculation is reduced and the motion vector is measured at high speed. can do. In addition, when a high-level frame is used, the accuracy of detecting a motion vector of an object having a low moving speed is lowered, but according to the present invention, an appropriate frame interval can be set according to the moving speed of an object in the frame. Therefore, it is possible to measure the motion vector with high accuracy and high speed regardless of the moving speed of the object.

  In addition, by changing the interval parameter so that the smaller the motion vector, the larger the interval between the target frame and the past frame, both the motion vector for an object with a low moving speed and the motion vector for an object with a high moving speed Can be detected with high accuracy.

  In addition, since the interval parameter can be accurately set according to the motion of the object by changing the interval parameter based on the plurality of motion vectors at the target pixel position and a plurality of pixel positions in the vicinity of the target pixel position, the motion vector is more accurately detected. Can be measured.

  In addition, the motion vector can be reliably measured by using a plurality of past frames.

  In this case, for a plurality of past frames, the motion vector is measured in order from the past frame having a frame interval close to the target frame or from the past frame having a frame interval distant from the target frame. When a motion vector is measured, it is not necessary to measure the motion vector for all of the plurality of frames by stopping the motion vector measurement using the past frames thereafter, so that the calculation speed can be improved. it can.

  In addition, by determining the motion vector of the target pixel position based on the target pixel position and a plurality of motion vectors of a plurality of pixel positions in the vicinity of the target pixel position, variations in the motion vectors in the target pixel position and the vicinity of the target pixel position are suppressed. Thus, the motion vector at the target pixel position can be measured with higher accuracy.

1 is a schematic block diagram showing the configuration of a motion vector measuring device according to an embodiment of the present invention. The figure for demonstrating the process which a hierarchization part performs Diagram for explaining setting of interval parameter Schematic block diagram showing the configuration of the motion vector measurement unit Diagram for explaining calculation of motion vector distribution The figure for demonstrating the reconstruction of a motion vector (the 1) Diagram for explaining reconstruction of motion vector (part 2) Diagram for explaining motion vector detection The figure for demonstrating the measurement of a 2nd motion vector Diagram for explaining calculation of self-motion vector distribution The figure which shows the state which allocated the score of self-motion vector distribution to the motion vector detection space of the object pixel position A flowchart showing processing performed in the present embodiment Diagram for explaining interval parameter update Flow chart of motion vector measurement processing

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a schematic block diagram showing a configuration of a motion vector measuring apparatus according to an embodiment of the present invention. As shown in FIG. 1, the motion vector measurement device 1 according to the present embodiment includes a frame memory 2, a hierarchization unit 3, an interval parameter setting unit 4, a motion vector measurement unit 5, a memory 6, and a control unit 7.

  The frame memory 2 temporarily stores image data of a digital moving image input from an image input device (not shown) in units of frames. When measuring a motion vector, a frame that is a measurement target of the motion vector is a target frame Frt, and a frame that is past the target frame Frt used for measuring the motion vector is a past frame Frt-i (i = 1 to n: n is a positive integer).

  The hierarchizing unit 3 hires each frame of the image data of the moving image temporarily stored in the frame memory 2 to create a low resolution hierarchical frame. Since the frame memory 2 sequentially stores the frames of the image data of the moving image, the hierarchizing unit 3 hierarchizes the frames sequentially stored in the frame memory 2 only once to generate a low-resolution hierarchical frame. Create FIG. 2 is a diagram for explaining processing performed by the hierarchizing unit 3. FIG. 2 shows the hierarchization of the target frame Frt. As shown in FIG. 2, the hierarchizing unit 3 performs a known process such as thinning out pixels or calculating an average value of four pixels for a non-hierarchical highest resolution frame (hereinafter referred to as a first hierarchical frame Frt0-1). The second layer frame Frt0-2 in which the resolution of the first layer frame Frt0-1 is 1/2 (the vertical and horizontal size is 1/2 that of the first layer frame Frt0-1) is created using the above method. Also, a third layer frame Frt0-3 having a resolution of 1/2 of the second layer frame Frt0-2 is created. Further, a fourth layer frame Frt0-4 having a resolution of 1/2 of the third layer frame Frt0-3 is created.

  As will be described later, when measuring a motion vector, the lower the frame resolution, the smaller the amount of calculation and the higher the processing speed, but the measurement accuracy decreases. Therefore, the number of layers of frames to be created may be set as appropriate according to the required calculation time and measurement accuracy.

  The interval parameter setting unit 4 sets an interval parameter for each pixel on the target frame Frt. The interval parameter represents the frame interval between the target frame Frt and the past frame Frt-i when the motion vector is measured, and the number of frames of the past frame Frt-i used for measuring the motion vector. Hereinafter, setting of the interval parameter will be described. FIG. 3 is a diagram for explaining the setting of the interval parameter. As shown in FIG. 3, it is assumed that the target frame Frt includes areas A1, A2, and A3 of three objects having different moving speeds. The moving speed is assumed to be A1> A2> A3.

  In the present embodiment, the target frame Frt and a plurality of past frames Frt-i that are earlier in time than the target frame Frt are stored in the frame memory 2, and either the target frame Frt or the past frame Frt-i is used. To measure the motion vector. Note that the number of past frames Frt-i to be used is set according to the hierarchy of frames used for motion vector measurement. For example, if the frame layer used for motion frame measurement is the third layer, the motion of one pixel of the object in the first layer can be detected if there are four past frames. If the frame layer used for measuring the motion frame is the fourth layer, the motion of one pixel of the object in the first layer can be detected if there are eight past frames. For this reason, the number of past frames used for measuring a motion vector may be set to be equal to or more than the number determined according to the hierarchy of frames used for measuring a motion vector. In the present embodiment, for example, a third layer frame is used, and a motion vector is measured using six past frames Frt-1 to Frt-6.

  In the present embodiment, the frame interval between the target frame Frt and the past frame Frt-i used for measuring the motion vector of the next target frame is set for each pixel in accordance with the size of the measured motion vector. To do. For example, for the pixel in the region A1 where the moving speed is the highest, the frame interval represented by the interval parameter is set to 1, and the target frame Frt and the past frame Frt-1 closest in time to the target frame Frt are obtained. To measure the motion vector. The number of frames is set to 2 so that the two past frames Frt-1 and Frt-2 are used for measuring the motion vector. The measurement of the motion vector using a plurality of past frames will be described later.

  For the pixels in the area A2 where the moving speed is the second highest after the area A1, the frame interval represented by the interval parameter is set to 2, and the motion vector is measured using the target frame Frt and the past frame Frt-2. To do. In this case, the number of frames is set to 2 so that the two past frames Frt-2 and Frt-3 are used for measuring the motion vector.

  For the region A3 where the moving speed is the lowest, the frame interval represented by the interval parameter is set to 3, and the motion vector is measured using the target frame Frt and the past frame Frt-3. In this case, the number of frames is set to 3 so that the past frames Frt-3, Frt-4, and Frt-5 can be used for motion vector measurement.

  Here, the number of past frames used for motion vector measurement increases as the moving speed of the object including the target pixel decreases, that is, as the frame interval represented by the interval parameter increases, the measurement accuracy increases. From the viewpoint of For example, in the present embodiment, when the frame interval represented by the interval parameter is 2 or less, the number of past frames to be used is 2, and when the frame interval is 3 or more, the number of past frames to be used is 3. It is preferable to set.

  The interval parameter setting unit 4 updates the interval parameter when measuring the motion vector for the next target frame, based on the measured motion vector. The update of the interval parameter based on the motion vector will be described later.

  Further, the interval parameter setting unit 4 measures the motion vector, and then, at each pixel position of the target frame Frt, the motion vector of the target pixel position that is the target of motion vector measurement and a plurality of pixel positions around the target pixel position. Compared with motion vectors, if all motion vectors are not pointing in the same direction or the size of all motion vectors is not the same, the measurement accuracy is improved. Therefore, the interval parameter may be updated so that the number of past frames used for motion vector measurement increases.

  Here, in the case where there is no moving object in an area composed of the target pixel position and a plurality of surrounding pixel positions, the average value of the motion vectors at the pixel positions in the area is zero. On the other hand, when a moving object moves in the region, the average value of the motion vector increases as the degree of movement intrusion into the region increases. Therefore, the interval parameter setting unit 4 measures the motion vector, and then, at each pixel position of the target frame Frt, the motion vector of the target pixel position that is the target of motion vector measurement and a plurality of pixel positions around the target pixel position. Calculate the average value with the motion vector, and the larger the average value, the smaller the frame interval and the smaller the number of past frames to be used. The smaller the average value, the larger the frame interval and the number of past frames to be used. The interval parameter may be updated so as to increase.

  Here, at the time of the first processing for measuring the motion vector, since the motion vector is not measured for all the pixels of the target frame Frt, the interval parameter cannot be set for each pixel. Therefore, at the first measurement of the motion vector, the motion vector is measured using the target frame Frt and all the past frames Frt-i without setting the interval parameter, and according to the size of the motion vector, An interval parameter may be set for each pixel. Then, when measuring the second motion vector, the motion vector may be measured using the interval parameter set by the first motion vector measurement.

  The motion vector measuring unit 5 measures a motion vector at each pixel position of the target frame Frt. Hereinafter, measurement of a motion vector will be described. Note that the motion vector measurement unit 5 measures a motion vector using the high-layer target frame and the past frame. In the following description, the high-layer target frame and the past frame are also used as reference symbols for Frt and Frt. -I shall be used. FIG. 4 is a schematic block diagram showing the configuration of the motion vector measuring unit 5. As illustrated in FIG. 4, the motion vector measurement unit 5 includes a motion vector distribution calculation unit 51, an averaging processing unit 52, a motion vector detection unit 53, and an erroneous measurement determination unit 54. Hereinafter, each process performed by the motion vector distribution calculation unit 51, the averaging processing unit 52, the motion vector detection unit 53, and the erroneous measurement determination unit 54 of the motion vector measurement unit 5 will be described.

First, the motion vector distribution calculation unit 51 calculates a motion vector distribution between the target frame Frt and the past frame Frt-i. FIG. 5 is a diagram for explaining the calculation of the motion vector distribution. In FIG. 5, the target frame Frt is indicated by a solid line, and the past frame Frt-i is indicated by a broken line. As shown in FIG. 5, the motion vector measurement unit 5 first matches the target frame Frt with the past frame Frt-i, and scores representing the correlation with the past frame Frt-i at each pixel position on the target frame Frt. C0 is calculated. Specifically, as shown in the following formula (1), a 3 × 3 block B0 centered on the target pixel for which the score is to be calculated is used, and the target frame Frt and the past frame Frt-i are used. While the block B0 is raster-scanned, the absolute value sum (average absolute error) of the difference values of the corresponding pixel values of the blocks B0 is calculated as the score C0. Here, the block size is not limited to 3 × 3, but being as small as possible can reduce the amount of calculation. In equation (1), f (x + p, y + q) represents the pixel value of the target frame Frt, and g (x + p, y + q) represents the pixel value of the past frame Frt-i. X and y indicate the x direction and the y direction, respectively. Further, a mean square error may be used as the score C0.

It should be noted that in order to prevent erroneous detection of motion vectors due to fluctuations in exposure values and unexpected brightness changes during moving image shooting, pixels in a 3 × 3 size block in the target frame Frt and the past frame Frt-i The average value fm, gm of the value may be calculated, and the score C0 may be normalized by the following formula (2) using the average value fm, gm.

  Further, the score C0 calculated by the above formulas (1) and (2) becomes smaller as the correlation between the pixels is higher. In the present embodiment, the following description will be given using the score C1 calculated by subtracting the score C0 from the maximum pixel value of the input moving image. For example, when the image is an 8-bit image, it can be calculated as C1 = 255−C0. The score C1 increases as the correlation between the pixels increases.

  Further, the motion vector distribution calculation unit 51 shifts the past frame Frt-i by 1 pixel in the range of ± 2 pixels in the horizontal direction and the vertical direction with respect to the target frame Frt, and on the target frame Frt every time the pixel is shifted. At each pixel position, a score C1 with the past frame Frt-i is calculated. Here, if the horizontal coordinate for shifting the past frame Frt-i with respect to the target frame Frt is h and the vertical coordinate is v, the pixel shift amount of the past frame Frt-i with respect to the target frame Frt is ( h, v) = (k, l) (k, l = integer of −2 to +2). Specifically, when the target frame Frt and the past frame Frt-i are matched, (h, v) = (0, 0), and the past frame Frt-i is set in the horizontal direction with respect to the target frame Frt. When shifted by +1 pixel, (h, v) = (1, 0). In FIG. 5, for the target frame Frt in the case of (h, v) = (0, 0), (1, 0), (2, 0), (1, 1) and (−2, −2). The shift amount of the past frame Frt-i is shown. In FIG. 5, the shift amount of the target frame Frt and the past frame Frt-i is shown larger than the actual shift amount for the sake of explanation.

  Thus, by calculating the score C1 while shifting the past frame Frt-i one pixel at a time with respect to the target frame Frt within a range of ± 2 pixels in the horizontal direction and the vertical direction, the shift is performed on the target frame Frt for each shift amount. It is possible to calculate the distribution of the score C1 in which the score C1 at each shifted position is assigned to all the pixel positions. Here, the score C1 can be regarded as a motion vector on the target frame Frt when the past frame Frt-i is shifted from the target frame Frt. For this reason, in this embodiment, the distribution of the score C1 is referred to as a motion vector distribution D. Since there are 5 × 5 = 25 ways of shifting the pixels, 25 motion vector distributions D (k, l) (k, l = integer of −2 to +2) are calculated. In other words, this means that 25 scores C1, that is, motion vectors corresponding to the shifted positions are calculated at each pixel position of the target frame Frt.

  The averaging processing unit 52 performs a spatial averaging filter process on the motion vector distribution D (k, l), thereby removing noise of the motion vector distribution D (k, l) and averaging. The motion vector distribution Dm (k, l) is calculated. As the averaging filter, a filter having a size of 3 × 3 can be used, for example. However, the averaging filter is not limited to this, and may be appropriately set according to a desired degree of noise removal. The filter to be used is not limited to the averaging filter, and a spatial median filter may be used.

  The motion vector detection unit 53 detects a motion vector at each pixel position of the target frame Frt using a motion vector distribution Dm (k, l) including 25 motion vectors. Hereinafter, detection of a motion vector will be described. The motion vector detection unit 53 sets a motion vector detection space at each pixel position of the target frame Frt, and reconstructs a motion vector in the motion vector detection space. 6 and 7 are diagrams for explaining the reconstruction of the motion vector. In the present embodiment, with respect to the target frame Frt, the past frame Frt-i is shifted by one pixel within a range of ± 2 pixels in the horizontal direction and the vertical direction, thereby obtaining 25 pixels at each pixel position of the target frame Frt. Since the score C1, that is, the motion vector is calculated, a motion vector detection space of 5 × 5 = 25 pixels is set as shown in FIG. 6 at the target pixel position that is the target of motion vector detection. Note that the positive and negative directions of the motion vector detection space are the same as the direction of pixel shift when the motion vector distribution is calculated.

  Then, the motion vector detection unit 43 assigns the score C1 of the corresponding pixel position of the motion vector distribution Dm (k, l) calculated based on the shift amount of the corresponding pixel to the 25 coordinate positions in the motion vector detection space. . That is, the motion vector distribution Dm (2, 2) is present at the (0, 0) coordinate position in the motion vector detection space, and the motion vector distribution Dm (2, 2) is present at the (2, 2) coordinate position. A pixel position score C1 is assigned. As a result, as shown in FIG. 6, for each pixel position of the target frame Frt, the score C1, that is, the motion vector is assigned to the motion vector detection space, and the motion vector at the target pixel position is reconstructed.

  Next, the motion vector detection unit 43 detects the coordinate position where the score C1 is maximum in the motion vector detection space. When the score C1 is calculated, the maximum coordinate position (hereinafter referred to as the maximum coordinate position) is detected. When the score C0 is calculated, the coordinate position where the score C0 is minimum is determined. It is to detect. Then, with reference to the center coordinate position (that is, (0, 0)) in the motion vector detection space, a vector from the center coordinate position to the maximum coordinate position is detected as a motion vector. For example, when the motion vector is reconstructed as shown in FIG. 7, the maximum coordinate position is (2, 2), so that the start point is (0, 0) and the end point is (2, 2) as shown in FIG. 2) can be detected. The magnitude of the motion vector can be detected as the magnitude from the center coordinate position to the maximum coordinate position. For example, when the start point of the motion vector is (0, 0) and the end point is (2, 2), the magnitude of the motion vector is 2√2.

  The magnitude of the detected motion vector differs depending on the frame interval between the target frame Frt and the past frame Frt-i. For example, if the frame interval between the target frame Frt and the past frame Frt-i is 1, the size of the detected motion vector can be directly used as the size of the motion vector at the target pixel position. On the other hand, if the frame interval is 2, the actual size of the motion vector is 1/2 times the size of the detected motion vector, and if the frame interval is 3, it is 1/3 times. Therefore, when detecting the motion vector, the motion vector detection unit 53 corrects the size of the motion vector according to the frame interval between the target frame Frt and the past frame Frt-i.

  The motion vector detection unit 53 detects the motion vectors at all the pixel positions of the target frame Frt by performing the above processing at all the pixel positions of the target frame Frt. Note that the motion vector distribution D (k, l) calculates a score C1 representing a motion vector while shifting the past frame Frt-i with respect to the target frame Frt within a range of ± 2 pixels in the horizontal direction and the vertical direction. Is calculated by For this reason, the score C1 may not be calculated in the range of two pixels around the target frame Frt. Therefore, the motion vector is detected at a pixel position excluding a range of two pixels around the target frame Frt.

  The erroneous measurement determination unit 54 determines whether or not the motion vector at each pixel position of the target frame Frt detected by the motion vector detection unit 43 is an erroneous measurement. Here, the motion vector detected by the motion vector detection unit 43 has a center coordinate position in the motion vector detection space as a start point and a coordinate position where the score C1 is maximum (or the score C0 is minimum) as an end point. However, in the present embodiment, 3 × 3, which is a smaller block than the 8 × 8 block as in the conventional block matching method, is used to calculate the motion vector distribution. There is a risk of detecting a motion vector that appears by chance due to noise. In addition, when the target pixel position where the motion vector in the target frame Frt is detected is not a moving object but corresponds to the background of the object, the motion vector may be generated in a random direction. In such a case, the difference between the score C1s of the start position of the detected motion vector and the score C1e of the end position is not so large.

  For this reason, the erroneous measurement determination unit 54 calculates the coordinates C1s of the coordinate position of the start point of the detected motion vector and the score C1e of the coordinate position of the end point for the target pixel position that is the target of the erroneous measurement determination in the target frame Frt. When the relationship is C1s> C1e · α, it is determined that the detected motion vector has been erroneously measured, and a determination result that the motion vector at the target pixel position has not been measured is output. On the other hand, if C1s ≦ C1e · α, it is determined that the detected motion vector is a true motion vector, and the motion vector is output as a determination result of the target pixel position. Note that, for the target pixel position where the motion vector is determined to be erroneous measurement, a determination result that the motion vector is 0 may be output. Here, the value of the coefficient α may be set according to the required measurement accuracy. For example, a value such as 0.99 can be used, but is not limited thereto.

  The detected motion vector is erroneously measured when the relationship between the score C1s of the coordinate position of the start point of the detected motion vector and the score C1e of the coordinate position of the end point is C1e> C1s> C1e · α. You may make it determine.

  On the other hand, in the above embodiment, the motion vector distribution D (k, l) is calculated by calculating the score C1 representing the motion vector while shifting the past frame Frt-i with respect to the target frame Frt. Conversely, consider a case where a motion vector distribution is calculated while shifting the target frame Frt with respect to the past frame Frt-i, and a motion vector at each pixel position of the target frame Frt is detected. In the following, for the sake of explanation, the motion vector detected when the past frame Frt-i is shifted with respect to the target frame Frt is the first motion vector, and the target frame Frt is shifted with respect to the past frame Frt-i. The motion vector detected in this case is referred to as a second motion vector. In this case, when the first and second motion vectors are not erroneously measured, the magnitudes of the first and second motion vectors match, and the direction of the first motion vector and the direction of the second motion vector The opposite is true. However, when the first and second motion vectors are erroneously measured, the magnitudes of the first and second motion vectors do not match, or the direction of the second motion vector is the first motion vector. It will not be the opposite of the direction.

  Therefore, the erroneous measurement determination unit 54 detects the second motion vector by calculating the motion vector distribution while shifting the target frame Frt with respect to the past frame Frt-i, as shown in FIG. Determining the pixel positions of the start and end points of the first and second motion vectors at the target pixel position of the target frame Frt, the start point of the first motion vector and the end point of the second motion vector match, and When the end point of the first motion vector coincides with the start point of the second motion vector, the first motion vector is determined to be a true motion vector, otherwise, the first motion vector May be determined to be erroneous measurement.

  Note that an object that moves between the target frame Frt and the past frame Frt-i does not necessarily move in units of one pixel, and may move in units of less than one pixel. For example, the moving object included in the past frame Frt-i may move in so-called sub-pixel units such that the object frame Frt is shifted by 1.5 pixels from the position of the past frame Frt-i. In such a case, even if the target pixel position corresponds to an object that is truly moving, the respective pixel positions of the start point and end point of the second motion vector measured at the target pixel position are the first The pixel positions of the end point and start point of the motion vector are not completely coincident. Therefore, when the erroneous measurement determination unit 54 determines the erroneous measurement using the first and second motion vectors, the difference between the end point coordinate position of the second motion vector and the start point coordinate position of the first motion vector. However, if it is in the range of ± 1 pixel in the horizontal and vertical directions, it may be determined that the magnitude of the second motion vector matches the magnitude of the first motion vector. As a result, the possibility that the true motion vector is determined to be erroneous measurement can be reduced.

  Further, it is assumed that the motion vector distribution is calculated by calculating a score that increases as the correlation increases, while shifting the target frame Frt itself with respect to the target frame Frt. The motion vector distribution calculated in this way is referred to as a self-motion vector distribution. The score of the self motion vector distribution is C1 ′. As shown in FIG. 10, when the moving object 20 is included in the target frame Frt, the position of the object 20 is also shifted when the overlapping target frames Frt are shifted. Accordingly, when 25 types of self-motion vector distributions are calculated in the same manner as described above, and the self-motion vector distribution score C1 ′ is assigned to the motion vector detection space at the target pixel position corresponding to the object 20, as shown in FIG. In addition, the score C1 ′ at the center coordinate position (0,0) increases, but the score C1 ′ at other coordinate positions decreases as the distance from the center coordinate position (0,0) increases.

  Conversely, when the self-motion vector distribution score C1 ′ is assigned to the motion vector detection space at the target pixel position corresponding to the monotonous background of the object 20, the pixel values around the target pixel position are the pixel values of the target pixel position. Since it is not so different from the pixel value, the score C1 ′ is relatively large regardless of the coordinate position of the motion vector detection space.

  Here, in the self-motion vector distribution, when the score C1 ′ becomes large at a coordinate position other than the center coordinate position (0, 0), there is a possibility that an erroneous measurement has occurred at the coordinate position having the large score. It is expensive. Therefore, in the self-motion vector distribution, it can be determined whether or not the detected motion vector is an erroneous measurement according to the value of the score C1 ′ at the position corresponding to the end position of the detected motion vector.

  Therefore, the erroneous measurement determination unit 54 calculates the self-motion vector distribution, assigns the self-motion vector distribution score C1 ′ to the motion vector detection space at the target pixel position, and the motion vector that is the end point of the motion vector at the target pixel position. The self-motion vector distribution score C1 ′ at the coordinate position in the detection space is measured. Then, the measured score C1 ′ is compared with a predetermined threshold value Th1, and when the score C1 ′ is less than the threshold value Th1, it is determined that the motion vector is a true motion vector. If the threshold value is equal to or greater than Th1, the motion vector may be determined to be erroneous measurement. Thereby, the erroneous measurement of a motion vector can be determined similarly to the said embodiment.

  In addition, it is possible to determine whether or not the motion vector is a true motion vector by using a discriminator that determines whether or not the motion vector is a true motion vector, which is generated using machine learning. Also good. In this case, the discriminator uses the score distribution in the motion vector detection space in which the motion vector is a true motion vector as positive teacher data, and the score distribution in the motion vector detection space in which the motion vector is false detection as negative teacher data. And can be created by learning.

  The memory 6 stores the interval parameter set by the interval parameter setting unit 4.

  The control unit 7 controls the operations of the hierarchization unit 3, the interval parameter setting unit 4, and the motion vector measurement unit 5.

  Next, the operation of this embodiment will be described. FIG. 12 is a flowchart showing processing performed in the present embodiment. In this embodiment, when measuring a motion vector for each frame that is continuously input, an interval parameter is set for each pixel of the frame. Since the motion vector of each pixel is unknown, the interval parameter cannot be set. For this reason, in the present embodiment, at the time of measuring the first motion vector, the motion parameter distribution is calculated between all past frames Frt-i and the target frame Frt without using the interval parameter. Assume that a motion vector is measured and an interval parameter is set. In the flowchart shown in FIG. 12, it is assumed that the measurement of the first motion vector is completed, and an interval parameter is set for each pixel of the target frame Frt and stored in the memory 6 based on the motion vector.

  When the target frame Frt and the number of past frames Frt-i necessary for processing are stored in the frame memory 2 from an image input device (not shown) (step ST1), the hierarchizing unit 3 determines that the target frame Frt and the past frame Frt- i is hierarchized (step ST2). Note that, after the motion vector measurement process for the current target frame Frt is completed, the motion vector measurement process is performed for the next target frame. In this case, the hierarchization unit 3 hierarchizes only the next target frame. It will do.

  Next, the motion vector measuring unit 5 reads out the interval parameters of all pixel positions in the higher-layer target frame Frt from the memory 6 (step ST3), and further, the target frame Frt and the past frame Frt− closest to the target frame Frt in time. i is read from the frame memory 2 (step ST4). And the control part 7 sets the object pixel position used as the object of the motion vector measurement in the object frame Frt of a high hierarchy to an initial position (step ST5). As the initial position, for example, the pixel position at the upper left corner of the target frame Frt can be used.

  Next, the control unit 7 refers to the interval parameter of the target pixel position, and the target pixel position is determined by using the target frame Frt and the past frame (currently past frame) Frt-i that is currently read out. It is determined whether or not the pixel position is to be measured (step ST6). If step ST6 is affirmed, the control unit 7 further determines whether or not a motion vector has already been measured for the target pixel position (step ST7). If step ST7 is negative, the motion vector measuring unit 5 measures the motion vector at the target pixel position using the target frame Frt and the current past frame Frt-i (step ST8). The motion vector measurement process will be described later. Subsequent to step ST8 and when step ST6 is denied or step ST7 is affirmed, the control unit 7 determines whether or not the target pixel position is the final pixel position of the target frame Frt (step ST9). ). If step ST9 is negative, the target pixel position is changed to the next pixel position (step ST10), the process returns to step ST6, and the processes after step ST6 are repeated.

  If step ST9 is affirmed, the control unit 7 determines whether or not the motion vector has been measured using the past frames Frt-i of all the frames represented by the interval parameter (step ST11). If step ST11 is negative, the past frame Frt-i is changed to the next past frame, the changed past frame is read from the frame memory 2 (step ST12), the process returns to step ST5, and the processes after step ST6 are repeated. .

  On the other hand, when step ST11 is affirmed, the motion vector measurement part 5 outputs the measurement result of a motion vector (step ST13). When the motion vector is measured, the measured motion vector is output. When the motion vector is not measured, the measurement result indicating that the motion vector is not measured or the size is 0. Output motion vector.

  The above processing will be specifically described. The motion vector measuring unit 5 measures a motion vector for the past frame Frt-i with reference to the interval parameters for all the pixels of the target frame Frt. For example, when the past frame is Frt-1, the motion vector is measured only at the pixel position where the frame interval is set to 1 by the interval parameter. Then, the past frame Frt-i is changed to the next past frame, and the motion vector is measured for the next past frame Frt-i with reference to the interval parameters for all the pixels of the target frame Frt.

  In the present embodiment, even when the target pixel position is a pixel position at which a motion vector is measured by the interval parameter, if the motion vector has already been measured at the target pixel position, the motion vector is measured. Will not be. For example, when there are three past frames Frt-i defined by the interval parameter of the target pixel position, for example, Frt-1 to Frt-3, the motion vector is determined using the target frame Frt and the past frame Frt-1. When measured, no motion vector is measured for the remaining past frames Frt-2 and Frt-3. On the other hand, if the motion vector is not measured, the motion vector is measured using the target frame Frt and the past frame Frt-2. If the motion vector is measured, the motion vector is not measured for the remaining past frame Frt-3. As described above, when a motion vector is measured, the motion vector is not measured for all past frames Frt-i represented by the interval parameter, thereby reducing the amount of calculation and performing processing at high speed. be able to.

  On the other hand, following step ST13, the interval parameter setting unit 4 updates the interval parameter of the target pixel position based on the measurement result of the motion vector (step ST14). Hereinafter, the update of the interval parameter will be described.

  FIG. 13 is a diagram for explaining the update of the interval parameter. Here, a case will be described in which the past frames Frt-i defined by the interval parameter of the target pixel position before processing are, for example, three of Frt-3 to Frt-5. In this case, the frame interval defined by the interval parameter is 3, and the number of frames is 3. When the motion vector is measured by processing using the past frame Frt-3, the moving speed of the object at the target pixel position may be faster than when the interval parameter is set earlier. One frame interval is made smaller than the set interval parameter. Further, since the frame interval becomes 2 by reducing the frame interval by 1, the number of past frames to be used is set to 2. Therefore, the interval parameter setting unit 4 updates the interval parameter so that the frame interval is 2 and the number of frames is 2. Thereby, in the next process, the past frames Frt-2 and Frt-3 are used.

  On the other hand, when the motion vector is measured by the process using the past frame Frt-5, there is a possibility that the moving speed of the object at the target pixel position is slower than when the previous interval parameter is set. The frame interval is increased by one more than the previously set interval parameter. Further, since the frame interval becomes 4 by increasing the frame interval by one, the number of past frames to be used remains three. Therefore, the interval parameter setting unit 4 updates the interval parameter so that the frame interval is 4 and the number of frames is 3. Thereby, in the next process, the past frames Frt-4, Frt-5, and Frt-6 are used.

  If no motion vector is measured using any past frame, the number of past frames used is increased by one. Therefore, the interval parameter setting unit 4 updates the interval parameter so that the frame interval is 3 and the number of frames is 4. Thus, in the next process, the past frames Frt-3, Frt-4, Frt-5, and Frt-6 are used. Note that if the number of past frames to be used is increased, the amount of calculation increases. Therefore, it is preferable to set the maximum number of past frames to be used to, for example, 4. The interval parameter may be periodically reset to an initial value (for example, frame interval 2 and frame number 3).

  Further, when the measured motion vector is 0, the frame interval is set to 6, and the motion vector is measured by sequentially using the past frame Frt-6 to the past frame Frt-1 until the motion vector is measured. Set the interval parameter to In this case, if the motion vector is measured using the past frame Frt-6, the measurement of the motion vector is ended there. If the motion vector is not measured, the motion vector is measured using the past frame Frt-5. Thereafter, the motion vector is measured until the motion vector is sequentially measured using the past frames Frt-4 to Frt-1.

  In this embodiment, since the motion vector is measured using the six past frames Frt-1 to Frt-6, the past frames after Frt-6 are not used. For this reason, in this embodiment, the upper limit value of the frame interval in the interval parameter is set so as not to use past frames after Frt-6. As described above, in the present embodiment, since the maximum value of the number of past frames to be used is 4, the upper limit value of the frame interval is set to 4. Therefore, in the case where the past frame Frt-i defined by the interval parameter of the target pixel position before processing is, for example, Frt-4 to Frt-6, if the motion vector is not measured, the interval is substantially reduced. The parameter will not be updated.

  Subsequent to step ST14, in order to measure a motion vector for the next target frame, the control unit 7 changes the target frame Frt to the next frame (t = t + 1, step ST15), and returns to step ST2. In step ST2, only the next target frame is hierarchized.

  FIG. 14 is a flowchart of the motion vector measurement process. The motion vector distribution calculating unit 51 of the motion vector measuring unit 5 sets the shift amount between the target frame Frt and the past frame Frt-i to an initial value (for example, (h, v) = (0, 0)) (step ST21). ) To calculate the motion vector distribution D (step ST22). Then, it is determined whether or not the motion vector distribution has been calculated for all the shift amounts of the target frame Frt and the past frame Frt-i (step ST23). If step ST23 is negative, the shift amount is changed (step ST24). ), The process returns to step ST22. If step ST23 is positive, the averaging processing unit 52 performs an averaging process on the calculated motion vector distribution D (step ST25).

  Subsequently, the motion vector detection unit 53 detects the motion vector (step ST26), and the erroneous measurement determination unit 54 determines the erroneous measurement of the motion vector (step ST27), and the motion vector measurement process ends.

  As described above, in the present embodiment, for each pixel on the target frame Frt, the interval parameter indicating the frame interval between the target frame Frt and the past frame Frt-i when measuring the motion vector is set, and the interval parameter is set as the interval parameter. Therefore, the motion vector is measured. Therefore, an appropriate frame interval can be set according to the moving speed of the object in the frame, and as a result, the motion vector can be measured with high accuracy regardless of the moving speed of the object.

  In addition, the target frame Frt and the past frame Frt-i are hierarchized, and the interval parameter is set for each pixel on a layer other than the lowest layer, that is, the low-resolution target frame Frt. Vectors can be measured. In addition, when a high-level frame is used, the accuracy of detecting a motion vector of an object having a low moving speed is lowered. According to this embodiment, an appropriate frame interval is set according to the moving speed of the object in the frame. Therefore, it is possible to measure the motion vector with high accuracy and high speed regardless of the moving speed of the object.

  In addition, by changing the interval parameter so that the interval between the target frame Frt and the past frame Frt-i becomes larger as the size of the motion vector is smaller, the motion vector and the movement velocity for an object with a lower movement velocity are obtained. Both motion vectors for a large object can be accurately detected.

  In addition, it is possible to reliably measure a motion vector by using a plurality of past frames. At that time, for a plurality of past frames, the motion vector is measured in order from a past frame having a frame interval close to the target frame. When the motion vector of the value is measured, the motion vector measurement using the past frames after that is stopped, so it is not necessary to measure the motion vector for all of the multiple frames. Speed can be improved. Of course, the motion vector may be measured in order from the past frame at a frame interval away from the target frame.

  In the above-described embodiment, the interval parameter is set for each pixel on the target frame Frt, but the moving object exists over a region composed of a plurality of pixels. For this reason, the interval parameter may be set for each region corresponding to the moving object on the target frame Frt.

  In the above embodiment, the motion vector measurement unit 5 calculates the motion vector by calculating the motion vector distribution. However, the motion vector measurement method is not limited to this, and the conventional block matching is performed. Of course, it is possible to use any method such as the method or the method described in Patent Documents 1 to 4 above. Even in the case where the conventional block matching method is used, the block size can be reduced in this embodiment, so that the amount of calculation can be reduced and processing can be performed at high speed.

  In the above embodiment, the hierarchizing unit 3 hierarchizes the target frame Frt and the past frame Frt-i, and measures the motion vector using a higher layer frame. However, the target frame Frt and the past frame Frt− are measured. Of course, the motion vector may be measured using the target frame Frt and the past frame Frt-i itself without hierarchizing i.

  In the above embodiment, the motion vector at the target pixel position is determined based on a plurality of motion vectors at a plurality of pixel positions near each pixel position for the motion vector at each pixel position on the target frame Frt. Also good. For example, an average value of motion vectors at a plurality of pixel positions may be used as the motion vector at the target pixel position. As a result, the motion vector at the target pixel position can be measured with higher accuracy while suppressing variations in the motion vector.

  In the above embodiment, when a motion vector is measured, the motion vector is not measured for all past frames Frt-i represented by the interval parameter. However, all past frames Frt-i are not measured. Of course, the motion vector may be measured for. In this case, an average value of motion vectors measured using all past frames may be used as the motion vector of the target pixel position.

  In the above embodiment, when the motion vector is measured, the amount of smoothing between the target frame Frt and the past frame Frt-i is ± 2 pixels. However, the present invention is not limited to this. Alternatively, the smoothing amount may be set as appropriate according to the required measurement accuracy, such as ± 3 pixels.

  In the above embodiment, the motion vector distribution D is averaged, but it is needless to say that the motion vector may be measured without averaging the motion vector distribution D.

  Although the apparatus 1 according to the embodiment of the present invention has been described above, the computer functions as means corresponding to the hierarchization unit 3, the interval parameter setting unit 4, the motion vector measurement unit 5, and the control unit 7. A program for performing the processing as shown in FIG. 12 and FIG. 14 is also one embodiment of the present invention. A computer-readable recording medium in which such a program is recorded is also one embodiment of the present invention.

DESCRIPTION OF SYMBOLS 1 Motion vector measurement apparatus 2 Frame memory 3 Hierarchical part 4 Space | interval parameter setting part 5 Motion vector measurement part 51 Motion vector distribution calculation part 52 Averaging process part 53 Motion vector detection part 54 Incorrect measurement determination part

Claims (9)

In a motion vector measurement device that measures a motion vector at a target pixel position on a target frame based on a target frame in a plurality of frames included in a moving image and a past frame that is separated from the target frame by a predetermined frame interval,
Interval parameter setting means for setting an interval parameter representing a frame interval between the target frame and the past frame when measuring the motion vector for each region or pixel on the target frame;
A motion vector measuring device comprising: motion vector measuring means for measuring the motion vector according to the interval parameter. Further comprising hierarchizing means for hierarchizing the target frame and the past frame;
2. The motion vector measuring apparatus according to claim 1, wherein the interval parameter changing unit is a unit that sets the interval parameter for each region or for each pixel on the target frame in a layer other than the lowest layer. Storage means for storing the interval parameter;
The interval parameter setting unit is a unit that updates the interval parameter stored in the storage unit so that the interval between the target frame and the past frame becomes larger as the measured motion vector is smaller. The motion vector measuring device according to claim 1 or 2. Storage means for storing the interval parameter;
The interval parameter setting means is means for updating the stored interval parameter based on the plurality of motion vectors at the target pixel position and a plurality of pixel positions near the target pixel position. The motion vector measuring device according to any one of 1 to 3.   5. The interval parameter setting unit is a unit that sets the interval parameter that represents the number of frames of the past frame used for measurement of the motion vector together with the frame interval. The motion vector measuring device according to claim 1.   The motion vector measuring means is a means for measuring the motion vector in order from the past frame at a frame interval close to the target frame or from the past frame at a frame interval away from the target frame. 6. The motion vector measurement device according to claim 5, wherein when the motion vector having a value other than 0 is measured, the motion vector measurement unit stops measurement of the motion vector using the past frame thereafter. .   The motion vector measuring means is means for determining a motion vector at the target pixel position based on the plurality of motion vectors at the target pixel position and a plurality of pixel positions in the vicinity of the target pixel position. Item 7. The motion vector measuring device according to any one of Items 1 to 6. In a motion vector measurement method for measuring a motion vector at a target pixel position on the target frame based on a target frame in a plurality of frames included in the moving image and a past frame separated from the target frame by a predetermined frame interval,
For each region or pixel on the target frame, set an interval parameter representing a frame interval between the target frame and the past frame when measuring the motion vector,
A motion vector measuring method comprising measuring the motion vector according to the interval parameter. A motion vector measuring method for measuring a motion vector at a target pixel position on a target frame based on a target frame in a plurality of frames included in the moving image and a past frame separated from the target frame by a predetermined frame interval In the program to be executed,
The program sets, for each region or pixel on the target frame, a procedure for setting an interval parameter representing a frame interval between the target frame and the past frame when measuring the motion vector;
A program for causing a computer to execute a procedure for measuring the motion vector according to the interval parameter.

Images