JP2005259027A - Apparatus and method for detecting motion vector, program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately detect motion vectors at a low cost. <P>SOLUTION: A vector position detecting part 61 detects the pointing positions of each inputted vector. A block extraction part 62 extracts a block with predetermined size, centered around a pixel to be an origin of the vector, block extracting parts 63-1 to 63-4 extract blocks with the same size, centered around four pixels around the pointing positions of the vectors, differential absolute sum calculating parts 64-1 to 64-4 calculate four block difference absolute sums, respectively and output them to a result output part 65. The result output part 65 calculates the average value, the maximum value or the minimum value of the four block differential absolute sums, associates the values with each inputted vector and outputs it as the evaluation value for each vector. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a motion vector detection device and method, a program, and a recording medium, and more particularly, to a motion vector detection device and method, a program, and a recording medium that can accurately detect a motion vector at low cost.

  In recent years, in the image processing technology for moving images, it is important to accurately detect a motion vector representing the motion of an image in order to improve the resolution of the image or track the motion of an object in the image. Yes.

For this reason, in order to suppress erroneous detection of a motion vector, in the image data, when a part centering on a pixel that is the starting point of the motion vector of the previous frame in time and the part moves according to the motion vector, It is necessary to compare a portion centered on a point that is the pointing position of a motion vector of a subsequent frame in time, and in this comparison, a predetermined size (predetermined about a pixel that is the starting point of the motion vector) And a block of the same size centered on the pixel at which the motion vector is pointed, and the sum of absolute values of the corresponding pixel values of each block (block difference absolute value) A method has been proposed in which, when the sum of the block difference absolute values is small, the accuracy of the motion vector is determined to be high (see, for example, Patent Document 1).
JP-A-7-87495

  However, the pointing position of the motion vector in a temporally subsequent frame does not necessarily match the pixel position in that frame. That is, in many cases, points located between pixels arranged at a predetermined interval in a frame are the motion vector pointing positions. If the motion vector pointing position does not match the pixel position, the conventional technique rounds off the coordinate value of the motion vector pointing position to make the appropriate pixel the motion vector pointing position, or The calculation of the block difference absolute value sum is performed by, for example, generating a virtual pixel block by performing predetermined weighting on the values of pixels around the vector pointing position.

  However, even if the coordinates of the motion vector pointing position are rounded off and an appropriate pixel is used as the motion vector pointing position, it is guaranteed whether the extracted block is appropriate as a block corresponding to the motion vector. In addition, when a virtual pixel block is generated by performing predetermined weighting on the values of pixels around the position indicated by the motion vector, the processing required for pixel value calculation becomes complicated, The cost of the device increases. Furthermore, in any case, there is a possibility that the value of the sum of block difference absolute values is locally reduced, and as a result, there is a problem that an accurate motion vector may not be detected.

  The present invention has been made in view of such a situation, and makes it possible to accurately detect a motion vector at low cost.

  An image processing apparatus according to the present invention is a motion vector detection apparatus that detects a vector representing an image motion, and a position detection unit that detects a pointing position of a plurality of candidate vectors that are candidates for a vector representing an image motion; A first extracting unit that extracts a pixel block of a preset size area centered on a pixel that is a starting point of the candidate vector; and a plurality of pixels that are close to the pointing position of the candidate vector detected by the position detecting unit. A second extraction means for extracting a pixel block of a plurality of predetermined size areas as a center, a value of a pixel in the pixel block extracted by the first extraction means, and a second extraction means Comparison result output means for comparing the corresponding pixel values in the plurality of extracted pixel blocks and outputting a plurality of comparison results, and a plurality of outputs outputted by the comparison result output means Evaluation value output means for outputting an evaluation value of a candidate vector based on the comparison result, and selection means for selecting a vector having high accuracy from a plurality of candidate vectors based on the evaluation value output by the evaluation value output means It is characterized by providing.

  The second extraction means may extract four pixel blocks centered on the four pixels located closest to the up / down / left / right direction of the point representing the pointing position of the candidate vector.

  The comparison result output means outputs four difference absolute value sums as comparison results for the pixel block extracted by the first extraction means and the four pixel blocks extracted by the second extraction means. The evaluation value output means can output the average value, the maximum value, or the minimum value of the four sums of absolute differences as the evaluation value.

  As the plurality of candidate vectors, an initial vector, a first or second vector obtained by performing an operation by a gradient method once or twice on the initial vector is input, and the selection means includes an initial vector, The first or second vector having high accuracy can be selected as the motion vector of the image.

  The plurality of candidate vectors are initial vector candidate vectors, and the selection means may select a plurality of candidate vectors with high accuracy as initial vectors.

  The motion vector detection method of the present invention is a motion vector detection method for a motion vector detection device that detects a vector representing image motion, and detects the pointing positions of a plurality of candidate vectors that are candidates for vectors representing image motion. A position detecting step, a first extracting step for extracting a pixel block of a preset size area centered on a pixel that is a starting point of the candidate vector, and a candidate vector detected by the processing of the position detecting step A second extraction step of extracting a plurality of pixel blocks of a predetermined size area centered on a plurality of pixels close to the position, and pixels in the pixel block extracted by the processing of the first extraction step Is compared with the corresponding pixel values in the plurality of pixel blocks extracted by the processing of the second extraction step. A plurality of comparison result output steps, an evaluation value output step for outputting the evaluation value of the candidate vector based on the plurality of comparison results output by the processing of the comparison result output step, and output by the processing of the evaluation value output step And a selection step of selecting a vector with high accuracy from a plurality of candidate vectors based on the evaluated value.

  The program according to the present invention is a program for a motion vector detection device that detects a vector that represents image motion, and that controls the detection of the pointing positions of a plurality of candidate vectors that are candidates for the vector representing image motion. Candidates detected by processing of a step, a first extraction control step for controlling extraction of a pixel block of a preset size region centered on a pixel serving as a starting point of a candidate vector, and a position detection control step A second extraction control step for controlling to extract a plurality of pixel blocks of a predetermined size area centered on a plurality of pixels close to the pointing position of the vector, and processing of the first extraction control step And the pixel values in the pixel block extracted by the second extraction control step and the values in the plurality of pixel blocks extracted by the processing of the second extraction control step. Based on the comparison result output control step for comparing the values of the corresponding pixels and controlling to output a plurality of comparison results and the plurality of comparison results output by the processing of the comparison result output control step, Based on the evaluation value output control step for controlling the evaluation value to be output and the evaluation value output by the processing of the evaluation value output control step, control is performed so that a vector with high accuracy is selected from a plurality of candidate vectors. The selection control step is performed by a computer.

  The recording medium of the present invention is a recording medium on which a program of a motion vector detecting device that detects a vector representing image motion is recorded, and the pointing positions of a plurality of candidate vectors that are candidates for vectors representing the image motion A position detection control step for controlling detection, a first extraction control step for controlling so as to extract a pixel block of a preset size area centered on a pixel serving as a starting point of a candidate vector, and position detection control A second extraction control step for performing control to extract a plurality of pixel blocks of a predetermined size area centered on a plurality of pixels close to the pointing position of the candidate vector detected by the processing of the step; The value of the pixel in the pixel block extracted by the process of the first extraction control step and a plurality of values extracted by the process of the second extraction control step Based on the comparison result output control step for controlling to output a plurality of comparison results by comparing each corresponding pixel value in the elementary block and the comparison result output control process. Based on the evaluation value output control step for controlling to output the evaluation value of the candidate vector and the evaluation value output by the processing of the evaluation value output control step, a vector with high accuracy is selected from the plurality of candidate vectors. A program for causing a computer to execute a selection control step for controlling selection is recorded.

  In the motion vector detection apparatus and method, the program, and the recording medium of the present invention, the input position of a plurality of candidate vectors that are candidates for the vector representing the motion of the image is detected, and the pixel that is the starting point of the candidate vector A pixel block of a region of a preset size centered on the pixel is extracted, and a plurality of pixel blocks of a region of a predetermined size centering on a pixel close to the pointed position of the detected candidate vector The value of the pixel in the extracted and extracted pixel block is compared with the value of the corresponding pixel in the plurality of pixel blocks around the plurality of pixels around the pointed position of the extracted candidate vector, Multiple comparison results are output. Based on the calculated multiple comparison results, the evaluation value of the candidate vector is output. Based on the evaluation value output. Probable vector is selected from a plurality of candidate vectors.

  According to the present invention, a motion vector can be detected. In particular, it is possible to accurately detect a motion vector at a low cost.

  Embodiments of the present invention will be described below. The correspondence relationship between the invention described in this specification and the embodiments of the invention is exemplified as follows. This description is intended to confirm that the embodiments supporting the invention described in this specification are described in the specification. Therefore, even if there is an embodiment which is described in the specification but is not described here, this means that the embodiment does not correspond to the invention. It is not a thing. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean that all the inventions described in the specification are claimed. In other words, this description is for the invention described in the specification and not claimed in this application, i.e., for the invention that will be filed in division or applied or added in the future. It does not deny existence.

  The motion vector detection device according to claim 1 is a motion vector detection device (for example, the motion vector detection device 1 in FIG. 1) that detects a vector that represents the motion of an image, and a vector candidate that represents the motion of the image. Position detection means (for example, the vector position detection unit 61 in FIG. 4) that detects the pointing positions of a plurality of candidate vectors, and pixels in a preset size region centered on the pixel that is the starting point of the candidate vector First extracting means for extracting a block (for example, the block extracting unit 62 in FIG. 4), and a plurality of preliminarily centered on a plurality of pixels close to the indicated position of the candidate vector detected by the position detecting means. A second extraction unit (for example, the block extraction units 63-1 to 63-4 in FIG. 4) that extracts pixel blocks in the set size region and the first extraction unit extract the pixel blocks. The value of the pixel in the pixel block thus obtained is compared with the value of each corresponding pixel in the plurality of pixel blocks extracted by the second extraction means, and the comparison result (for example, sum of absolute block differences) Based on a plurality of comparison results output by the comparison result output means (for example, the difference absolute value sum calculation units 64-1 to 64-4 in FIG. 4) and the comparison result output means. Based on the evaluation value output means (for example, the result output unit 65 of FIG. 4) for outputting the evaluation value (for example, the average value, the maximum value, or the minimum value), and the evaluation value output by the evaluation value output means , And a selection unit (for example, the selection unit 43 in FIG. 1 or the selection unit 124 in FIG. 11) that selects a vector having high accuracy from the plurality of candidate vectors.

  In the motion vector detection device according to claim 2, the second extraction unit is located at a position closest to the vertical and horizontal directions of a point (for example, point 82 in FIG. 6) representing the pointing position of the candidate vector. Four pixel blocks (for example, block 100A to block 100D in FIG. 7) centering on one pixel can be extracted.

  The motion vector detection apparatus according to claim 3, wherein the comparison result output unit includes the pixel block extracted by the first extraction unit and the four pixel blocks extracted by the second extraction unit. Each of the four difference absolute value sums (for example, DFD (A) to DFD (D)) is output as the comparison result, and the evaluation value output means has an average value, maximum value, or The minimum value can be output as the evaluation value.

  The motion vector detection device according to claim 4 obtains an initial vector (for example, an initial vector V0) as the plurality of candidate vectors, and performs an operation by a gradient method once or twice on the initial vector. The first or second vector (vector V1 or vector V2) to be input is input, and the selecting means uses the initial vector or the first or second vector having a high accuracy as the motion vector of the image. Can be selected.

  The motion vector detection apparatus according to claim 5, wherein the plurality of candidate vectors are candidate vectors of the initial vector, and the selection unit uses a plurality of candidate vectors having high accuracy as the initial vector. Can be selected.

  The motion vector detection method according to claim 6 is a motion vector detection method of a motion vector detection device (for example, the motion vector detection device 1 in FIG. 1) that detects a vector representing a motion of an image, and the motion of the image is detected. A position detection step (for example, step S21 in FIG. 5) for detecting the pointing positions of a plurality of candidate vectors that are candidates for the vector, and a preset size region centered on the pixel that is the starting point of the candidate vector A first extraction step (for example, step S22 in FIG. 5) for extracting the pixel block and a plurality of pixels centering on a plurality of pixels close to the pointing position of the candidate vector detected by the processing of the position detection step A second extraction step (for example, step S23 in FIG. 5) for extracting a pixel block of a predetermined size area, and the first extraction step. The pixel value in the pixel block extracted by the processing of the group is compared with the corresponding pixel value in the plurality of pixel blocks extracted by the processing of the second extraction step, and the comparison result (for example, The comparison result output step (for example, step S24 in FIG. 5) for outputting a plurality of block difference absolute value sums, and evaluation of the candidate vector based on the plurality of comparison results output by the processing of the comparison result output step Based on the evaluation value output step (for example, step S25 in FIG. 5) for outputting a value (for example, average value, maximum value, or minimum value) and the evaluation value output by the processing of the evaluation value output step, A selection step (for example, step S3 in FIG. 3 and steps S104 and S10 in FIG. 13) for selecting a vector having high accuracy from among a plurality of candidate vectors. 9 or step S126) of FIG.

  The program according to claim 7 is a program of a motion vector detection device (for example, the motion vector detection device 1 in FIG. 1) that detects a vector that represents the motion of an image, and a vector candidate that represents the motion of the image. A position detection control step (for example, step S21 in FIG. 5) for controlling the detection of the pointing positions of the plurality of candidate vectors, and a pixel block of a preset size region centered on the pixel that is the starting point of the candidate vector A first extraction control step (for example, step S22 in FIG. 5) for controlling to extract the pixel, and a plurality of pixels close to the pointing position of the candidate vector detected by the processing of the position detection control step. A second extraction control step (for example, the step of FIG. 5) that controls to extract a plurality of pixel blocks of a predetermined size area. S23), the value of the pixel in the pixel block extracted by the process of the first extraction control step, and the corresponding pixel in the plurality of pixel blocks extracted by the process of the second extraction control step And a comparison result output control step (for example, step S24 in FIG. 5) for performing control to output a plurality of comparison results (for example, sum of block difference absolute values), and processing of the comparison result output control step An evaluation value output control step (for example, the step of FIG. 5) that controls to output an evaluation value (for example, an average value, a maximum value, or a minimum value) of the candidate vector based on the plurality of comparison results output by A vector having high accuracy is selected from the plurality of candidate vectors based on S25) and the evaluation value output by the processing of the evaluation value output control step. Control selecting control step (e.g., for example, step S3 of FIG. 3, step S104 of FIG. 13, S109, or step S126 in FIG. 14) so that the causes the computer to execute.

  The recording medium according to claim 8 is a recording medium in which a program of a motion vector detection device (for example, the motion vector detection device 1 in FIG. 1) for detecting a vector representing a motion of an image is recorded. A position detection control step (for example, step S21 in FIG. 5) for controlling the detection of the pointing positions of a plurality of candidate vectors that are candidates for vectors representing the motion of the vector, and a preset centered on the pixel that is the starting point of the candidate vector A first extraction control step (for example, step S22 in FIG. 5) that controls to extract a pixel block of the size region that has been set, and the pointing position of the candidate vector detected by the processing of the position detection control step A second extraction control step for controlling to extract a plurality of pixel blocks of a predetermined size area centered on a plurality of adjacent pixels; For example, in step S23) of FIG. 5, the value of the pixel in the pixel block extracted by the process of the first extraction control step, and the plurality of pixel blocks extracted by the process of the second extraction control step. A comparison result output control step (for example, step S24 in FIG. 5) for comparing the values of the corresponding pixels with each other and controlling to output a plurality of comparison results (for example, sum of block difference absolute values), and the comparison An evaluation value output control step for controlling to output an evaluation value (for example, an average value, a maximum value, or a minimum value) of the candidate vector based on a plurality of comparison results output by the processing of the result output control step ( For example, based on the evaluation value output in step S25) of FIG. 5 and the processing of the evaluation value output control step, the accuracy is selected from the plurality of candidate vectors. Selection control step of controlling to select a higher vector (e.g., for example, step S3 of FIG. 3, step S104 of FIG. 13, S109, or step S126 in FIG. 14) programs to be executed and to a computer is recorded.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a configuration example of a motion vector detection apparatus to which the present invention is applied. The motion vector detection device 1 outputs, for example, a motion vector that represents the motion of an image from input image data of the moving image. In this example, an initial vector that detects (selects) an initial vector from the image data. A selection unit 21 and a motion vector selection unit 22 that generates (selects) a motion vector based on the initial vector output from the initial vector selection unit 21 are provided.

  FIG. 2 is a block diagram illustrating a detailed configuration example of the initial vector detection unit 21. The candidate vector detection unit 41 detects a plurality of candidate vectors as initial vector candidates based on the input image data, and outputs the detected candidate vectors to a DFD (Displaced Frame Difference) calculation unit 42. For the detection of the candidate vector, for example, a vector close to the motion of the image is detected based on a past detection result or the like.

  Based on the candidate vector output from the candidate vector detection unit 41, the DFD calculation unit 42 sets a block having a preset size in a temporally previous frame in the image data and a temporally subsequent frame. Blocks having the same size are extracted, a block difference absolute value sum (described later) between frames is calculated, and the calculation result is output to the selection unit 43.

  The selection unit 43 selects and outputs the initial vector V0 from among the plurality of vectors detected by the candidate vector detection unit 41 based on the calculation result of the block difference absolute value sum between frames by the DFD calculation unit 42.

  Next, the operation of the initial vector detection unit 21 will be described with reference to the flowchart of FIG.

  In step S1, the candidate vector detection unit 41 detects a plurality of candidate vectors. In step S2, the DFD calculation unit 42 performs block difference absolute value sum calculation processing described later. Thereby, the block difference absolute value sum between frames of the temporally previous frame and temporally subsequent frame in the image data is calculated.

  Here, the block difference absolute value sum calculation processing will be described in detail with reference to FIGS. 4 to 7. FIG. 4 is a block diagram illustrating a more detailed configuration example of the DFD calculation unit 42. The plurality of candidate vectors output from the candidate vector detection unit 41 are input to the vector position detection unit 61 together with the image data. The vector position detection unit 61 detects a pixel position (a vector pointing position) corresponding to the candidate vector based on the target pixel and the input candidate vector. Specifically, the motion of the candidate vector is calculated for the pixel (target pixel) that is the starting point of the candidate vector in the temporally previous frame (eg, frame n−1) in the image data. The pixel position (vector pointing position) of the pixel of interest in the temporally subsequent frame (for example, frame n) is detected.

  The block extraction unit 62 is a predetermined block around the pixel of interest in the frame n−1 where the pixel of interest exists in the image data, for example, three in the vertical direction (vertical direction in the figure) centered on the pixel of interest, A block of nine pixels is extracted in the horizontal direction (horizontal direction in the figure). The block extraction units 63-1 to 63-4 have the same size as the block extracted by the block extraction unit 62 (9 pixels in this case) for the pixels existing around the pixel of interest in the frame n. Four blocks are extracted. A method for extracting four blocks around the pixel of interest in frame n will be described later.

  The difference absolute value sum calculation units 64-1 to 64-4 are configured to obtain the corresponding pixels between the blocks extracted by the block extraction units 63-1 to 63-4 and the blocks extracted by the block extraction unit 62. The sum of absolute values of the differences is calculated, and the result of each calculation is output to the result output unit 65.

  The result output unit 65 calculates the average value, maximum value, or minimum value of the four block difference absolute value sums based on the output results from the difference absolute value sum calculation units 64-1 to 64-4, and the calculation The result is output as a calculation result of the DFD calculation unit 42 in association with the input candidate vector.

  Next, details of the block difference absolute value sum calculation process in step S2 of FIG. 3 will be described with reference to the flowchart of FIG.

  In step S21, the vector position detector 61 detects the pointing position of the input vector.

  Here, the pointing position of the vector will be described with reference to FIG. FIG. 6A and FIG. 6B show the arrangement of pixels in a predetermined portion in frame n-1 and frame n, respectively. In FIG. 6A, pixels m11 to m14 are arranged in order from the upper left. . Below the pixels m11 to m14, pixels m21 to m24, pixels m31 to m34, and pixels m41 to m44 are arranged. Similarly in FIG. 6B, pixels n11 to 14, pixels n21 to n24, pixels n31 to n34, and pixels n41 to n44 are arranged in order from the upper left. For example, in the frame n−1 of the input image, when the starting point 81 of the candidate vector is the pixel m22 indicated by the black circle in FIG. 6A, the motion of the candidate vector is calculated for this pixel, A pixel position corresponding to the target pixel is detected as a point 82 indicated by a cross in FIG. 6B.

  In step S <b> 22, the block extraction unit 62 extracts a block of pixels centered on the pixel of interest (the pixel that is the starting point of the candidate vector in the frame n−1 of the input image). At this time, 3 × 3 = 9 pixels m11 to m33 are extracted as a block 80 with the pixel m22 of FIG.

  In step S23, the block extraction units 63-1 to 63-4 extract a block of pixels centered on four pixels close to the point 82 that is the vector pointing position. In FIG. 6B, the point 82 that is the pointing position of the vector does not coincide with the pixel position (positioned between the pixels indicated by the circles in the figure). 3 = 9 blocks cannot be extracted. For this reason, as shown in FIGS. 7A to 7D, a block centered on four pixels close to the pointing position of the vector is extracted.

  A block 100A in FIG. 7A is a block configured by 3 × 3 = 9 pixels n11 to n33 centering on the pixel n22 that is closest to the upper left side of the point 82 that is the pointing position of the vector. Extracted by the extraction unit 63-1. The block 100B in FIG. 7B is a block configured by 3 × 3 = 9 pixels n12 to n34 centering on the pixel n23 that is closest to the upper right side of the point 82 that is the vector pointing position. Extracted by the extraction unit 63-2.

  A block 100C in FIG. 7C is a block configured by 3 × 3 = 9 pixels n21 to n43 centering on the pixel n32 that is closest to the lower left side of the point 82 that is the pointing position of the vector. Extracted by the extraction unit 63-3. The block 100D in FIG. 7D is a block configured by 3 × 3 = 9 pixels n22 to n44 centering on the pixel n33 that is closest to the lower right side of the point 82 that is the pointing position of the vector, Extracted by the block extraction unit 63-4.

  When the pointing position of the vector in the frame n matches the pixel position, the block extracting units 63-1 to 63-4 extract the same block centered on the pixel.

  In step S24, the difference absolute value sum calculation units 64-1 to 64-4 calculate the sum of absolute differences between the block 80 and the four blocks 100A to 100D. For example, the values of the nine pixels m11 to m33 of the block 80 are P (1) to P (9), respectively, and the values of the nine pixels n11 to n33 of the block 100A are respectively In the case of Q (1) to Q (9), the difference absolute value sum calculation unit 64-1 calculates the block difference absolute value sum DFD (A) of the block 80 and the block 100A by the following equation.

  DFD (A) = │P (1) -Q (1) │ ＋ │P (2) -Q (2) │ ＋ │P (3) -Q (3) │ + ・ ・ ・ │P (9)- Q (9) │

  Similarly, the difference absolute value sum calculation unit 64-2 is the block difference absolute value sum DFD (B) of the block 80 and the block 100B, and the difference absolute value sum calculation unit 64-3 is the block of the block 80 and the block 100C. The difference absolute value sum DFD (C), and the difference absolute value sum calculation unit 64-4 calculate the block difference absolute value sum DFD (D) of the block 80 and the block 100D, respectively.

  In step S25, the result output unit 65 calculates an average value, a maximum value, and a minimum value of the four block difference absolute value sums DFD (A) to DFD (D).

  Note that the processing in steps S21 to S25 is executed for each input candidate vector. For example, when three candidate vectors are input, in step S25, the average value of the sum of absolute block differences corresponding to each vector. The maximum value and the minimum value are calculated, and three types of combinations of the average value, the maximum value, and the minimum value are calculated.

  In step S26, the result output unit 65 outputs the average value, the maximum value, and the minimum value of the block difference absolute value sum in association with the input candidate vector. Note that the average value, the maximum value, and the minimum value of the block difference absolute value sum are used as evaluation values of the candidate vectors.

  In this way, the block difference absolute value sum is calculated. As for the block of pixels in the temporally subsequent frame (frame n), a block of pixels centering on the four pixels close to the vector pointing position is extracted. Even if they do not match, the block difference absolute value sum can be obtained. In addition, the circuit scale becomes larger due to an increase in decimal precision bit length, etc., compared to the case of calculating the virtual pixel value based on the pixels around the vector pointing position and obtaining the block difference absolute value sum, Problems such as complexity are eliminated, and as a result, a high-quality device can be realized at low cost.

  Here, an example in which a block of pixels each composed of nine pixels is extracted in frame n and frame n−1 has been described, but the number of pixels in the extracted block is limited to nine. The number of pixels may be changed as necessary.

  Returning to FIG. 3, in step S3, the selection unit 43 executes a vector selection process. Here, the details of the vector selection process in step S3 in FIG. 3 will be described with reference to the flowcharts in FIGS.

  FIG. 8 is a flowchart for explaining vector selection processing 1 which is a first example of vector selection processing. In step S41, the selection unit 43 compares the average values of the block difference absolute value sums corresponding to the candidate vectors, and in step S42, selects the candidate vector having the smallest average value of the block difference absolute value sums as the initial vector V0. To do.

  As described above, the block difference absolute value sums DFD (A) to DFD (D) are the difference absolute value sums between the blocks in the frame n−1 and the frame n, respectively. Therefore, the average value of these block difference absolute value sums is It is considered that the smaller the value is, the more in common the image of the block of frame n-1 and the image of the block of frame n, and the candidate vector corresponding to the average value of the sum of absolute differences of the blocks is more suitable for the motion of the image. This is considered to be reflected (high accuracy). For this reason, in the vector selection process 1, the average value of the block difference absolute value sum is used as the evaluation value of the candidate vector.

  FIG. 9 is a flowchart for explaining vector selection processing 2 as a second example of vector selection processing. In step S61, the selection unit 43 compares the maximum value of the block difference absolute value sum corresponding to each candidate vector, and in step S62, selects the candidate vector having the smallest maximum value of the block difference absolute value sum as the initial vector V0. To do.

  Since the block difference absolute value sums DFD (A) to DFD (D) are the sum of absolute differences between the blocks in the frame n−1 and the frame n corresponding to the pointing positions of the candidate vectors, the sum of these block difference absolute values. The smaller the maximum value of, the smaller the difference between the block n-1 image and the frame n block image, and the candidate vector corresponding to the maximum block difference absolute value sum is It is thought that it accurately reflects the movement (high accuracy). For this reason, in the vector selection process 2, the maximum value of the block difference absolute value sum is used as the evaluation value of the candidate vector.

  FIG. 10 is a flowchart for explaining vector selection processing 3 as a third example of vector selection processing. In step S81, the selection unit 43 compares the minimum value of the block difference absolute value sum corresponding to each candidate vector, and in step S82, selects the candidate vector having the smallest minimum value of the block difference absolute value sum as the initial vector V0. To do.

  Since the block difference absolute value sums DFD (A) to DFD (D) are the sum of absolute differences between the blocks in the frame n−1 and the frame n corresponding to the pointing positions of the candidate vectors, the sum of these block difference absolute values. The smaller the minimum value of, the smaller the difference between the image of the block in frame n-1 and the image of the block in frame n (there was almost a match), corresponding to the minimum value of the sum of absolute block differences The candidate vector to be considered is considered to accurately reflect the motion of the image (highly accurate). Therefore, in the vector selection process 3, the minimum value of the block difference absolute value sum is used as the evaluation value of the candidate vector.

  In this way, the initial vector V0 is selected. Since the initial vector V0 is selected based on the average value, maximum value, or minimum value of the block difference absolute value sums DFD (A) to DFD (D), the initial vector V0 is incorrect based on the local solution that is locally minimum. It is possible to suppress selection and the like and select an accurate initial vector. It should be noted that which of vector selection processing 1 to vector selection processing 3 is executed may be selected by the user in consideration of image characteristics or the like, or may be set in advance.

  Returning to FIG. 3, in step S4, the selection unit 43 outputs the initial vector V0 selected in step S3.

  Next, the configuration and operation of the motion vector selection unit 22 will be described. FIG. 11 is a block diagram illustrating a detailed configuration example of the motion vector selection unit 22. In this example, the motion vector selection unit 22 performs the calculation by the gradient method once or twice for the initial vector V0 output from the initial vector selection unit 21, and finally outputs the motion vector V.

  The initial vector V0 is input to the first gradient method calculation unit 121, and the vector V1 is output as a result of the calculation of the first gradient method calculation unit 121. The pixel value used in the second gradient method is interpolated by selecting the pixel block of the pixel corresponding to the pointing position of the vector V1 by the pixel value interpolation unit 125 and the DFD calculation unit 123. Then, the vector V1 is input to the second gradient method calculation unit 122, and the vector V2 is output as a result of the calculation by the second gradient method calculation unit 122.

  Here, the gradient method will be described. In a moving image, a pixel value represented by coordinates (x, y, t) using a horizontal axis, a vertical axis, and a time axis is defined as g (x, y, t). When a pixel (x0, y0, t0) changes by (dx, dy, dt) during a very short time, the gradient in the horizontal, vertical, and time directions is changed to gx (x0, y0, t0), gy (x0, When expressed as y0, t0) and gt (x0, y0, t0), the value of the pixel after the change can be expressed as in Equation (1) using Taylor expansion.

  Here, if a pixel of interest in the image moves by vx in the horizontal direction and vy in the vertical direction after one frame,

となり、式（１）より、式（３）を導出することができる。

Thus, Equation (3) can be derived from Equation (1).

  Since the equation (3) is a two-variable equation of vx and vy, the solution cannot be obtained with a single equation for one pixel of interest. Therefore, the peripheral region of the target pixel is considered as one processing unit, and it is assumed that all the pixels in the region have the same movement (vx, vy), and the same formula is established for each pixel. Then, the motions (vx, vy) are obtained so that the sum of squares of the inter-frame differences of all the pixels in the region is minimized by simultaneously formulating the peripheral pixels.

  The inter-frame difference d when the pixel (x, y, t) moves by (vx, vy) between one frame is expressed by Expression (4).

  Here, Δx = g (x, y, t), which represents a horizontal gradient. The same applies to Δy and Δt. Using these, the square sum E of the interframe difference is expressed as shown in Equation (5).

Here, E is the smallest (vx, vy) when the partial differential value in each variable becomes 0, that is,
Since the condition of ∂E / ∂vx = ∂E / ∂vy holds, from equation (5),

It becomes. The movement (vx, vy) can be obtained as in Expression (8) from Expression (6) and Expression (7).

  In this way, the obtained vector V1 or V2 is output to the DFD computing unit 123 together with the initial vector V0 or vector V1, respectively. The DFD calculation unit 123 has the same configuration as the DFD calculation unit 42 shown in FIG. 4, and as shown in FIG. 12, the pointing position 171 of the initial vector V0 and the pointing of the vector V1 (V0 + V1) in the frame n A position 172 or a pointing position 173 of the vector V2 (V0 + V1 + V2) is detected, and a block centered on the pixel of interest 151 in the frame n−1 and four pixels around the pointing positions 171 to 173 are centered. Are calculated and outputted to the selection unit 124. The selection unit 124 selects, as the motion vector V, the vector that is estimated to have the highest accuracy among the initial vector V0, the vector V1 (V0 + V1), or the vector V2 (V0 + V1 + V2) and outputs the selected vector.

  Next, the operation of the motion vector selection unit 22 will be described with reference to the flowchart of FIG.

  In step S101, the first gradient method computing unit 121 performs a gradient method computation on the initial vector V0 to obtain a vector V1. In step S102, the pixel value interpolation unit 125 detects the pointing position (pointing position 172 in FIG. 12) of the vector V1 (V0 + V1).

  In step S103, the DFD calculation unit 123 executes block difference absolute value sum calculation processing. Since this process is the same as the process described above with reference to the flowchart of FIG. 5, detailed description is omitted, but here, unlike the above-described case, instead of a plurality of candidate vectors, an initial vector V0 and The vector V1 is input, and four block difference absolute value sums DFD (A) to DFD (D) are calculated (step S24), and the block difference absolute value sums DFD (A) to DFD (D) are average and maximum values. A value and a minimum value are respectively calculated (step S25). Then, a combination of the average value, the maximum value, and the minimum value of the block difference absolute value sum is output in association with the two vectors of the initial vector V0 and the vector V1 (step S26).

  In step S104, the vector selection unit 124 executes vector selection processing. Since this process is the same as the process described above with reference to FIGS. 8 to 10, detailed description thereof will be omitted. However, unlike the case described above, the initial vector V0 is used instead of the plurality of candidate vectors. The vector that is estimated to have the highest accuracy is selected from the two vectors V1.

  In step S105, the selection unit 124 determines whether or not the initial vector V0 has been selected as a result of the process in step S105. If it is determined in step S105 that the initial vector V0 has not been selected, the result is output to the DFD calculation unit 123, and the process proceeds to step S106 in order to perform the second gradient method calculation.

  In step S106, the pixel value interpolation unit 125 calculates the minimum value among the four block difference absolute value sums DFD (A) to DFD (D) of the vector V1 calculated by the block difference absolute value sum calculation process in step S103. Is selected as the pixel block at the position indicated by the vector V1, and is output to the second gradient method computing unit 122.

  For example, when the pixel blocks corresponding to the four block difference absolute value sums DFD (A) to DFD (D) of the vector V1 are the blocks 100A to 100D in FIGS. 7A to 7D, respectively, the four block difference absolute value sums When the smallest value among the DFD (A) to DFD (D) is the block difference absolute value sum DFD (A), the block 100A is selected as the pixel block corresponding to the minimum value, and the block 100A The center pixel n22 is interpolated as the pointing position of the vector V1, and is used for the second calculation of the gradient method.

  In step S107, the second gradient method computing unit 122 performs the second gradient method computation based on the pixel block selected in step S106 and the vector V1 obtained in step S101, and obtains the vector V2 as the vector V2. Ask.

  In step S108, the DFD calculation unit 123 executes block difference absolute value sum calculation processing. Since this process is the same as the process described above with reference to the flowchart of FIG. 5, detailed description thereof is omitted, but here, unlike the case described above, instead of a plurality of candidate vectors, the vector V1 and the vector V2 is input, and four block difference absolute value sums DFD (A) to DFD (D) are calculated (step S24), and the average value and the maximum value of the block difference absolute value sums DFD (A) to DFD (D) are calculated. , And the minimum value are respectively calculated (step S25). Then, a combination of the average value, the maximum value, and the minimum value of the block difference absolute value sum is output in association with the two vectors of the vector V1 and the vector V1 (step S26).

  In step S109, the vector selection unit 124 executes vector selection processing. Since this process is the same as the process described above with reference to FIGS. 8 to 10, detailed description thereof will be omitted. However, unlike the case described above, instead of a plurality of candidate vectors, the vector V1, Of the two vectors V2, the vector that is estimated to have the highest accuracy is selected.

  In step S110, the vector selection unit 124 outputs the vector selected by the process in step S109 as a motion vector V.

  If it is determined in step S105 that the initial vector V0 has been selected, the processes in steps S106 to S109 are skipped, and the initial vector V0 is directly output as the motion vector V in step S110.

  In this way, the motion vector V representing the motion of the image is detected. The motion vector V is the largest of the initial vector V0, the vector V1 obtained by performing an arithmetic operation on the initial vector, or the vector V2 obtained by further performing an arithmetic operation on the vector V1. Since the one with high accuracy is selected and output, the motion of the image can be detected more accurately.

  In this example, in order to perform the gradient method, block difference absolute value sum calculation processing is performed and pixel values (pixel blocks) used in the gradient method are interpolated. However, the pixel values used in the gradient method are other values. Interpolation may be performed by this method. FIG. 14 is a flowchart illustrating another example of the motion vector selection process.

  That is, in step S121, the first gradient method computing unit 121 performs a gradient method operation on the initial vector V0 to obtain a vector V1. In step S122, the pixel value interpolation unit 125 detects the pointing position (pointing position 172 in FIG. 12) of the vector V1 (V0 + V1), and in step 123, the pixel value at the pointing position is interpolated. At this time, the interpolation of the pixel value is performed as follows, for example.

  Assume that the pointing position of the vector V1 in the frame n is detected as a point q indicated by a black circle in FIG. Around the point q, there is a pixel l0 on the upper left, a pixel l1 on the upper right, a pixel l2 on the lower left, and a pixel l3 on the lower right. If the distance between the pixels is 1, the point q is separated from the vertical line (parallel to the y-axis) in the figure connecting the pixels l0 and l2 by the distance α in the horizontal direction (x-axis direction), and the pixel l1 Is separated from the vertical line in the figure connecting L3 and l3 by a distance 1-α in the x-axis direction. The point q is separated from the horizontal line (parallel to the x-axis) in the figure connecting pixels l0 and l1 by a distance β in the vertical direction (y-axis direction), and is horizontal in the figure connecting pixels l2 and l3. It is a distance 1-β away from the line in the y-axis direction.

  Assuming that the pixel values of the pixels l0 to l3 are L0 to L3, the pixel value interpolation unit 125 calculates and interpolates the value Q of the pixel at the pointing position by the following equation.

  Q = (1-α) (1-β) L0 + α (1-β) L1 + (1-α) βL2 + αβL3

  Similarly, in FIG. 16, pixel values at predetermined positions around a point q indicated by a black circle in the drawing are also interpolated based on the pixel values of the pixels indicated by white circles in the drawing.

  Returning to FIG. 14, in step S124, the second gradient method computing unit 122 performs the second gradient method computation based on the pixel value interpolated in step S123 and the vector V1 obtained in step S121. To obtain the vector V2.

  In step S125, the DFD calculation unit 123 executes a block difference absolute value sum calculation process. Since this process is the same as the process described above with reference to the flowchart of FIG. 5, detailed description thereof is omitted, but here, unlike the case described above, the initial vectors V0, Three vectors of vectors V1 and V2 are input, and four block difference absolute value sums DFD (A) to DFD (D) corresponding to each vector are calculated (step S24), and block difference absolute value sum DFD (A ) To DFD (D), the average value, the maximum value, and the minimum value are respectively calculated (step S25). Then, a combination of the average value, the maximum value, and the minimum value of the block difference absolute value sum is output in association with the initial vector V0 and the three vectors V1 and V2 (step S26).

  In step S126, the selection unit 124 executes vector selection processing. Since this process is the same as the process described above with reference to FIGS. 8 to 10, detailed description thereof will be omitted. However, unlike the case described above, the initial vector V0 is used instead of the plurality of candidate vectors. The vector estimated to have the highest accuracy among the three vectors V1 and V2 is selected as the motion vector V.

  In step S127, the selection unit 124 outputs the motion vector V.

  In this way, the motion vector V representing the motion of the image is detected.

  It does not matter whether the above-described series of processing is realized by hardware or software. When the above-described series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, a general-purpose personal computer as shown in FIG. 17 is installed from a network or a recording medium.

  In FIG. 17, a CPU (Central Processing Unit) 901 executes various processes according to a program stored in a ROM (Read Only Memory) 902 or a program loaded from a storage unit 908 into a RAM (Random Access Memory) 903. To do. The RAM 903 also appropriately stores data necessary for the CPU 901 to execute various processes.

  The CPU 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input / output interface 905 is also connected to the bus 904.

  The input / output interface 905 includes an input unit 906 including a keyboard and a mouse, a display (display unit) including a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal display), an output unit 907 including a speaker, a hard disk, and the like A storage unit 908 composed of a communication unit 909 composed of a modem and a terminal adapter is connected. The communication unit 909 performs communication processing via a network such as the Internet.

  A drive 910 is also connected to the input / output interface 905 as required. For example, a removable medium 911 is mounted on the drive 910 as a recording medium on which the program of the present invention is recorded, and read from them. A computer program is installed in the storage unit 908 as necessary.

  Note that the steps of executing the series of processes described above in this specification are performed in parallel or individually even if they are not necessarily processed in time series, as well as processes performed in time series in the order described. The processing to be performed is also included.

It is a block diagram which shows the structural example of the motion vector detection apparatus to which this invention is applied. It is a block diagram which shows the detailed structural example of the initial vector selection part of FIG. It is a flowchart explaining an initial vector selection process. FIG. 3 is a block diagram illustrating a detailed configuration example of a DFD calculation unit in FIG. 2. It is a flowchart explaining block difference absolute value sum calculation processing. It is a figure which shows the starting point and pointing position of a vector. It is a figure which shows the block centering on the pixel around the pointing position of a vector. 10 is a flowchart for explaining vector selection processing 1; It is a flowchart explaining the vector selection process 2. FIG. 10 is a flowchart for explaining vector selection processing 3; FIG. 2 is a block diagram illustrating a detailed configuration example of a motion vector selection unit in FIG. 1. It is a figure which shows the pointing position of the vector calculated | required by the gradient method. It is a flowchart explaining a motion vector selection process. It is a flowchart explaining the other example of a motion vector selection process. It is a figure explaining the interpolation of a pixel value. It is a figure which shows the position where a pixel value is interpolated. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Motion vector detection apparatus, 21 Initial vector selection part, 22 Motion vector selection part, 41 Candidate vector detection part, 42 DFD calculation part, 43 Selection part, 61 Vector position detection part, 62 Block extraction part, 63-1 thru | or 63- 4 block extraction unit, 64-1 to 64-4 absolute difference sum calculation unit, 65 result output unit, 123 DFD calculation unit, 124 selection unit, 125 pixel value interpolation unit

Claims (8)

A motion vector detection device for detecting a vector representing a motion of an image,
Position detecting means for detecting the pointing positions of a plurality of candidate vectors that are candidates for vectors representing the movement of the image;
First extraction means for extracting a pixel block of a preset size region centered on a pixel serving as a starting point of the candidate vector;
Second extraction means for extracting a plurality of pixel blocks of a predetermined size area centered on a plurality of pixels close to the position indicated by the candidate vector detected by the position detection means;
The pixel value in the pixel block extracted by the first extraction unit is compared with the corresponding pixel value in the plurality of pixel blocks extracted by the second extraction unit, and the comparison result is obtained. Comparison result output means for outputting a plurality of results,
Evaluation value output means for outputting an evaluation value of the candidate vector based on a plurality of comparison results output by the comparison result output means;
A motion vector detection apparatus comprising: selection means for selecting a vector having high accuracy from the plurality of candidate vectors based on the evaluation value output by the evaluation value output means. The second extraction unit extracts four pixel blocks centering on four pixels located closest to the up, down, left, and right directions of a point representing the pointing position of the candidate vector. 2. The motion vector detection device according to 1. The comparison result output means outputs four difference absolute value sums for the pixel block extracted by the first extraction means and the four pixel blocks extracted by the second extraction means, respectively, and the comparison result. Output as
The motion vector detection device according to claim 2, wherein the evaluation value output means outputs an average value, a maximum value, or a minimum value of the four difference absolute value sums as the evaluation value. As the plurality of candidate vectors, an initial vector, a first or second vector obtained by performing an operation by a gradient method once or twice on the initial vector, are input,
The motion vector detection apparatus according to claim 1, wherein the selection unit selects a highly accurate one of the initial vector and the first or second vector as a motion vector of the image. The plurality of candidate vectors are candidate vectors of the initial vector;
The motion vector detection apparatus according to claim 4, wherein the selection unit selects, as the initial vector, a highly accurate one of the plurality of candidate vectors. A motion vector detection method of a motion vector detection device for detecting a vector representing motion of an image, comprising:
A position detecting step of detecting a pointing position of a plurality of candidate vectors that are candidates for vectors representing the movement of the image;
A first extraction step of extracting a pixel block of a preset size region centered on a pixel serving as a starting point of the candidate vector;
A second extraction step of extracting a plurality of pixel blocks of a predetermined size area centered on a plurality of pixels close to the pointing position of the candidate vector detected by the processing of the position detection step;
The value of the pixel in the pixel block extracted by the process of the first extraction step is compared with the value of the corresponding pixel in the plurality of pixel blocks extracted by the process of the second extraction step. A comparison result output step for outputting a plurality of comparison results;
An evaluation value output step of outputting an evaluation value of the candidate vector based on a plurality of comparison results output by the processing of the comparison result output step;
And a selection step of selecting a vector having high accuracy from the plurality of candidate vectors based on the evaluation value output by the processing of the evaluation value output step. A motion vector detection device program for detecting a vector representing a motion of an image,
A position detection control step for controlling detection of pointing positions of a plurality of candidate vectors that are candidates for vectors representing the motion of the image;
A first extraction control step for controlling to extract a pixel block of a preset size region centered on a pixel that is a starting point of the candidate vector;
A second control is performed to extract a plurality of pixel blocks of a predetermined size area centered on a plurality of pixels close to the pointing position of the candidate vector detected by the processing of the position detection control step. An extraction control step;
The value of the pixel in the pixel block extracted by the process of the first extraction control step and the value of the corresponding pixel in the plurality of pixel blocks extracted by the process of the second extraction control step A comparison result output control step for comparing and controlling to output a plurality of comparison results;
An evaluation value output control step for controlling to output an evaluation value of the candidate vector based on a plurality of comparison results output by the processing of the comparison result output control step;
And a selection control step for controlling the computer to select a vector having high accuracy from the plurality of candidate vectors based on the evaluation value output by the processing of the evaluation value output control step. Program to do. A recording medium on which a program of a motion vector detecting device for detecting a vector representing a motion of an image is recorded,
A position detection control step for controlling detection of pointing positions of a plurality of candidate vectors that are candidates for vectors representing the motion of the image;
A first extraction control step for controlling to extract a pixel block of a preset size region centered on a pixel that is a starting point of the candidate vector;
A second control is performed to extract a plurality of pixel blocks of a predetermined size area centered on a plurality of pixels close to the pointing position of the candidate vector detected by the processing of the position detection control step. An extraction control step;
The value of the pixel in the pixel block extracted by the process of the first extraction control step and the value of the corresponding pixel in the plurality of pixel blocks extracted by the process of the second extraction control step A comparison result output control step for comparing and controlling to output a plurality of comparison results;
An evaluation value output control step for controlling to output an evaluation value of the candidate vector based on a plurality of comparison results output by the processing of the comparison result output control step;
Based on the evaluation value output by the processing of the evaluation value output control step, a program for causing the computer to execute a selection control step for controlling to select a vector with high accuracy from the plurality of candidate vectors is recorded. A recording medium.

Images