JP2017027363A - Video processing device, video processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve tracking accuracy of a feature point of a video frame.SOLUTION: A video processing device comprises: an extraction part for extracting a feature point from a first frame; a tracking part for performing block matching with a block containing the feature point, with respect to the second frame, for specifying a tracking point corresponding to the feature point; a detection part for detecting a position of a first pixel which is the most similar to the feature point, similarity between the first pixel and the feature point, a position of a second pixel which is the second most similar to the feature point, and similarity between the second pixel and the feature point; a first determination part for, when the position of the first pixel and the tracking point position match, and difference between the similarity of the first pixel and the similarity of the second pixel is equal to or larger than a first threshold, determining that the first condition is satisfied; and a second determination part for, calculating similarity between the tracking point and the feature point, and similarity between each pixel out of the plural pixels on the surrounding of the tracking point, and the feature point, and when difference between the similarity between the tracking point and the feature point, and the similarity between each pixel out of the plural pixels on the surrounding of the tracking point and the feature point, is equal to or larger than a second threshold, determining that the second condition is satisfied. When the first and second conditions are satisfied, it is determined that the feature point corresponds to the tracking point.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a video processing apparatus, a video processing method, and a program.

  One of the technologies used for autonomous driving of automobiles is a technology for estimating self-location (SLAM: Simulatively Localization And Mapping), and a method for accurately estimating self-location using SLAM has been widely studied. . In SLAM, a feature point is extracted from video data photographed by a camera or the like, and the self-position is estimated by tracking the feature point.

  As a feature point tracking method, block matching for tracking feature points by comparing similarities of predetermined regions between frames of a video is widely known.

  In block matching, tracking of feature points may be mistaken due to a calculation error in comparison of similarities. For this reason, various methods for improving the accuracy of tracking feature points using pixel attributes such as luminance have been proposed (for example, Patent Document 1).

  However, even if the proposed method is used, if similar pixels are distributed around the tracked feature point, there is a problem that the feature point cannot be accurately tracked.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to improve the accuracy of tracking feature points of video frames.

  The video processing apparatus according to the present embodiment includes an extraction unit that extracts a feature point from a first video frame, and a second video frame that is a video frame temporally later than the first video frame. Tracking for identifying a tracking point corresponding to the feature point by performing block matching processing with a block that is a part of the frame of the first video and includes the feature point for a predetermined region of And the position of the first pixel most similar to the feature point in the predetermined area, the similarity between the first pixel and the feature point, and the second pixel second closest to the feature point And the detection unit for detecting the similarity between the second pixel and the feature point, the tracking point position matches the first pixel position, and the first pixel similarity And the difference between the degree of similarity and the second pixel similarity is greater than or equal to a first threshold value In some cases, a first determination unit that determines that the first condition is satisfied, a similarity between the tracking point and the feature point, and a similarity between each of a plurality of surrounding pixels of the tracking point and the feature point When the difference between the calculated similarity of the tracking points and the similarity of each of the plurality of surrounding pixels is equal to or greater than a second threshold, it is determined that the second condition is satisfied. When the first condition and the second condition are satisfied, the tracking unit determines that the feature point corresponds to the tracking point.

  According to the present embodiment, it is possible to provide a video processing apparatus, a video processing method, and a program with improved accuracy of tracking feature points of video frames.

It is a functional block diagram of the video processing device concerning one embodiment. It is a functional block diagram of the tracking part of the video processing device concerning one embodiment. It is a flowchart which shows the operation | movement procedure which concerns on one Embodiment. It is a figure (the 1) which shows the process of the block matching which concerns on one Embodiment. It is FIG. (2) which shows the process of the block matching which concerns on one Embodiment. It is a figure which shows the calculation formula of the process of the block matching which concerns on one Embodiment. It is FIG. (1) for demonstrating the operation | movement of the 1st determination which concerns on one Embodiment. It is FIG. (1) which shows the calculation formula of the operation | movement of the 1st determination which concerns on one Embodiment. FIG. 10 is a diagram (No. 2) for explaining the operation of the first determination according to the embodiment; It is FIG. (2) which shows the calculation formula of the operation | movement of the 1st determination which concerns on one Embodiment. It is FIG. (3) for demonstrating the operation | movement of the 1st determination which concerns on one Embodiment. It is FIG. (The 3) which shows the calculation formula of the operation | movement of the 1st determination which concerns on one Embodiment. It is FIG. (4) for demonstrating the operation | movement of the 1st determination which concerns on one Embodiment. It is a flowchart which shows the operation | movement of the 1st determination which concerns on one Embodiment. It is FIG. (The 3) which shows the process of the block matching which concerns on one Embodiment. It is a figure for demonstrating the operation | movement of the 2nd determination which concerns on one Embodiment. It is a figure which shows the calculation formula of the operation | movement of the 2nd determination which concerns on one Embodiment. It is a hardware block diagram of the video processing apparatus which concerns on one Embodiment.

[First Embodiment]
<Functional configuration>
(1) Overall Configuration The functional configuration of the video processing apparatus 1 according to the present embodiment will be described using FIG. The video processing apparatus 1 includes a photographing unit 100, a video input unit 110, a video correction unit 120, a feature point extraction unit 130, a tracking unit 140, a frame buffer 150, and an overlapping point removal unit 160.

  The photographing unit 100 photographs a video with a camera. The captured video is output to the video input unit 110. The camera may be a monocular camera or a stereo camera. The camera of the photographing unit 100 may be an in-vehicle camera attached to a car.

  The video input unit 110 receives an input of a video shot by the shooting unit 100. Further, the video input unit 110 may accept video input from other than the photographing unit 100. For example, the video input unit 110 may accept input of video stored in a storage medium, or may accept video input via a network.

  The video correction unit 120 acquires a video frame from the video input unit 110 and corrects distortion of the video frame.

  The feature point extraction unit 130 extracts feature points of the video frame. A feature point is a point on a video frame that can be detected prominently, such as a corner of an object.

  The frame buffer 150 stores the video frame acquired from the video correction unit 120. The frame buffer 150 transmits the video frame to the tracking unit 140.

  The tracking unit 140 tracks the feature points extracted by the feature point extraction unit 130. Specifically, the tracking unit 140 extracts a tracking point corresponding to the feature point from a video frame temporally after the video frame from which the feature point is extracted. The tracking unit 140 acquires a video frame from the video correction unit 120.

  The tracking unit 140 acquires information on the feature points extracted in the video frame from the feature point extraction unit 130. The information on the feature point is, for example, the luminance at the feature point and coordinate information indicating the position of the feature point in the video frame. The tracking unit 140 acquires a video frame temporally prior to the video frame from which the feature points are extracted from the frame buffer 150. The tracking unit 140 tracks feature points from the acquired video frame and information on the feature points.

  A case where the tracking unit 140 tracks feature points extracted in the video frame F (M) at time M will be described.

  The tracking unit 140 acquires information on the feature points in the video frame F (M) at the time M from the feature point extraction unit 130. The tracking unit 140 acquires the video frame F (M + 1) at the time M + 1 from the video correction unit 120. The tracking unit 140 acquires the video frame F (M) from the frame buffer 150.

  The tracking unit 140 extracts a tracking point corresponding to the feature point in the video frame F (M + 1) from the video frame F (M) and information about the feature point in the video frame.

  When the feature point of the video frame F (M + 1) extracted by the feature point extraction unit 130 and the tracking point of the video frame F (M + 1) extracted by the tracking unit 140 overlap, Remove one.

(2) Configuration of Tracking Unit The tracking unit 140 and functional units related to the tracking unit 140 will be described in detail with reference to FIG.

  The tracking unit 140 includes a block matching unit 141, an extreme value detection unit 142, a unity determination unit 143, and a curved surface shape determination unit 145.

  The feature point extraction unit 130 extracts feature points from the frame of the first video.

  The block matching unit 141 performs block matching processing on a predetermined region of the second video frame that is a video frame temporally subsequent to the first video frame, and performs tracking points corresponding to the feature points. Is identified. When executing the block matching process, the block matching unit 141 uses a block that is a part of the frame of the first video and includes a feature point.

  The extreme value detection unit 142 is second in the predetermined region of the second video frame, the position of the first pixel most similar to the feature point, the similarity between the first pixel and the feature point, and the feature point. The position of the similar second pixel and the similarity between the second pixel and the feature point are detected.

  In the following description, the similarity means the similarity to the feature point. Therefore, for example, “similarity of pixel A” means “similarity between feature points and pixel A”.

  The extreme value detection unit 142 detects the first pixel and the second pixel, in addition to the similarity with the feature point of the pixel itself, the similarity between the pixel and the feature point, The first pixel and the second pixel may be detected in consideration of the difference in similarity between adjacent pixels and feature points.

  The unity determination unit 143 matches the position of the tracking point with the position of the first pixel, and the difference between the similarity of the first pixel and the similarity of the second pixel is equal to or greater than the first threshold. If there is, it is determined that the first condition is satisfied.

  The curved surface shape determination unit 145 calculates the similarity between the tracking point and the feature point, the similarity between each of the plurality of pixels around the tracking point and the feature point, and the calculated similarity between the tracking points, When the difference between each of the plurality of neighboring pixels is equal to or greater than the second threshold, it is determined that the second condition is satisfied.

  When the first condition and the second condition are satisfied, the tracking unit 140 determines that the feature point of the frame of the first video corresponds to the tracking point in the frame of the second video. In this case, the tracking unit 140 may determine that the feature point of the frame of the first video has moved to the tracking point in the frame of the second video.

  In addition to the block matching process, when the first condition and the second condition are satisfied, it is determined that the feature point corresponds to the tracking point. Therefore, the video processing apparatus 1 can accurately specify the tracking point. it can.

  The block matching unit 141 is an example of a tracking unit. The extreme value detection unit 142 is an example of a detection unit. The unity determination unit 143 is an example of a first determination unit. The curved surface shape determination unit 145 is an example of a second determination unit.

  When detecting the first pixel and the second pixel, the extreme value detection unit 142 may detect a plurality of similar pixels similar to the feature points in a predetermined region of the frame of the second video. . The plurality of similar pixels are more similar to feature points than neighboring pixels. These similar pixels may be expressed as pixels having extreme values. For example, a pixel that is more similar to the feature point than the adjacent eight pixels may be a pixel having an extreme value. The extreme value detection unit 142 may detect the first pixel and the second pixel from the pixels having the extreme value. In this case, the pixel having the extreme value most similar to the feature point is the first pixel, and the pixel having the extreme value second most similar to the feature point is the second pixel.

  When the video frame is a black and white video, the similarity to the feature point may be represented by the difference between the luminance at the feature point and the luminance of the pixel whose similarity is to be calculated. Pixels with such smaller differences are more similar to feature points.

  When the video frame is a color video, the similarity with the feature point may be represented by the difference between the total luminance value of each color at the feature point and the total luminance value of the pixels for which similarity is calculated. . Pixels with a smaller difference from the total value are more similar to feature points.

  The block matching unit 141 may specify a tracking point by performing a block matching process with the accuracy of a sub-pixel that is a unit smaller than a pixel. Further, the extreme value detection unit 142 may detect the first pixel and the second pixel with the accuracy of the sub-pixel.

  Since the tracking point is specified with the accuracy of the sub-pixel, the tracking point corresponding to the feature point can be accurately specified.

  The pixel may be expressed as a pixel, and the sub-pixel may be expressed as a sub-pixel.

  The extreme value detection unit 142 may interpolate the similarity between subpixels and feature points existing between adjacent pixels from the similarity between each of the pixels around the adjacent pixels and the feature points.

  For example, the extreme value detection unit 142 calculates the coordinates of the sub-pixel based on the change in the similarity between the pixel in the predetermined area and the pixel in the area corresponding to the predetermined area of the frame of the first video, The similarity between the sub-pixel and the feature point may be interpolated. When calculating the coordinates of the sub-pixel from the change tendency, the extreme value detection unit 142 may calculate the coordinates using a gradient method. The “pixels in the region corresponding to the predetermined region of the first video frame” are pixels in the same position as the pixels in the predetermined region.

  A specific interpolation procedure will be described later.

  The curved surface shape determination unit 145 forms a similarity distribution of the quadric surface from the similarity between the tracking point and the feature point and the similarity between the surrounding pixels and the feature point, and the similarity of the formed quadric surface Whether the second condition is satisfied may be determined from the degree distribution.

  For example, the curved surface shape determination unit 145 does not satisfy the second condition when the quadratic curved surface has a nearly flat shape, that is, when the point similar to the feature point is not distinguished from the surrounding pixels. You may judge. Thereby, erroneous detection of tracking points can be avoided.

  When at least one of the quadratic coefficients of the equation representing the distribution of the quadratic curved surface is equal to or greater than the third threshold value, the curved surface shape determining unit 145 may determine that the second condition is satisfied. This is because as the secondary coefficient increases, the curved surface becomes more conspicuous and the tracking point is more likely to correspond to the feature point.

  The extreme value detection unit 142 arranges pixels having extreme values in the order similar to the feature points, the pixel having the Nth similar extreme value to the feature points, and the pixel having the N + 1th extreme value similar to the feature points. The difference in similarity with the feature point is calculated. Here, N is a natural number equal to or greater than 2, and the maximum value of N + 1 is the number of pixels having extreme values included in a predetermined region of the frame of the second video.

  The unity determination unit 143 determines that the first condition is satisfied even when the difference between the similarity of the first pixel and the similarity of the second pixel is not equal to or greater than the first threshold. There are cases where it is good.

  For example, the difference between the similarity of the first pixel and the similarity of the second pixel is a pixel and a feature having an extreme value similar to the feature point included in a predetermined region of the frame of the second video. If the difference between the points and the pixel having the N + 1th most similar extreme value is greater than the similarity difference, it may be determined that the first condition is satisfied.

  The first pixel is the most similar to the feature point, and the difference in similarity between the first pixel and the second pixel is the N + 1th extreme point similar to the feature point, the feature point, and the N + 1th feature point. This is because the difference in degree of similarity with a pixel having a similar extreme value is larger, so that it can be determined that the feature point is remarkably similar to other pixels.

  When the first condition is not satisfied, the video processing device 1 may execute block matching processing and a determination as to whether or not the first condition is satisfied in an area obtained by enlarging a predetermined area of the second video frame. .

  The “predetermined area of the second video frame” may be referred to as a first search window.

<Operation procedure>
An example of the operation procedure of this embodiment will be described with reference to FIG.

  When specifying a tracking point corresponding to a feature point, the video processing device 1 executes processing in the order of block matching processing, first determination processing, and second determination processing.

  In step S300, the block matching unit 141 executes a block matching process. The block matching unit 141 extracts an image of an arbitrary size centering on the feature point extracted in the first video frame from the first video frame. The extracted image may be referred to as a template patch. Next, the block matching unit 141 moves the template patch within the first search window of the second video frame.

  The block matching unit 141 moves the template patch in the first search window, calculates the similarity between the template patch and a portion overlapping the template patch, and sets the center of the template patch at the most similar position as the tracking point. To do. A specific calculation method will be described later.

  If the block matching process is successful (step S301 YES), the process proceeds to step S302. On the other hand, if the block matching process is not successful (NO in step S301), the process proceeds to step S308.

  Steps S <b> 302 to S <b> 304 are an operation procedure of a first condition determination process.

  In step S302, the extreme value detection unit 142 performs a unity comparison value calculation process. Specifically, the extreme value detection unit 142 calculates the similarity between the pixel in the search window and the feature point. The similarity may be, for example, a difference in luminance between a feature point and a pixel in the search window. In this case, the smaller the similarity is, the more similar the pixel is to the feature point.

  The extreme value detection unit 142 extracts a pixel having a smaller degree of similarity than the surrounding pixels and sets it as an extreme value. For example, the extreme value detection unit 142 may extract an extreme value by comparing an arbitrary pixel with eight pixels around the arbitrary pixel.

  In the search window, the pixel having the smallest extreme value is the first pixel, and the pixel having the second smallest extreme value is the second pixel.

  In step S303, the extreme value detection unit 142 compares the similarity of the first pixel with the similarity of the second pixel, and compares whether the difference in similarity is equal to or greater than the first threshold. The first threshold is a value that can be set by the user of the video processing device 1. For example, a value such as 10% or 20% of the similarity of the first pixel may be set as the first threshold.

  In step S304, when the difference in similarity is equal to or greater than the first threshold and the position of the first pixel matches the tracking point extracted by block matching, the unity determination unit 143 performs unity determination. It determines with having succeeded and progresses to step S305 (step S304 YES). “Success in unity determination” may be expressed as “first condition is satisfied”.

  On the other hand, if the difference in similarity is not greater than or equal to the first threshold value, or the position of the first pixel does not match the tracking point, the unity determination unit 143 determines that unity determination has failed and step S308. (Step S308 NO).

  Steps S <b> 305 to S <b> 307 are operation procedures for determining the second condition.

  In step S305, the curved surface shape determination unit 145 calculates the similarity between the tracking point and the feature point and the similarity between the pixel around the tracking point and the feature point, and the calculated tracking point similarity, A quadric surface distribution is formed from the similarity of surrounding pixels.

  In step S306, the curved surface shape determination unit 145 determines whether the shape stands out. For example, if at least one of the quadratic coefficients of the equation representing the distribution of the quadratic curved surface is equal to or greater than the third threshold, the curved surface shape determining unit 145 determines that the shape is outstanding.

  This is because when the secondary coefficient is large, the shape of the curved surface is not flat and the shape stands out. The secondary coefficient represents the difference between the similarity of the surrounding pixels and the similarity of the tracking points. The third threshold is a value that can be set by the user of the video processing apparatus 1 and is set to a value such as 0.1 or 0.2.

  If the curved surface shape determination unit 145 determines in step S306 that the shape is outstanding, that is, if it is determined that the second condition is satisfied, the process proceeds to step S307. On the other hand, if the curved surface shape determination unit 145 determines that the shape is not outstanding, that is, if it is determined that the second condition is not satisfied, the process proceeds to step S308.

  When the process proceeds to step S307, the tracking unit 140 determines that the tracking is successful. On the other hand, when the process proceeds to step S308, the tracking unit 140 determines that the tracking has failed.

  Note that the video processing apparatus 1 may execute the block matching processing in steps S300 and S301 after executing steps S302 and S303 to extract the first pixel and the second pixel.

<Block matching process>
The block matching process will be described with reference to FIGS.

  4 and 5 are diagrams showing an outline of block matching processing. As shown in (1) of FIG. 4, the block matching unit 141 extracts the feature point 400 from the frame 410 of the nth video. The block matching unit 141 extracts a region having a size of N (pixels) × N (pixels) around the feature point 400 from the frame 410 of the nth video, and generates a template patch 430.

  As shown in (2) of FIG. 4, the block matching unit 141 extracts an area having a size of H (pixel) × W (pixel) in the frame 420 of the nth and subsequent images, and the second search window 440. Is generated.

  Here, the size of the template patch 430 and the size of the second search window 440 are values that can be set by the user of the video processing apparatus 1.

  As shown in (1) of FIG. 5, the block matching unit 141 moves the center of the template patch 430 within the second search window 440. That is, the block matching unit 141 extracts an area having a size of (H + N) (pixel) × (W + N) (pixel) from the frame 420 of the nth and subsequent images, and moves the template patch 430 into the area. . This area may be expressed as a first search window 450.

  The block matching unit 141 moves the template patch 430 in the first search window 450, and calculates the similarity between the overlapping portion with the template patch 430 and the template patch 430.

  For example, as shown in (1) of FIG. 5, the block matching unit 141 may move the template patch 430 in the horizontal direction within the first search window 450. The block matching unit 141 calculates the similarity between the template patch 430 and the overlapping portion while moving the template patch 430 pixel by pixel vertically or horizontally. The block matching unit 141 may calculate the similarity by moving the template patch 430 vertically or horizontally in a sub-pixel unit that is a unit smaller than one pixel.

FIG. 6 (Formula 1) shows an example of calculating the similarity. When the difference _ESAD between the total value of the luminance values of the pixels in the template patch 430 and the total value of the luminance values of the overlapping portions is the smallest or smaller than a predetermined threshold, the block matching unit 141 Then, it is determined that the center of the overlapping portion is a tracking point corresponding to the feature point.

In the example of (2) of FIG. 5, the block matching unit 141 determines that Vn (X, Y) is a tracking point 560 from _ESAD calculated when the center of the template patch 430 is moved to Vn (X, Y). It is judged that.

  The block matching unit 141 may determine that the feature point 400 of the nth frame has moved to the tracking point 560 in the frame 420 of the nth and subsequent images. The block matching unit 141 may determine that the feature point has moved along the movement locus 570. The block matching unit 141 may repeatedly perform block matching processing on the nth and subsequent video frames to calculate a trajectory of the feature point movement. The trajectory of feature point movement may be referred to as an optical flow.

In the above-described embodiment, a case has been described in which _ESAD , which is the difference between the sum of the brightness values of the template patch 430 and the sum of the brightness values of overlapping portions (Sum of Absolute Difference), is used. The block matching unit 141 may execute the block matching process using other methods. For example, the block matching unit 141 may use SSD (Sum of Squared Differences) that calculates the sum of squares of the difference between the luminance values of the template patch 430 and the overlapping portion, and the template patch 430 and the luminance value of the overlapping portion are mutually reciprocal. NCC (Normalized Cross Correlation) for calculating the correlation may be used.

  When the video frame is color, the block matching unit 141 may determine the similarity between the template patch and the overlapping portion using a value obtained by summing the luminances of the respective colors.

<Determination of the first condition>
The operation for determining the first condition will be described with reference to FIGS.

(1) Extreme value detection method (1)
A procedure for detecting an extreme value will be described with reference to FIGS.

  The extreme value detection unit 142 calculates the degree of similarity between each pixel in the first search window and the feature point. For example, the extreme value detection unit 142 calculates a luminance difference between each pixel in the first search window and the feature point. FIG. 7 (1) shows an example of a graph of similarity distribution between each pixel and a feature point. Pixels with smaller similarity values are more similar to feature points.

Next, the extreme value detection unit 142 extracts pixels having extreme values using the calculated similarity to the feature points. Pixels having extreme values are pixels that are more similar to feature points than neighboring pixels. For example, in FIG. 7B, the similarity R (X _i , Y _i ) between the pixel at the position (X _i , Y _i ) and the feature point is more similar to the feature point than the surrounding eight pixels. In this case, the extreme value detection unit 142 determines that the pixel at the position (X _i , Y _i ) is a pixel having an extreme value, and R (X _i , Y _i ) is the extreme value of the pixel. Specifically, the extreme value detection unit 142 determines that the pixel at the position (X _i , Y _i ) is a pixel having an extreme value when (Expression 2) in FIG. 8 is satisfied.

  The extreme value detection unit 142 uses, as the first pixel, a pixel having the extreme value most similar to the feature point among the extracted pixels having the extreme value, and a pixel having the second extreme value similar to the feature point. The second pixel.

  FIG. 7 (3) shows an example of the distribution of similarity between the extreme value pixels and the feature points. The smaller the similarity value, the more similar to the feature point. The extreme value detection unit 142 detects a pixel 720 having an extreme value with the smallest similarity value and a pixel 730 having an extreme value with the second smallest similarity value. The extreme value detection unit 142 sets the pixel 720 as the first pixel and the pixel 730 as the second pixel.

  The unity determination unit 143 has the same position of the first pixel as that of the tracking point specified by the block matching process, and the similarity between the first pixel and the feature point, and the second If the difference in similarity between the pixel and the feature point is greater than or equal to the first threshold, it is determined that the first condition is satisfied.

  When the similarity between the first pixel and the feature point and the similarity between the second pixel and the feature point are not sufficiently separated, the unity determination unit 143 may extract the tracking point by mistake. High nature. For example, when similar features such as a striped pattern repeatedly appear in the image in the search window, the unity determination unit 143 is likely to erroneously extract tracking points corresponding to the feature points.

  The unity determination unit 143 tracks the first pixel only when the difference between the similarity between the first pixel and the feature point and the similarity between the second pixel and the feature point is greater than or equal to the first threshold. Candidate points. Therefore, the unity determination unit 143 can extract an appropriate tracking point candidate.

(2) Method for detecting extreme values in subpixels A method for detecting extreme values in subpixels will be described with reference to FIG. In the above-described extremum detection method, the extremum is calculated with accuracy in units of pixels. However, the extremum detection unit 142 may detect the extremum in units of sub-pixels that are finer than pixels.

In the detection method described above, the unity determination unit 143 compares the similarity between the central pixel (X _i , Y _i ) and the feature point with the similarity between the eight neighboring pixels and the feature point, and determines the poles. The value is detected. In such a detection method, when the pixels having the similarity between the central pixel and the feature point and the similarities with the substantially same feature point exist in the pixels in the surrounding eight points, (Equation 2) in FIG. 8 is not satisfied. .

When (Equation 2) in FIG. 8 is not satisfied, there are a case where the image has no feature and cannot be a tracking point, and an extreme value exists between pixels. FIG. 9 shows an example in the case where extreme values exist between pixels. FIG. 9 shows the similarity R (X _i , Y _i ) and R (X _{i + 1} , Y _i ) between feature points in adjacent pixels (X _i , Y _i ) and (X _{i + 1} , Y _i ). In the case where the subpixels (a, b) 900 are substantially the same, an example in which extreme values exist is shown.

  The extreme value detection unit 142 may detect the extreme value with the accuracy of the sub-pixel in order to detect the extreme value more accurately than the accuracy of the pixel.

  When detecting the extreme value with the accuracy of the sub-pixel, the extreme value detection unit 142 detects the extreme value with the accuracy of the sub-pixel by interpolating the similarity between the center pixel and the surrounding pixels and the feature point. To do.

  An example of detecting an extreme value in the subpixel (a, b) 900 will be described with reference to (3) of FIG. Here, an example in which the luminance similarity L between the sub-pixel (a, b) 900 and the feature point is interpolated from the luminance difference L between the pixels and the feature point (hereinafter, luminance similarity L) will be described. However, it is needless to say that similarity other than luminance may be used.

In (3) of FIG. 9, the distance between the pixels is β (for example, (X _i , Y _i ) to (X _i , Y _i-1 )), and the interpolation target sub-pixel (a, b) 900 The case where the distance from the center pixel (X _i , Y _i ) is α is shown.

  The luminance similarity L (a, b) in the sub-pixel (a, b) 900 is calculated from (Equation 3) in FIG. The luminance similarity L in pixels around the sub-pixel (a, b) 900 is weighted by the distance between the sub-pixel (a, b) 900 and each pixel, whereby the luminance in the sub-pixel (a, b) 900 is determined. The similarity is calculated.

(3) Extreme Value Calculation Method Using the Gradient Method When performing extreme value detection with subpixel accuracy, the extreme value detection unit 142 may identify the position of a subpixel having an extreme value using the gradient method. Good.

  Here, a method for specifying the position of a sub-pixel using the KLT (Kanade-Lucas Tomasi) method will be described with reference to FIGS.

  FIG. 11 shows image I of frame 1100 of the Nth video, image J of frame 1110 of the (N + 1) th video, first search window 1120 on image J, and first search window 1130 on image I. , A luminance value I (x, y) 1140 on the image I and a luminance value J (x, y) 1150 on the image J are represented.

  A method for specifying the coordinates of the sub-pixel on the image J from the luminance value of each pixel of the image I and the image J will be described.

The extreme value detection unit 142 calculates the difference (δI _k ) of the luminance values in the corresponding pixels of the image I and the image J using (Equation 4) in FIG.

The extreme value detection unit 142 uses “Equation 5” of FIG. 12 to set each pixel of the images I and J to the “difference in luminance value (δI _k )” of each pixel in the first search window 1120 of the image J. B _k which is “a value obtained by multiplying the differentiated value” is _obtained . Here, the value obtained by differentiating each pixel of the image I at the position (x, y) is represented by I _x (x, y), and the value obtained by differentiating each pixel of the image J at the position (x, y) is I _y (x, y).

Also, the extreme value detector 142, with reference of FIG. 12 (6), the differential value of the x-direction at each pixel in the first search window 1120 _{(I x)} and y-direction of the differential value _(I y) And a matrix G of 2 rows and 2 columns calculated by multiplying.

The extreme value detection unit 142 calculates the amount of movement η ^k in the (x, y) direction from b _k and the inverse matrix of G using (Expression 7) in FIG.

The extreme value detection unit 142 calculates the coordinates of the sub-pixel using (Equation 8) in FIG. (Equation 8) represents the distance from the position of the pixel having the extreme, an initial value of [nu ^k is 0. K represents the number of calculations.

The extreme value detection unit 142 may repeat the calculation of ν ^k a predetermined number of times. For example, the extreme value detection unit 142 may repeat the calculation of ν ^k about 20 times. Alternatively, the extreme value detection unit 142 may repeat the calculation of ν ^k until the movement amount η ^k becomes equal to or less than a predetermined threshold value. For example, the extreme value detection unit 142 may repeat the calculation of ν ^k until η ^k becomes 0.03 subpixel or less.
The extreme value detection unit 142 calculates the degree of similarity between the subpixel at the calculated position and the feature point. For example, the extreme value detection unit 142 may calculate the luminance difference between the sub-pixel at the position and the feature point using (Expression 3) in FIG.

  The extreme value detection unit 142 compares the degree of similarity between the subpixel and the feature point at the calculated position with the degree of similarity between the neighboring extreme value candidate pixel and the feature point calculated in units of pixels. The pixel that is more similar to the point is defined as a pixel having an extreme value.

  In addition, although the method of calculating the subpixel coordinates by the KLT method has been described, the subpixel coordinates may be calculated using other methods. In addition, the range in which the extreme value detection unit 142 calculates the coordinates of the sub-pixels is not limited to the first search window.

  Although the method for calculating the coordinates of the subpixels in the frame of the (N + 1) th image has been described, the above-described calculation method can also be applied to the calculation of the coordinates of the subpixels in the frames of the N + 1th and subsequent images.

(4) Extreme value detection method (2)
In the extreme value detection method described above, if the difference between the similarity between the first pixel and the feature point and the similarity between the second pixel and the feature point are not equal to or greater than the first threshold, the unity determination unit 143 determined that the first condition was not satisfied. However, even when the difference in similarity is not equal to or greater than the first threshold value, the first pixel may be a tracking point depending on the similarity between a pixel having another extreme value and a feature point. In such a case, it is preferable that the unity determination unit 143 sets the first pixel as a tracking point candidate. An operation procedure of extreme value detection in such a case will be described with reference to FIGS.

  As described above, the unity determination unit 143 determines the first when the difference between the similarity between the first pixel and the feature point and the similarity between the second pixel and the feature point is equal to or greater than the first threshold. It is determined that the above condition is satisfied.

  (1) in FIG. 13 shows a distribution 1300 indicating the similarity between each pixel in the first search window and the feature point, and (2) in FIG. 13 shows the distribution of (1) in FIG. FIG. 2 is a diagram showing two-dimensionally the pixel indicated by, and the relationship of similarity between the pixel and the feature point. In these figures, the smaller the similarity, the more similar to the feature point.

  In the case of (2) in FIG. 13, the difference 1330 between the similarity between the first pixel 1310 and the feature point and the similarity between the second pixel 1320 and the feature point is equal to or greater than the first threshold. The uniqueness determination unit 143 sets the first pixel 1310 as a tracking point candidate.

  Next, a case where the difference between the similarity between the first pixel and the feature point and the difference between the similarity between the second pixel and the feature point is not greater than or equal to the first threshold will be described.

  (3) in FIG. 13 is a distribution 1340 indicating the similarity between each pixel in the first search window and the feature point, and (4) in FIG. 13 is indicated by the distribution in (3) in FIG. It is a figure which shows the relationship of the similarity of a pixel and this pixel and a feature point in two dimensions.

  In FIG. 13 (4), the first pixel 1350, the second pixel 1360, and the similarity difference 1355 between the feature points of the first pixel 1350 and the second pixel 1360 are indicated.

  When the similarity difference 1355 is smaller than the first threshold, the unity determination unit 143 determines the similarity difference 1365 between the second pixel 1360 and the pixel 1370 having the third most similar feature point. , And a similarity difference 1375 between a pixel 1370 having an extreme value third most similar to the feature point and a pixel 1380 having an extreme value fourth similar to the feature point are calculated.

  Next, the unity determination unit 143 compares the calculated similarity difference (such as 1365 and 1375) with the similarity difference 1355. That is, the unity determination unit 143 determines the similarity between the feature point and the pixel having the Nth most similar extreme value to the feature point, and the similarity degree between the feature point and the pixel having the N + 1th extreme value. A difference (hereinafter, “difference in similarity between N and N + 1”) is calculated.

  Then, the unity determination unit 143 compares the calculated similarity difference between N and N + 1 with the similarity difference 1355 (N is a natural number of 2 or more).

  As a result of such comparison, if the similarity difference 1355 is greater than the difference in similarity between N and N + 1, the unity determination unit 143 indicates that the similarity difference 1355 is smaller than the first threshold. Also, the first pixel 1350 is set as a tracking point candidate.

  The processing flow of the unity determination unit 143 will be described with reference to FIG.

  In step S1401, the unity determination unit 143 determines whether or not the difference between the similarity between the first pixel and the feature point and the similarity between the second pixel and the feature point is greater than or equal to the first threshold. To do. When the similarity difference is equal to or larger than the first threshold value, the process proceeds to step S1404 (YES in step S1401). On the other hand, if the difference in similarity is not equal to or greater than the first threshold, the process proceeds to step S1402 (NO in step S1401).

  In step S1402, the unity determination unit 143 calculates the similarity between the first pixel and the feature point, the difference between the second pixel and the feature point, and the difference between the similarity between N and N + 1. Compare. When the difference in similarity between the first pixel and the second pixel is greater than the difference in similarity between N and N + 1, the process proceeds to step S1404 (YES in step S1402). On the other hand, if the difference in similarity between the first pixel and the second pixel is not greater than the difference in similarity between N and N + 1, the process proceeds to step S1403 (NO in step S1402).

  In step S1403, the unity determination unit 143 determines that the unity determination is unsuccessful and ends the process.

  In step S1404, when the position of the first pixel matches the position of the tracking point specified by the block matching process, the unity determination unit 143 determines that the unity determination is successful and ends the process. To do.

  Note that when only one pixel having an extreme value is detected in the search window, the unity determination unit is used when the position of the pixel matches the position of the tracking point specified by the block matching process. 143 determines that the unity determination is successful.

(5) Expansion of search window for second video frame When unity determination unit 143 determines that the determination of unity has failed, video processing apparatus 1 expands the search window for the second video frame. Block matching processing and first condition determination processing may be performed.

  Processing contents of the video processing device 1 when the search window of the second video frame is enlarged will be described with reference to FIG.

  (1) of FIG. 15 is a diagram illustrating a relationship among the first search window 1530, the second search window 1500, and the template patch 1510 in the second video frame 1550 before the search window is enlarged.

  The second video frame 1550 shows an image in which a house is present behind the fence. The area where the first and second search windows (1530, 1500) are set is a part corresponding to the fence, and a similar pattern is repeated. It is difficult for the unity determination unit 143 to extract the first pixel that is a tracking point candidate. In addition, since similar patterns are repeated, the block matching unit 141 may erroneously specify the tracking point.

  In such a case, the video processing apparatus 1 may enlarge the search window and execute block matching processing and unity determination.

  (2) of FIG. 15 is an example when the search window is enlarged. In (2) of FIG. 15, the size of the second search window 1500 is constant, but the video processing device 1 enlarges the first search window 1540. Also, the video processing apparatus 1 increases the size of the template patch 1520 that moves in the first search window.

  By enlarging the template patch 1520, the overlapping portion of the second video frame 1550 of the template patch 1520 becomes wider. As a result, the overlapping portion includes a portion other than the fence, and when the template patch 1520 is moved in the first search window, the value for evaluating the similarity between the overlapping portion and the template patch 1520 changes. Therefore, it becomes easy to extract tracking points.

  In addition, since the enlarged first search window 1540 is an image having a change from before the enlargement, the unity determination unit 143 can easily determine the most similar first pixel as a tracking point candidate. .

  Note that the video processing apparatus 1 does not need to enlarge the template patch 1510 when the search window for the second video frame is enlarged. Further, the video processing device 1 may enlarge the size of the second search window together with the first search window.

<Determination of the second condition>
(1) Method for calculating curved surface shape A method for calculating a curved surface shape will be described with reference to FIGS. 16 and 17. The curved surface shape determination unit 145 determines whether the extracted tracking point candidate pixel is a tracking point as a result of the block matching processing and the unity determination (first condition determination) processing. (Determination of the second condition).

  When determining whether or not a tracking point candidate pixel is a tracking point, the curved surface shape determination unit 145 determines whether or not the tracking point candidate pixel has a distinctive feature with respect to surrounding pixels. Do. In this embodiment, the curved surface shape determination unit 145 compares the similarity between the tracking point candidate pixel and the feature point with the similarity between the 8 pixels around the candidate pixel and the feature point, and the tracking point. It is determined whether or not.

  Specifically, the curved surface shape determination unit 145 calculates the shape of the secondary curved surface from the positions of the candidate pixel (x, y) and the surrounding eight pixels and the similarity between these pixels and feature points.

  FIG. 16 (1) shows the positional relationship between the candidate pixel (x, y) and the surrounding eight pixels. R represents the similarity between each pixel and the feature point. For example, R may represent the degree of luminance similarity with the feature point.

  (2) of FIG. 16 is a diagram illustrating a state in which a quadric surface is formed from the candidate pixel (x, y), the positions of the surrounding eight pixels, and the similarity value. (2) of FIG. 16 shows a state in which a quadric surface is cut out with the similarity R (x, y) of the candidate pixel (x, y) as the center.

  The quadric surface is obtained from (Equation 9) in FIG. Here, the secondary coefficient a of x and the secondary coefficient c of y are obtained from (Expression 10) and (Expression 11), respectively.

  Here, (Equation 10) is expressed by two columns of 6 pixels ((x-1, y-1), (x-1, y), (x-1, y + 1)) at both ends of (1) in FIG. (X + 1, y-1), (x + 1, y), (x + 1, y + 1)) similarity and three pixels ((x, y-1), middle column including candidate pixel (x, y), The difference between (x, y) and (x, y + 1)) is twice the similarity. Further, (Equation 11) is expressed by 6 pixels ((x−1, y−1), (x, y−1), (x + 1, y−1)) in the upper and lower rows of (1) in FIG. , (X-1, y + 1), (x, y + 1), (x + 1, y + 1)) and three pixels ((x-1, y) in the middle row including the candidate pixel (x, y)) , (X, y), (x + 1, y)) is twice the similarity. A quadratic surface formed from the similarity between a candidate pixel (x, y) and surrounding pixels when the value of the quadratic coefficient a of x and the value of the quadratic coefficient c of y are smaller than the third threshold. The shape of is nearly flat and does not stand out in the candidate pixel (x, y). For this reason, there is a high possibility that tracking points corresponding to feature points cannot be correctly identified. In this case, the curved surface shape determination unit 145 determines that the second condition is not satisfied, and the video processing device 1 determines that the tracking point extraction has failed.

  On the other hand, when at least one of the secondary coefficients a and c is larger than the third threshold, the candidate pixel (x, y) has a remarkable similarity to the feature point with respect to surrounding pixels. . Therefore, the curved surface shape determination unit 145 determines that the second condition is satisfied, and the video processing device 1 determines that the tracking point has been successfully extracted.

  When the block matching process and the first determination process are executed with the accuracy of the sub-pixel, the curved surface shape determination unit 145 selects the pixel closest to the sub-pixel that is the tracking point candidate, The second determination process may be executed using the surrounding pixels. Alternatively, the curved surface shape determination unit 145 forms a secondary curved surface from the similarity between the sub-pixels that are tracking point candidates and the feature points, and the similarity between the surrounding pixels and the feature points, and performs the second determination. Processing may be executed.

  In the above-described embodiment, the shape of the quadric surface is calculated from the candidate pixel (x, y) and the adjacent eight neighboring pixels, but the neighboring pixels are not adjacent to the candidate pixel (x, y). It should be noted that may be included. For example, the peripheral pixels may include two pixels or three pixels of the candidate pixel (x, y) up to the neighbor.

<Hardware configuration>
The video processing apparatus 1 is realized by a hardware configuration as shown in FIG. 18, for example.

  The video processing apparatus 1 includes an input / output unit 201, an external I / F 203, a RAM 204, a ROM 205, a CPU 206, a communication I / F 207, an HDD 209, a camera module 210, and the like.

  The input / output unit 201 displays a frame of video captured by the camera module 210. The input / output unit 201 displays the state of the video processing device 1 and the like. Further, the input / output unit 201 may accept various settings of the video processing device 1 from the user of the video processing device 1.

  A communication I / F 207 communicates with a server, a PC (Personal Computer), or the like via a wired or wireless network. The video processing device 1 may receive an instruction for the video processing device 1 from a terminal such as a PC via the communication I / F 207.

  The HDD 209 is an example of a non-volatile storage device that stores programs and data. The stored programs and data include an OS that is basic software for controlling the entire video processing apparatus 1 and application software that provides various functions on the OS. Note that the video processing apparatus 1 may use a drive device (for example, a solid state drive: SSD) that uses a flash memory as a storage medium instead of the HDD 209.

  The external I / F 203 is an interface with an external device. The external device includes a recording medium 203a. Thereby, the video processing apparatus 1 can read and / or write the recording medium 203a via the external I / F 203. Examples of the recording medium 203a include a flexible disk, a CD, a DVD, an SD memory card, and a USB memory.

  The ROM 205 is an example of a nonvolatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. The ROM 205 stores programs and data such as BIOS, OS settings, and network settings that are executed when the video processing apparatus 1 is started. The RAM 204 is an example of a volatile semiconductor memory (storage device) that temporarily stores programs and data.

  The camera module 210 has a camera, performs shooting based on an instruction from the CPU 206, and transmits the captured video data to a storage device such as the RAM 204. A storage device such as the RAM 204 stores the received photographing data and reads it in accordance with an instruction from the CPU 206.

  The CPU 206 reads programs and data from a storage device such as the ROM 205 and the HDD 209 onto the RAM 204 and executes the processing, thereby realizing the control of the entire video processing device 1 and the functions of the video processing device 1 shown in FIGS. 1 and 2. It is a computing device. The video processing apparatus 1 can implement the various processes described above with the hardware configuration shown in FIG.

[Others]
In the embodiment described above, the video processing apparatus 1 executes the first determination process and the second determination process using the pixels in the first search window. These processes may be executed using these pixels. As a result, the video processing apparatus 1 can narrow down the pixels to be subjected to the first determination process and the second determination process to pixels around the tracking point candidate.

  A storage medium that records a program code of software that implements the functions of the above-described embodiments may be supplied to the video processing apparatus 1. Needless to say, the above-described embodiment can also be achieved by the video processing apparatus 1 reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes any of the embodiments. Here, the storage medium is a recording medium or a non-transitory storage medium.

  The functions of the above-described embodiments are not only realized by executing the program code read by the computer device. An operating system (OS) or the like operating on the computer device may perform part or all of the actual processing in accordance with the instruction of the program code. Furthermore, it goes without saying that the functions of the above-described embodiments may be realized by the processing.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to such embodiment, A various deformation | transformation and substitution can be added in the range which does not deviate from the summary of this invention.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 110 Image | video input part 120 Image | video correction | amendment part 130 Feature point extraction part 140 Tracking part 141 Block matching part 142 Extreme value detection part 143 Singleness determination part 145 Curved surface shape determination part 150 Frame buffer 160 Duplicate point removal part

JP 2003-0788807 A

Claims (10)

An extraction unit for extracting feature points from the frame of the first video;
A part of the frame of the first video with respect to a predetermined area of the frame of the second video that is a video frame temporally after the frame of the first video, and the feature point A tracking unit that identifies a tracking point corresponding to the feature point by performing a block matching process with a block including the block;
A position of a first pixel that is most similar to the feature point in the predetermined region; a similarity between the first pixel and the feature point; a position of a second pixel that is second closest to the feature point; And a detection unit for detecting the similarity between the second pixel and the feature point;
When the position of the tracking point matches the position of the first pixel, and the difference between the similarity of the first pixel and the similarity of the second pixel is equal to or greater than a first threshold, A first determination unit that determines that one condition is satisfied;
The similarity between the tracking point and the feature point, the similarity between each of a plurality of surrounding pixels of the tracking point and the feature point, and the calculated similarity between the tracking point and the plurality A second determination unit that determines that the second condition is satisfied when the difference between the respective similarities of pixels around the pixel is equal to or greater than a second threshold value,
When the first condition and the second condition are satisfied, the tracking unit determines that the feature point corresponds to the tracking point.   The detection unit detects a plurality of similar pixels that are similar to the feature point in the predetermined region, and the plurality of similar pixels are more similar to the feature point than pixels adjacent thereto, and are detected. The video processing according to claim 1, wherein among the plurality of similar pixels, a pixel that is most similar to the feature point is the first pixel, and a pixel that is second most similar to the feature point is the second pixel. apparatus.   The video processing apparatus according to claim 1, wherein the similarity with the feature point is represented by a difference between a luminance at the feature point and a luminance of a pixel for which the similarity is calculated.   The degree of similarity with the feature point is represented by a difference between a total value of luminance of each color at the feature point and a total value of luminance of pixels for which the similarity is calculated. Video processing device. The tracking unit performs the block matching process with the accuracy of a subpixel that is a unit smaller than a pixel, specifies the tracking point,
The video processing apparatus according to claim 1, wherein the detection unit detects the first pixel and the second pixel with accuracy of the sub-pixel.   The detection unit interpolates the similarity between the feature point and a sub-pixel existing between adjacent pixels from the similarity between each of the pixels around the adjacent pixel and the feature point. The video processing apparatus described.   The detection unit calculates coordinates of the sub-pixel based on a change in similarity between pixels of the predetermined area and pixels of an area corresponding to the predetermined area of the frame of the first video, and the coordinates The video processing apparatus according to claim 6, wherein the similarity with the feature point is interpolated.   The second determination unit is formed by forming a similarity distribution of a quadric surface from the similarity between the tracking point and the feature point and the similarity between the surrounding pixel and the feature point. The video processing apparatus according to claim 1, wherein whether or not the second condition is satisfied is determined from a distribution of similarity of quadric surfaces. Extracting feature points from a frame of the first video;
A part of the frame of the first video with respect to a predetermined area of the frame of the second video that is a video frame temporally after the frame of the first video, and the feature point Identifying a tracking point corresponding to the feature point by performing block matching processing with a block including
A position of a first pixel that is most similar to the feature point in the predetermined region; a similarity between the first pixel and the feature point; a position of a second pixel that is second closest to the feature point; And detecting the degree of similarity between the second pixel and the feature point;
When the position of the tracking point matches the position of the first pixel, and the difference between the similarity of the first pixel and the similarity of the second pixel is equal to or greater than a first threshold, Determining that condition 1 is satisfied;
The similarity between the tracking point and the feature point, the similarity between each of a plurality of surrounding pixels of the tracking point and the feature point, and the calculated similarity between the tracking point and the plurality Determining that the second condition is satisfied if the difference between each of the neighboring pixels of the pixel is equal to or greater than a second threshold value;
And a step of determining that the feature point corresponds to the tracking point when the first condition and the second condition are satisfied.   A program causing a video processing apparatus to execute the video processing method according to claim 9.

Images