JP2010039520A - Feature point detection device and moving image processor with the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To detect a feature point by simple arithmetic processing while securing predetermined accuracy. <P>SOLUTION: A secondary feature value calculation part 10 calculates a secondary differential value in a diagonally lower right direction and a secondary differential value in a diagonally upper right direction in pixels of interest in an image. An evaluation value calculation part 20 calculates an absolute value difference between the secondary differential value in the diagonally lower right direction and the secondary differential value in the diagonally upper right direction, the secondary differential values being calculated by the secondary differential value calculation part 10, as a first evaluation value. A decision part 30 compares a first evaluation value calculated by the evaluation value calculation part 20 with the first threshold, and decides whether or not the pixels of interest correspond to a corner or an oblique edge. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a feature point detection device that detects feature points in an image, such as corners and edges, and a moving image processing device equipped with the feature point detection device.

  As preprocessing for image processing, processing for detecting feature points from a target image is widely used. The amount of calculation can be reduced by limiting the calculation target in the target image to pixels detected as feature points instead of all pixels. For example, feature point detection processing is used as preprocessing for optical flow calculation processing. The optical flow is a vector indicating in what direction and how far a certain point or figure in the image moves at the next moment.

  A technique often used for detecting feature points from an image is a KLT (Kanada-Lucas-Tomashi) method. In the KLT method, a feature point is obtained by evaluating a feature amount H (x, y) of a coordinate c (x, y) on an input image. In the KLT method, first, an input image is smoothed by a Gaussian operator. Next, the matrix M shown in the following formula 1 is calculated using the gradient of the luminance I on the image (Ix = ∂I / ∂x, Iy = ∂I / ∂y) in the square window W having a certain size. . Next, the feature value H (x, y) is obtained by obtaining the minimum values of the eigenvalues λ1 and λ2 of the matrix M as shown in the following equation 2. Finally, the feature amount H (x, y) is evaluated to obtain a feature point.

Patent Document 1 discloses an image processing apparatus that calculates an evaluation value for detecting a feature point in an image. This image processing apparatus estimates a noise amount by calculating a primary spatial differential value and a secondary spatial differential value of a luminance value for each of a plurality of directions in each pixel.
JP 2007-231565 A

  Although the above-described KLT method can detect feature points with relatively high accuracy, it requires complicated arithmetic processing such as obtaining minimum values of eigenvalues λ1 and λ2 of the matrix M. Therefore, it is difficult to configure the feature point detection process by the KLT method with hardware resources.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a feature point detection device capable of detecting feature points by simple arithmetic processing while maintaining a certain accuracy, and a moving image equipped with the feature point detection device. It is to provide a processing apparatus.

  A feature point detection apparatus according to an aspect of the present invention includes a second-order differential value calculation unit that calculates a second-order differential value in a diagonally downward direction and a second-order differential value in a diagonally right direction at a target pixel in an image; An evaluation value calculation unit that calculates an absolute value difference between a second-order differential value in the diagonally descending right direction and a secondary differential value in the diagonally right-up direction calculated by the secondary differential value calculating unit; and an evaluation value A determination unit that determines whether or not the target pixel corresponds to a corner or an oblique edge in the image by comparing the first evaluation value calculated by the calculation unit and the first threshold value;

  Another aspect of the present invention is also a feature point detection apparatus. The apparatus divides an image into a plurality of partial areas, and for each pixel of each partial area divided by the dividing section, a second-order differential value in a right-down diagonal direction and a second-order differential in a right-up diagonal direction. Value, horizontal differential value and vertical differential value calculation unit, and for each pixel in each partial area, the right-down diagonal differential value and the right-up diagonal Evaluation that calculates absolute value difference from secondary differential value in direction as first evaluation value, and calculates absolute value difference between secondary differential value in horizontal direction and secondary differential value in vertical direction as second evaluation value A value calculation unit; and a feature point detection unit that detects at least one feature point for each partial region. The feature point detection unit compares the first evaluation value of each pixel in a certain partial region with the first threshold value, and at least one pixel whose first evaluation value exceeds the first threshold value exists in the partial region. In this case, when at least one pixel is determined as a feature point candidate of the partial region, and there is no pixel in the partial region that has a first evaluation value exceeding the first threshold value, the second evaluation value of each pixel in the partial region The second threshold value is compared, and if there is at least one pixel in the partial area whose second evaluation value exceeds the second threshold value, the at least one pixel is determined as a feature point candidate for the partial area.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, it is possible to detect a feature point by simple arithmetic processing while ensuring a certain accuracy.

  FIG. 1 is a block diagram showing a configuration of a feature point detection apparatus 100 according to Embodiment 1 of the present invention. Feature point detection apparatus 100 according to Embodiment 1 includes secondary differential value calculation unit 10, evaluation value calculation unit 20, and determination unit 30. These configurations can be realized by an arbitrary processor, memory, or other LSI in terms of hardware, and can be realized by a program loaded in the memory in terms of software, but here by their cooperation. Draw functional blocks. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The secondary differential value calculation unit 10 has a second-order differential value in the diagonally lower right direction, a second-order differential value in the right-up diagonal direction, a second-order differential value in the horizontal direction, and a vertical differential value in the pixel of interest in the input image. Calculate the second derivative value. Note that all the pixels in the image may correspond to the target pixel. The secondary differential value calculation unit 10 may calculate using pixels adjacent to the target pixel, or may calculate using pixels that are two or more pixels away from the target pixel, such as every other pixel. An operator for calculating the secondary differential value in the present embodiment will be described later.

  The evaluation value calculation unit 20 calculates, as the first evaluation value, the absolute value difference between the second-order differential value in the lower right diagonal direction and the second-order differential value in the upper right diagonal direction calculated by the secondary differential value calculation unit 10. Then, an absolute value difference between the secondary differential value in the horizontal direction and the secondary differential value in the vertical direction is calculated as the second evaluation value. Further, the evaluation value calculation unit 20 calculates, as the third evaluation value, the sum of the absolute value of the second-order differential value in the diagonally lower right direction and the absolute value of the second-order differential value in the diagonally right upward direction at the target pixel. The sum of the absolute value of the secondary differential value in the horizontal direction and the absolute value of the secondary differential value in the vertical direction is calculated as the fourth evaluation value.

  The determination unit 30 determines whether the target pixel corresponds to a corner or an oblique edge in the image by comparing the first evaluation value calculated by the evaluation value calculation unit 20 with the first threshold value. More specifically, when the first evaluation value is larger than the first threshold, it is determined that the target pixel corresponds to a corner or an oblique edge in the image. Alternatively, assuming that the target pixel is highly likely to correspond to a corner or a diagonal edge in the image, the target pixel is set as a corner or diagonal edge candidate in the image. The principle of this determination will be described later.

  The determination unit 30 determines whether the target pixel corresponds to a horizontal edge or a vertical edge in the image by comparing the second evaluation value calculated by the evaluation value calculation unit 20 with the second threshold value. . More specifically, when the second evaluation value is larger than the second threshold value, it is determined that the target pixel corresponds to a horizontal edge or a vertical edge in the image. Alternatively, assuming that the pixel of interest is likely to correspond to a horizontal edge or a vertical edge in the image, the pixel of interest is set as a candidate for a horizontal edge or a vertical edge in the image.

  Corners, diagonal edges, horizontal edges, and vertical edges in the image appear in the outline of the object in the image. Since the contour is a portion where the brightness in the image changes abruptly, the contour can be detected by detecting the change in the brightness in the image. In the present embodiment, the change in brightness is detected as a secondary differential value.

  The determination unit 30 determines the strength of the corner or the oblique edge by comparing the third evaluation value with the third threshold value. It is determined that the strength is stronger as the third evaluation value is larger than the third threshold value. The determination unit 30 determines the strength of the horizontal edge or the vertical edge by comparing the fourth evaluation value with the fourth threshold value. It is determined that the strength is stronger as the fourth evaluation value is larger than the fourth threshold value. The first threshold value, the second threshold value, the third threshold value, and the fourth threshold value can be set to values obtained by the designer through experiments or simulations.

  FIG. 2 shows the operator 15 necessary for calculating the secondary differential value according to the first embodiment. The operator 15 has a coefficient of interest pixel of −2 and a peripheral pixel coefficient of +1. Using the four one-dimensional filters based on the operator 15, the secondary differential value calculation unit 10 uses a right-down diagonal direction (−45 ° direction), a right-up diagonal direction (+ 45 ° direction), and a horizontal direction (0 ° direction). ) And vertical differential values in the vertical direction (+ 90 ° direction). That is, the secondary differential value is calculated as shown in the following formulas 3-6.

Second-order differential value in the −45 ° direction Ya = I (i−1, j−1) −2 * I (i, j) + I (i + 1, j + 1) (Equation 3)
+ 45 ° direction secondary differential value Yb = I (i−1, j + 1) −2 * I (i, j) + I (i + 1, j−1) (Formula 4)
Second-order differential value in the 0 ° direction Yc = I (i-1, j) -2 * I (i, j) + I (i + 1, j) (Formula 5)
Second-order differential value in the + 90 ° direction Yd = I (i, j + 1) −2 * I (i, j) + I (i, j−1) (Formula 6)
I is a value indicating the brightness of the pixel. In the case of a color image, it may be an average value of R, B, and G values, or a YCbCr luminance value. i is a variable that increments from left to right as shown in FIG. j is a variable that increments from top to bottom as shown in FIG.

  FIG. 3 is a diagram illustrating the one-dimensional filter 15a for calculating the second-order differential value Ya in the −45 ° direction according to the first embodiment. The one-dimensional filter 15a includes an adder 16, a multiplier 17, and a subtracter 18. The adder 16 adds I (i−1, j−1) and I (i + 1, j + 1). Multiplier 17 doubles I (i, j). The subtracter 18 calculates the difference between the output value of the adder 16 and the output value of the multiplier 17. The output value of the subtracter 18 becomes the second derivative value Ya in the −45 ° direction. The secondary differential value Yb in the + 45 ° direction, the secondary differential value Yc in the 0 ° direction, and the secondary differential value Yd in the + 90 ° direction can be obtained with the same configuration.

  FIG. 4 shows the transition of the secondary differential value Ya on the spatial axis S in the −45 ° direction in the image according to the first embodiment. That is, the transition of the secondary differential value Ya of each pixel continuous in the −45 ° direction is shown. When the brightness of each pixel is classified into light and dark, it is classified into dark when the secondary differential value Ya is positive and bright when the secondary differential value Ya is negative. A portion where the secondary differential value Ya greatly crosses from positive to negative zero crossing (e1 in FIG. 4) is an edge boundary from dark to bright. Similarly, a portion where the second-order differential value Ya greatly crosses from negative to positive greatly (e2 in FIG. 4) becomes an edge boundary from light to dark. Since the secondary differential value Ya is a value reflecting two brightness changes between the target pixel and adjacent pixels on both sides thereof, the secondary differential value Ya is in the positive and negative directions before and after the edge boundary. Shakes greatly. The same argument applies to the secondary differential value Yb in the + 45 ° direction, the secondary differential value Yc in the 0 ° direction, and the secondary differential value Yd in the + 90 ° direction.

  FIG. 5 is a diagram for explaining the principle of detecting a corner or diagonal edge from the first evaluation value and a horizontal edge or vertical edge from the second evaluation value according to the first embodiment. The first pattern P1 indicates a 3 × 3 pixel pattern that appears at a corner. The second pattern P2 also indicates a 3 × 3 pixel pattern that appears at the corner. The first pattern P1 indicates a corner that appears at the upper right of the pixel pattern, and the second pattern P2 indicates a corner that appears at the lower right of the pixel pattern.

  The third pattern P3 is a 3 × 3 pixel pattern that appears with a diagonally downward sloping edge. The fourth pattern P4 indicates a 3 × 3 pixel pattern that appears at the horizontal edge. The fifth pattern P5 indicates a 3 × 3 pixel pattern that appears at a vertical edge. In FIG. 5, black circles indicate darkness and white circles indicate light when the brightness of each pixel is classified into light and dark.

  FIG. 6 shows a table 25 in which the secondary differential values Ya to Yd in the four directions of the patterns P1 to P5 in FIG. 5 are collected. In FIG. 6, the average value of the difference between the pixel value of the black circle pixel and the pixel value of the white circle pixel is denoted by α. In the first pattern P1, only the secondary differential value Ya in the −45 ° direction is 2α, and the remaining secondary differential values Yb, Yc, and Yd in the three directions are all α. The same result is obtained when a corner appears in the lower left corner of the pixel pattern. In the second pattern P2, only the secondary differential value Yb in the + 45 ° direction is 2α, and the remaining secondary differential values Ya, Yc, and Yd in the three directions are all α. The same result is obtained when a corner appears in the upper left of the pixel pattern.

  2α is a value that appears when the target pixel is dark and adjacent pixels on both sides thereof are bright. As described above, when a corner exists in the pixel pattern, either the −45 ° -direction secondary differential value Ya or the + 45 ° -direction secondary differential value Yb becomes 2α, and the −45 ° -direction secondary differential value. The difference between Ya and the secondary differential value Yb in the + 45 ° direction is α. Further, the difference between the secondary differential value Yc in the 0 ° direction and the secondary differential value Yd in the + 90 ° direction is zero.

  In the third pattern P3, only the secondary differential value Ya in the −45 ° direction is 0, and the remaining secondary differential values Yb, Yc, and Yd in the three directions are all α. In the pixel pattern that appears at the diagonally upward edge, only the secondary differential value Yb in the + 45 ° direction is 0, and the remaining secondary differential values Ya, Yc, and Yd in the three directions are all α. As described above, when a diagonal edge exists in the pixel pattern, either the −45 ° -direction secondary differential value Ya or the + 45 ° -direction secondary differential value Yb becomes 0, and the −45 ° -direction secondary differential value. The difference between the value Ya and the secondary differential value Yb in the + 45 ° direction is α. Further, the difference between the secondary differential value Yc in the 0 ° direction and the secondary differential value Yd in the + 90 ° direction is zero.

  In the fourth pattern P4, only the secondary differential value Yc in the 0 ° direction is 0, and the remaining secondary differential values Ya, Yb, Yd in the three directions are all α. In the fifth pattern P5, only the secondary differential value Yd in the 90 ° direction is 0, and the remaining secondary differential values Ya, Yb, and Yc in the three directions are all α. As described above, when a horizontal edge or a vertical edge exists in the pixel pattern, either the secondary differential value Yc in the 0 ° direction or the secondary differential value Yd in the + 90 ° direction is 0, and the secondary differential value in the 0 ° direction is zero. The difference between the differential value Yc and the secondary differential value Yd in the + 90 ° direction is α. Also, the difference between the secondary differential value Ya in the −45 ° direction and the secondary differential value Yb in the + 45 ° direction is zero.

  Here, the calculation formulas of the first evaluation value E1, the second evaluation value E2, the third evaluation value E3, and the fourth evaluation value E4 described above are shown in the following formulas 7 to 10.

First evaluation value E1 = | Ya−Yb | (Expression 7)
Second evaluation value E2 = | Yc−Yd | (Expression 8)
Third evaluation value E3 = | Ya | + | Yb | (Formula 9)
Fourth evaluation value E4 = | Yc | + | Yd | (Expression 10)

  Thus, if the first evaluation value E1 exceeds α, it can be recognized that there is a high possibility that a corner or an oblique edge has appeared. Further, if the third evaluation value E3 is calculated, it is possible to classify the corner or the diagonal edge. If the third evaluation value E3 exceeds 3α, it can be determined as a corner. If the 3rd evaluation value E3 is less than (alpha), it can determine with an oblique edge. The designer can set a predetermined value within α to 3α to the fifth threshold value that separates the corner and the oblique edge.

  When the value of α is large, the value of the third evaluation value E3 is also large. A large value of α indicates that the change in brightness between the pixel of interest and its adjacent pixels is large, indicating a sharper contour. A larger third evaluation value E3 indicates a clearer corner. That is, the corner is strong.

  If the second evaluation value E2 exceeds α, it can be recognized that there is a high possibility that a horizontal edge or a vertical edge has appeared. Further, by referring to the secondary differential value Yc in the 0 ° direction and the secondary differential value Yd in the + 90 ° direction, it is possible to classify the horizontal edge or the vertical edge. Similar to the third evaluation value E3, the larger the fourth evaluation value E4, the clearer the horizontal edge or the vertical edge. That is, it indicates that the strength is a horizontal edge or a vertical edge.

  Since the first evaluation value E1 to the fourth evaluation value E4 are expressed as absolute values, in the first pattern P1 to the fifth pattern P5 shown in FIG. 5, the pixel pattern in which the white circle pixels and the black circle pixels are all inverted. The same evaluation value as that of the original pattern is also obtained. Further, even if the white circle pixels and the black circle pixels on both sides of the target pixel are interchanged, the same secondary differential value is calculated, but it can be said that such a pixel pattern is very few in a natural image.

  FIG. 7 is a flowchart illustrating an example of processing for detecting a feature point from a frame image using the feature point detection apparatus 100 according to the first embodiment. Since adjacent pixels are required on both sides to obtain the secondary differential value, the top and bottom horizontal pixel columns and the left and rightmost vertical pixel columns in the frame image No second derivative value is calculated for pixels belonging to. All pixels in a frame image to be described later refer to pixels excluding these pixels.

  First, an arbitrary pixel in the frame image is set as an initial value of the target pixel. For example, the second pixel from the top and the second pixel from the left are set as initial values. The determination unit 30 compares the first evaluation value E1 = | Ya−Yb | of the target pixel with the first threshold value Yth1 (S10). When the first evaluation value E1 is larger than the first threshold value Yth1 (Y in S10), the determination unit 30 compares the third evaluation value E3 = | Ya | + | Yb | with the third threshold value Yth3 (S12). When the third evaluation value E3 is larger than the third threshold Yth3 (Y in S12), the target pixel is set as a feature point (S18).

  When the first evaluation value E1 is less than or equal to the first threshold value Yth1 in step S10 (N of S10), and when the third evaluation value E3 is less than or equal to the third threshold value Yth3 in step S12 (N of S12), the determination unit 30 Compares the second evaluation value E2 = | Yc−Yd | of the target pixel with the second threshold value Yth2 (S14). When the second evaluation value E2 is larger than the second threshold value Yth2 (Y in S14), the determination unit 30 compares the fourth evaluation value E4 = | Yc | + | Yd | with the fourth threshold value Yth4 (S16). When the fourth evaluation value E4 is larger than the fourth threshold Yth4 (Y in S16), the target pixel is set as a feature point (S18).

  When the second evaluation value E2 is equal to or smaller than the second threshold value Yth2 in step S14 (N in S14), and when the fourth evaluation value E4 is equal to or smaller than the fourth threshold value Yth4 in step S16 (N in S16), the determination unit 30 Does not determine the target pixel as a feature point.

  The determination unit 30 determines whether or not the determination as to whether or not the feature point should be set has been completed for all the pixels in the frame image (S20). If not completed (N in S20), the determination unit 30 sets the next pixel as a target pixel (S22). Thereafter, the process proceeds to step S10 and the determination process described above is continued. If completed (Y in S20), the feature point detection process in the frame image ends.

  The algorithm shown in this flowchart gives priority to corners as feature points. This is because corners are usually less frequent and more valuable than edges. Those skilled in the art can easily implement an algorithm that prioritizes corners over edges, an algorithm that detects only corners, an algorithm that detects only diagonal edges, and an algorithm that detects only horizontal and vertical edges.

  In addition, although the process of the secondary differential value calculation part 10 and the evaluation value calculation part 20 is not explicitly described in the said flowchart, there exist the following two processing methods. In the first processing method, the secondary differential value calculation unit 10 calculates the quadratic secondary differential values Ya, Yb, Yc, and Yd for all the pixels, and the evaluation value calculation unit 20 uses the first evaluation values E1 to E1. This is a processing method for calculating the fourth evaluation value E4.

  The second processing method is a processing method in which the secondary differential value calculation unit 10 and the evaluation value calculation unit 20 calculate those values within the limits necessary for determining feature points for all pixels. For example, when the first evaluation value E1 is larger than the first threshold value Yth1 and the third evaluation value E3 is larger than the third threshold value Yth3, the secondary differential value calculation unit 10 calculates the secondary differential value Yc in the 0 ° direction and + 90 °. The secondary differential value Yd in the direction is not calculated, and the evaluation value calculation unit 20 does not calculate the second evaluation value E2 and the fourth evaluation value E4. The second processing method can eliminate useless calculations.

  As described above, according to the first embodiment, it is possible to detect feature points by a simple calculation process while maintaining a certain accuracy. That is, a corner or a diagonal edge can be detected by evaluating the difference between the second-order differential value in the diagonally downward direction and the secondary differential value in the diagonally right-up direction. Further, by evaluating the difference between the secondary differential value in the horizontal direction and the secondary differential value in the vertical direction, a horizontal edge or a vertical edge can be detected. In either case, detection can be performed only by simple four arithmetic operations and threshold determination processing. Further, as shown in FIGS. 9A and 9B described later, the detection accuracy is high. In addition, since complicated arithmetic processing is not required, it is easy to implement with hardware resources.

  Further, the strength of corners and edges can be detected, and the strength can be used for improving the detection accuracy of feature points and adjusting the number of feature points. Specifically, the detection accuracy of the feature points can be improved by setting the third threshold value and the fourth threshold value high. That is, erroneous detection can be reduced. In addition, the number of feature points detected from the frame image can be reduced. On the other hand, by setting the third threshold value and the fourth threshold value low, the number of feature points detected from within the frame image can be increased.

  FIG. 8 is a block diagram showing a configuration of feature point detection apparatus 150 according to Embodiment 2 of the present invention. The feature point detection apparatus 150 according to Embodiment 2 includes a dividing unit 5, a secondary differential value calculation unit 10, an evaluation value calculation unit 20, and a feature point detection unit 40.

  The dividing unit 5 divides the input frame image into a plurality of partial areas. This partial region may be a rectangular window region of vertical n pixels × horizontal m pixels (n and m are integers of 3 or more). Note that the partial area is not necessarily rectangular, and may be a circular area set by a SUSAN operator.

  For each pixel of each partial area divided by the dividing unit 5, the secondary differential value calculation unit 10 has a second-order differential value in the right-down diagonal direction, a second-order differential value in the right-up diagonal direction, and a horizontal second-order differential value. The differential value and the secondary differential value in the vertical direction are calculated. The calculation method of these secondary differential values is the same as the calculation method described in the first embodiment.

  The evaluation value calculation unit 20 calculates, as the first evaluation value, the absolute value difference between the second-order differential value in the lower right diagonal direction and the second-order differential value in the upper right diagonal direction for each pixel in each partial region. The absolute value difference between the secondary differential value in the direction and the secondary differential value in the vertical direction is calculated as the second evaluation value.

  The evaluation value calculation unit 20 calculates the absolute value of the second-order differential value in the lower right diagonal direction and the absolute value of the second-order differential value in the upper right diagonal direction in each pixel or a pixel in which the first evaluation value exceeds the first threshold value. Is calculated as the third evaluation value. In addition, the evaluation value calculation unit 20 sums the absolute value of the secondary differential value in the horizontal direction and the absolute value of the secondary differential value in the vertical direction at each pixel or a pixel whose second evaluation value exceeds the second threshold value. Is calculated as the fourth evaluation value. Note that the third evaluation value and the fourth evaluation value may be calculated for all the pixels, or may be calculated to the limit necessary for detecting the feature points.

  The feature point detection unit 40 detects at least one feature point for each partial region. This will be specifically described below. The feature point detection unit 40 compares the first evaluation value of each pixel in a certain partial region with the first threshold value. When at least one pixel whose first evaluation value exceeds the first threshold exists in the partial area, the feature point detection unit 40 determines the at least one pixel as a feature point candidate of the partial area.

  When the pixel whose first evaluation value exceeds the first threshold does not exist in the partial region, the feature point detection unit 40 compares the second evaluation value of each pixel in the partial region with the second threshold. . When at least one pixel whose second evaluation value exceeds the second threshold exists in the partial region, the feature point detection unit 40 determines the at least one pixel as a feature point candidate for the partial region.

  When there is a pixel whose first evaluation value exceeds the first threshold value in the partial area, the feature point detection unit 40 selects a pixel having the largest third evaluation value or a plurality of upper pixels having a large third evaluation value as a feature of the partial area. Decide on a point. When two feature points are set for each partial region, the pixel having the third highest evaluation value and the second largest pixel are determined as the feature points. When there is no pixel whose first evaluation value exceeds the first threshold in the partial area and there is a pixel whose second evaluation value exceeds the second threshold, the feature point detection unit 40 has the largest fourth evaluation value. A pixel or a plurality of higher-order pixels having a large fourth evaluation value are determined as feature points of the partial region.

  When there is no pixel whose first evaluation value exceeds the first threshold in the partial area and no pixel whose second evaluation value exceeds the second threshold, the feature point detection unit 40 sets in advance in the partial area. The pixel at the fixed position is set as a temporary feature point of the partial area, and a flag indicating that no regular feature point exists is set. The pixel at the fixed position may be a pixel located at the center of each partial area, a pixel at the upper left corner, or a pixel at another position.

  The feature point detection unit 40 specifies the position information (for example, coordinate values) and pixel values of the feature points of the partial areas in which the feature points are detected, and effective feature points are set in the partial areas. You may raise the flag which shows. The feature point detection unit 40 cannot detect a feature point, and for the partial area in which the pixel at the fixed position is set, specifies the position information and the pixel value of the pixel, and sets an invalid feature point in the partial area. A flag may be set to indicate that this is done.

  FIG. 9A is a diagram illustrating a frame image in which feature points are set by the feature point detection apparatus 150 according to the second embodiment. In the frame image shown in FIG. 9A, a rectangular object is drawn at the center. A plurality of square windows are set as the partial areas in the frame image shown in FIG. Here, one feature point is set for each square window. Square window W (1,1) to Square window W (6,1), Square window W (1,1) to Square window W (1,5), Square window W (1,5) to Square window W (3 , 5), and the square window W (6, 1) to square window W (6, 5) do not include the object. In those square windows, the corners and edges of the object are not detected. Accordingly, dummy feature points at fixed positions are set in these square windows. In FIG. 9A, a dummy feature point is set at the center position of each square window.

  In addition, the square window W (3, 3) and the square window W (4, 3) are included in the object, and do not include the corners and edges of the object. Accordingly, dummy feature points at fixed positions are also set in these square windows.

  The square window W (2, 2), the square window W (5, 2), the square window W (2, 4), and the square window W (5, 4) include the corners of the object. Therefore, in each square window, the feature point F (2, 2), feature point F (5, 2), feature point F (2, 4) and feature point F (5, 4) are detected at the corners, respectively. Is done. Square window W (3,2), Square window W (4,2), Square window W (2,3), Square window W (5,3), Square window W (3,4) and Square window W (4 , 4), the square window W (4, 5) and the square window W (5, 5) include the horizontal edge or the vertical edge of the object. Therefore, in each square window, the feature point F (3,2), feature point F (4,2), feature point F (2,3), feature point F (5,3), feature point F (3,4), feature point F (4,4), feature point F (4,5) and feature point F (5,5) are detected.

  FIG. 9B is a diagram showing a frame image in which feature points are set by the KLT method. In the KLT method, a Harris operator is used. The object drawn in the frame image of FIG. 9B is the same as the object drawn in the frame image of FIG. Although feature points are detected at corners and edges of the object, there are three misdetections (see E1, E2, and E3). These feature points are detected in regions other than corners and edges.

  FIG. 10 is a flowchart illustrating an example of processing for detecting a feature point from a frame image using the feature point detection apparatus 150 according to the second embodiment. The algorithm shown in this flowchart is for setting one feature point in one partial area. Therefore, this algorithm is executed for all the partial areas set in the frame image.

  First, an arbitrary pixel in the target partial area is set as an initial value of the target pixel. For example, the pixel at the upper left corner is set as the initial value. The feature point detection unit 40 compares the first evaluation value E1 = | Ya−Yb | of the target pixel with the first threshold value Yth1 (S30). When the first evaluation value E1 is larger than the first threshold Yth1 (Y in S30), the feature point detection unit 40 sets the pixel as a candidate for a corner and an oblique edge (S32).

  When the first evaluation value E1 is equal to or less than the first threshold value Yth1 (N in S30), the feature point detection unit 40 compares the second evaluation value E2 = | Yc−Yd | of the target pixel with the second threshold value Yth2 ( S34). When the second evaluation value E2 is larger than the second threshold Yth2 (Y in S34), the feature point detection unit 40 sets the pixel as a candidate for a horizontal edge and a vertical edge (S36). When the second evaluation value E2 is equal to or smaller than the second threshold Yth2 (N in S34), the feature point detection unit 40 does not set the pixel as a feature point candidate.

  The feature point detection unit 40 determines whether or not the determination as to whether or not the feature point should be set has been completed for all the pixels in the partial region (S38). If not completed (N in S38), the feature point detection unit 40 sets the next pixel as a target pixel (S40). Thereafter, the process proceeds to step S30 and the determination process described above is continued. If completed (Y in S38), the following processing is executed.

  The feature point detection unit 40 determines whether there are corner and oblique edge candidates (S42). If it exists (Y in S42), the feature point detection unit 40 sets a feature point having a maximum third evaluation value E3 = | Ya | + | Yb | among corner and oblique edge candidates (S44). The process is terminated. When there are no corner and oblique edge candidates (N in S42), the feature point detection unit 40 determines whether there are horizontal edge and vertical edge candidates (S46). If it exists (Y in S46), the feature point detection unit 40 sets a candidate having the maximum fourth evaluation value E4 = | Yc | + | Yd | as a feature point among candidates for the horizontal edge and the vertical edge (S48). ), The process is terminated. If there is no candidate for a horizontal edge or a vertical edge (N in S46), the feature point detection unit 40 sets a preset fixed pixel as a dummy feature point (S50), and ends the process.

  The algorithm shown in this flowchart sets one feature point in one partial area, but an algorithm for setting a plurality of feature points in one partial area can be easily implemented by those skilled in the art. .

  As described above, according to the second embodiment, the same effects as those of the first embodiment can be obtained. In addition to the effects of the first embodiment, the following effects can be obtained. In the second embodiment, the frame image is divided into a plurality of partial areas, and a fixed number of feature points are set in each partial area. Also, dummy feature points are set for the partial areas where feature points could not be detected. As a result, the density or distribution of feature points in the frame image can be made uniform. Therefore, it becomes easy to design the subsequent processing block of the feature point detection apparatus 150 with hardware resources.

  Next, a moving image processing system to which the feature point detection apparatus 150 according to Embodiment 2 is applied will be described. FIG. 11 shows a configuration of a moving image processing system 200 according to the third embodiment. The moving image processing system 200 includes an imaging unit 210 and a moving image processing device 220. The imaging unit 210 includes an imaging element 60 and a signal processing unit 62. The moving image processing apparatus 220 includes a frame buffer 70, a target frame selection unit 72, a reference frame selection unit 74, an optical flow detection unit 76, and a post-processing unit 80. The optical flow detection unit 76 includes a feature point detection device 150 and an optical flow calculation unit 78.

  The imaging unit 210 is mounted on a moving body such as a vehicle or a building such as a wall or a pillar, captures a moving image, and outputs the moving image to the moving image processing apparatus 220. The imaging device 60 uses a CCD (Charge Coupled Devices) sensor, a CMOS (Complementary Metal-Oxide Semiconductor) image sensor, or the like, converts incident light into an electrical signal, and outputs the electrical signal to the signal processing unit 62.

  The signal processing unit 62 converts the analog signal input from the image sensor 60 into a digital signal. Further, a smoothing filter may be provided, and smoothing processing may be performed on the image data output to the moving image processing apparatus 220. Further, other noise removal processing may be performed.

  The moving image processing device 220 detects an optical flow from the image data input from the imaging unit 210. The frame buffer 70 has an area for storing q (q is a natural number) frame images, and temporarily stores image data input from the imaging unit 210. When the area of the frame buffer 70 overflows, the frame image input first is discarded in order. The frame image stored in the frame buffer 70 is output to the target frame selection unit 72 and the reference frame selection unit 74.

  The target frame selection unit 72 sequentially outputs the frame images in the frame buffer 70 to the optical flow detection unit 76. Here, the frame images in the frame buffer 70 may be appropriately thinned out and output. The reference frame selection unit 74 selects a reference frame image of the frame image selected by the target frame selection unit 72 and acquires it from the frame buffer 70.

  The optical flow detection unit 76 detects an optical flow using the target frame image and the reference frame image. The feature point detection device 150 is the device described in the second embodiment, and detects a plurality of feature points from the target frame image selected by the target frame selection unit 72. At this time, as described above, feature points are detected in units of partial areas. Note that the feature point detection process may be executed in a plurality of layers having different resolutions. The feature point detection device 150 outputs position information and pixel values of the detected feature points to the optical flow calculation unit 78.

  The optical flow calculation unit 78 calculates optical flows of a plurality of feature points detected by the feature point detection device 150, respectively. The optical flow calculation unit 78 includes a target frame image selected by the target frame selection unit 72, position information and pixel values of feature points input from the feature point detection device 150, and a reference frame selected by the reference frame selection unit 74. An optical flow is calculated based on the image. More specifically, a point corresponding to each of a plurality of feature points detected in the target frame image is searched for in the reference frame image.

  The post-processing unit 80 performs predetermined post-processing on the optical flow output from the optical flow detection unit 76. For example, the post-processing unit 80 invalidates the optical flow of the feature point of the partial area where the flag indicating that the regular feature point does not exist is set.

  Further, the post-processing unit 80 may perform a process of subtracting a component due to the movement of the imaging unit 210 itself from the input optical flow. Further, in order to remove noise, the post-processing unit 80 may perform a process of limiting the length of the optical flow to be validated by comparing the length of the input optical flow with the sixth threshold value. The sixth threshold value for noise removal may be a preset value, or may be a value obtained by adjusting the average value or the average value. Further, the post-processing unit 80 may calculate an average value of the lengths of the optical flows output from the optical flow detection unit 76.

  As described above, according to the third embodiment, as a pre-process for optical flow calculation, the feature point detection device 150 according to the second embodiment is used to detect a feature point that is to be an optical flow calculation target. As a result, the calculation objects can be greatly reduced. Therefore, it is possible to speed up the optical flow calculation process. Further, since the density of feature points is uniform, it is easy to configure the optical flow detection unit 76 and the post-processing unit 80 with hardware resources. In addition, since the optical flow can be masked in the partial region where the feature point cannot be detected, erroneous detection of the optical flow can be reduced.

  The present invention has been described based on some embodiments. It is understood by those skilled in the art that these embodiments are exemplifications, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. By the way.

  In the first and second embodiments, the example in which the pixel of interest and the neighboring pixels on both sides thereof are in direct contact with each other is shown as the basis for calculating the second derivative value. One pixel or several pixels may be inserted between them.

  In the third embodiment, the example in which the feature point detection device 150 according to the second embodiment is mounted in the optical flow detection unit 76 has been described. However, the feature point detection device 100 according to the first embodiment may be mounted. . In the third embodiment, the configuration in which the optical flow based on the dummy feature point is masked by the post-processing unit 80 in the partial region where the feature point cannot be detected has been described. However, the calculation of the optical flow itself is invalid in the partial region. It is also possible to use a configuration.

It is a block diagram which shows the structure of the feature point detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the operator required for calculation of a secondary differential value based on Embodiment 1. FIG. 6 is a diagram illustrating a one-dimensional filter for calculating a second derivative value in the −45 ° direction according to Embodiment 1. FIG. It is a figure which shows transition of the 2nd derivative value in the space axis of the -45 degree direction in an image based on Embodiment 1. FIG. It is a figure for demonstrating the principle which detects a corner or a diagonal edge from the 1st evaluation value, and a horizontal edge or a vertical edge from the 2nd evaluation value based on Embodiment 1. FIG. It is a figure which shows the table which put together the secondary differential value of four directions of each pattern in FIG. 5 is a flowchart illustrating an example of processing for detecting a feature point from a frame image using the feature point detection apparatus according to the first embodiment. It is a block diagram which shows the structure of the feature point detection apparatus which concerns on Embodiment 2 of this invention. FIG. 9A is a diagram illustrating a frame image in which feature points are set by the feature point detection apparatus according to the second embodiment. FIG. 9B is a diagram showing a frame image in which feature points are set by the KLT method. 10 is a flowchart illustrating an example of processing for detecting a feature point from a frame image using the feature point detection device according to the second embodiment. FIG. 10 is a diagram illustrating a configuration of a moving image processing system according to a third embodiment.

Explanation of symbols

  5 division unit, 10 secondary differential value calculation unit, 15 operator, 16 adder, 17 multiplier, 18 subtractor, 20 evaluation value calculation unit, 30 determination unit, 40 feature point detection unit, 60 image sensor, 62 signal processing Unit, 70 frame buffer, 72 target frame selection unit, 74 reference frame selection unit, 76 optical flow detection unit, 78 optical flow calculation unit, 80 post-processing unit, 200 moving image processing system, 210 imaging unit, 220 moving image processing device .

Claims (8)

A second-order differential value calculation unit that calculates a second-order differential value in the diagonally lower right direction and a second-order differential value in the diagonally right-up direction at the target pixel in the image;
An evaluation value calculation unit that calculates, as a first evaluation value, an absolute value difference between the second-order differential value in the diagonally descending right direction and the secondary differential value in the diagonally right-up direction calculated by the secondary differential value calculating unit. When,
A determination unit that determines whether the target pixel corresponds to a corner or an oblique edge in the image by comparing the first evaluation value calculated by the evaluation value calculation unit with a first threshold value; ,
A feature point detection apparatus comprising: The secondary differential value calculation unit calculates a secondary differential value in the horizontal direction and a secondary differential value in the vertical direction at the target pixel,
The evaluation value calculation unit calculates an absolute value difference between the secondary differential value in the horizontal direction and the secondary differential value in the vertical direction, calculated by the secondary differential value calculation unit, as a second evaluation value;
The determination unit compares the second evaluation value calculated by the evaluation value calculation unit with a second threshold value to determine whether the target pixel corresponds to a horizontal edge or a vertical edge in the image. The feature point detection apparatus according to claim 1, wherein: The evaluation value calculation unit calculates, as a third evaluation value, a sum of an absolute value of the second-order differential value in the diagonally lower right direction and an absolute value of the second-order differential value in the diagonally right upward direction at the target pixel. And calculating the sum of the absolute value of the secondary differential value in the horizontal direction and the absolute value of the secondary differential value in the vertical direction as a fourth evaluation value,
The determination unit determines the strength of the corner or the oblique edge by comparing the third evaluation value with a third threshold, and compares the fourth evaluation value with a fourth threshold to The feature point detection apparatus according to claim 2, wherein a strength of a horizontal edge or the vertical edge is determined. A dividing unit for dividing the image into a plurality of partial regions;
For each pixel in each of the partial areas divided by the dividing unit, a second-order differential value in the right-down diagonal direction, a second-order differential value in the right-up diagonal direction, a second-order differential value in the horizontal direction, and a second-order differential value in the vertical direction A second derivative calculation unit for calculating a value;
For each pixel in each partial region, an absolute value difference between the second-order differential value in the diagonally lower right direction and the second-order differential value in the diagonally right upward direction is calculated as a first evaluation value, and the second value in the horizontal direction is calculated. An evaluation value calculation unit that calculates an absolute value difference between the second derivative value and the second derivative value in the vertical direction as a second evaluation value;
A feature point detection unit for detecting at least one feature point for each partial region,
The feature point detection unit includes:
When the first evaluation value of each pixel in a certain partial region is compared with a first threshold value, and when at least one pixel having the first evaluation value exceeding the first threshold value exists in the partial region, Determining at least one pixel as a feature point candidate of the partial region;
If there is no pixel in the partial area where the first evaluation value exceeds the first threshold, the second evaluation value of each pixel in the partial area is compared with a second threshold, and the second When at least one pixel having an evaluation value exceeding the second threshold exists in the partial area, the feature point detection apparatus determines at least one pixel as a feature point candidate of the partial area. The evaluation value calculation unit
The sum of the absolute value of the second-order differential value in the right-down diagonal direction and the absolute value of the second-order differential value in the right-up diagonal direction at each pixel or the pixel where the first evaluation value exceeds the first threshold value. Calculated as the third evaluation value,
The fourth evaluation value is the sum of the absolute value of the secondary differential value in the horizontal direction and the absolute value of the secondary differential value in the vertical direction at each pixel or the pixel where the second evaluation value exceeds the second threshold value. As
The feature point detector
When there is a pixel having the first evaluation value exceeding the first threshold in the partial region, the pixel having the largest third evaluation value or the plurality of upper pixels having the large third evaluation value is selected as a feature point of the partial region Decided on
In the partial area, when there is no pixel whose first evaluation value exceeds the first threshold and there is a pixel whose second evaluation value exceeds the second threshold, the pixel having the largest fourth evaluation value The feature point detection apparatus according to claim 4, wherein a plurality of upper pixels having a large fourth evaluation value are determined as feature points of the partial region. The feature point detector
In the partial area, when there is no pixel in which the first evaluation value exceeds the first threshold value and no pixel in which the second evaluation value exceeds the second threshold value, a preset value in the partial area is set. 6. The feature point detection device according to claim 4, wherein a pixel at a fixed position is set as a temporary feature point of the partial area, and a flag indicating that no regular feature point exists is set. The feature point detection device according to any one of claims 4 to 6, wherein a plurality of feature points are detected from a frame image included in the moving image;
An optical flow calculator that calculates optical flows of a plurality of feature points detected by the feature point detection device;
A moving image processing apparatus comprising: The feature point detection apparatus according to claim 6, wherein a plurality of feature points are detected from within a frame image included in the moving image,
An optical flow calculator that calculates optical flows of a plurality of feature points detected by the feature point detection device;
A post-processing unit for invalidating the optical flow of the feature points of the partial area where the flag is set;
A moving image processing apparatus comprising: