JP2011043922A - Device and method for recognizing traveling object, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To quickly and precisely detect a traveling object, concerning a device and method for recognizing a traveling object, and a program. <P>SOLUTION: Before template matching, a feature point for matching is selected in advance from a retrieval region set in advance in an image. Then, the template matching is performed only on an image including the feature point selected in advance. Thus, it is possible to selectively perform matching only on the region assumed to have high correlation value which is obtained by matching. Therefore, it is possible to quickly and precisely detect the traveling object. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a mobile object recognition apparatus, a mobile object recognition method, and a program. More specifically, the present invention relates to a mobile object recognition apparatus that recognizes a mobile object that moves within a visual field, and a mobile object that recognizes a mobile object that moves within a visual field. The present invention relates to a recognition method and a program.

  2. Description of the Related Art In recent years, driving support systems that detect a moving body around the own vehicle and a moving body approaching the own vehicle from a driver's blind spot have been put into practical use. In this type of system, it is necessary to detect a moving body with high accuracy in a short time due to its nature. Therefore, various methods for performing image processing with high accuracy and speed have been proposed (see, for example, Patent Documents 1 to 3).

Japanese Patent Laid-Open No. 9-18863 Japanese Patent Laid-Open No. 8-030792 JP-A-4-207790

  In the method described in Patent Literature 1, an image obtained by imaging a moving body is divided into, for example, a first image and a second image. Then, one image is selected from the first image and the second image according to the degree of risk, and image processing is executed only for the selected image. By using this method, the moving direction of the moving body can be detected in a short time.

  Further, in the method described in Patent Document 2, when the moving body moves, the optical flow caused by an obstacle that periodically appears in an image such as a railing of a bridge or a guardrail, and the moving body. An optical flow is identified. By using this method, an optical flow caused by an obstacle can be excluded, and a moving object can be detected based only on the optical flow caused by the moving object that is the detection target. Therefore, the amount of information used for detecting the moving body can be reduced, and as a result, the moving body can be detected quickly.

  Further, in the method described in Patent Document 3, the vector with the lowest matching degree is detected from the optical flows between images that are different in time by one frame. By using this method, it is possible to detect the moving object based only on the vector having the most movement information of the moving object. Therefore, the amount of information used for detecting the moving body can be reduced, and as a result, the moving body can be detected quickly.

  Each of the above-described methods limits the area to be calculated in order to reduce the amount of calculation in image processing. However, in order to search for a corresponding region between images, a convolution operation is required. This convolution calculation has a large calculation amount, and if the search area is wide, the convolution calculation takes a long time. For this reason, when trying to execute the above three methods, a high-speed arithmetic device is required.

  In addition, when the luminance of the pixels constituting the search target image is low or when the difference between the luminance of the pixels constituting the corresponding region and the luminance of the pixels in the surrounding region is small, the corresponding region is selected. It may be difficult to search accurately, and errors may be included in the optical flow.

  The present invention has been made under the above circumstances, and an object thereof is to provide a moving body recognition apparatus and the like that can detect a moving body with high accuracy in a short time.

  In order to achieve the above object, the mobile object recognition apparatus of the present invention calculates a feature value for pixels constituting each of the first image and the second image obtained by sequentially imaging a moving mobile object. A calculation means, a feature point extraction means for extracting each feature point of the first image and the second image based on the feature quantity, a feature quantity of the feature point of the first image, and a feature point of each feature point of the second image A selection means for calculating a difference between the second image and a feature point of the second image in which the difference is within a predetermined range; a feature point of the first image; and a feature point of the selected second image. Correlation value calculation means for calculating a correlation value, and optical flow generation for generating an optical flow having a feature point of the first image as a start point and a feature point of the second image having the highest correlation value with respect to the feature point as the start point as an end point Means.

  The mobile object recognition apparatus further includes search area setting means for setting search areas respectively corresponding to the feature points of the first image on the second image, and the selection means includes the second image in the search area. The selection of feature points may be executed.

  Further, the feature point extracting means may extract the feature points based on the feature amount of the calculation target pixel that is the target of calculation of the feature amount and the average luminance of pixels around the calculation target pixel.

  In addition, the feature point extraction unit corresponds to the calculation target pixel when the ratio of the feature quantity of the calculation target pixel that is the calculation target of the feature quantity to the average luminance of the pixels around the calculation target pixel is equal to or greater than a predetermined value. The feature points to be extracted may be extracted.

  The correlation value calculating means may calculate an correlation value between the template and an image in an area including the feature point of the second image using an image including the feature point of the first image as a template.

  In order to achieve the above object, a moving object recognition method of the present invention includes a step of calculating feature amounts for pixels constituting each of a first image and a second image obtained by sequentially imaging moving moving objects, and Based on the feature amount, the step of extracting the feature points of the first image and the second image, and the difference between the feature amount of the feature point of the first image and the feature amount of each feature point of the second image is calculated. And selecting each of the feature points of the second image whose difference is within a predetermined range, and calculating a correlation value between each of the feature points of the first image and each of the feature points of the selected second image. Generating an optical flow having a feature point of the first image as a start point and a feature point of the second image having the highest correlation value with respect to the feature point as the start point as an end point.

  In order to achieve the above object, a program according to the present invention comprises means for calculating a feature amount for pixels constituting each of a first image and a second image obtained by sequentially imaging a moving body. Based on the feature amount, a means for extracting the feature point of each of the first image and the second image, and the difference between the feature amount of the feature point of the first image and the feature amount of each feature point of the second image is calculated. And means for selecting each feature point of the second image within which the difference falls within a predetermined range; and means for calculating a correlation value between each feature point of the first image and each feature point of the selected second image And a means for generating an optical flow having a feature point of the first image as a start point and a feature point of the second image having the highest correlation value with respect to the feature point as the start point as an end point.

  According to the present invention, it is possible to accurately detect a moving body in a short time.

It is a block diagram of the mobile body recognition system which concerns on 1st Embodiment. It is a figure which shows the arrangement | positioning of a camera roughly. FIG. 3 is a first diagram illustrating an image captured by an imaging device. It is FIG. (2) which shows the image imaged by the imaging device. It is FIG. (1) for demonstrating operation | movement of a feature point extraction part. It is FIG. (2) for demonstrating operation | movement of a feature point extraction part. It is a figure which shows an example of the search area | region set on the image. It is a figure which shows the template prescribed | regulated on the image. It is a figure for demonstrating operation | movement of a correlation value calculating part. It is a figure for demonstrating operation | movement of an optical flow production | generation part. It is a block diagram of the mobile body recognition system which concerns on 2nd Embodiment. It is a flowchart (the 1) for demonstrating operation | movement of a mobile body recognition apparatus. It is a flowchart (the 2) for demonstrating operation | movement of a mobile body recognition apparatus. It is a flowchart (the 3) for demonstrating operation | movement of a mobile body recognition apparatus.

<< First Embodiment >>
Hereinafter, a first embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing a schematic configuration of a moving object recognition system 10 according to the present embodiment. The mobile object recognition system 10 is a system that recognizes and detects a mobile object included in images sequentially captured by the imaging device 20, for example. As illustrated in FIG. 1, the moving body recognition system 10 includes an imaging device 20 and a moving body recognition device 30 that recognizes a moving body that moves within the field of view of the imaging device 20.

  The imaging device 20 is a device that converts an image acquired by imaging a subject into an electrical signal and outputs the electrical signal. For example, as illustrated in FIG. 2, the imaging device 20 captures an image of a vehicle as a moving body that travels behind the vehicle 100 via a camera 20 a attached to the rear of the vehicle 100, and relates to an image obtained by imaging. Information is output to the moving body recognition apparatus 30.

  For example, when the imaging device 20 sequentially images the moving body 101 traveling in the right lane at a speed faster than the own vehicle, the image P1 shown in FIG. 3 and the image P2 shown in FIG. 4 are sequentially imaged. Information about the image P1 and the image P2 is output to the moving body recognition apparatus 30.

  Returning to FIG. 1, the mobile object recognition device 30 includes a storage unit 31, a feature point extraction unit 32, a search region setting unit 33, a feature point selection unit 34, a correlation value calculation unit 35, an optical flow generation unit 36, and a grouping processing unit 37. , And a moving body detection unit 38.

  The storage unit 31 stores information on images sequentially output from the imaging device 20 in time series. Further, information as a processing result of each of the units 32 to 38 is sequentially stored.

  The feature point extraction unit 32 calculates a feature amount for each pixel constituting the image stored in the storage unit 31, and extracts a feature point included in the image based on the feature amount. The feature quantity f (x, y) of the pixel M (x, y) indicated in the XY coordinate system defined on the image is expressed by the following equation (1) using the function I (x, y) indicating the luminance gradient of the image: Represented by 1).

  However, Ixx, Iyy, and Ixy are respectively represented by the following formulas (2) to (3), and k is a constant.

The feature point extraction unit 32 first calculates a feature amount f (x, y) for each pixel M (x, y) constituting the image using Expression (1). Next, the feature point extraction unit 32 calculates the average value AVG (x, y) of the brightness of the pixels around the pixel M (x, y), and calculates the feature value f (x, y) as the average brightness. Divide the value AVG (x, y) by the fourth power. The feature point extraction unit 32 compares the comparison value V (= f (x, y) / AVG (x, y) ⁴ ) obtained thereby with a preset threshold value, and the comparison value V is equal to or greater than the threshold value. In this case, the pixel M (x, y) at this time is extracted as a feature point. Here, the reason why the feature quantity f (x, y) is divided by the fourth power of the average luminance value AVG (x, y) is to normalize the feature quantity f (x, y) with respect to the brightness. .

FIG. 5 is a diagram schematically showing feature points a _{1 to} a ₆ related to the moving object 101 extracted from the image P1. FIG. 6 is a diagram schematically illustrating feature points b _{1 to} b ₁₈ related to the moving object 101 extracted from the image P2. In the image P1 and the image P2, a discontinuous point where the outline of the moving body 101 changes sharply, a point where the shape of the parts constituting the moving body 101 changes discontinuously, and the like are extracted.

  When the feature point extraction is completed, the feature point extraction unit 32 outputs information about the extracted feature points to the storage unit 31 and notifies the search region setting unit 33 that the feature point extraction is completed.

  Returning to FIG. 1, the search region setting unit 33 sequentially selects the feature points of the first image captured first among the set of images sequentially captured, and selects the search region corresponding to the selected feature points. The second image captured after the first image is set.

For example, FIG. 7 shows a search area F ₁ corresponding to the feature point a ₁ on the first image P1 of the pair of images P1 and P2. The search area F ₁ is an area where a feature point corresponding to the feature point a ₁ on the image P ₁ is expected to be included. Specifically, the search area F ₁ is predicted using an optical flow having the feature point a ₁ on the image P ₁ as the end point and the feature point on the image captured before the image P 1 is captured as the start point. Can be determined in consideration of the moving speed, moving direction, etc. The search area setting unit 33 sets search areas F _{1 to} F ₆ (partial areas of the image P2) for all the feature points a _{1 to} a ₆ on the image P1 on the image P2.

  When the setting of the search area is completed, the search area setting unit 33 outputs information related to the search area to the storage unit 31 and notifies the feature point selection unit 34 that the setting of the search area is completed.

  Returning to FIG. 1, the feature point selection unit 34 has a feature amount equivalent to the feature amount of the feature point of the first image from the search area set in the second image so as to correspond to the feature point of the first image. Select feature points.

For example, the feature point selection unit 34 includes the feature amount of the feature point a ₁ shown in FIG. 5 and the four feature points b ₁ , b ₃ , b ₄ of the image P2 in the search area F ₁ shown in FIG. , B ₉ are compared. The difference between the feature quantity of the feature point a ₁ selects the feature points to be smaller than the predetermined threshold value. The feature point selection unit 34 selects feature points from the search areas F _{2 to} F ₆ set for the feature points a _{2 to} a _{6 in} the same manner.

  Returning to FIG. 1, the correlation value calculation unit 35 sequentially selects feature points of the first image captured first from among a set of images sequentially captured. Then, a correlation value between the selected feature point and the feature point selected from the search area of the second image set for the feature point is calculated.

Specifically, the correlation value calculation section 35, first, as shown in FIG. 8, to define a rectangular image around the feature point a ₁ of the image P1 (the partial image of the image P1) as the template T ₁ . The template T ₁ is an image consisting of M rows and N matrix to pixels arranged in columns. In the following description, the coordinate template T _1, it is assumed to refer to the coordinates of the center of the template.

Next, the correlation value calculation unit 35 matched the template T ₁ with the image including the feature point selected by the feature point selection unit 34 among the feature points b ₁ , b ₃ , b ₄ , and b ₉ . The correlation value R is calculated. The correlation value R can be calculated using, for example, the following equation (5) indicating normalized cross-correlation. Note that I (i, j) is the luminance of the pixel located in the i-th row and the j-th column of the image to be matched. T (i, j) is the luminance of the pixel located in the i-th row and the j-th column of the template T. Further, I _AVG is an average value of the luminance of the pixels constituting the image to be matched, and _TAVG is an average value of the luminance of the pixels constituting the template. I _AVG and T _AVG are represented by the following formulas (6) and (7).

The correlation value calculation unit 35 matches the feature point a ₁ included in the template T ₁ with the feature point selected from the feature points b 1, b 3, b 4, and b 9 included in the search region F ₁ . , template T _1, calculates a correlation value R between the target image overlapping with the template T _1. For example, the feature point selection unit 34, when the feature point b ₁ in the search area F ₁ is selected, the correlation value calculation unit 35, a feature point a ₁ of the template T ₁ is coincident to the feature point b ₁ as such, overlaying the template _{T 1} to the image P2. Then, the target image overlaps with the template T ₁ of the image P2, a correlation value R with the template T _1, is calculated based on the equation (5).

Next, as can be seen with reference to FIG. 9, the correlation value calculator 35 defines a matching region MA larger than the template T ₁ around the feature point b ₁ in the search region F ₁ . Then, the template T _1, while moving at a predetermined pitch in the matching area MA, the template T _1, the correlation value R between the target image overlapping with the template T _1, sequentially based on the equation (5) is calculated To do.

The correlation value calculator 35 also performs the same processing as described above for the search points F _{2 to} F ₆ set for the feature points a _{2 to} a ₆ and the feature points a _{2 to} a ₆ of the image P1.

  When the calculation of the correlation value is completed, the correlation value calculation unit 35 outputs information on the calculated correlation value to the storage unit 31 and notifies the optical flow generation unit 36 that the calculation of the correlation value is completed.

  Returning to FIG. 1, based on the correlation value calculated by the correlation value calculator 35, the optical flow generator 36 uses the feature point of the first image as the start point and the feature point of the second image or its vicinity as the end point. Generate an optical flow.

For example, the optical flow generator 36 in FIG. 9 shows the position coordinates (X2 ₁ , Y2) of the template T ₁ on the image P2 when the correlation value between the template T ₁ and the image in the search area F ₁ is the highest. ₁ ) is specified as a new feature point. An optical flow having the position coordinates (X2 ₁ , Y2 ₁ ) as the end point E ₁ and the position coordinates (X1 ₁ , Y1 ₁ ) on the image P of the feature point a ₁ as the start point S ₁ is displayed on the images P1 and P2. It is defined in a common XY coordinate system. The optical flow generation unit 36 performs the same processing as described above for the feature points a _{2 to} a ₆ on the image P1, and as shown in FIG. 10 as an example, the start points S _{1 to} S ₆ and the end points E ₁ to E ₆ are performed. generating an optical flow of a group defined by the E _6.

  When the optical flow is generated, the optical flow generation unit 36 outputs information related to the optical flow to the storage unit 31 and notifies the grouping processing unit 37 that the generation of the optical flow is completed.

  Returning to FIG. 1, the grouping processing unit 37 performs grouping of the generated group of optical flows. As shown in FIG. 10, in the present embodiment, the case where there are six optical flows related to the moving body 101 has been described for convenience of explanation. However, actually, tens or hundreds of feature points can be extracted from an image obtained by capturing the moving object 101, and tens or hundreds of optical flows can be defined.

  The grouping processing unit 37 excludes optical flows including a large amount of noise components from among tens or hundreds of optical flows, and groups the remaining optical flows. For example, when the moving body 101 has a complete linear motion, the straight lines that coincide with each optical flow should intersect at one point (vanishing point). Therefore, the grouping processing unit 37 excludes the optical flow when the straight line that matches the optical flow is significantly away from the vanishing point where the straight line that matches the other optical flow converges, and the remaining optical flow is removed. Grouping is considered as an optical flow related to the same moving object.

  When the optical flow grouping is completed, the grouping processing unit 37 outputs information on the grouped optical flows to the storage unit 31 and notifies the moving body detection unit 38 that the grouping is completed.

  The moving body detection unit 38 detects and outputs a distance between the own vehicle and the moving body, a relative speed and a relative moving direction between the own vehicle and the moving body, based on the grouped optical flows. For example, the moving body detection unit 38 detects and outputs the relative speed and the relative movement direction of the moving body from the difference between the start point arrangement and the end point arrangement of the group of optical flows.

  As described above, in the first embodiment, before the matching process using the template T including the feature point of the image P1 is performed by the correlation value calculation unit 35, the feature point selection unit 34 performs the image P2. A feature point to be subjected to the matching process is selected from the search region set to. Then, the matching process using the template T is executed only for the image including the feature point selected by the feature point selection unit 34. For this reason, it is possible to execute the matching process only for the region where the correlation value obtained by executing the matching process is expected to be high, and as a result, it is possible to detect the moving object with high accuracy in a short time. Become.

  In the first embodiment, the matching process is executed only for an area where the correlation value obtained by executing the matching process is expected to be high. Therefore, it is possible to avoid the feature points of the image P1 and the feature points of the image P2 having low mutual feature amounts from being associated with each other due to an error in the matching process.

  Further, in the first embodiment, the feature point extraction unit 32 includes the feature amount of the pixels constituting each of the image P1 and the image P2, and the average luminance of the pixels existing around the pixel whose feature amount is to be calculated. In consideration of the above, feature points are extracted from the image P1 and the image P2. Therefore, it is possible to accurately extract feature points regardless of the brightness of the captured image, that is, the brightness of each pixel constituting the image.

Specifically, in general, a bright region in an image has a high contrast and a feature amount tends to increase. On the other hand, in a dark region, the contrast is low and the feature amount tends to be small. For this reason, when the feature amount calculated for the pixel is simply compared with a predetermined threshold, a large number of feature points are extracted in a bright region, and a small number of feature points are extracted in a dark region. In the present embodiment, since the feature amount is calculated in consideration of the average luminance of the pixels existing around the pixel whose feature amount is to be calculated, an appropriate feature is obtained regardless of the brightness of the captured image. It becomes possible to extract points.
<< Second Embodiment >>
Next, a second embodiment of the present invention will be described with reference to FIGS. In addition, about the structure same or equivalent to 1st Embodiment, while using an equivalent code | symbol, the description is abbreviate | omitted or simplified.

  The mobile object recognition system 10 according to the present exemplary embodiment is such that the mobile object recognition apparatus 30 is realized by the same configuration as a general computer or a device such as a microcomputer. This is different from the body recognition system 10.

  FIG. 11 is a block diagram showing a physical configuration of the mobile object recognition system 10. As shown in FIG. 11, the moving body recognition system 10 includes an imaging device 20 and a moving body recognition device 30 including a computer.

  The moving body recognition device 30 includes a central processing unit (CPU) 30a, a main storage unit 30b, an auxiliary storage unit 30c, a display unit 30d, an input unit 30e, an interface unit 30f, and a system bus 30g that interconnects the above units. It consists of

  CPU30a performs the process mentioned later with respect to the image acquired by the imaging device 20 according to the program memorize | stored in the auxiliary storage part 30c.

  The main storage unit 30b includes a RAM (Random Access Memory) and the like, and is used as a work area for the CPU 30a.

  The auxiliary storage unit 30c includes a nonvolatile memory such as a ROM (Read Only Memory), a magnetic disk, and a semiconductor memory. The auxiliary storage unit 30c stores programs executed by the CPU 30a, various parameters, and the like. In addition, information about an image output from the imaging device 20 and information including a processing result by the CPU 30a are sequentially stored.

  The display unit 30d includes a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), or the like, and displays the processing result of the CPU 30a.

  The input unit 30e includes a pointing device such as a keyboard and a mouse. The operator's instruction is input via the input unit 30e and notified to the CPU 30a via the system bus 30g.

  The interface unit 30f includes a serial interface or a LAN (Local Area Network) interface. The imaging device 20 is connected to the system bus 30g via the interface unit 30f.

The flowchart in FIG. 12 corresponds to a series of processing algorithms of a program executed by the CPU 30a. Hereinafter, the process performed by the mobile object recognition device 30 will be described with reference to FIG. This process is executed after the moving body recognition system 10 is activated and information related to the image captured by the imaging device 20 is output. Moreover, as a premise, from the imaging device 20, an image P1 shown in FIG. 5, and the image P2 are successively outputted as shown in FIG. 6, the image P1 is the characteristic point a ₁ ~a ₆ has already been extracted Shall.

  First, in the first step S101, the CPU 30a calculates a feature amount for each pixel constituting the image P2 captured by the imaging device 20, and extracts a feature point included in the image P2 based on the feature amount. FIG. 13 is a diagram showing the subroutine 1 executed in step S101. The CPU 30a implements the processing in step S101 by executing the subroutine 1 shown in FIG.

  In the first step S201, the CPU 30a sets the coordinates of the pixel for calculating the feature amount. This coordinate is defined on the XY coordinate system on the image P2 imaged by the imaging device 20.

  In the next step S202, the CPU 30a calculates an average value AVG (x, y) of the luminance of pixels around the pixel M (x, y) constituting the image P2.

  In the next step S203, the CPU 30a calculates a feature quantity f (x, y) for the pixel M (x, y) using Expression (1).

In the next step S204, the CPU 30a divides the feature value f (x, y) by the fourth power of the average luminance value AVG (x, y), and compares the value V (= f (x, y). / AVG (x, y) ⁴ ) is calculated.

  In the next step S205, the CPU 30a determines whether or not the comparison value V is greater than or equal to a preset threshold value. If the comparison value V is greater than or equal to the threshold value, the determination in step S205 is affirmed, and the CPU 30a proceeds to step S206. On the other hand, if the comparison value V is less than or equal to the threshold value, the determination in step S205 is denied, and the CPU 30a proceeds to step S207.

  In the next step S206, the CPU 30a extracts the pixel M (x, y) as a feature point of the image P2.

  In the next step S207, it is determined whether or not the processing for the pixels constituting the image P2 is completed. When the processing from step S201 to S206 is not completed for all the pixels P2, the determination in step S207 is denied and the CPU 30a returns to step S201.

In step S201, the CPU 30a resets the coordinates of the pixel for calculating the feature amount, and thereafter repeatedly executes the processing in steps S201 to S207 until the determination in step S207 is affirmed. Thereby, as shown in FIG. 6, feature points b _{1 to} b ₁₈ are extracted from the image P2. When the processes from step S201 to S206 are completed for all the pixels P2 (step S207: Yes), the CPU 30a ends the subroutine 1 and proceeds to the next step S102.

Returning to FIG. 12, in the next step S102, the CPU 30a sets search areas respectively corresponding to the feature points a _{1 to} a ₆ extracted from the image P1 on the image P2. For example, FIG. 7 shows a search area F ₁ corresponding to the feature point a ₁ on the first image P1 of the pair of images P1 and P2. The search area F ₁ is an area where a feature point corresponding to the feature point a ₁ on the image P ₁ is expected to be included. The CPU 30a sets the search areas F _{1 to} F ₆ for all the feature points a _{1 to} a ₆ on the image P1 on the image P2.

  In the next step S103, the CPU 30a calculates a correlation value. FIG. 14 is a diagram showing the subroutine 2 executed in step S103. The CPU 30a implements the processing in step S103 by executing the subroutine 2 shown in FIG.

In the first step S301, CPU 30a selects the feature point _{a 1} of the image P1.

In the next step S302, CPU 30a includes a feature quantity of the feature point _{a 1,} 4 single feature point _b 1 in the search area _{F 1} corresponding to the feature points _{a 1, b 3, b 4} , b 9 , respectively, wherein Compare the amount. The difference between the feature quantity of the feature point a ₁ selects the feature points to be smaller than the predetermined threshold value. Here, for example, the following description will be made assuming that the feature point b ₁ and the feature point b ₄ are selected.

In the next step S303, CPU 30a includes a feature point _{a 1,} the search area _F 4 single feature point _b 1 in the _1, b _3, b 4, the selected feature points _{b 1} and feature point b from among _{b 9} A correlation value with ₄ is calculated. Specifically, CPU 30a, first, as shown in FIG. 8, to define a rectangular image around the feature point _{a 1} of the image P1 as a template _{T 1.} Then, CPU 30a overlaps the characteristic point _{a 1} included in the template _{T 1,} when matched with a feature point _{b 1} included in the search area _{F 1,} the template _{T 1,} and the template _{T 1} target The correlation value R ₁ with the image is calculated using the above equation (5).

In the next step S304, it is determined whether or not the calculated correlation value R is the maximum. When the calculated correlation value R is larger than the correlation value calculated in the past, the determination in step S304 is affirmed, and the CPU 30a proceeds to next step S305. If the calculated correlation value R is smaller than the correlation value calculated in the past, the determination in step S304 is denied, and the CPU 30a proceeds to step S306. Here, because there is no correlation value calculated in the past, the correlation value R ₁ is maximized. Therefore, the CPU 30a proceeds to the next step S305.

In the next step S305, the CPU 30a uses the position coordinates (X1 ₁ , Y1 ₁ ) on the image P of the feature point a ₁ as the start point S _1, and the position coordinates on the image P2 of the template T ₁ in step S303 (here, prescribed in the XY coordinate system that is common to optical flow of feature points _{b 1} of coordinates _(X2 1, _Y2 1)) and the end point _{E 1} to the image P1 and the image P2.

In the next step S306, CPU 30a determines whether the search of the peripheral region of the feature point _{b 1} has been completed. If the search for the peripheral area is not completed, the determination in step S306 is denied, and the CPU 30a proceeds to the next step S307. On the other hand, if it is completed, the determination in step S306 is affirmed, and the CPU 30a proceeds to the next step S308. Here, since the search for the peripheral region of the feature point b ₁ is not completed, the CPU 30a proceeds to the next step S307.

In the next step S307, CPU 30a, as seen with reference to FIG. 9, in the search area _{F 1,} centered on the feature point _{b 1,} defining a template _{T 1} is greater than the matching area MA. Then, an offset region having the same shape as the template T ₁ and having the center shifted from the feature point b ₁ is set in this region.

  Thereafter, the CPU 30a repeatedly executes the processing from steps S303 to S307 on the matching area MA. Thereby, correlation values for the offset areas sequentially set in the matching area MA are sequentially calculated. When the matching is completed for the entire matching area MA, the determination in step S306 is affirmed, and the CPU 30a proceeds to the next step S308. If the correlation value between the template T and the offset region is determined to be the maximum in step S304, the CPU 30a sets a point corresponding to the coordinates at which the template T is located in step S305 as a new image P2. The end point of the optical flow is changed to a new feature point.

In the next step S308, CPU 30a determines whether or not the matching process for all the feature points selected from the search area F ₁ is completed. If the matching process for all of the feature points selected from the search area F ₁ is not completed, the determination in step S308 is denied. In this case, CPU 30a returns to step S302, selects the next characteristic point _{b 4,} similarly executes the processes of steps S302～S308. Thereafter, the CPU 30a repeatedly executes the processes from step S302 to S308 until the determination in step S308 is affirmed.

On the other hand, when the matching process for all of the feature points selected from the search area F ₁ is completed, the determination in step S308 is affirmative, CPU 30a proceeds to the next step S309.

In the next step S309, CPU 30a, for the feature point _a 1 ~a ₆ images P1, it is determined whether the processing of steps S302~S309 are performed. If the processing for the feature points a _{1 to} a ₆ of the image P1 has not been completed, the process returns to step S301, the next feature point is selected, and the processing from step S302 to S309 is executed in the same manner. Thereafter, the CPU 30a repeatedly executes the processes from step S302 to S309 until the determination in step S309 is affirmed. On the other hand, when the processing for the feature points a _{1 to} a ₆ of the image P1 is completed, the determination in step S309 is affirmed, and the CPU 30a ends the subroutine 2 and proceeds to the next step S104. When the subroutine 2 is completed, a group of optical flows is defined as shown in FIG. In FIG. 10, optical flows related to six moving bodies 101 are shown. Actually, tens or hundreds of optical flows are defined.

  Returning to FIG. 12, in the next step S104, the CPU 30a excludes optical flows including a large amount of noise components from several tens or hundreds of optical flows, and groups the remaining optical flows. For example, when the moving body 101 has a complete linear motion, the straight lines that coincide with each optical flow should intersect at one point (vanishing point). Therefore, the CPU 30a excludes the optical flow when the straight line that matches the optical flow is significantly away from the vanishing point where the straight line group that matches the other optical flows converges, and the remaining group of optical flows is removed. Grouping is considered as an optical flow related to the same moving object.

  In the next step S105, the CPU 30a detects and outputs the distance between the own vehicle and the moving body, the relative speed and the relative moving direction of the own vehicle and the moving body, based on the grouped optical flows. For example, the CPU 30a detects and outputs the relative speed and the relative movement direction of the moving body from the difference between the start point arrangement and the end point arrangement of the group of optical flows.

  As described above, in the second embodiment, the matching process is performed from the search region set in the image P2 before the matching process using the template T including the feature point of the image P1 is performed. A feature point is selected. Then, the matching process using the template T is executed only for the image including the selected feature point. For this reason, it is possible to execute the matching process only for the region where the correlation value obtained by executing the matching process is expected to be high, and as a result, it is possible to detect the moving object with high accuracy in a short time. Become.

  In the second embodiment, the matching process is executed only for an area where the correlation value obtained by executing the matching process is expected to be high. Therefore, it is possible to avoid the feature points of the image P1 and the feature points of the image P2 having low mutual feature amounts from being associated with each other due to an error in the matching process.

  In the second embodiment, the image P1 is considered in consideration of the feature amount of the pixels constituting each of the images P1 and P2 and the average luminance of the pixels existing around the pixel whose feature amount is to be calculated. Then, feature points are extracted from the image P2. Therefore, it is possible to accurately extract feature points regardless of the brightness of the captured image, that is, the brightness of each pixel constituting the image.

  As mentioned above, although embodiment of this invention was described, this invention is not limited by the said embodiment.

  For example, in each of the above embodiments, the feature amount f (x, y) is calculated using the equation (1), but the present invention is not limited to this. For example, the feature amount may be a so-called KLT feature amount min (λ1, λ2). In this case, the comparison value V can be calculated by dividing the feature amount by the square of the average value AVG (x, y) described above. Note that λ1 and λ2 are represented by the following equations (8) and (9), respectively.

  A to C are represented by the following formula (10). Here, Ix and Iy indicate gradients in the X-axis direction and Y-axis direction of the luminance I (x, y) at the position (x, y) on the image. Specifically, it is represented by the following formula (11) and formula (12).

  Moreover, the function of the moving body recognition apparatus 30 according to each of the above embodiments can be realized by dedicated hardware or by a normal computer system.

  In the second embodiment, the program stored in the auxiliary storage unit 30c of the moving body recognition apparatus 30 is a flexible disk, a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk), an MO ( A device that executes the above-described processing may be configured by storing and distributing in a computer-readable recording medium such as a Magneto-Optical disk) and installing the program in the computer.

  Further, the program may be stored in a disk device or the like included in a predetermined server device on a communication network such as the Internet, and may be downloaded onto a computer by being superimposed on a carrier wave, for example.

  Further, the program may be activated and executed while being transferred via a communication network.

  The program may be executed entirely or partially on the server device, and the above-described image processing may be executed while transmitting / receiving information regarding the processing via the communication network.

  Note that when the above functions are realized by sharing an OS (Operating System) or when the functions are realized by cooperation between the OS and an application, only the part other than the OS may be stored in a medium and distributed. It may also be downloaded to a computer.

  It should be noted that the present invention can be variously modified and modified without departing from the broad spirit and scope of the present invention. Further, the above-described embodiment is for explaining the present invention, and does not limit the scope of the present invention.

  The mobile object recognition apparatus, mobile object recognition method, and program of the present invention are suitable for recognition of a mobile object.

DESCRIPTION OF SYMBOLS 10 Moving body recognition system 20 Imaging apparatus 20a Camera 30 Moving body recognition apparatus 30a CPU
30b Main storage unit 30c Auxiliary storage unit 30d Display unit 30e Input unit 30f Interface unit 30g System bus 31 Storage unit 32 Feature point extraction unit 33 Search region setting unit 34 Feature point selection unit 35 Correlation value calculation unit 36 Optical flow generation unit 37 Grouping Processing unit 38 moving object detection unit 100 vehicle 101 moving object P1, P2 image a _{1 to} a ₆ feature points b _{1 to} b ₁₈ feature points MA matching region F search region T template

Claims (7)

A feature amount calculating means for calculating a feature amount for pixels constituting each of the first image and the second image obtained by sequentially capturing moving moving bodies;
Feature point extraction means for extracting feature points of the first image and the second image based on the feature amount; and
The difference between the feature amount of the feature point of the first image and the feature amount of each of the feature points of the second image is calculated, and the feature of the second image is within a predetermined range. A selection means for selecting each of the points;
Correlation value calculating means for calculating a correlation value between the feature point of the first image and each of the feature points of the selected second image;
Optical flow generation means for generating an optical flow starting from the feature point of the first image and having the feature point of the second image having the highest correlation value as the start point as the end point;
A mobile object recognition apparatus comprising: Search area setting means for setting search areas respectively corresponding to the feature points of the first image on the second image;
The moving body recognition apparatus according to claim 1, wherein the selection unit performs selection of the feature point of the second image within the search area.   The feature point extraction unit extracts the feature points based on the feature amount of a calculation target pixel that is a target of calculation of the feature amount and an average luminance of the pixels around the calculation target pixel. Or the moving body recognition apparatus of 2.   The feature point extraction means, when the ratio of the feature quantity of the calculation target pixel that is the calculation target of the feature quantity to the average luminance of the pixels around the calculation target pixel is equal to or greater than a predetermined value, The mobile object recognition apparatus according to claim 3, wherein the feature point corresponding to the calculation target pixel is extracted. The correlation value calculating means uses an image including the feature points of the first image as a template,
5. The moving object recognition apparatus according to claim 1, wherein the correlation value between the template and the image in an area including the feature point of the second image is calculated. A step of calculating a feature amount for pixels constituting each of the first image and the second image obtained by sequentially capturing moving moving bodies;
Extracting a feature point of each of the first image and the second image based on the feature amount;
The difference between the feature amount of the feature point of the first image and the feature amount of each of the feature points of the second image is calculated, and the feature of the second image is within a predetermined range. Selecting each of the points;
Calculating a correlation value between the feature point of the first image and each of the feature points of the selected second image;
Generating an optical flow having the feature point of the first image as a start point and the feature point of the second image having the highest correlation value with respect to the feature point as the start point as an end point;
A moving body recognition method. Computer
Means for calculating a feature amount for pixels constituting each of the first image and the second image obtained by sequentially capturing moving moving bodies;
Means for extracting feature points of each of the first image and the second image based on the feature amount;
The difference between the feature amount of the feature point of the first image and the feature amount of each of the feature points of the second image is calculated, and the feature of the second image is within a predetermined range. Means for selecting each of the points;
Means for calculating a correlation value between the feature point of the first image and each of the feature points of the selected second image;
Means for generating an optical flow having the feature point of the first image as a start point and the feature point of the second image having the highest correlation value with respect to the feature point as the start point as an end point;
Program to function as.

Images