JP4367475B2 - Moving object recognition apparatus, moving object recognition method, and computer program - Google Patents

Description

  The present invention relates to a moving object recognition apparatus, a moving object recognition method, and a computer program that recognize a moving object using an image.

  In recent years, in order to inform the driver of the approach of other vehicles so that the vehicle can be driven more safely, a technique for detecting a moving object approaching the vehicle, particularly a vehicle approaching from behind in the traveling direction has been proposed. Yes.

  For example, the technique of Patent Document 1 detects a traveling vehicle obliquely behind based on the coincidence or mismatch of the optical flow derived from the captured image and the optical flow corresponding to the traveling speed of the vehicle by the vehicle speed sensor. It is.

  Patent Document 2 describes a technique for recognizing another vehicle even when traveling in a dark place. The technique of Patent Document 2 detects, as an optical flow, movement in a direction that diverges from an infinite point at the same point in two images before and after a predetermined time phase obtained by imaging the rear side of the host vehicle. And the position of the infinity point of an optical flow is changed with the movement amount computed based on the traveling direction information and traveling distance information of the own vehicle.

Patent Document 3 describes a technique for recognizing a moving object around a vehicle from an optical flow obtained even when the vehicle is not traveling straight ahead. The technique of Patent Literature 3 classifies optical flows calculated from time-series images into a plurality of groups in which changes are continuous. On the other hand, the optical flow is corrected based on the movement direction of the vehicle estimated from the vehicle speed sensor and the yaw rate sensor. Then, a moving object is extracted from the corrected optical flow group, and the relative velocity is calculated.
JP-A-6-130076 Japanese Patent Laid-Open No. 9-86314 JP-A-6-282655

  The conventional technology recognizes an approaching vehicle from the rear side using an image, but these are based on a technology that extracts an optical flow from two time-series images and detects an approaching vehicle from the optical flow. Yes.

  In the technique of Patent Document 1, a moving body is detected from the difference between the optical flow and the own vehicle speed. The vehicle speed corresponds to the optical flow of the background on the image plane. An optical flow different from the background can be recognized as a moving object, but since the optical flows are not grouped, it is not possible to cope with each moving object.

  Although Patent Document 2 detects a moving body from the direction of an optical flow from an infinite point, it is also impossible to group an apparent flow (on the image plane).

  In Patent Document 3, optical flows with continuous changes are divided into a plurality of groups. However, grouping may not be possible because the geometric size differs depending on the position on the image only by the continuity of changes. In particular, when a wide-angle lens is used, even if the optical flow of the same moving object is large, the spread is large when approaching, so the direction and size change, making grouping difficult and sometimes wrong.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a moving object recognition apparatus that can accurately detect a moving object from an image.

  In order to achieve the above object, a moving object recognition apparatus according to a first aspect of the present invention extracts an image capturing unit that captures a time-series image and a feature point of each time-series image captured by the image capturing unit. A feature point extracting unit that compares the feature points of the time-series images between different images, and generates an optical flow that is a vector connecting the feature points having the same pattern, and the optical flow The optical flow extension line intersects at one vanishing point within a predetermined error range, and the optical flow extension line has one end point of the optical flow as an outer dividing point, and the other of the optical flow. An optical flow in which the external division ratio between the end point of the line and the vanishing point is equal within a predetermined error range is assigned to an object belonging to one moving object. It characterized in that it comprises a grouping means for selecting as the optical flow, a.

  Preferably, image correction means for correcting an image picked up by the image pickup means into a perspective view in accordance with the characteristics of the lens of the image pickup means is provided, and the feature point extraction means is an image corrected by the image correction means. The feature point is extracted.

Further, assuming that the lowest point in the image of the optical flows grouped by the grouping unit is located on a plane in which the object including the imaging unit and the moving object exist in common, the imaging unit You may provide the distance calculation means which calculates the direction cosine to the imaging direction of the distance to the feature point of a moving object as a distance from the said imaging means to the said moving object.

  A moving object recognition method according to a third aspect of the present invention includes an imaging step of inputting a time-series image, a feature point extraction step of extracting a feature point of each of the time-series images input in the imaging step, Comparing the feature points of the time-series images between different images, generating an optical flow that is a vector connecting the feature points having the same pattern, and of the optical flows, the optical flow of the optical flow An extension line intersects at one vanishing point within a predetermined error range, and one end point of the optical flow is an outer dividing point in the extension line of the optical flow, and the other end point of the optical flow and the vanishing point A grouping step for selecting an optical flow in which the external ratio of the line segments connecting is equal within a predetermined error range; and Characterized in that it obtain.

  Preferably, the image correction step includes an image correction step for correcting the image input in the imaging step into a perspective view in accordance with the characteristics of the lens of the imaging means, and the feature point extraction step is an image corrected in the image correction step. The feature point is extracted.

  According to a fourth aspect of the present invention, there is provided a computer program comprising: an image input unit that inputs a time-series image from an imaging unit; and a feature point that extracts a feature point of each time-series image input by the image input unit. An extraction means, and an optical flow generation means for comparing the feature points of the time-series images between different images and generating an optical flow that is a vector connecting the feature points having the same pattern whose position is changed on the image; In the optical flow, the extension line of the optical flow intersects at one vanishing point within a predetermined error range, and one end point of the optical flow in the extension line of the optical flow is an outer dividing point. An optical flow in which the external division ratio between the other end point of the optical flow and the vanishing point is equal within a predetermined error range. Over the, characterized in that to function grouping means for selecting as, as optical flows belonging to one moving object.

  According to the moving object recognition apparatus of the present invention, it is possible to detect a moving object by correctly grouping the optical flow of feature points extracted from a time-series image for each moving object.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals, and description thereof will not be repeated. As an embodiment of a moving object recognition apparatus according to the present invention, a moving object recognition apparatus for a vehicle will be described with reference to the drawings.

(Embodiment 1)
FIG. 1 is a block diagram showing a logical configuration of a moving object recognition apparatus according to an embodiment of the present invention. The moving object recognition apparatus 1 includes a camera 2, an image input unit 21, an image correction unit 22, a feature point extraction unit 23, an optical flow generation unit 24, a grouping processing unit 25, a data holding unit 5, a moving object determination unit 26, and a display process. The unit 27, the display device 7, and the like. The data holding unit 5 stores and holds collected time-series image data 51, corrected time-series image data 52, feature point data 53, optical flow data 54, and grouping data 55. The moving object recognition apparatus 1 recognizes whether there is a moving object around the vehicle and what kind of moving object it is. In particular, a vehicle approaching from the rear in the vehicle traveling direction is detected.

Here, the optical flow and its grouping method will be described. FIG. 3 is a diagram for explaining the optical flow. FIG. 3 schematically shows a case where the vehicle approaches the camera 2 as a moving object. The feature points in the current image [pi ₁ and x1i (i = 1 ... n) . A feature point corresponding to x1i in the past image ₂ is assumed to be x2i. When expressed in the current image [pi ₁ overlapping previous image [pi ₂ feature points x2i, that the optical flow vector toward the x1i from x2i.

  When the image is a perspective view, if the optical flow of the feature point of an object that moves parallel to the coordinates (camera coordinates) fixed to the camera 2 (goes straight in the camera coordinates) is extended, one point on the image Intersect (Figure 3). This is because each feature point draws a straight line parallel to the camera coordinates, and the parallel lines intersect at a single point when viewed in perspective. This point is called a vanishing point (FOE: Focus Of Expansion). If different moving objects are not moving in parallel with each other, their optical flows meet at different vanishing points.

The optical flow of one object moving parallel to the camera coordinates has the following relationship when the image is a perspective view.

Here, x1i the coordinates on the image of the current image [pi ₁ feature point, x2i the coordinates on the previous image of the [pi ₂ feature points image, Xfoe the coordinates on the image of the vanishing point, C is some constant .

  That is, outside the line segment connecting the feature point x2i (the other end point) of the past image in the optical flow and the vanishing point xfoe with the feature point x1i of the current image as one end point of the optical flow as the outer dividing point. The fraction is constant for one translating object. Therefore, an optical flow that intersects at one vanishing point and has the same external division ratio can be grouped as an optical flow of one object. In addition to the above, the external division ratio is constant for one parallel moving object even if the external division ratio of the optical flow is taken with the vanishing point as the external division point, for example.

  Returning to FIG. 1, the operation of each part of the moving object recognition apparatus 1 will be described. The camera 2 captures an image and converts it into digital image data. The image input unit 21 inputs time-series image data from the camera 2 at regular time intervals, and stores them as collected time-series image data 51 in the data holding unit 5.

  For example, a camera 2 for monitoring the rear of the vehicle 4 as shown in FIG. 4 can be used as the camera 2 that captures a moving object. By using the rear monitor camera 2, it is not necessary to provide a new camera 2, and the moving object recognition apparatus 1 can be configured at low cost.

  Since the parking support rear monitor camera is a wide-angle lens, the periphery of the image is distorted and the resolution in the distance is low. The mounting position is low. These conditions are inconvenient for processing an image as real world coordinates, making it difficult to recognize an approaching moving object.

  Therefore, distortion of the captured image is corrected. The image correction unit 22 in FIG. 1 corrects the distortion of the captured image in accordance with the characteristics of the lens, and converts the image into a perspective image. The corrected image is stored in the data holding unit 5 as corrected time-series image data.

The coordinates (x, y) of the perspective view image and the corresponding coordinates (x ′, y ′) of the pre-correction image have the following relationship.


Here, (x0, y0) is the distortion center coordinate of the corrected image, and (x′0, y′0) is the distortion center coordinate of the image before correction. fn (r) is a camera lens distortion characteristic function.

Furthermore, you may convert into the image when the direction of the camera 2 is made horizontal. The conversion for changing the orientation of the camera 2 is expressed by the following equation, where the coordinates before conversion are (x ′, y ′) and the coordinates after conversion are (x ′, y ′).



Here, β is the tilt angle of the camera 2, z is a scale factor, e0 is the distance between the vanishing point before image conversion and the image center, and H = [hij] is the image conversion matrix.

  FIG. 5 is an example of an image captured by a rear monitor camera with a wide-angle lens. FIG. 6 is an example of an image in which the distortion in FIG. 5 is corrected. In FIG. 5, the white line on the road surface is curved, but in the corrected image in FIG. 6, the white line on the straight road is corrected to be a straight line. FIG. 7 shows an image obtained by normalizing the image of FIG. This is converted into an image captured with the camera 2 oriented horizontally. In FIG. 6, the vertical line converges to one point below, but in FIG. 7, since the direction of the camera 2 is corrected to be horizontal, the vertical lines are parallel.

  Next, feature points for comparison between images are extracted from the corrected time-series image data. The feature point extraction unit 23 extracts feature points from the corrected time-series image data 52 and stores them as feature point data 53 in the data holding unit 5. The optical flow generation unit 24 compares feature points between images and generates a vector connecting feature points having high pattern correlation as an optical flow. The optical flow generator 24 stores the generated optical flow in the data holding unit 5 as optical flow data 54. A method of feature point extraction and optical flow generation will be described next.

  A motion vector is calculated in order to detect an approaching object in the search region obtained by the background difference. There are various methods for calculating the motion vector, but template matching that is relatively resistant to environmental changes can be employed. For template matching, it is important to select an image block that is easy to track. For example, a KLT feature point calculation method can be used.

The moment matrix of the luminance gradient ∇I = [Ix, Iy] T is

far. here,

It is. The KLT feature point calculation method is expressed by the following formula, where the KLT feature quantity is λ2 / λ1 (the condition number of the matrix M (x, y) = the eigenvalue ratio) or min (λ2, λ1).

A template is set around the obtained feature points, and an area having a high correlation with the template is searched from images after time series. The movement vector is obtained as a motion vector (optical flow). A normalized correlation method (NCC) can be adopted as a template matching method. The similarity RNCC of the normalized correlation method is obtained by the following equation.



Here, I is a brightness value vector of an area of N rows × M columns, and T is a brightness value vector of a template area of N rows × M columns. The bar above the variable represents the average value within each region.

  If the entire search of the area is performed, the amount of calculation is large. Therefore, an amount of movement restriction of the object is assumed according to the situation, and only the restricted area is searched to reduce the amount of calculation. For example, no search is made except in the approach direction. Or, an area that is physically unlikely based on the past movement amount is not searched. Furthermore, since the object close to the camera 2 has a large movement amount on the image, the amount of calculation can be reduced by performing a hierarchization process and narrowing down the search area by matching in the upper layer and performing a detailed search.

  FIG. 8 shows an example of an optical flow. The white square represents the feature point of the previous frame, and the white circle represents the point of the latest frame having a high correlation value. Even with an optical flow of one moving object, for example, the optical flow of the front grille and the optical flow of the rear wheel have different directions and lengths, and it is difficult to group them as an optical flow of one moving object.

  Therefore, as described above, optical flows that intersect at one vanishing point and have the same external division ratio of Equation 1 are grouped as optical flows of one object. The grouping processing unit 25 groups the optical flows as follows, for example.

Equation 1 has two parameters (vanishing point and external ratio constant), and the condition for grouping is that the parameters have the same value for each optical flow. Therefore, Equation 1 is transformed into a linear equation for two variables.
Here, dxi = x1i-x2i and C1 = C · xfoe.

  For each optical flow, dxi and x1i are plotted on a plane with respect to the x-coordinate and y-coordinate, respectively, as shown in FIG. FIG. 9A shows the x component of dxi with respect to the x coordinate of x1i. FIG. 9B shows the y component of dxi with respect to the y coordinate of x1i.

  Among points plotted as shown in FIG. 9, a point that is on a straight line while allowing a certain error can be grouped as an optical flow belonging to the same moving object. There are various methods for determining whether or not the object is on a straight line. For example, RANSAC (RANdom SAmple Consensus) can be used. RANSAC is an algorithm that selects any two points, counts the number of points within a certain allowable width of the straight line connecting the two points, and selects the straight line with the largest number of points. Points within a certain allowable range of the selected straight line are set as one optical flow group.

  An optical flow belonging to another moving object can be grouped by selecting a point that is on a straight line while allowing a certain error, except for the points classified into one group. By repeating this, a plurality of moving objects can be recognized.

  The optical flow processing unit stores the grouped optical flow data in the data holding unit 5 as grouping data 55. The moving object determination unit 26 determines a moving object from the grouping data 55. For example, the size of the moving object, the vehicle type, etc. are determined from the arrangement of the feature points. It is also possible to calculate the distance to the moving object.

Assuming that the lowest point of the optical flow is the road surface (on the same plane as the vehicle), the distance to the moving object can be calculated by the following equation. The distance Z here is a direction cosine of the distance from the camera to the feature point of the object in the imaging direction. The imaging direction is the direction of the optical axis of the camera, but when the image is normalized in the horizontal direction, it is in the horizontal direction (parallel to the road surface). In the following, the imaging direction is horizontal (parallel to the road surface). The image plane coordinates are (u, v), the world coordinates are (X, Y, Z), and the image center coordinates are (c _u , c _v ). Let U = u−c _u and V = v−c _v .



Here, f is the focal length of the camera 2, δu and δv are the physical intervals of the pixels in the horizontal and vertical directions, respectively, and h is the camera height from the road surface. R = [r _ij ] is a rotation matrix of the projective transformation matrix coefficient, and T = [t _k ] ^T is a translation vector of the projective transformation matrix coefficient.

  The display processing unit 27 displays the result determined by the moving object determination unit 26 on the display device 7. For example, the relationship between the position of the approaching vehicle behind and the host vehicle is shown in a plan view. Further, the display of the display device 7 may be accompanied by sound such as voice or chime.

  FIG. 2 is a block diagram illustrating an example of a physical configuration of the moving object recognition apparatus 1. As shown in FIG. 2, the moving object recognition apparatus 1 of the present invention shown in FIG. 1 includes a control unit 11, a main storage unit 12, an external storage unit 13, an operation unit 14, a screen display unit 15, and a transmission / reception unit. 16, a camera 2, a distance measuring device 3, and a display device 7. The control unit 11, the main storage unit 12, the external storage unit 13, the operation unit 14, the screen display unit 15, and the transmission / reception unit 16 are all connected to the control unit 11 via the internal bus 10.

  The control unit 11 includes a CPU (Central Processing Unit) and the like. According to a program stored in the external storage unit 13, an image input unit 21, an image correction unit 22, a feature point extraction unit 23, an optical flow generation unit 24, a grouping. The processing of the processing unit 25, the moving object determination unit 26, and the display processing unit 27 is executed. The image input unit 21, the image correction unit 22, the feature point extraction unit 23, the optical flow generation unit 24, the grouping processing unit 25, the moving object determination unit 26, and the display processing unit 27 are the control unit 11 and a program executed thereon. It is realized with.

  The main storage unit 12 includes a RAM (Random-Access Memory) and the like, and is used as a work area for the control unit 11. The data holding unit 5 is stored and held in a part of the main storage unit 12 as a storage area structure.

  The external storage unit 13 includes a nonvolatile memory such as a flash memory, a hard disk, a DVD (Digital Versatile Disc), a DVD-RAM (Digital Versatile Disc Random-Access Memory), a DVD-RW (Digital Versatile Disc ReWritable), and the like. A program for causing the control unit 11 to perform the above process is stored in advance, and data of the program is supplied to the control unit 11 according to an instruction from the control unit 11, and the data supplied from the control unit 11 is stored. For example, the time series image data may be stored in the external storage unit 13.

  The operation unit 14 includes a pointing device such as a key switch, a jog dial, a keyboard and a mouse, and an interface device for connecting them to the internal bus 10 in order for the operator to give a command to the moving body recognition apparatus 1. Commands such as input of moving object recognition determination conditions and start of moving object recognition are input via the operation unit 14 and supplied to the control unit 11.

  The display device 7 is composed of a CRT (Cathode Ray Tube) or an LCD (Liquid Crystal Display), etc., and collects time-series image data 51 and correction time according to a command of the control unit 11 according to a command input from the operation unit 14. The series image data 52, the moving object determination result, and the like are displayed. The screen display unit 15 converts screen data to be displayed on the display device 7 into a signal for driving the display device 7.

  The display device 7 may include a speaker or a buzzer. In that case, the screen display unit 15 causes the display device 7 to output sound such as voice or chime.

  The camera 2 converts the image of the lens into an electrical signal using a CCD, for example, and outputs image data digitized for each pixel. The distance measuring device 3 is, for example, a millimeter wave radar, a laser distance meter, an ultrasonic distance meter, or the like, and outputs a distance to an object as an electrical signal. In the first embodiment, the distance measuring device 3 is not used. This is used to measure the distance to the object in the second embodiment to be described later.

  The transmission / reception unit 16 includes a modem or network termination device, and a serial interface or LAN (Local Area Network) interface connected thereto. The control unit 11 inputs time series data from the camera 2 or the distance measuring device 3 via the transmission / reception unit 16.

  Next, the operation of the moving object recognition device 1 will be described. As described above, the operation of the moving object recognition device 1 is performed by the control unit 11 in cooperation with the main storage unit 12, the external storage unit 13, the operation unit 14, the screen display unit 15, and the transmission / reception unit 16.

  FIG. 11 is a flowchart illustrating an example of the operation of moving object recognition. The control unit 11 inputs time-series images from the camera 2 via the transmission / reception unit 16 (step A1). Then, as described above, the image is corrected according to the lens characteristics of the camera 2 (step A2). Furthermore, the tilt angle of the camera 2 may be corrected and normalized to a horizontal image.

  The control unit 11 extracts the feature points of the corrected image, for example, by the above-described KLT feature point extraction method (step A3). The control unit 11 compares the extracted feature points between time-series images, and generates an optical flow by the template matching described above (step A4).

  Next, the optical flows are grouped (step A5). As described above, among points plotted as shown in FIG. 9, a point that is on a straight line while allowing a certain error is grouped as an optical flow belonging to the same moving object. For example, RANSAC is used to determine whether or not the object is on a straight line. Then, a moving object is determined from the pattern of feature points belonging to the grouped optical flows (step A6). Further, assuming that the lowest point of the optical flow is the road surface (on the same plane as the own vehicle), the distance to the moving object may be calculated by the method shown in Equation 9.

  When a moving object is recognized (step A7; Yes), the control part 11 displays the information on the display apparatus 7 via the screen display part 15 (step A8). For example, the position of the approaching vehicle behind is shown in a plan view in relation to the host vehicle. Further, the driver may be notified by sound such as voice or chime. Then, returning to the time-series image data input (step A1), the moving object recognition is repeated. If the moving object is not recognized (step A7; No), the moving object recognition is repeated without displaying (step A1).

  According to the moving object recognition apparatus 1 of the present invention, optical flows generated from time-series image data can be correctly grouped for each moving object. As a result, the moving object can be accurately recognized. In addition, since the image is corrected according to the lens characteristics of the camera 2, the optical flow generated from the corrected image data is correctly grouped for each moving object even in a rear camera using a wide-angle lens, for example. it can.

(Embodiment 2)
Next, the moving object recognition apparatus 1 in the case where the coordinates of feature points in the optical flow are calculated and the feature points having the same displacement between two images are grouped as the feature points belonging to one moving object will be described. FIG. 12 is a block diagram showing an example of a logical configuration of the moving object recognition apparatus 1 according to Embodiment 2 of the present invention.

  The moving object recognition device 1 includes a camera 2, a distance measuring device 3, a data acquisition unit 28, an image correction unit 22, a feature point extraction unit 23, an optical flow generation unit 24, an equal displacement point extraction processing unit 29, a data holding unit 5, The moving object determination unit 26, the display processing unit 27, the display device 7, and the like are included. The data holding unit 5 stores and holds collected time-series image data 51, distance data 56, corrected time-series image data 52, feature point data 53, optical flow data 54, and grouping data 55.

  The camera 2, the image correction unit 22, the feature point extraction unit 23, the optical flow generation unit 24, the moving object determination unit 26, the display processing unit 27, and the display device 7 are the same as those in the first embodiment.

  The distance measuring device 3 is a measuring device that measures a distance by a physical method such as a millimeter wave radar, a laser distance meter, an ultrasonic distance meter, and measures a distance to an object in the direction of imaging with the camera 2. To do. The data acquisition unit 28 inputs time-series images from the camera 2 to obtain collected time-series image data 51. Further, the data of the distance measuring device 3 is input and stored in the data holding unit 5 as the distance data 56.

  The equal displacement point extraction processing unit 29 calculates the displacement of the feature point in the optical flow from the distance to the feature point and the optical flow including the feature point. Further, feature points having the same calculated displacement are grouped as feature points belonging to one moving object. Then, the equal displacement point extraction processing unit 29 stores the grouped optical flow data in the data holding unit 5 as the grouping data 55.

From the distance Z to the feature point that is the end point of the optical flow, the world coordinates (X, Y) of the feature point can be calculated by the following formula. The distance Z here is a direction cosine of the distance from the camera to the feature point of the object in the imaging direction. The imaging direction is the direction of the optical axis of the camera, but when the image is normalized in the horizontal direction, it is in the horizontal direction (parallel to the road surface). In the following, the imaging direction is horizontal (parallel to the road surface). The coordinates of the feature point on the image plane are (u, v), the world coordinates are (X, Y, Z), and the image center coordinates are (c _u , c _v ). (Xnum, Ynum) are the world coordinates of the starting point of the optical flow. The previously calculated value can be used as the world coordinate of the start point.







Here, S _u and S _v are scale factors, R = [r _ij ] is a rotation matrix of a projective transformation matrix coefficient, and T = [t _k ] ^T is a translation vector of the projective transformation matrix coefficient. That is, θ _Pan , θ _Tilt , and θ _Roll are camera mounting angles, and T _X , T _Y , and T _Z are camera mounting positions, and projective transformation matrix coefficients are expressed by the following equations.

  Regarding the Z of the end point, the Z coordinate of the start point can be a value calculated previously, and the Z coordinate of the end point can be a value calculated by Equation 9 assuming that the lowest point of the optical flow is the road surface. That is, the distance Z calculated in the first embodiment can be used. In that case, the distance measuring device 3 may not be used.

  If the distance Z of each feature point is measured and the world coordinates of the feature point are calculated, the feature points having the same displacement (vector) in the world coordinate can be grouped as belonging to one moving object. Approximately, the distance Z to the feature point of the processing region may be represented by one value, and the world coordinates may be calculated assuming that the distance Z (direction cosine in the imaging direction) is the same. Approximately, for example, grouping is performed as follows.

  When the calculated end point is a vector from the start point to the end point and the (X, Y) coordinates are plotted on a plane, the result is as shown in FIG. The optical flows having the same direction and length, that is, the optical flows having the same displacement within a predetermined error range, are grouped as feature points belonging to one moving object. Similar to the first embodiment, the moving object is determined from the grouped optical flows and displayed.

  Next, the operation of the moving object recognition apparatus 1 when calculating the coordinates of feature points in the optical flow and grouping the feature points having the same displacement between two images as the feature points belonging to one moving object will be described. The physical configuration of the moving object recognition apparatus 1 in the second embodiment is shown in FIG. 2, for example. FIG. 13 is a flowchart illustrating an example of the moving object recognition operation according to the second embodiment.

  The control unit 11 inputs time-series images from the camera 2 via the transmission / reception unit 16 (step B1). Further, distance data 56 is input from the distance measuring device 3 (step B2). When the distance Z is a value calculated by Equation 9 assuming that the lowest point of the optical flow is the road surface, the previous value is used. As described above, the image is corrected according to the lens characteristics of the camera 2 (step B3). Furthermore, the tilt angle of the camera 2 may be corrected and normalized to a horizontal image.

  The control unit 11 extracts the feature points of the corrected image, for example, by the above-described KLT feature point extraction method (step B4). The control unit 11 compares the extracted feature points between time-series images, and generates an optical flow by the template matching described above (step B5).

  Next, the world coordinates of the feature point are calculated from the optical flow and the distance Z (step B6). When the distance Z is assumed to be the road surface, assuming that the lowest point of the optical flow is the road surface, the distance Z is calculated from the optical flow of the feature point that was the last lowest point. Then, the equal displacement points are grouped (step B7). As described above, among the vectors plotted as shown in FIG. 10, vectors having the same direction and size are grouped as optical flows belonging to the same moving object while allowing a certain error.

  A moving object is determined from the pattern of feature points belonging to the grouped optical flows (step B8). Further, assuming that the lowest point of the optical flow is the road surface (on the same plane as the own vehicle), the distance to the moving object may be calculated by the method shown in Equation 9.

  When a moving object is recognized (step B9; Yes), the control part 11 displays the information on the display apparatus 7 via the screen display part 15 (step B10). For example, the position of the approaching vehicle behind is shown in a plan view in relation to the host vehicle. Moreover, you may make a driver | operator sound, such as an audio | voice or a chime. Then, returning to the time-series image data input (step B1), the moving object recognition is repeated. If the moving object is not recognized (step B9; No), the moving object recognition is repeated without displaying (step B1).

  According to the moving object recognition apparatus 1 of the present invention, optical flows generated from time-series image data can be correctly grouped for each moving object. As a result, the moving object can be accurately recognized. In addition, since the image is corrected according to the lens characteristics of the camera 2, the optical flow generated from the corrected image data is correctly grouped for each moving object even in a rear camera using a wide-angle lens, for example. it can.

  In addition, the hardware configuration and the flowchart described above are merely examples, and can be arbitrarily changed and modified.

  The central part that performs processing of the moving object recognition apparatus 1 including the control unit 11, the main storage unit 12, the external storage unit 13, the operation unit 14, the screen display unit 15, the transmission / reception unit 16, and the internal bus 10, It can be realized by using a normal computer system regardless of a dedicated system. For example, a computer program for executing the above operation is stored and distributed on a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.), and the computer program is installed in the computer. Thus, the moving object recognition apparatus 1 that executes the above-described processing may be configured. Alternatively, the moving object recognition device 1 may be configured by storing the computer program in a storage device included in a server device on a communication network such as the Internet and downloading the computer program by a normal computer system.

  Further, when the function of the moving object recognition apparatus 1 is realized by sharing an OS (operating system) and an application program, or by cooperation between the OS and the application program, only the application program part is stored in a recording medium or a storage device. It may be stored.

  It is also possible to superimpose a computer program on a carrier wave and distribute it via a communication network. For example, the computer program may be posted on a bulletin board (BBS, Bulletin Board System) on a communication network, and the computer program distributed via the network. The computer program may be activated and executed in the same manner as other application programs under the control of the OS, so that the above-described processing may be executed.

It is a block diagram which shows the logical structure of the moving object recognition apparatus which is one embodiment of this invention. It is a block diagram which shows an example of a physical structure of a moving object recognition apparatus. It is a figure explaining an optical flow. It is a figure which shows the example of the camera attachment position of a vehicle. It is an example of the image imaged with the rear monitor camera of a wide angle lens. It is an example of the image which correct | amended distortion of FIG. It is a figure which shows the image which normalized the image of FIG. It is a figure which shows the example of an optical flow. It is the graph which plotted the x component (a) and y component (b) of the linear type | formula which deform | transformed the external division ratio of an optical flow and a vanishing point. It is the figure which projected the world coordinate of the optical flow on XY plane. It is a flowchart which shows an example of the operation | movement of moving object recognition. 6 is a block diagram illustrating an example of a logical configuration of a moving object recognition apparatus according to Embodiment 2. FIG. 12 is a flowchart illustrating an example of an operation of moving object recognition according to the second embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Moving object recognition apparatus 2 Camera (imaging means)
3 Distance measuring device (distance detection means)
5 Data holding unit 7 Display device 10 Internal bus 11 Control unit (feature point extraction means, optical flow generation means, grouping means, displacement calculation means, equal displacement point selection means, image correction means)
DESCRIPTION OF SYMBOLS 12 Main memory | storage part 13 External memory | storage part 14 Operation part 15 Screen display part 16 Transmission / reception part 21 Image input part 22 Image correction part (image correction means)
23 feature point extraction unit (feature point extraction means)
24 Optical flow generator (optical flow generator)
25 Grouping processing unit (grouping means)
26 moving object determination unit 27 display processing unit 28 data acquisition unit 29 equal displacement point extraction processing unit (displacement calculation means, equal displacement point selection means)
51 Collected time-series image data 52 Corrected time-series image data 53 Feature point data 54 Optical flow data 55 Grouping data 56 Distance data

Claims (6)

An imaging means for capturing time-series images;
Feature point extracting means for extracting feature points of each time-series image captured by the imaging means;
Optical flow generation means for comparing the feature points of the time-series images between different images and generating an optical flow that is a vector connecting the feature points having the same pattern;
Among the optical flows, the extension line of the optical flow intersects at one vanishing point within a predetermined error range, and the optical flow extension line has one end point of the optical flow as an outer dividing point. A grouping means for selecting, as an optical flow belonging to one moving object, an optical flow in which the outer division ratio of the line connecting the other end point of the flow and the vanishing point is equal within a predetermined error range;
A moving object recognition device comprising: An image correction unit that corrects an image captured by the imaging unit to a perspective view according to the characteristics of the lens of the imaging unit,
The moving object recognition apparatus according to claim 1, wherein the feature point extraction unit extracts a feature point of the image corrected by the image correction unit. Assuming that the lowest point in the image of the optical flows grouped by the grouping unit is located on a plane in which the object including the imaging unit and the moving object exist in common, the imaging unit to the moving object The moving object recognition apparatus according to claim 1, further comprising a distance calculating unit that calculates a direction cosine of the distance to the feature point in the imaging direction as a distance from the imaging unit to the moving object. An imaging step for inputting time-series images;
A feature point extracting step of extracting feature points of each time-series image input in the imaging step;
An optical flow generation step of comparing the feature points of the time-series images between different images and generating an optical flow that is a vector connecting the feature points having the same pattern;
Among the optical flows, the extension line of the optical flow intersects at one vanishing point within a predetermined error range, and the optical flow extension line has one end point of the optical flow as an outer dividing point. A grouping step of selecting an optical flow in which the external division ratio between the other end point of the flow and the vanishing point is equal within a predetermined error range;
A moving object recognition method comprising: An image correction step of correcting the image input in the imaging step into a perspective view in accordance with the characteristics of the lens of the imaging means;
5. The moving object recognition method according to claim 4 , wherein the feature point extraction step extracts feature points of the image corrected in the image correction step. Computer
Image input means for inputting time-series images from the imaging means;
Feature point extraction means for extracting a plurality of feature points of each time-series image input by the image input means;
Optical flow generation means for comparing the feature points of the time-series images between different images and generating an optical flow that is a vector connecting the feature points having the same pattern whose position is changed on the images;
Among the optical flows, the extension line of the optical flow intersects at one vanishing point within a predetermined error range, and one end point of the optical flow in the extension line of the optical flow is defined as an outer dividing point. A grouping means for selecting, as an optical flow belonging to one moving object, an optical flow in which an external division ratio between the other end point of the flow and the vanishing point is equal within a predetermined error range;
A computer program that functions as a computer program.