JPH10222679A - Vehicle image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To identify whether an object on a road, which is obtained by a CCD camera, is the planar one or the stereoscopic one. SOLUTION: The anterior images at different times T0 and T1 are obtained by an on-vehicle CCD camera 10 so as to be supplied to an image recognizing device 16. A vehicle action measuring device 14 detects vehicle speed and a yaw rate and supplies them to the image recognizing device 16. The image recognizing device 16 calculates a position which is assumed to be obtained at the time T1 when the object in the image is assumed to be the planar object (a print, etc., on the road) on the road by the image obtained at the time T0 and the running state of a vehicle and comparaes it with the image which is actually obtained at the time T1. When they coincide, the object is judged to be the planar one and it is judged to be the stereoscopic one at the time of non-coincidence. The height of the stereoscopic object is calculates from the deviation quantity of both.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for a vehicle, and more particularly, to an apparatus for identifying whether or not a photographed object is a plane object on a road.

[0002]

2. Description of the Related Art Conventionally, a technology has been developed in which a vehicle is mounted with a CCD camera or the like to photograph a traveling direction of a vehicle and an obstacle on a road is detected.

For example, Japanese Patent Application Laid-Open No. Hei 5-2333813 discloses a technique for detecting a moving object ahead of a vehicle using an optical flow (movement vector) of an image photographed by a vehicle-mounted camera.

[0004]

Although the optical flow is effective as a method for detecting a moving object on a road, an image obtained at a certain time T0 and an image obtained at a next time T1 are certainly effective.
The pattern matching (for all images) with the image obtained in step (1) must be performed, and there is a problem in that the processing amount is large and it takes time to finally extract an object that becomes an obstacle. Further, even if an optical flow is calculated by performing pattern matching, a grouping process (whether the optical flow belongs to the same object or not) is required to determine whether the optical flow indicates a certain moving object. Therefore, there is also a problem that the size of the object that can be detected in order to secure the accuracy of the grouping process is limited to a certain value or more. Of course, it is conceivable to detect a smaller object by extracting the optical flow itself in, for example, a pixel unit, but the processing time is significantly increased.

The present invention has been made in view of the above-mentioned problems of the prior art, and determines whether an object on a road is a three-dimensional object or a two-dimensional object without using an optical flow, that is, whether an object on a road can be an obstacle for a vehicle. An object of the present invention is to provide a vehicular image processing device that can be quickly identified.

[0006]

In order to achieve the above object, a first aspect of the present invention provides an image pickup means for photographing a front of a vehicle, a detection means for detecting a running state of a vehicle, and an image pickup means. A conversion unit for converting the image into a plane image on the road surface, a plane image at a certain time T0 and a plane image at the next time T1 are obtained, and the plane image at the time T0 and the running state at the time T1−T0 are obtained. Processing means for comparing the calculated predicted plane image at the next time T1 with the plane image actually obtained at the time T1 to identify whether or not the object in the image is a plane object on the road; It is characterized by the following.

According to a second aspect of the present invention, in the first aspect, when the processing means determines that the object in the image is not a plane object on the road, the predicted plane image and the time T1 are determined.
And a calculating means for calculating the height of the object from the road surface based on the amount of difference from the plane image obtained in step (1).

In a third aspect based on the second aspect, the calculation means determines that the object is a virtual image on a road when the height is negative.

[0009]

Embodiments of the present invention will be described below with reference to the drawings.

<First Embodiment> FIG. 1 is a block diagram showing the configuration of the first embodiment. A CCD camera 10 is provided at the front of the vehicle as an imaging means for photographing the front of the vehicle, and photographs the front of the vehicle at a predetermined photographing timing. C
Images obtained by the CD camera 10 are sequentially supplied to an on-vehicle image capture board 12. Image capture board 1
2 has an A / D converter and a VRAM;
Images from the CD camera 10 are sequentially stored in the VRAM and supplied to the image recognition device 16 at a predetermined timing. Also,
The vehicle behavior measuring device 14 includes a vehicle speed sensor, a yaw rate sensor, and the like, detects a traveling state (speed and traveling direction) of the vehicle, and supplies the detected state to the image recognition device 16. Image recognition device 1
Reference numeral 6 includes a CPU, a ROM, a RAM, a data bus, and the like, and performs predetermined coordinate conversion on an image supplied from the image capture board 12 to convert the image into plane coordinates on a road. It has a function of calculating the predicted plane coordinates based on the traveling state data supplied from the computer and comparing the calculated plane coordinates with the actual plane image. Note that the conversion to the plane image and the calculation of the predicted plane coordinates based on the traveling state data are all performed based on a predetermined calculation formula.
This is performed by executing a calculation program stored in M. The contents of the calculation program will be described later. The result recognized by the image recognition device 16, that is, whether the object in the image is a flat object or a three-dimensional object, is determined by, for example, a vehicle control ECU.
(Electronic control unit) and the like, and are used for brake control, warning to a driver, and the like.

2 to 5, in this embodiment, the image recognition device 16 identifies an object in an image obtained by the CCD camera 10, that is, a flat object (a printed sign or the like) on a road. The basic principle for discriminating between an object and a three-dimensional object is shown.

First, FIG. 2 shows a case in which a plane object 100 (for example, a diamond-shaped sign) on a road is photographed. The photographing is performed at time T0 and then at time T1.
When the vehicle is traveling, the vehicle position at time T0 and the time T1, that is, the photographing position is different, so that the images obtained at both times are naturally different. FIG. 3 schematically shows an image (A) obtained by the CCD camera 10 at time T0 and an image (B) obtained at time T1. Here, if the height of the CCD camera 10, its line-of-sight direction, and photographing timing are known, and the running state (speed and traveling direction) of the vehicle is known, the plane object 100 in the image obtained at time T0 will be at time T1.
Then, it is possible to predict how it changes, and the position of the plane object 100 in the image obtained by prediction and the actual time T1
The position of the plane object 100 in the image obtained by the above will coincide.

On the other hand, FIG. 4 shows a three-dimensional object 2 existing on the road.
A case where 00 is photographed at time T0 and time T1 is shown. The height and line of sight of the CCD camera 10 are fixed,
Even if the running state of the vehicle is the same as that of FIG. 2, the moving distance per unit time in the image differs depending on whether the object to be photographed is a flat object or a three-dimensional object. If the position at T1 is predicted in the same manner as in the case of a plane object, the position obtained by prediction does not match the actual position. Conversely, if the position obtained by assuming that the object to be photographed is a plane object does not match the position actually obtained, the object to be photographed is not a plane object on the road. Become.

This will be described in more detail with reference to FIG. FIG. 5A is an image of the three-dimensional object 200 obtained at the time T0. In order to process the three-dimensional object 200 as a planar object, a coordinate system viewed from directly above the road, that is, a three-dimensional coordinate system (X-
In the YZ coordinate system, the data is converted into a plane coordinate system (Y = constant) set in the road surface. FIG. 5B schematically shows a planar image obtained by the conversion. Then, the image (plane coordinates) obtained by performing coordinate conversion on the image is compared with the time T1 using the running state of the vehicle at the time (T1-T0).
Predict the position of the object in the plane coordinate system at. FIG.
(C) schematically shows the position of the object obtained by prediction. A black square in the figure is a characteristic portion (preferably a corner portion) unique to the object, and is a region to be compared. In the figure, the plane coordinate system is an XZ system, and the position of the black square is (X0
', Z0'). Note that the suffix indicates an image obtained at time T0, and the dash (') indicates a predicted image. FIG. 5D shows time T
1 shows an actual camera image obtained by the CCD camera 10, which corresponds to FIG. Then, the plane image is calculated by converting the road into a coordinate system viewed from directly above the image. FIG. 7E schematically shows a plane image obtained by the conversion. In the figure, the position of the black square is (X1,
Z1). The subscript means that the image is obtained at time T1. Here, the object in the image is a plane object 100
In the case of, the object position in FIG.
, The position of the object coincides with that of X0 '= X1, Z0'
= Z1, but since the object is a three-dimensional object 200, FIG.
The black square position in (E) and the black square position in FIG. 5C, especially the Z coordinate, do not coincide with each other and are shifted. Therefore, X
By comparing the values of 0 'and X1, and the values of Z0' and Z1,
It is possible to easily identify whether the object is a plane object or a three-dimensional object on a road.

FIG. 6 shows a flowchart of a process performed by the image recognition device 16 according to the basic principle described above. First, the image recognition device 16 inputs the image obtained at time T0 (this is referred to as image 1 for convenience) from the image capture board 12 (S101), and converts this image 1 into a plane image on the road surface ( S102). This conversion is
More specifically, as shown in FIG. 7, the Y-coordinate of the road surface in the three-dimensional space XYZ system of the real space having the origin at the center of the CCD camera is shown in FIG. Is Y = −H (H is the height of the CCD camera 10 from the road surface), the plane coordinates (X, Z) in the road surface are:

(Equation 1)

(Equation 2) Is calculated. F is the focal length of the CCD camera 10. The plane image converted in this manner, that is, an image when the road is viewed from directly above in a real three-dimensional space having the origin at the camera center is referred to as a pattern 1 for convenience. On the other hand, the image recognition device 16 records the vehicle behavior from the time when the image 1 was obtained, that is, from time T0 (S103). Vehicle behavior is
Specifically, it means a vehicle speed and a moving direction (for example, a yaw rate). Then, the current position, that is, the position and orientation of the vehicle at time T1 is calculated from the recorded behavior history (S10).
4). FIG. 8 shows an example of the position and the posture, from the position at time T0 (the origin in the figure) to (dX, dZ).
And the attitude angle (yaw angle) becomes θ. Then, the position of the vehicle at the current time T1 (dX, d
Z) and the pattern at time T0 to the pattern at the current time T1 using the attitude angle θ (this pattern is referred to as pattern 1 'for convenience).
Is predicted (S105). Specifically, when the coordinates of pattern 1 are (X0, Z0) and the coordinates of pattern 1 'are (X0', Z0 '),

(Equation 3) Is converted using. The obtained planar image is stored in the image recognition device 1
6 is stored in the RAM. After this processing is completed, the image at the current time T1 (referred to as image 2 for convenience) is input from the image capture board 12 (S106), and S1
The image is converted into a planar image in the same manner as in step S02 (S107). The plane image is referred to as pattern 2 for convenience. This pattern 2 has a current time T
This is a plane image actually obtained in 1 and corresponds to the pattern 1 '. Then, the pattern 1 'and the pattern 2 are compared (S108), and it is determined whether or not the shapes match (S109). In the actual comparison process, the coordinates of the corresponding feature points P of the pattern 1 'and the pattern 2 may be compared as described in the description of FIG. Then, when the shapes match, that is, when the coordinates of the corresponding feature points P of both patterns match, it is determined that the object in the image is a flat object on the road and does not become an obstacle (S110), and if not, It is determined that the object is not a plane object but a three-dimensional object and can be an obstacle (S111). It should be noted that whether or not they coincide with each other may be determined based on whether or not the coordinates of the two feature points P are shifted by a predetermined amount (for example, two pixels) or more.

As described above, in the present embodiment, the plane image at time T1 is predicted from the camera image acquired at time T0 based on the running state of the vehicle, and the predicted plane image and the plane image actually obtained at time T1 are predicted. By comparing the object with the image, it is possible to easily identify whether the object in the image is a planar object or a three-dimensional object. Further, when comparing the two plane images, it is only necessary to determine whether or not the coordinates of the characteristic points P of the two-dimensional images match, so that the processing can be speeded up. Note that the number of feature points does not need to be one, and a plurality of feature points are extracted as necessary, and the correspondence between the plurality of feature points is comprehensively determined to determine whether the object is a three-dimensional object or a planar object. Is also good. In other words, if a point in contact with the road is extracted as a feature point even for a three-dimensional object, there is no difference between the predicted plane image and the actual image. Is extracted, and when at least one of the plurality of feature points is displaced between the two images, it is preferable to determine that the object is a three-dimensional object. Of course, if there is room in the processing speed of the image recognition device, pattern matching between the predicted plane image and the actual plane image may be performed to determine whether the two shapes match.

In this embodiment, the step of calculating the predicted plane image at the time T1 from the plane image at the time T0 based on the running state of the vehicle is omitted, and only the characteristic points of the plane image at the time T0 are actually obtained. It may be determined whether or not the feature point of the planar image at the time T1 satisfies the relationship of the expression (3). Of course, even in such a process, the predicted plane image is substantially calculated at the time of using the expression (3), and it is needless to say that the process is equivalent to the process of the present embodiment.

<Second Embodiment> In the first embodiment,
Although the object in the image is identified as a flat object or a three-dimensional object, this embodiment shows a case where the height of the three-dimensional object is also detected.
The configuration of this embodiment is the same as that of FIG. 1 except that the CPU in the image recognition device 16 calculates the height of the object from the road surface by executing the height calculation program stored in the ROM. It is. FIG. 9 shows a processing flowchart of the present embodiment. The processing from S201 to S209 is the same as the processing from S101 to S109 shown in FIG. Then, as a result of comparing the pattern 1 'and the pattern 2, if it is determined that the shapes do not match (S2
(NO at 09), and a search is made for a corresponding point between pattern 1 'and pattern 2 (S210). As the corresponding point, the feature point P when the shape coincidence is determined may be used, and if necessary, another corresponding point (a feature point at which a deviation has been detected) may be searched.
Then, the coordinates of the characteristic point P of the pattern 1 'are represented by (X0', Z0
'), Assuming that the coordinates of the feature point P of the pattern 2 are (X1, Z1), the height of the object is calculated based on the amount of displacement in the Z-axis (see FIG. 7) (S211).

FIG. 10 shows the basic principle of calculating the height of an object. Height Hc from the road surface at time T0
Assume that an object is photographed by the in-vehicle CCD camera 10 at ar, and the apparent distance of the feature point P of the object is L1 (point Q in the figure). Next, it is assumed that the vehicle has moved by Lmove by time T1, and the apparent distance to the feature point P has become L2 (point R in the figure). At this time, the line of sight when the object is a flat object (a road print or the like) at the time T1 is a point Q as indicated by a dotted line in the figure. This corresponds to a point that would be obtained at time T1 if the object was a plane object. Therefore, the deviation between the Q point and the R point is equal to the deviation ΔZ (= Z1−Z0 ′) between the Z coordinate Z0 ′ of the characteristic point P of the pattern 1 ′ and Z1 of the characteristic point P of the pattern 2, and the height of the object H is from the figure

(Equation 4) Is required. However,

(Equation 5) ΔZ is usually negative (the R point is Q when viewed from the vehicle)
Before the point). As described above, the height of the object can be calculated from the shift amount ΔZ between the pattern 1 ′ that is the predicted plane image calculated from the time T0 and the pattern 2 that is the plane image actually obtained at the time T1 ( Lmove can be calculated based on the vehicle speed data from the vehicle behavior measuring device 14).

After calculating the height of the object in S211, it is determined whether or not the obtained height H is positive (S212).
This determination is for determining whether the object is a real image (ΔZ is negative) or a virtual image (ΔZ is positive).

FIG. 11 is a principle diagram when ΔZ is positive, that is, when an object is reflected in a puddle on a road or the like. In the puddle on the road, a virtual image of a real object (a tree is shown as an example in the figure) is visible, and at time T0, the apparent distance to the feature point P of this virtual image is L1 (point Q in the figure). If there is, at time T1, the apparent distance becomes L2 (point R in the figure), and point R is located farther from the vehicle than point Q. At this time, ΔZ = Z1−Z0 ′
Is positive, and the net H in equation (4) is also negative. Therefore, if the obtained H is negative, the object in the image can be determined to be a virtual image instead of a real image (S213), and if H is positive, the object is a three-dimensional object that can be an obstacle in the real image. Can be determined (S214). In the case of a virtual image, it can be determined as a non-obstacle as in the case where the shapes match (a planar object) (S215), and no alarm processing or the like is performed.

As described above, in the present embodiment, the height of a three-dimensional object can be calculated, rather than simply discriminating between a planar object and a three-dimensional object, so that the driver is notified of the height. More detailed processing is possible, for example, by changing the traveling control according to the height (for example, when the height is smaller than 0.5 m, the vehicle is passed as it is).

When the height of the object is calculated based on the difference between the predicted plane image and the actual plane image as in this embodiment, the sampling timing of the CCD camera 10 is important. The reason is that, for example, in order to reliably detect an object having a height (for example, 0.5 m or more) that can be an obstacle, it is necessary to always ensure a sampling interval that can detect a deviation amount regardless of the vehicle speed. (The same is true for the above-described first embodiment from the viewpoint of reliably detecting the shift amount.)

FIG. 12 shows that the number of pixels of the CCD camera 10 is 512 × 512 pixels, the focal length F = 7.5 mm, and the obstacle finding distance (the distance between the vehicle and the obstacle at time T0) =
30m, CCD camera height Hcar = 1.4m, "shift"
The relationship between the height H (cm) of the obstacle and the moving distance Lmove (m) of the vehicle at the sampling interval is shown when the minimum amount of difference to be determined is two pixels. From this figure, it can be seen that when an object having a height of H = 0.5 m or more is to be detected as an obstacle, it is sufficient to sample two images to be compared while the vehicle moves 7.2 m.

FIG. 13 shows the relationship between the sampling time (sec) required to capture two images with the CCD camera 10 while the vehicle moves 7.2 m and the vehicle speed (km / h). I have. From this figure, for example, 40 km / h
When the vehicle travels at h, if the sampling time of the CCD camera 10 is approximately 650 msec, an obstacle having a height of 0.5 m or more can be reliably detected.

[0026]

As described above, according to the present invention,
It is possible to quickly identify whether the object on the road is a three-dimensional object or a two-dimensional object, that is, whether the object can be an obstacle for the vehicle.

According to the present invention, when the object on the road is a three-dimensional object, the height of the object can be detected, and furthermore, the object in the image can be identified as a real image or a virtual image. More appropriate traveling control can be executed using the result of the image processing.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an embodiment of the present invention.

FIG. 2 is a plan view showing a positional relationship between a plane object 100 and a vehicle.

FIG. 3 is an explanatory diagram of an image of a plane object 100 obtained at times T0 and T1.

FIG. 4 is an explanatory diagram of an image of a three-dimensional object 200 obtained at times T0 and T1.

FIG. 5 is an explanatory diagram of a basic principle of processing according to the embodiment;

FIG. 6 is a processing flowchart of the embodiment.

FIG. 7 is an explanatory diagram of a coordinate system used for coordinate conversion.

FIG. 8 is an explanatory diagram showing the movement of the vehicle from time T0 to T1.

FIG. 9 is a processing flowchart of another embodiment.

FIG. 10 is an explanatory diagram showing a relationship between a vehicle and an image position in the case of a real image.

FIG. 11 is an explanatory diagram showing a relationship between a vehicle and an image position in the case of a virtual image.

FIG. 12 is a graph showing a relationship between an obstacle height and a moving distance.

FIG. 13 is a graph showing a relationship between a vehicle speed and a sampling time.

[Explanation of symbols]

10 CCD camera, 12 image capture board, 1
4 Vehicle behavior measurement device, 16 image recognition device.

Claims (3)

[Claims] 1. An image capturing means for capturing an image in front of a vehicle, a detecting means for detecting a running state of the vehicle, a converting means for converting an image obtained by the image capturing means into a plane image on a road surface, and a time T0. A plane image at time T1 and a plane image at the next time T1 are obtained, and a plane image at time T0 and a predicted plane image at the next time T1 calculated based on the traveling state at time T1−T0 are actually obtained at time T1. Processing means for comparing the obtained plane image with the obtained plane image to determine whether the object in the image is a plane object on a road or not. 2. When the processing means determines that the object in the image is not a plane object on the road, the road of the object is determined based on the difference between the predicted plane image and the plane image obtained at time T1. 2. The image processing apparatus for a vehicle according to claim 1, further comprising a calculating unit that calculates a height from a plane. 3. The vehicular image processing apparatus according to claim 2, wherein said calculating means determines that said object is a virtual image on a road when said height is negative.