JP2014089547A - Road surface level difference detection method, road surface level difference detection device, and vehicle equipped with the road surface level difference detection device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a road surface level difference detection device capable of detecting not only an obstacle in front but also a level difference of a road surface.SOLUTION: A road surface level difference detection method is configured to: project at least a first image and a second image in which a road surface is stereoscopically imaged on an XY plane coordinate system; calculate a parallax when an image is on a road surface position, for each piece of image row data in a prescribed Y-axis direction; generate a third image corrected through shifting the parallax of the second image for each Y axis; and compare the first image with the third image for each detection area of a level difference to obtain a height from a road surface.

Description

  The present invention relates to a road surface step detection method, a road surface step detection device, and a vehicle including a road surface step detection device that detect a road surface step from a plurality of image data obtained by imaging a road surface with a stereo camera.

  Recently, the target scenery outside the vehicle is imaged with a camera mounted on the car, and the captured image is processed to determine the distance from the car to the object, predicting the risk of collision with the vehicle or guardrail ahead. Thus, a method of performing control such as braking the vehicle has been put into practical use.

  The distance measurement technology using such images is based on the principle of triangulation from the technology that estimates the distance to a target object from the monocular image using the relationship with the camera position and the stereo images captured by multiple cameras. It is roughly divided into the technology that calculates the distance. Among these, the technique for obtaining the distance from the stereo image based on the principle of triangulation obtains the distance from the relative shift of the position of the same object in the left and right images, and therefore, an accurate distance can be obtained.

  For example, Patent Literature 1 discloses an image storage unit that stores images input by a plurality of cameras, a feature extraction unit that extracts a plurality of white lines existing on a road surface, and an arbitrary on a road surface from the extracted white lines. An obstacle detection apparatus comprising a parameter calculation unit that obtains a relational expression that is established between projection positions of each point on each image, and a detection part that detects an object having a height from the road surface using the relational expression Disclosure.

  According to the obstacle detection device of Patent Document 1, even when there is a change in the slope of the road surface, the road surface is recognized from the movement of the two white lines, and the obstacle present on the road surface is detected at high speed and with high accuracy. Can be detected.

JP 2001-76128 A

  However, the obstacle detection device of Patent Document 1 is applied to a vehicle that travels on a road surface without a lane, such as a senior car or an electric wheelchair, in order to recognize a road surface by extracting a white line such as a lane. In this case, there is a problem that the road surface cannot be recognized correctly and it is difficult to detect an obstacle.

  Further, when measuring the distance to the obstacle by triangulation parallax as in the obstacle detection device of Patent Document 1, as shown in FIG. 12, a solid object on the road surface and a concave portion of the road surface are imaged together. In the image, the three-dimensional object has the same shape in the left and right images because the front is reflected, and the parallax can be obtained. This makes it difficult to accurately measure the distance to the top surface of the convex portion and the bottom surface of the concave portion by the same measurement method as that for the three-dimensional object.

  This invention is made | formed in view of such a subject, and provides the road surface level | step difference detection method which can detect level | step differences, such as a recessed part parallel to a road surface, correctly.

  In order to solve the above-described problem, the road surface level difference detection method of the present invention projects at least a first image and a second image obtained by stereo shooting a road surface onto XY plane coordinates, and row data of the image in a specific Y-axis direction. The parallax when the image is at the road surface position is calculated every time, a third image is generated by correcting the second image by shifting the parallax for each Y axis, and the first image and the third image are stepped. The height from the road surface is detected in comparison with each detection region.

  Further, the height difference between adjacent detection areas is obtained from the heights of a plurality of detection areas, and it is determined that there is a step between the detection areas if the height difference is equal to or greater than a threshold value. .

  In addition, the height difference between adjacent detection areas is obtained from the heights of the plurality of detection areas, and there is an inclination between the detection areas when the height difference continuously changes between the plurality of detection areas. It is characterized by determining.

  The road surface level difference detection apparatus of the present invention projects a stereo camera that stereo-shoots at least a first image and a second image of a road surface, and projects the first and second images shot in stereo on XY plane coordinates, Calculating a parallax when the image is at a road surface position for each row data of the image in a specific Y-axis direction, generating a third image corrected by shifting the parallax for each Y-axis, A height detection unit that compares the first image and the third image for each step detection region and detects the height from the road surface is provided.

  A vehicle according to the present invention includes the road surface level difference detecting device described above.

  According to the present invention, a step can be easily detected from each area of a road surface in an image using an image captured by a stereo camera.

It is a block diagram of a road surface level | step difference detection apparatus. It is a layout diagram of a stereo camera. It is the image imaged with the selethio camera. It is a figure which shows the correction method of a stereo image. It is a flowchart of the height detection process in a calculating part. It is explanatory drawing which shows the distance calculation method. It is explanatory drawing which shows the parallax calculation method. It is explanatory drawing which shows the parallax calculation method. It is explanatory drawing which shows the road surface height calculation method. It is an example of a road surface height calculation result. It is a senior car which is a vehicle provided with a road surface level difference detection device. It is a figure which shows the distortion of a stereo image.

  An embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration of a road surface level difference detecting device 1 according to the present invention. The road surface level difference detection device 1 according to the present invention includes two cameras 11 and 12 that capture a stereo image, and a calculation unit 20 that calculates a stereo image. In addition, the calculation unit 20 includes a height detection unit 30 that calculates the height of a region in which a step in the stereo image is to be detected, and a step detection unit 40 that determines the presence or absence of a step between detection regions from the height of each detection region. It is composed of Furthermore, an output device 50 such as an audio speaker or a display device for notifying the operator of the presence or absence of a step is provided as necessary.

  FIG. 2A is a top view showing the arrangement of the stereo camera, and FIG. 2B is a side view showing the mounting position of the stereo camera. The two cameras 11 and 12 have the same specifications, and have a predetermined horizontal angle of view as shown in FIG. 2A. For example, a predetermined interval g is set to the left and right in the front part of a senior car or an electric wheelchair vehicle. Installed separately. Also, as shown in FIG. 2B, the cameras 11 and 12 are installed at a predetermined height hc from the road surface, have a predetermined vertical field angle and depression angle, and the optical axis of the lens images the road surface side. So that it is fixed downward.

  If the horizontal and vertical angles of view of the cameras 11 and 12 are too large, the step of the road surface to be imaged becomes small and the detection accuracy of the step is lowered. Conversely, if the angle of view is too small, the detection range of the step becomes narrow. It is necessary to set appropriately according to conditions. The depression angle is preferably set so that the proportion of the road surface is increased. The mounting height hc of the cameras 11 and 12 is preferably set as high as possible in order to widen a detectable range from a small step to a large step.

  The specifications and arrangement of the cameras 11 and 12 have, for example, a horizontal field angle and a vertical field angle corresponding to a 35 mm lens, a mounting interval g is 15 to 25 cm, a mounting height hc is 60 to 80 cm, 10 It is fixed at a depression angle of ˜25 °. In the following description of the present embodiment, half of the horizontal and vertical angles of view are θ1 and θ2, respectively, and the depression angle is θ3. In addition, as shown in FIG. 2, an embodiment in which the cameras 11 and 12 are installed on the left and right will be described. However, the cameras 11 and 12 can be installed in the vertical and diagonal directions. The detection method is the same.

  FIG. 3 is an image of two stereo cameras, and a road surface including a sidewalk is captured. FIG. 3A is a first image captured by the left camera 11, and FIG. 3B is a second image captured by the right camera 12. Further, FIG. 3C is obtained by extracting only the boundary line between the sidewalk and the road surface by superimposing the first image and the second image. As shown in FIG. 3C, the image is taken at a position where the boundary line is shifted in the left image and the right image.

  The amount of deviation in the left-right direction is parallax, and the road surface level difference detecting device of the present invention extracts such parallax v1 and detects the height and height of the level difference from the road surface. As described above, since the bottom surface of the concave portion parallel to the road surface is distorted and imaged unlike a three-dimensional object, the height from the road surface is detected using a third image in which this distortion is corrected.

  4A and 4B are diagrams for explaining correction of image distortion, in which FIG. 4A shows a first image, FIG. 4B shows a second image, and FIG. 4C shows a third image after correction. Yes. As shown in FIG. 4, in the road surface level difference detection method of the present invention, the first image and the second image are taken in the first image and the second image after the first image and the second image obtained by viewing the road surface from different directions with a stereo camera. 4, the line data obtained by collecting pixel values in the same Y coordinate are compared with each other, and the parallax v1 when the object in the line data is assumed to be on the road surface is expressed as a coordinate value in the Y axis direction. And camera information. After calculating the parallax v1 for all the Y axes, as shown in FIG. 4C, a third image corrected by each parallax v1 in the Y axis direction is generated in the second image. Then, the corrected third image and the first image are compared for each detection area, and the height of the detection area from the road surface is detected based on the shift amount v2 between the third image and the first image.

  FIG. 5 is a flowchart of the height detection process in the calculation unit 20 of the road surface level difference detection device 1 of the present invention. The computing unit 20 first obtains the parallax v1 for each row data by the processing from step 1 to step 3.

  In order to obtain the parallax v1, it is first necessary to calculate the distance d1 from the camera to the focal plane of the image. FIG. 6 is an explanatory diagram illustrating a distance calculation method. In FIG. 6A, the first image is converted into a coordinate space 13 in which the vertical center is ± w pixels and the horizontal direction is ± h pixels with the origin (P) of the coordinates (0, 0) as the origin, and there is row data. Coordinate points (X, Y) are shown in the coordinate space 13. FIG. 6B is a side view showing the focal plane A1 of the cameras 11 and 12 and camera position information.

  In step 1, row data including coordinate points (X, Y) of an arbitrary detection area for detecting a step in the coordinate space 13 is selected. The coordinate space 13 corresponds to a focal plane that is a plane perpendicular to the optical axis of the camera 11 shown in FIG. When there is no distortion of the lenses of the cameras 11 and 12, since all the objects on the focal plane have the same parallax, the coordinate point (X, Y) in the coordinate space 13 and the origin P also have the same parallax.

  In step 2, the distance d1 from the camera to the origin P of the focal plane A1 when the coordinate point (X, Y) is assumed to be on the road surface is obtained. Here, since it is complicated to calculate the distance d1 using the coordinate point (X, Y) as a base point, the same focal point as the coordinate point (X, Y) is used by utilizing that the coordinate points on the focal plane all have the same parallax. A distance d1 to the origin P of the focal plane A1 is calculated using the coordinate point Q (0, Y) on the plane A1 as a base point.

  FIG. 7 is an explanatory diagram illustrating a parallax calculation method. FIG. 7A is a side view of the camera 11 and shows a downward angle θy when the coordinate point Q is viewed from the camera 11. Assuming that half of the vertical angle of view of the camera 11 is θ2, the height of the coordinate point Q in FIG. 6A is Y, so θy can be obtained by the following equation 1.

θy = arctan (tan θ2 × Y / h) (Formula 1)
As shown in FIG. 6B, when the camera mounting height is hc and the camera is mounted downward at a depression angle θ3, the distance d1 ′ from the camera 11 to the coordinate point Q is
d1 ′ = hc / sin (θ3 + θy) (Formula 2)
It can be obtained more.

Therefore, the distance d1 from the camera 11 to the origin P of the focal plane A1 is
d1 = d1 ′ × cos (θy) (Formula 3)
It can be obtained more.

  Next, in step 3, the parallax v1 between the first image and the second image when the road surface is on the focal plane A1 is obtained. In the case of a lens without distortion, all the points on the focal plane A1 are considered to have the same parallax. Therefore, the parallax v1 may be obtained using the distance d1 obtained in the above (Equation 3). The distance d1 may use another value or need to be corrected depending on lens distortion or the like.

  FIG. 7B is a top view of the left and right cameras 11 and 12. As shown in FIG. 7B, the origin P located at the focal plane A1 of the left camera 11 is seen by the right camera 12 in the direction of the angle θx from the center. This θx is obtained by the following equation, where g is the distance between the left and right cameras.

θx = arctan (g / d1) (Formula 4)
At this time, the origin P looks like the coordinates of the origin (0, 0) in the first image on the left side as shown in FIG. 8A, and the parallax in the second image on the right side as shown in FIG. 8B. When the number of pixels is v1, it appears at the point of (−v1, 0). As shown in FIG. 6A, v1 is obtained by the following equation when θ1 is half of the horizontal field angle of the camera.

v1 = w · tan θx / tan θ1 (Formula 5)
Since the number of parallax pixels v1 is the same at the point P and the point (X, Y), it appears at the position (Xr, Y) in the right image.
here,
Xr = X-v1 (Formula 6)
It is.

  Next, in step 4, after obtaining the parallax v1 between the first image and the second image when there is a road surface for each row data, the second image is as shown in FIG. to correct. In the correction method, each row data of the second image is moved in the X coordinate direction by a parallax v1 pixel corresponding to the Y coordinate. When the parallax v1 is a decimal, it is complemented by two nearby points. For example, when v1 is 5.5, correction is performed by writing half of the sum of the fifth and sixth pixels from the left in the position of the zeroth pixel.

  The third image shown in FIG. 4C is obtained by correcting the row data of the second image over the entire Y coordinate with the parallax v1. As a result, the corrected object of the third image has the same shape as the first image.

In step 5, it is determined whether or not the object shown in the coordinates (X, Y) of the first image is at the same height as the road surface. Here, the detection area centered on (X, Y) of the first image and the comparison area centered on (X, Y) of the third image are compared, and the object is the same as the detection area and the comparison area. It can be judged by whether it is reflected in the position of.

  There are various methods for confirming whether or not they are the same object. For example, the luminance of several pixels around the target point of the left and right images can be extracted and compared. If they match within the range of error factors such as camera noise, the point can be determined to be the same height as the road surface. If it is determined that the two do not match and are shifted to the left or right, it can be determined that the position is higher or lower than the road surface according to the shift amount.

For example, when it is determined that the object at the position (X, Y) in the first image on the left side is at the coordinates (Xr−v2, Y) in the second image on the right side, When the distance is d2, from (Expression 4) and (Expression 5),
v1 = (w × g) / (d1 × tan θ1)
v1 + v2 = (w × g) / (d2 × tan θ1)
If v1 is deleted,
d2 = (d × w × g) / (w × g + v2 × tan θ1) (Expression 7)
As shown in FIG. 9, the height hs of the object on the road surface is
hs = hc × (d1−d2) / d1 (Formula 8)
Thus, the height hs of the road level difference can be known. That is, when the parallax v2 of the object is positive (larger than the parallax v1 of the road surface), the distance d2 to the object is smaller than the distance d1 to the road surface as shown in FIG. 9A, and hs is a positive value. It can be judged that it is higher than the road surface. Conversely, when v2 is negative (smaller than the parallax v1 on the road surface), the distance d2 to the object is larger than the distance d1 to the road surface as shown in FIG. 9B, and hs is a negative value. It can be judged that it is lower than the road surface. In this way, the height difference from the road surface of the coordinate point (X, Y) in the first image is known.

  In step 6, the above procedure is repeated at other coordinate points at appropriate intervals, and the process ends when the detection of the height from the road surface within the required range in the image is completed.

  The details of the flowchart shown in FIG. 5 are as described above, and these processes are processed by the arithmetic unit 20 shown in FIG. Specifically, it may be realized as software on a PC or a microcomputer, or may be realized as hardware using an FPGA or ASIC. A configuration in which the remainder is partially processed by hardware and software is also possible.

  FIG. 10 is an example of a result of applying the above method to the stereo image of FIG. The gradation is displayed according to the height of each detection area, the area having the same height as the road surface is displayed in gray, and the area lower than the road surface is displayed in black.

  Next, the level difference detection unit 40 of the calculation unit 20 will be described. In the detection result of the height detection unit 30 as shown in FIG. 10, the height from the road surface is compared between the detection areas adjacent to each other in the vertical and horizontal directions. It is judged that. The height difference threshold for determining a step is set so as to ensure safety even when a senior car or a wheelchair falls, for example. As an example, in the detection result of FIG. 10, a boundary determined as a step is indicated by a broken line portion.

  Further, when it is assumed that the height difference between adjacent detection areas is small and a road surface, it is possible to obtain the inclination angle of the road surface from the distance between the detection areas and the height difference. Thereby, when the road surface is inclined uphill, downhill, or left and right, it is possible to determine whether the road surface can be secured.

  FIG. 11 shows an application example of a senior car 60 as an example of a vehicle provided with the road surface level difference detecting device 1 of the present invention. The road surface level difference detection device 1 is provided in front of the handle 61 of the senior car 60 at a height hc from the road surface. The road surface level difference detection device 1 includes an output device 30 such as a speaker 31 and a display device 32 in order to notify the driver of the senior car 60 of the level difference detection result. When a road level difference is detected by the road level difference detection device 1, a buzzer sound or voice guidance is output from the speaker 53, or a character or a graphic is displayed on the display device 54 to notify the existence of the level difference on the road surface. It is possible to avoid dangers such as falling wheels and falling of the senior car 60.

  Further, the step detecting device 1 of the present invention may be provided not only in front of the senior car 60 but also in the rear. As a result, it is possible to avoid danger such as falling wheels even when reversing with poor visibility. Further, the use of the level difference detection device 1 of the present invention is not limited to the senior car 60, and is suitably used in vehicles that need to detect road level differences, for example, various vehicles ranging from wheelchairs to forklifts. It is possible.

  For example, in a wheelchair, by providing the step detection device 1 of the present invention in the front and rear and left and right, even if the direction is changed on the spot, the risk of falling or falling over the step around the wheelchair is prevented. can do. In addition, with a forklift, even if the forward field of view may be blocked during cargo transportation, it is possible to detect a short cargo placed on the road surface as a step and avoid a collision.

DESCRIPTION OF SYMBOLS 11, 12 Camera 20 Calculation part 30 Height detection part 40 Level difference detection 50 Output device

Claims (5)

Projecting at least a first image and a second image obtained by stereo shooting of a road surface onto XY plane coordinates;
Calculating the parallax when the image is at the road surface position for each row data of the image in a specific Y-axis direction;
Generating a third image obtained by correcting the second image by shifting the parallax for each Y axis;
A road level difference detection method, wherein the height from the road surface is detected by comparing the first image and the third image for each level difference detection area. From the height of the plurality of detection areas, to determine the height difference between the adjacent detection areas,
The road surface level detection method according to claim 1, wherein if the level difference is equal to or greater than a threshold value, it is determined that there is a level difference between the detection areas. From the height of the plurality of detection areas, to determine the height difference between the adjacent detection areas,
The road surface level difference detection method according to claim 1, wherein it is determined that there is an inclination between the detection areas when the difference in height continuously changes between the plurality of detection areas. A stereo camera for stereo shooting at least a first image and a second image of the road surface;
The first and second images captured in stereo are projected onto XY plane coordinates, and parallax is calculated when the image is at the road surface position for each row data of the image in a specific Y-axis direction. A third image is generated by correcting the parallax for each Y-axis by shifting the parallax, and the height from the road surface is compared by comparing the first image and the third image for each step detection area. A road surface level difference detection device comprising: a height detection unit for detection. A vehicle comprising the road surface level difference detecting device according to claim 4.