JPH06266828A - Outside monitoring device for vehicle - Google Patents

Abstract

PURPOSE:To surely detect a sidewall to be a continuous solid object as a boundary of a road such as a guard rail, a bush, and a pylon string and to detect the existence, position and direction of the sidewall by a data format to be easily processed. CONSTITUTION:A stereoscopic optical system 10 picks up an image of an object existing in a set range on the outside of a vehicle 1 and inputs the image to a stereoscopic image processor 20. The processor 20 processes the image picked up by the optical system 10 and calculates the distance distribution of the whole picture. A road/sidewall detector 100 calculates the three-dimensional positions of respective parts of the object correspondingly to the information of the distance distribution and detects a road shape and a sidewall by a sure and data format to be easily processed based on the information of these three- dimensional positions.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a vehicle exterior monitoring device for detecting a side wall as a continuous three-dimensional object which becomes a boundary of a road such as a guide rail.

[0002]

2. Description of the Related Art Vehicles such as automobiles, which are one of means for transporting people and goods, have become indispensable in modern society because of their convenience and comfort. Therefore, technology has been developed to determine the risk of accidents such as car collisions and deviations from the road, and to automatically avoid these accidents, without sacrificing the advantages of cars. Therefore, recognizing the route along which the vehicle travels and the range of the road on which the vehicle can travel is one of the important issues.

As a technique for recognizing the shape of the road and the range in which the vehicle can travel, a white line on the road is obtained by taking an image of a target landscape outside the vehicle with an image pickup device such as a camera mounted on the vehicle and processing the picked-up image. , A technique for detecting an object around another vehicle or a road has been developed, and further, a measurement technique for obtaining a shape of the road and a distance to the object has been developed.

This image recognition / measurement technique is based on a technique of estimating the distance to an object using the relationship between a monocular image and a camera position, and by using a plurality of cameras or changing the position of one camera. Take multiple images,
It is roughly divided into the so-called stereo method, which is the technique of finding the distance to an object based on the principle of triangulation.

Various techniques for detecting obstacles and road shapes from a two-dimensional image by a monocular view have been proposed. For example, in Japanese Patent Laid-Open No. 1-2242916, one TV camera is installed in a front window of a vehicle. It is installed near the center upper part, and using the image obtained from this, it detects obstacles and white lines on the road from the brightness distribution pattern on a survey line in the image and the two-dimensional brightness distribution pattern, and mounts the TV camera. A technique is disclosed in which a camera position is assumed from parameters such as a position, a direction, and a visual field, and a three-dimensional position of an obstacle or a white line is estimated.

However, various objects and backgrounds such as surrounding buildings and trees are reflected in the image taken on the actual road, and the white line of the road is defined by only the two-dimensional characteristic pattern among them. It is difficult to detect accurately.
Furthermore, there are many places where white lines are not maintained on ordinary roads, and in this case, this technology cannot be used.

As described above, on a road without a white line, a side wall as a continuous three-dimensional object, such as a guardrail, a plant, or a pylon row, which is a boundary of the road, can be detected to detect the shape of the road and the travelable range. Is considered to be effective. However, it is difficult to detect such a side wall with a technique using a monocular image.

On the other hand, the technique of the stereo method for obtaining the distance from a plurality of images by the principle of triangulation obtains the distance from the relative displacement of the positions of the same object in the left and right images, so that the accurate distance can be obtained.

For example, in JP-A-48-2927 and JP-A-55-27708, two TV cameras are attached to the front of the vehicle, and first, each image is spatially differentiated so that only the light-dark change points are present. Is extracted, and the scanning of the image of one camera is delayed by a predetermined time and then superimposed on the other image,
A technique of extracting only a three-dimensional object from the characteristics of the change in light and shade of the three-dimensional object among the portions that match in two images is disclosed. 21,
No. 2 (February 1985), "A technique for recognizing a two-dimensional distribution of obstacles", which describes a technique for detecting a guardrail on the side of a road by a passage pattern recognition device is described.

However, according to this technique, the detected guardrail is merely output as a collection of data of the positions of each fine portion, and the side wall is not recognized as a three-dimensional object that continuously exists. Therefore, if it is attempted to use the output of this detection result to determine the risk of accident occurrence or to avoid the accident as described above, complicated data processing is required.

Further, a technique has been developed in which ultrasonic sensors are attached to the side surfaces and four corners of a vehicle to detect a guardrail or the like, but the ultrasonic sensor has a short measurable distance and can detect only the guardrail in the vicinity of the vehicle. For this reason,
When traveling on a general road at a normal speed, the detection ability is insufficient, and furthermore, it is impossible to deal with a side wall that is difficult to detect with ultrasonic waves such as implantation.

[0012]

As described above, in an actual complicated situation, various objects are shown in an image on an actual road, and an object to be detected such as a guardrail, a plant, a pylon row, etc. There is also a case where another object in front is partially hiding. Under such circumstances, it is extremely difficult to accurately detect only the desired side wall.

The present invention has been made in view of the above circumstances, and it is possible to reliably detect a side wall as a continuous three-dimensional object that becomes a boundary of a road such as a guardrail, a plant, a pylon row, and the presence or absence of the side wall. An object of the present invention is to provide a vehicle exterior monitoring device capable of detecting the position, the direction, and the like in a data format that can be easily processed.

[0014]

According to a first aspect of the present invention, there is provided a measuring means for detecting an object in a set range outside the vehicle and outputting position information for the object, and a road boundary based on the position information from the measuring means. The side wall straight line detecting means for detecting the presence or absence of a side wall as a continuous three-dimensional object and a linear type approximating the position of this side wall, and a space area is set around the straight line type detected by the side wall straight line detecting means. And a sidewall range detecting means for extracting only the data in the spatial region and detecting the range in which the sidewall exists.

A second aspect of the invention is to determine a road model based on measuring means for detecting an object within a setting range outside the vehicle and outputting three-dimensional position information for this object, and three-dimensional position information from the measuring means. Then, the road detection means for detecting this road model as a road shape, the data extraction means for extracting only the data above the road surface from the three-dimensional position information based on the road model, and the data extraction. From the data extracted by the means, the presence or absence of a side wall as a continuous three-dimensional object that becomes the boundary of the road, and a side wall straight line detecting means for detecting a straight line formula approximating the position of this side wall, and the side wall straight line detecting means are detected It is characterized by further comprising: a side wall range detecting means for setting a space area around the straight line type, extracting only data in the space area, and detecting a range in which the side wall exists.

In a third aspect based on the second aspect, the road detecting means includes a road shape estimating means for estimating a position and a shape of a white line of the road based on the three-dimensional position information.
A three-dimensional window setting means for setting a three-dimensional space area that includes the white line of the road estimated by the road shape estimation means as a three-dimensional window, and only data in the three-dimensional window from the three-dimensional position information. A linear element detecting means for detecting a three-dimensional linear element forming the road model, and determining the validity of the linear element detected by the linear element detecting means, and if the determination criteria do not match, And a road shape determining means for determining the road model by correcting or changing the straight line element.

In a fourth aspect based on the first or second aspect, the side wall straight line detecting means sets a search area for searching for the existence of the side wall, and extracts only data in this search area. It is characterized in that the presence / absence of the side wall and the linear equation are detected by performing the Hough transform.

In a fifth aspect based on the first or second aspect, the side wall straight line detecting means divides and sets a search region for searching the presence of the side wall in a grid pattern, and the search region is set within the search region. The number of data contained in each grid is obtained by extracting only the data of (1), and the Hough transform is performed assuming that the data in each grid exists at the central position of each grid. And the linear equation are detected.

According to a sixth aspect of the invention, in the first or second aspect of the invention, the side wall range detecting means has a vertical axis that represents the number of data in the spatial area and a horizontal axis that represents the distance from the vehicle. A histogram is created, and the range in which the side wall exists is detected from the magnitude of the frequency of this histogram.

In a seventh aspect based on the first or second aspect, the side wall straight line detecting means sets a search area for searching for the existence of the side wall, and extracts only data in this search area. By performing a Hough transform by detecting the presence or absence of the side wall and the linear equation,
The side wall range detection means creates a histogram with the number of data in the spatial region as the vertical axis and the distance to the vehicle as the horizontal axis, and detects the range in which the side wall exists from the magnitude of the frequency of this histogram. It is characterized by being

In an eighth aspect based on the first or second aspect, the side wall straight line detecting means sets a search area for searching the existence of the side wall in a grid pattern and sets the search area within the search area. The number of data contained in each grid is obtained by extracting only the data of (1), and the Hough transform is performed assuming that the data in each grid exists at the central position of each grid. And the linear equation, the side wall range detection means creates a histogram with the vertical axis representing the number of data in the spatial region and the horizontal axis representing the distance to the vehicle, and this histogram The range in which the side wall exists is detected from the magnitude of the frequency.

In a ninth aspect based on the first or second aspect, the measuring means outputs the position information for an object outside the vehicle by picking up an image of a set range outside the vehicle and processing the image. Is characterized in that.

According to a tenth aspect of the present invention, in the first or second aspect of the invention, the measuring means scans a set range outside the vehicle and projects / receives a laser beam to obtain the position information for the object outside the vehicle. Is output.

[0024]

In the first aspect of the present invention, the presence or absence of the side wall as a continuous three-dimensional object that becomes the boundary of the road and the linear expression approximating the position of the side wall are detected based on the position information for the object in the set range outside the vehicle. Then, a space region is set around this linear equation, and only the data in this space region is extracted to detect the range where the side wall exists.

In the second invention, the road model is determined based on the three-dimensional position information for the object within the set range outside the vehicle, and the road model is selected from the three-dimensional position information above the road surface based on the road model. Extract only certain data. Then, from the extracted data, the presence or absence of a side wall as a continuous three-dimensional object that becomes the boundary of the road and a linear equation approximating the position of this side wall are detected, and further, a spatial region is set around this linear equation. The range in which the side wall exists is extracted by extracting only the data in this spatial region.

According to a third invention, in the second invention, the position and shape of the white line of the road are estimated based on the three-dimensional position information, and the three-dimensional spatial area that includes the estimated white line of the road is cubically ordered. While setting as the original window, only the data in the three-dimensional window is extracted from the three-dimensional position information to detect a three-dimensional linear element that constitutes the road model, and the validity of the detected linear element is determined. Judge,
If it does not match the determination criteria, the road shape is detected by correcting or changing the linear element to determine the road model.

According to a fourth aspect of the present invention, in the first or second aspect of the present invention, a search area for searching for the existence of the side wall is set, and only the data in the search area is extracted to perform the Hough transform. The presence or absence of the side wall and the linear equation are detected.

According to a fifth aspect of the present invention, in the first or second aspect of the present invention, a search area for searching for the existence of the side wall is divided into grids and set, and only the data within the search area is extracted. The number of data included in each grid is obtained, and the data in each grid is subjected to a Hough transform assuming that the data exist in a central position of each grid, and thereby the presence or absence of the side wall and the linear equation are determined. To detect.

According to a sixth aspect of the present invention, in the first or second aspect of the present invention, a histogram is created in which the vertical axis represents the number of data in the spatial area and the horizontal axis represents the distance to the vehicle. The range in which the side wall exists is detected from the magnitude of the frequency.

According to a seventh invention, in the first or second invention, a search area for searching for the existence of the side wall is set, and only the data in the search area is extracted to perform the Hough transform. The presence or absence of the side wall and the linear equation are detected, and a histogram is created with the number of data in the spatial region as the vertical axis and the distance to the vehicle as the horizontal axis, and the frequency of this histogram. The range in which the side wall is present is detected from the size of.

According to an eighth invention, in the first or second invention, the search area for searching for the existence of the side wall is divided and set in a grid pattern, and only the data in this search area is extracted. The number of data included in each grid is obtained, and the data in each grid is subjected to a Hough transform assuming that the data exist in a central position of each grid, and thereby the presence or absence of the side wall and the linear equation are determined. In addition to the detection, a histogram is created in which the vertical axis represents the number of data in the spatial area and the horizontal axis represents the distance to the vehicle, and the range in which the sidewall exists is detected from the magnitude of the histogram.

In a ninth aspect based on the first or second aspect, the position information for the object outside the vehicle is output by imaging the set range outside the vehicle and performing image processing.

According to a tenth aspect of the present invention, in the first or second aspect of the invention, the position information for an object outside the vehicle is output by scanning a set range outside the vehicle and projecting / receiving laser light.

[0034]

Embodiments of the present invention will be described below with reference to the drawings. 1 to 38 relate to the first embodiment of the present invention, and FIG.
2 is an overall configuration diagram of the vehicle exterior monitoring device, FIG. 2 is a front view of the vehicle, FIG. 3 is a circuit block diagram of the vehicle exterior monitoring device, FIG. 4 is an explanatory diagram showing a relationship between a camera and a subject, and FIG. 5 is a stereo image processing device. FIG. 6 is a circuit configuration diagram showing a specific example, FIG. 6 is an explanatory diagram of a city block distance calculation circuit, FIG. 7 is a block diagram of a minimum value detection circuit, and FIG. 8 is an explanatory diagram showing an example of an image captured by a vehicle-mounted CCD camera. 9 is an explanatory view showing an example of a distance image, FIG. 10 is a top view of the vehicle, FIG. 11 is a side view of the vehicle, and FIG.
FIG. 13 is an explanatory diagram showing an example of a road model, FIG. 14 is an explanatory diagram showing the shape of a three-dimensional window, FIG. 15 is an explanatory diagram showing the shape of a two-dimensional window, and FIG. 16 is a straight line. Explanatory diagram showing the amount of deviation between elements and data,
FIG. 17 is an explanatory diagram showing the relationship between the shift amount and the weighting coefficient, and FIG.
Is an explanatory view showing an example of a detected road shape, FIG. 19 is an explanatory view showing a shape of a search region in the side wall detection, FIG. 20 is an explanatory view showing an example of an image in the side wall detection, and FIG. 21 is a distribution state of three-dimensional object data. 22 is an explanatory view showing the assumption of a straight line in the Hough transform, FIG. 23 is an explanatory view showing the voting area of the parameter space, FIG. 24 is an explanatory view showing the voting result in the parameter space, and FIG. 25 is Explanatory view showing the side wall candidate region,
FIG. 26 is an explanatory diagram showing the relationship between the histogram and the side wall existence range, FIG. 27 is a flowchart showing the operation of the stereo image processing device, FIG. 28 is an explanatory diagram showing the storage order in the shift register, and FIG. 29 is a city block distance calculation. 32 is a timing chart showing the operation of the circuit, FIG. 30 is a timing chart showing the operation of the deviation amount determining unit, FIG. 31 is a timing chart showing the operation of the stereo image processing apparatus, and FIGS.
5 is a flowchart showing the operation of the road detection unit, FIG. 32 is a flowchart of road shape estimation processing, FIG. 33 is a flowchart of three-dimensional window generation processing, FIG. 34 is a flowchart of linear element detection processing, and FIG. 35 is road shape determination. 36 to 38 are flowcharts showing the operation of the sidewall detection unit, FIG. 36 is a flowchart of solid object data extraction processing, FIG. 37 is a flowchart of sidewall straight line detection processing, and FIG. 38 is a sidewall range detection processing. It is a flowchart.

In FIG. 1, reference numeral 1 is a vehicle such as an automobile, and an on-vehicle monitoring device 2 for recognizing and monitoring an object within an installation range outside the vehicle is mounted on the vehicle 1. The vehicle exterior monitoring device 2 processes a stereo optical system 10 for capturing an image of an object within a setting range outside the vehicle and a stereo image processing device 20 for processing an image captured by the stereo optical system 10 and calculating three-dimensional distance distribution information. And the distance information from the stereo image processing device 20, and based on the distance information, the road shape and the three-dimensional position of the side wall as a continuous three-dimensional object that becomes the boundary of the road such as a guardrail, a plant, and a pylon row. In the case where there is a risk that the vehicle 1 collides with or touches the recognized side wall by connecting an external device for controlling actuators (not shown) It is possible to perform operations such as warning of the driver and avoidance of automatic collision of the vehicle body.

The stereo optical system 10 is constituted by a CCD camera using a solid-state image pickup device such as a charge coupled device (CCD) as an image pickup device for converting a picked-up image into an electric signal. As shown in FIG. 2, two CCD cameras 11a and 11b for long-distance left and right images are provided.
(The CCD camera 11 may be described as a representative)
Are attached to the front of the ceiling in the passenger compartment at regular intervals, and are used for short-distance left and right images.
The CCD cameras 12a and 12b (sometimes referred to as the CCD camera 12 as a representative) are mounted inside the long distance CCD cameras 11a and 11b at regular intervals.

Further, as shown in FIG. 3, the stereo image processing device 20 inputs two image signals on the left and right from the stereo optical system 10, and the two images are the same for each minute area of the image. A distance detection circuit 20a that detects a part in which an object is captured and executes a process for calculating the distance to the object from the amount of displacement of the position on the image over the entire image, and the distance output by the distance detection circuit 20a. A measuring means including a distance image memory 20b for storing information and the like, and the stereo optical system 10 and the stereo image processing device 20 detect an object in a setting range outside the vehicle and output position information for the object. Is configured.

The road / sidewall detecting device 100 is mainly composed of a microprocessor 100a which reads out the distance information written in the distance image memory 20b and performs various calculation processes, and is a read-only memory for storing a control program ( ROM) 100b, read / write memory (RAM) 100 for storing various parameters during calculation processing
c, an interface circuit 100d, an output memory 100e for storing parameters of processing results, and the like. The interface circuit 100d includes the vehicle 1
A vehicle speed sensor 3 attached to the vehicle and a steering angle sensor 4 for detecting the steering angle of the steering wheel are connected.

The microprocessor 100a inputs the distance image through the distance image memory 20b, executes the calculation process, and outputs the road shape and side wall parameters, which are the processing results, to the output memory 100e. The external device connected to the road / sidewall detection device 100 receives these parameters via the output memory 100e.

Here, as the stereo optical system 10,
When measuring the distance from the closest distance to 100 m, for example, assuming that the mounting positions of the CCD cameras 11 and 12 in the vehicle interior are, for example, 2 m from the tip of the hood of the vehicle 1, the position from 2 m to 100 m in front is actually measured. I wish I could.

That is, as shown in FIG. 4, a point P located at a distance Z from the installation surface of the two cameras 11a and 11b is photographed with the mounting interval of the two CCD cameras 11a and 11b for long distances set to r. When doing, two cameras 11a, 11b
If the focal lengths of and are both f, the image of the point P appears on the projection plane that is away from the focal position by f for each camera.

At this time, the distance from the position of the image on the right CCD camera 11b to the position of the image on the left CCD camera 11a is r + δ, and if this δ is the shift amount, the distance Z to the point P shifts. It can be obtained from the amount δ by the following formula.

Z = r × f / δ (1) In order to detect the shift amount δ of the left and right images, it is necessary to find the image of the same object in the left and right images. In the present invention, the stereo image processing device described below is used. At 20, the image is divided into small areas, and the luminance or color patterns in each small area are compared in the left and right images to find a matching area, and the distance distribution is obtained over the entire screen. Therefore,
As in the prior art, it is possible to avoid a decrease in the amount of information by extracting some feature such as an edge, a line segment, and a special shape and finding a portion where those features match.

The degree of coincidence between the left and right images is i of the right and left images.
The luminance of the th pixel (color may be used) is A
If i and Bi are used, for example, the city block distance H shown in the following equation (2) can be used for evaluation, and there is no reduction in the amount of information due to the adoption of the average value, and there is no multiplication, and therefore the calculation speed is improved. You can

H = Σ | Ai−Bi | (2) If the size of the small area to be divided is too large, there is a high possibility that a distant object and a nearby object are mixed in the area. The detected distance becomes ambiguous. It is preferable that the area is small in order to obtain the distance distribution of the image, but if it is too small, the amount of information for checking the degree of coincidence is insufficient.

Therefore, for example, the number of pixels divided into four is set so that a vehicle with a width of 1.7 m located 100 m ahead is not included in the same region as a vehicle with an adjacent lane. Then, there are four pixels for the stereo optical system 10. As a result of trying an optimum number of pixels in an actual image with this value as a reference, the number of pixels is 4 in both length and width.

In the following description, it is assumed that the image is divided into small regions of 4 × 4 and the degree of coincidence between the left and right images is examined, and the stereo optical system 10 uses the CCD cameras 11a and 11b for long distance.
Shall be represented.

A concrete circuit example of the stereo image processing apparatus 20 is shown in FIG. 5. In this circuit example, the distance detection circuit 20a is used to convert an analog image picked up by the stereo optical system 10 into a digital image. A conversion unit 30, a city block distance calculation unit 40 that sequentially calculates the city block distance H for determining the displacement amount δ of the left and right images with respect to the image data from the image conversion unit 30 while shifting pixels one by one. Is the minimum / maximum value detection unit 50 that detects the minimum value HMIN and maximum value HMAX of the city block distance H, and whether the minimum value HMIN obtained by this minimum / maximum value detection unit 50 indicates a match between the left and right small areas? A deviation amount determination unit 60 that checks whether or not the deviation amount δ is determined is provided,
A dual port memory 90 is used as the distance image memory 20b.

In the image conversion section 30, C for left and right images is displayed.
A / D converters 31a and 31b are provided corresponding to the CD cameras 11a and 11b, respectively.
a and 31b, lookup tables (LUTs) 32a and 32b as data tables, the CCD camera 1
Image memory 33 for storing the images captured by 1a and 11b
a and 33b are connected to each other.

The A / D converters 31a and 31b have a resolution of, for example, 8 bits, and the left and right CCD cameras 11 are provided.
The analog image from a, 11b is converted into a digital image having a predetermined brightness gradation. That is, if the image is binarized in order to speed up the processing, the information for calculating the degree of coincidence between the left and right images is significantly lost.
It is converted into a gray scale of gradation.

Further, the LUTs 32a and 32b are constructed on the ROM, and the contrast of the low luminance part is increased or the left and right CCD cameras 11a and 11b are increased with respect to the image converted into the digital amount by the A / D converters 31a and 31b. Correct the difference in the characteristics of. Then, the LUTs 32a and 32b
The signals converted by the above are temporarily stored in the image memories 33a and 33b.
Memorized in.

As will be described later, the image memories 33a and 33b can be composed of relatively low speed memories because the city block distance calculation unit 40 repeatedly fetches and processes a part of the image, thus reducing the cost. Can be planned.

In the city block distance calculator 40,
In the image memory 33a for the left image of the image conversion unit 30,
Two sets of input buffer memories 41 via the common bus 80
a and 41b are connected, and two sets of input buffer memories 42a and 42b are connected to the image memory 33b for the right image via the common bus 80.

Each input buffer memory 41 for the left image
Two sets of shift registers 43a and 43b having, for example, eight stages are connected to a and 41b, and two sets of eight shift registers 43a and 43b are similarly provided to the input buffer memories 42a and 42b for the right image.
The shift registers 44a and 44b having a stage configuration are connected. Further, these four sets of shift registers 43a, 43
A city block distance calculation circuit 45 that calculates a city block distance is connected to b, 44a, and 44b.

The right image shift register 44 is also provided.
a and 44b have two sets of 1 of a shift amount determination unit 60 described later.
The shift registers 64a and 64b of 0-stage configuration are connected, and when the data transfer of the next small area starts, the old data for which the calculation of the city block distance H has finished is sent to these shift registers 64a and 64b, and the shift occurs. Used in determining the quantity δ.

Also, the city block distance calculation circuit 45
6 is a combination of an adder / subtractor and an input / output latch, which is combined with a high-speed CMOS type arithmetic unit 46 which is integrated into one chip.
As will be described in detail, a pipeline structure in which 16 calculators 46 are connected in a pyramid shape, for example, 8 pixels are simultaneously input and calculated. The first stage of the pyramid structure is an absolute value calculator, the second to fourth stages are first adder, second adder, and third adder, respectively, and the final stage is a sum adder. .

In FIG. 6, absolute value calculation and 1, 2
Only half of the adders in the first row are displayed.

Further, each of the input buffer memories 41a, 41a,
Reference numerals 41b, 42a, and 42b are high-speed types having a relatively small capacity according to the speed of the city block distance calculation, the input and output are separated, and generated by the # 1 address controller 86 according to the clock supplied from the clock generation circuit 85. The address to be given is commonly given. The transfer with the four sets of shift registers 43a, 43b, 44a, 44b is controlled by the # 2 address controller 87.

When the calculation of the city block distance H is performed by software of a computer, small areas of the left image are searched one after another for one small area of the right image, and this is searched for all the small areas of the right image. It is necessary to perform this calculation, and if this calculation is performed in, for example, 0.08 seconds, 500 MIPS (Mega
Instruction Per Second) ability is required. This is a number that cannot be realized by the current common SISC (CISC) type microprocessor, and the risk (RI
SC) processor, digital signal processor (DS)
P), or a parallel processor will have to be used.

The minimum / maximum value detecting section 50 includes a minimum value detecting circuit 51 for detecting the minimum value HMIN of the city block distance H and a maximum value detecting circuit 52 for detecting the maximum value HMAX of the city block distance H. In this configuration, two arithmetic units 46 used in the city block distance calculation circuit 45 are used for detecting the minimum value and the maximum value, and are synchronized with the output of the city block distance H. .

As shown in FIG. 7, the minimum value detection circuit 51
Are, specifically, A register 46a and B register 46
b, and an arithmetic unit 46 composed of an arithmetic logic operation unit (ALU) 46c, which is configured by connecting a C latch 53, a latch 54, and a D latch 55, and the output from the city block distance calculation circuit 45 is an A register 46a. And C latch 5
3 to the B register 46b and the ALU46
The most significant bit (MSB) of the output of is output to the latch 54. The output of the latch 54 is output to the B register 46b and the D latch 55, and the value in the middle of the minimum value calculation in the computing unit 46 is stored in the B register 46b.
The shift amount δ at that time is stored in the D latch 55.

Regarding the maximum value detection circuit 52, except that the logic is reversed and the deviation amount δ is not stored.
It has the same configuration as the minimum value detection circuit 51.

As described above, the city block distance H is
With respect to one right image small area, the left image small area is sequentially calculated by shifting the pixel by one pixel. Therefore, every time the value of the city block distance H is output, by comparing and updating the maximum value HMAX and the minimum value HMIN of the values so far, almost simultaneously with the output of the last city block distance H,
Maximum value HMA of city block distance H in the small area
X and the minimum value HMIN can be obtained.

The shift amount determining unit 60 is constructed as a RISC processor of a relatively small scale and has two 16-bit width data buses 62a and 62 centering on the arithmetic unit 61.
b, a latch 63a for holding the shift amount δ, a latch 63b for holding the threshold value Ha as a first specified value, a latch 63c for holding a threshold value Hb as a second specified value, a third specified value 63 for holding the threshold value Hc as
d, two sets of shift registers 64a and 64b for holding the luminance data of the right image, and the shift amount δ upon receiving the output of the arithmetic unit 61.
Alternatively, a switch circuit 65 that outputs "0", and an output buffer memory 66a that temporarily stores the output result,
66b, a ROM 67 of 16-bit width in which a control program of the operation timing of the circuit and the function of the arithmetic unit 61 is written
Is provided.

The arithmetic unit 61 has an ALU 70 as a center, and an A register 71, a B register 72, and an F register 7
3 and a selector 74, and the data bus 62
A (hereinafter, referred to as A bus 62a) is connected to the A register 71, and the data bus 62b (hereinafter, referred to as B bus 62b) is connected to the B register 72.
The switch circuit 65 is operated according to the calculation result of 70, and the shift amount δ or “0” is determined by the output buffer memories 66a and 66a.
6b is stored.

The A bus 62a has threshold values Ha,
Latches 63b, 63c, 63d for holding Hb, Hc,
The maximum value detection circuit 52 is connected to the B bus 62b.
The minimum value detection circuit 51 is connected to. Further, the shift registers 64a and 64b are connected to the A bus 62a and the B bus 62b.

Further, the switch circuit 65 is connected to the arithmetic unit 61 and the minimum value detection circuit 51 via the latch 63a, and the arithmetic unit 61 determines three check conditions described later. The output to the output buffer memories 66a and 66b is switched according to the determination result.

The shift amount determining unit 60 checks whether or not the obtained minimum value HMIN of the city block distances H really indicates a match between the left and right small areas, and only those satisfying the conditions are output buffer memory. 66a, 66b
The shift amount δ is output to the position of the corresponding pixel of.

That is, the shift amount that minimizes the city block distance H is the shift amount δ to be obtained. However, when the following three check conditions are satisfied, the shift amount δ is output, and when it is not satisfied, the shift amount δ is output. , Outputs "0" without adopting data.

(1) HMIN ≤ Ha (When HMIN> Ha, the distance cannot be detected.) (2) HMAX-HMIN ≥ Hb (The obtained minimum value HMIN is obviously lower than the fluctuation due to noise. It is a condition for checking, and by detecting the difference with the maximum value HMAX instead of the difference with the value near the minimum value HMIN, the distance can be detected even for objects such as curved surfaces where the brightness changes gently. (3) Brightness difference between adjacent pixels in the horizontal direction in the small area of the right image> Hc (when the threshold value Hc is increased, edge detection is performed, but when the brightness changes gently The threshold value Hc is set to be much lower than the normal edge detection level, which is based on the basic principle that distance detection cannot be performed in a portion where there is no brightness change. , For each pixel in the small area Since this is performed, only the pixels whose distances have been actually detected are adopted even in the small area, and a natural result is obtained.) Note that the processing for determining the amount of deviation is also software-like with an ordinary microprocessor. , It requires a speed of, for example, 27 MIPS, which is infeasible.

The distance distribution information, which is the final result output from the deviation amount determining unit 60, is written to the dual port memory 90 as the distance image memory 20b via the common bus 80.

The distance distribution information output from the stereo image processing device 20 described above is in the form of an image (distance image), and two CCD cameras 11a, 1 on the left and right sides are provided.
When an image captured by 1b, for example, an image as shown in FIG. 8 (FIG. 8 shows an image captured by one camera) is processed by the stereo image processing apparatus 20, an image as shown in FIG. 9 is obtained.

In the image example shown in FIG. 9, the image size is horizontal 4.
The number of pixels is 00 pixels × 200 pixels in the vertical direction, and the portion having the distance data is a black dot portion. This is a portion of each pixel of the image in FIG. 8 in which the brightness change between adjacent pixels in the left-right direction is large. As shown in FIG. 9, the coordinate system on the image has an upper left corner as an origin, a horizontal direction as an i coordinate axis, and a vertical direction as a j coordinate axis.
The unit is a pixel.

The distance image is read by the road / side wall detecting device 100, and the white line on the road and the side walls existing around the road are detected. In this case, the road / sidewall detection device 100 uses the three-dimensional position information of the object,
The road and the side wall are distinguished by the height from the road surface,
The side wall and the background are distinguished by the distance value.

Therefore, the road / side wall detecting device 100
Then, first, the coordinate system of the distance image from the stereo image processing device 20 is converted into the coordinate system of the real space surrounding the own vehicle (vehicle 1), and the position and size are detected with respect to the detected road shape and side wall. calculate.

That is, as shown in FIGS. 10 and 11, the coordinate system of the real space is a fixed coordinate system of the vehicle 1, the X axis is on the right side of the vehicle 1, the Y axis is above the vehicle 1, and the Z axis is. Vehicle 1
In front of, the origin is two CCD cameras 11a (12b),
Assuming the road surface directly below the center of 11b (12b), XZ
When the road is flat, the plane (Y = 0) coincides with the road surface, and the three-dimensional position (X, Y, Z) of the subject is calculated from the distance information (i, j, Z) in the image. To do so, a kind of coordinate conversion is performed by the following equations (3) and (4).

Y = CH-Z * PW * (j-JV) (3) X = r / 2 + Z * PW * (i-IV) (4) where: CH: Attachment of CCD camera 11 (CCD camera 12) Height, PW: viewing angle per pixel, JV, IV: coordinates on the image of the point at infinity in front of the vehicle 1.

Also, three-dimensional coordinates (X, Y, Z) in the real space
The equation for calculating the position (i, j) on the image from is also the following by modifying the equations (3) and (4).

J = (CH−Y) / (Z × PW) + JV (5) i = (X−r / 2) / (Z × PW) + IV (6) In addition, the mounting position of the CCD camera 11 is as described above. In the XYZ coordinate system of the real space, for example, the CCD camera 1 on the right side
1b has X = 0.45 m, Y = 1.24 m, Z = 0.0
m, and the CCD camera 11a on the left side has X = −0.4.
5m, Y = 1.24m, Z = 0.0m.

As shown in FIG. 12, the recognition function of the road / sidewall detecting device 100 is roughly divided into a road detecting unit 110 and a sidewall detecting unit 120, and the processing result is stored in the road / sidewall parameter storage unit 130. Is stored in the output memory 100e of FIG.

The road detection unit 110 functions as road detection means for determining a road model based on the distance distribution information (three-dimensional position information) included in the distance image and detecting the road model as a road shape. And has a road shape estimation unit 111, a three-dimensional window generation unit 112, a linear element detection unit 113, and a road shape determination unit 114. The side wall detection unit 120 includes a three-dimensional object data extraction unit 121, a side wall straight line detection unit 122, and a side wall range detection unit 123.

The road shape estimating unit 111 has a function as a road shape estimating unit for estimating the position and shape of the white line of the road based on the information of the distance distribution included in the distance image, and the three-dimensional window generating unit. Reference numeral 112 has a function as a three-dimensional window setting means for setting a three-dimensional space area that includes the estimated white line of the road as a three-dimensional window.

Further, the linear element detecting unit 113 extracts only the data in the three-dimensional window from the information on the distance distribution, and detects the three-dimensional linear elements constituting the road model. The road shape determining unit 114 determines the adequacy of the detected straight line element, and corrects or changes the straight line element if the straight line element does not meet the determination criterion, and determines the road model. It has a function as a determination means.

Further, the three-dimensional object data extracting unit 121 has a function as a data extracting means for extracting only the data above the road surface from the information of the distance distribution based on the determined road model, The sidewall straight line detection unit 122
From the extracted three-dimensional object data, only the data within the preset side wall search area is extracted, and this is processed by Hough transform to detect the presence or absence of the side wall and the linear expression indicating the position of the side wall. It has a function as a detection means.

Further, the side wall range detection unit 123 sets a side wall candidate area in which it is estimated that the side wall exists based on the linear expression indicating the position of the side wall, and based on the distribution state of the three-dimensional object data in this side wall candidate area. , And has a function as a side wall range detecting means for detecting the positions of the front and rear ends of the side wall.

The road detection unit 110 utilizes the three-dimensional position information based on the distance image stored in the distance image memory 20b, separates and extracts only the white line on the actual road, and stores the built-in road model. The road shape is recognized by modifying / changing the parameters so that they match the actual road shape.

In the actual image, the white line on the road is overlapped by the preceding vehicle and the like, but in many conventional devices that detect the white line of the road reflected in the image by relying on the two-dimensional characteristics, the white line and the stereo It is often difficult to separate an object from a two-dimensional feature, but in the present invention, the white line and a three-dimensional object can be reliably separated by using the three-dimensional position information of the white line. .

That is, in the three-dimensional space, the white line is on the plane of the road, while the three-dimensional object such as the preceding vehicle is located higher than the plane of the road. Therefore, the white line and the three-dimensional object are distinguished by the height from the road surface.

Further, the road detection unit 110 has a built-in road model, and this road model divides the own lane of the road to the recognition target range into a plurality of sections according to the set distance, and for each section. The left and right white lines are connected in a polygonal line shape by approximating them by a three-dimensional linear equation described later, and the range enclosed by the left and right polygonal lines is determined to be the driving lane.
It can be said that the recognition of the road shape is a process of deriving the three-dimensional linear equation parameters.

FIG. 13 is an example of a road model. For example,
0m section R0, 1st section R1,
The left curve is approximated by being divided into seven sections of the second section R2, ..., And the sixth section R6. In this road model, 7
By approximating roads with individual sections, not only straight roads but also curves and S-shaped roads can be expressed with sufficient accuracy.
Since each section is represented by a straight line, calculation processing and handling are easy. Further, as will be described later, each section is represented by a linear expression in the horizontal direction and the vertical direction, and the shape in the up-down direction of the road such as up and down and unevenness of the road can also be expressed.

The value of the distance separating each section of the road model needs to be changed according to the curvature of the curve of the road on which the vehicle travels. In general highways, the radius of the curve is designed to be a minimum of about 230m, so in such a case,
The division distance of each section is 10m, 17m, 25m, 35
Good results are obtained with m, 48 m, 64 m and 84 m.

Next, the function of the road detector 110 will be described in detail. In the road shape estimation unit 111, the last time (Δtsec
Based on the result of the road shape recognition described above, the movement of the vehicle 1 for Δt seconds is calculated using the output signals from the vehicle speed sensor 3 and the steering angle sensor 4, and the road viewed from the position of the vehicle 1 after Δt seconds. Estimate the shape.

That is, the output signal of the vehicle speed sensor 3 is set to V
(M / sec), assuming that the output signal (steering angle) of the steering angle sensor 4 attached to the steering column is η (rad), the amount of forward movement ΔZ (m) and the rotation angle (yaw angle) of the vehicle 1 for Δt seconds. Δθ (rad) can be generally estimated by the following equation.

ΔZ = V × Δt (7) Δθ = ΔZ × tan (η / rs) × 1 / wb (8) Where, rs is the rotation ratio between the steering wheel and the front wheels, and wb is the wheel base of the vehicle.

Therefore, the road shape detected in the previous process is moved forward by ΔZ, and further the road shape is rotated in the opposite direction to the rotation of the vehicle 1 by Δθ, whereby the approximate position of the road after Δt seconds is reached. The shape can be estimated.

In the three-dimensional window generating unit 112, a rectangular parallelepiped three-dimensional space area, that is, a three-dimensional window as shown in FIG. 14 is centered on one straight line element Ld of the left and right broken lines representing the estimated road shape RD. WD3A is set, and how the set three-dimensional window WD3A looks on a two-dimensional image is calculated as shown in FIG. 15, and the inside of the window contour line (hatched portion in FIG. 15) is calculated.
Is a two-dimensional window WD2A, and only the data in this is the detection target.

To obtain the two-dimensional window WD2A from the three-dimensional window WD3A, 8 of the three-dimensional window WD3A is used.
From the coordinates (Xn, Yn, Zn) of the respective vertices, the coordinates (in, jn) on the image are calculated by using the formulas (5) and (6) described above, and the polygon enclosing these points is calculated. To do.

This three-dimensional window WD3A has a length that divides each section (for example, the front 10 in the first section R1).
-17 m), while the height and width are changed according to the situation such as the vehicle speed, but if there is an error in the estimation of the road shape and a deviation from the actual white line position is expected,
Increase the height and width to widen the detection range. However, if the window is made too large, curbs and vegetation around the road will also be detected, which may cause misrecognition. Therefore, it is important to properly select the size of the window.
When driving on general highways, the test results show that the height is 0.4 m.
It has been found that it is preferable to change the width within the range of 0.8 m and the width of 0.4 m to 1.6 m.

As described above, even if the white line of the road and the three-dimensional object are overlapped on the two-dimensional image, the three-dimensional window is set to extract only the data near the surface of the road so that the white line becomes the three-dimensional object. It can be detected separately. Although there are curbs and vegetation around the road, white lines on the road can be removed by setting up a three-dimensional window and extracting only the data near the position where the white line is estimated. Etc. can be detected separately. Furthermore, by setting a two-dimensional window, it is possible to shorten the processing time by reducing the area to be searched and the number of data.

The linear element detection unit 113 calculates the deviation amount ΔX in the X direction and the deviation amount ΔY in the Y direction of the three-dimensional position of the subject with respect to the previously estimated linear element Ld of the road shape, and the deviation amount ΔX. , ΔY, each data is multiplied by a weighting coefficient, and linear equations in the horizontal direction (XZ direction) and the vertical direction (YZ) direction are derived by the least square method to obtain the parameters.

Specifically, first, the two-dimensional window WD2A
The pixels in the area are sequentially surveyed, and for the pixels having distance data, the three-dimensional position (X, Y, Z) of the object is calculated using the above equations (3) and (4), and the distance Z Is a range of the length of the three-dimensional window WD3A (for example, the first
Distance data outside Z = 10 to 17 m in the section R1 is excluded from the detection target.

That is, the image of the object on the other side or the front side of the three-dimensional window WD3A is the two-dimensional window W.
Since the image is reflected in D2A, the subject surveyed in the two-dimensional window WD2A is not always included in the three-dimensional window WD3A. Therefore, the three-dimensional position (X, Y, Z) of the subject of each pixel is calculated, and the three-dimensional window WD3A
It is determined whether or not it is included in.

Then, the previously estimated linear element Ld of the road shape is compared with the three-dimensional position of the object to calculate the shift amounts ΔXi and ΔYi of the data Di in the X and Y directions as shown in FIG. After selecting only the data within the width and height of the three-dimensional window WD3A, the weighting coefficient of the data Di corresponding to the deviation amounts ΔXi and ΔYi in the X and Y directions is determined.

As shown in FIG. 17, the weighting factor is, for example, parabolic with the center being 1.0 and the periphery being 0.0,
The product of the weighting factor fx in the X direction and the weighting factor fy in the Y direction is
The data Di is used as a weighting coefficient. The range in the X and Y directions where the weighting factor is 0.0 or more is the same as the width and height of the three-dimensional window WD3A, or
Make it larger than these.

After multiplying each data Di by the weighting coefficient, the following (9) and (10) are obtained by using the least squares method.
The linear equations in the horizontal and vertical directions shown in the equation are derived, the parameters a, b, c, d are obtained, and the new linear elements L are obtained.
It is a candidate for d.

Horizontal direction: X = a × Z + b (9) Vertical direction: Y = c × Z + d (10) At the same time, the weighting factor is a set value (for example, 0.05 to 0.1).
For the above data, the number and the range of the distance Z over which the data are distributed are examined. When the number of data is less than a set value (for example, about 10), or the range of the distance Z is the three-dimensional window WD3A. Is less than 1/2 of the length (for example, the length 7 m of Z = 10 m to 17 m in the first section R1), it is determined that an accurate candidate for the linear element Ld has not been obtained, and the above is calculated as described above. The straight line formula is rejected and there is no candidate.

The above processing is sequentially performed from the left and right sides and from the front side to the distant side section to obtain candidates for all the linear elements Ld constituting the road model. In this case, if the setting of the width of the three-dimensional window is too large, curbstones or vegetation around the road may hang on the edges of the three-dimensional window. Multiplying the weight of the peripheral part of the three-dimensional window to reduce the influence of curbs and vegetation, and to derive a stable straight line straight line formula. is there.

The road shape determination unit 114 determines the validity of each of the left and right linear element Ld candidates for each section from the parallelism in the horizontal and vertical directions. If the result of the determination is that it is appropriate, both are adopted as candidates for the new linear element Ld, while if it is determined that the candidate for either the left or right linear element Ld is not correct, the linear element Ld Substitute and complement. Then, the obtained parameters of each linear element Ld are output to the road / sidewall parameter storage unit 130.

Specifically, first, the parameters of the expression (9) for the left-side linear element Ld (hereinafter, L indicating the left side and R indicating the right side are added to each parameter) aL and the right-side linear element The parallelism in the horizontal direction is checked from the difference from the parameter aR of the equation (9) with respect to Ld, and when the value is equal to or more than the set value (for example, about 5 °), the linear element Ld on either the left or right side is inaccurate. judge. Similarly, the parallelism in the vertical direction is checked from the difference between the parameter cL and the parameter cR, and if it is equal to or larger than the set value (for example, about 1 °), it is determined that any of the linear elements is inaccurate.

If the result of this determination is that both the horizontal and vertical parallelisms satisfy the conditions, both are adopted as new linear elements, but either the left or right linear element L
When d is determined to be inaccurate, the left and right linear elements Ld
And the previously estimated position of the road shape are compared, and the one with the smaller deviation is adopted as the new linear element Ld, and the other is rejected and there is no candidate.

If the straight line element Ld on either the left or right is determined as no candidate by the parallelism determination, or the white line on the road has a broken line shape or is hidden by an obstacle and cannot be seen, data is insufficient. And then either the left or right linear element Ld
If it is determined that there is no candidate, the linear element Ld on the detected side is moved in parallel by the width of the lane to substitute it. Furthermore, when there is no candidate for both the left and right linear elements Ld, the previously estimated linear element Ld of the road shape is substituted. As a result, a stable road shape can be obtained as a whole even if a detection failure or an erroneous detection of a linear element occurs partially.

FIG. 18 is an explanatory diagram showing the road shape detected by the road detection unit 110 in a schematic form, in which straight line elements are detected along the left and right white lines. The horizontal line between the left and right linear elements is the boundary of each section.

Next, the side wall detector 120 will be described. In the side wall detection unit 120, the side wall is distinguished from the road by the height from the road surface, and the side wall and the distant background are distinguished by the front-rear direction and the lateral direction. Only the peripheral data to be extracted are extracted, and then the side wall data is detected by the Hough transform by paying attention to the features that are linearly aligned in the horizontal direction, and the position is obtained.

In detail, since the side wall is a kind of three-dimensional object, first, the three-dimensional object data extraction unit 121 reads the parameters of the detection result of the road detection unit 110 described above and sets the height H of the road surface. Data of a three-dimensional object located above the height H of the road surface is extracted from the distance image. In addition, on a road without a white line, the height of the road surface is set on the assumption that the road surface is horizontal to the vehicle 1.

The object in the distance image is obtained from the coordinates (i, j) on the image and the distance data Z from the above (3) and (4).
The three-dimensional position (X, Y, Z) in the real space is calculated using the formula, and the height Yr of the road surface at the distance Z is calculated using the formula (10) of the road shape detected previously. Calculated.
The height H of the subject from the road surface can be calculated by the following equation (11).

H = Y−Yr (11) Since a subject whose height H is about 0.1 m or less is considered to be a white line, dirt, shadow, etc. on the road, the data of this subject is rejected. Also, subjects above the height of the vehicle are considered to be pedestrian bridges, signs, etc., and are therefore rejected, and only the data of three-dimensional objects on the road are selected.

The three-dimensional object data extraction unit 121 extracts a wide range of three-dimensional object data shown on the screen. Since it is not rational to process all of these, the side wall straight line detection unit 122 limits the region for searching the side wall.

In this case, when the range in which the distance image is measured is viewed from above, the field of view of the CCD camera 11 is limited and the range of FIG.
When the vehicle is normally traveling on the road, the side walls are present on the left side and the right side of the vehicle 1 and substantially parallel to the vehicle 1. On the other hand, the far side wall is difficult to detect from the viewpoint of accuracy of distance data, and the necessity of detection is small.
Therefore, in consideration of these, two left and right search regions SL and SR as shown in the figure are set, and the left and right side walls are detected separately.

That is, in order to extract the three-dimensional object data included in each of the search areas SL and SR, the three-dimensional position (X, Z coordinates) of the subject of each data extracted by the three-dimensional object data extracting unit 121 is calculated. Then, the three-dimensional position (X, Z) and the respective search areas SL, SR are compared and determined.

For example, in the situation as shown in FIG. 20, the search areas SL and SR are shown on the image as a broken line frame, and the pedestrians other than the desired side wall are also included in these search areas. There are various three-dimensional objects such as electric poles. further,
The range image also contains noise-like fake data, and only the data exists in a space where no object actually exists. FIG. 21 schematically shows these data, and the side wall is characterized in that the data is linearly arranged. Therefore, the side wall is detected by using the Hough transform to detect the linear expression of the data column.

The detection of the linear equation by the Hough transform will be described. First, the three-dimensional object data Pi (coordinate X
For i, Zi), assume a straight line Fi passing through the point of this data Pi. The equation of this straight line is shown by the following equation (12).

X = afi × Z + bfi (12) Next, as shown in FIG. 23, the vertical axis indicates the slope a of the equation (12).
f, the horizontal axis is the parameter space of the intercept bf, and the equation (1
Votes are made at positions corresponding to the parameters afi and bfi in 2).

Here, the side wall of the slope afi is considered to be substantially parallel to the vehicle 1 as described above. Therefore, on a highway, for example, about ± 10 ° (afi: ± 0.18), and on a general road, For example, it is practically sufficient to change it within a range of about ± 20 ° (afi: ± 0.36). Also, the intercept b
The value of fi is, for example, in the range of X = -1 m to -10 m, which is the left side of the vehicle 1 when detecting the left side wall, and when detecting the right side wall, for example, X = + 1 m to +1.
Limit to a range of about 0 m. Thus, the reason why the limit range is set to, for example, about ± 10 m is that there is little practical need to detect a side wall that is too far away.

Due to such restrictions, the range of voting in the parameter space is a rectangular area as shown in FIG. 23, and this rectangular area is further divided into grids and votes are made for each grid. The slope afi of the equation (12) is within a predetermined change range (for example, ± 10 ° to ± 20 °), and is set by sequentially changing it for each lattice interval Δaf. Intercept bfi
Is the set inclination afi and the coordinates (X
i, Zi) is substituted into the equation (12), and if this is within the limit range, the corresponding lattice in the parameter space is voted.

The position of the side wall to be detected, that is, the accuracy of detecting the slope and intercept of the linear equation is determined by the lattice spacings Δaf and Δbf, and the lattice spacings Δaf and Δbf are set by the external device utilizing the side wall information. It is done on demand. For example, when it is used to detect a danger such as a collision when traveling on a road normally, the lattice spacing Δaf is 1 to 2
And the lattice spacing Δbf is preferably about 0.3 to 0.6 m.

As described above, when voting is performed on the parameter space for all the three-dimensional object data in the search area, as shown in FIG.
As shown in 2, if there are linearly arranged data,
The lattice of the parameter space corresponding to the linear parameters afi and bfi set so as to match the column of this data obtains many votes, and the local maximum value appears in each of the left and right voting regions SL and SR. FIG. 24 shows a result of processing the example of the three-dimensional object data shown in FIG. 22 and voting in the parameter space,
In this example, the local maximum is 43.

If there is a side wall and there is a clear row of three-dimensional object data, the local maximum of the parameter space shows a large value,
On the other hand, the local maximum shows a small value in the state where there are no side walls and a plurality of objects are dispersed. Therefore, the local maximum value is detected for each of the left and right voting regions SL and SR of the parameter space, and if the detected local maximum value is equal to or larger than the determination value, it can be determined that the sidewall exists. The judgment value is set in consideration of the size of the search area to be set, the interval of the grid, and the like.

Next, when the side wall straight line detecting section 122 determines that there is a side wall, the side wall range detecting section 123 detects the positions of the front and rear ends of the side wall. When the parameters af and bf corresponding to the grid of local maximum values are read, it is estimated that the side wall exists along the following linear equation (13), and the linear equation detected in the example of FIGS. 22 and 24 is illustrated. A straight line Ff shown in FIG. 25 is obtained.

X = af × Z + bf (13) First, assuming that a region having a width of about 0.3 m to 1.0 m centering on the straight line Ff is a sidewall candidate region Tf, this region is further arranged in the Z direction as shown in FIG. Be divided. Sidewall candidate region T
The width of f is set in consideration of a data error or the like in the interval Δbf of the grid of the parameter space.

Next, the three-dimensional object data in the search area are sequentially surveyed, only the data in the side wall candidate area Tf are extracted, and then the number of three-dimensional object data is counted for each section to create a histogram. This is schematically shown in FIG. 26, and shows a large frequency in the portion where the side wall exists. Therefore, it is possible to determine that the side wall exists in this range by detecting the section whose frequency is equal to or higher than the determination value, and the three-dimensional positions of both ends of the side wall are calculated and used as the front and rear end positions of the side wall.

To detect the left and right side walls, first, the side wall straight line detection unit 122 sets the left side search area SL to perform the side wall straight line detection processing and the side wall range detection processing to detect the left side wall. The side wall straight line detection unit 122 sets the right side search region SR again, and the same process is repeated to detect the right side wall.

The parameters such as the shape of the road, the presence / absence of the side wall, and the position obtained as described above are output to and stored in the road / side wall parameter storage unit 130.

Next, the calculation of distance information by the stereo image processing apparatus 20 and the operation of the road / sidewall detecting apparatus 100 will be described.

First, in the stereo image processing apparatus 20, when the images picked up by the left and right CCD cameras 11a and 11b are input in step S101 of the program shown in FIG. 27, the input image is A / D converted in step S102. , LUTs 32a, 32b, and image memory 33
It is stored in a and 33b.

The images stored in these image memories 33a and 33b are only the lines necessary for subsequent processing out of all lines of the CCD elements of the CCD cameras 11a and 11b, for example, once every 0.1 seconds. Percentage (3 on TV image
It is rewritten at a ratio of 1 to 1.

Next, in step S103, the image memories 33a and 33b for the left and right images are input to the input buffer memory 4
1a, 41b, 42a, 42b are read via the common bus 80, for example, left and right image data by 4 lines,
The read left and right images are matched, that is, the matching degree is evaluated.

At this time, for each of the left and right images, from the image memories 33a and 33b to the input buffer memories 41a and 4b.
The read operation to 1b, 42a and 42b and the write operation to shift registers 43a, 43b, 44a and 44b are alternately performed. For example, in the left image, while the image data is being read from the image memory 33a to the one input buffer memory 41a, the image data read from the other input buffer memory 41b to the shift register 43b is written, and the right image is written. Then, while the image data is being read from the image memory 33b to the one input buffer memory 42a, the image data read from the other input buffer memory 42b to the shift register 44b is written.

Then, as shown in FIG. 28, the shift register 43a, 43b, 44a, 44b has image data (1, 1) ... (4) in the left and right small regions of 4 × 4 pixels.
4) is stored, and one shift register 43a (44
a) has 1 or 2 lines of data, and the other shift register 43b (44b) has 3 or 4 lines of data.
Each pixel has an odd line and an even line.

Each of the shift registers 43a, 43b, 4
Each of 4a and 44b has an independent transfer line, and 4 × 4 pixel data is transferred at, for example, 8 clocks. Then, these shift registers 43a, 43b,
44a and 44b simultaneously output the contents of even-numbered stages of eight stages to the city block distance calculation circuit 45, and when the calculation of the city block distance H starts, the data of the right image is held in the shift registers 44a and 44b. The data of the odd line and the even line are alternately output every clock, while the data of the left image is continuously transferred to the shift registers 43a and 43b, and the data of the odd line and the even line are alternately output and 2 It is replaced by data that is shifted to the right by one pixel every clock. This operation is
Repeat until there is a shift of 00 pixels (200 clocks).

After that, when the transfer to one small area is completed, the contents of the right image address counter (the leading address of the next 4 × 4 pixel small area) are stored in the left image address counter in the # 2 address controller 87. Set,
Processing of the next small area begins. In the city block distance calculation circuit 45, as shown in the timing chart of FIG. 29, first, eight pixels of data are input to the absolute value calculator at the first stage of the pyramid structure, and the absolute value of the brightness difference between the left and right images is calculated. . That is, if the brightness of the corresponding left pixel is subtracted from the brightness of the right pixel, and if the result is negative, the subtraction and subtraction are reversed by changing the operation instruction, and subtraction is performed again. Calculate the value. Therefore, the subtraction may be performed twice in the first stage.

Next, when passing through the first stage, 4 from the second stage
The first to third adders up to the second stage add two simultaneous input data and output. Then, in the final-stage total sum adder, two consecutive data are added together to calculate the total sum, and the required city block distance H for 16 pixels is output to the minimum / maximum value detection unit 50 every two clocks.

Then, the process proceeds to step S104, and the step
The maximum value HMAX of the city block distance H calculated in S103,
Detect the minimum value HMIN. As described above, the detection of the maximum value HMAX and the detection of the minimum value HMIN are exactly the same except that the logics are opposite to each other and the deviation amount is not stored, and therefore, the minimum value will be representatively described below. The detection of the value HMIN will be described.

First, the initially output city block distance H (deviation amount δ = 0) is input to the B register 46b of the arithmetic unit 46 via the C latch 53 of the minimum value detection circuit 51 shown in FIG. To be done. The city block distance H (deviation amount δ = 1) output at the next clock is the C latch 5
3 and the A register 46a of the arithmetic unit 46, and the arithmetic unit 46 simultaneously starts the comparison operation with the B register 46b.

If the contents of the A register 46a are smaller than the contents of the B register 46b as a result of the comparison operation in the arithmetic unit 46, the contents of the C latch 53 (that is, the contents of the A register 46a at the next clock). (Contents) is sent to the B register 46b, and the shift amount δ at this time is stored in the D latch 55. At the same time with this clock, the next city block distance H (deviation amount δ = 2) is entered into the A register 46a and the C latch 53, and the comparison operation is started again.

In this way, the minimum value during the calculation is always stored in the B register 46b, and the shift amount δ at that time is stored in the D latch 55, and the calculation is continued until the shift amount δ reaches 100. When the calculation is completed (1 clock after the last city block distance H is output), the contents of the B register 46b and the D latch 55 are read into the deviation amount determination unit 60.

In the meantime, the city block distance calculation circuit 45 described above reads the initial value of the next small area so as not to waste time, and to calculate one city block distance H, For example, it takes 4 clocks, but since it has a pipeline structure, a new calculation result can be obtained every 2 clocks.

In step S105, when the minimum value HMIN and maximum value HMAX of the city block distance H are determined in step 104, the deviation amount determining unit 60 checks the above-mentioned three conditions to determine the deviation amount δ. It

That is, as shown in the timing chart of FIG. 30, the minimum value HMIN is latched in the B register 72 of the arithmetic unit 61 via the B bus 62b and is compared with the value of the B register 72. The value Ha is latched in the A register 71 via the A bus 62a. Then, the two are compared by the ALU 70, and if the minimum value HMIN is larger than the threshold value Ha, the switch circuit 65 is reset and 0 is always output regardless of the subsequent checks.

Next, the maximum value HMAX is latched in the A register 71, and the maximum value H latched in this A register 71.
The difference between MAX and the minimum value HMIN stored in the B register 72 is calculated, and the result is output to the F register 73. At the next clock, the threshold value Hb is latched in the A register 71 and compared with the value in the F register 73. If the content of the F register 73 is smaller than the threshold value Hb latched in the A register 71, the switch circuit 65 is similarly reset.

From the next clock, the calculation of the brightness difference between adjacent pixels starts. The two sets of shift registers 64a and 64b in which the luminance data is stored have a 10-stage configuration, and the shift register 44a for the 1st and 2nd lines and the shift register for the 3rd and 4th lines of the city block distance calculation unit 40, respectively. It is connected to the subsequent stage of 44b. The outputs of the shift registers 64a and 64b are taken out from the last stage and the stage immediately before the last stage, and are output to the A bus 62a and the B bus 62b, respectively.

When the calculation of the brightness difference is started, the brightness data of each place in the small area is held in each stage of the shift registers 64a and 64b, and the brightness data of the fourth area of the previous small area is initially stored.
The brightness data of the first row and the first column and the brightness data of the first row and the first column of the small area this time are latched in the A register 71 and the B register 72 of the arithmetic unit 61.

Then, the absolute value of the difference between the contents of the A register 71 and the contents of the B register 72 is calculated, and the result is stored in the F register 73. At the next clock, the threshold value Hc is latched in the A register 71 and compared with the value in the F register 73.

If the content of the F register 73 (absolute value of brightness difference) is larger than the content of the A register (threshold value Hc) as a result of the comparison in the arithmetic unit 61, the shift amount δ from the switch circuit 65 or "0" is output, A
If the content of the F register 73 is smaller than the content of the register, "0" is output and the output buffer memory 66
It is written in a position corresponding to the first row, first column of the corresponding small area of a, 66b.

While the arithmetic unit 61 compares the luminance difference between adjacent pixels and the threshold value Hc, the shift registers 64a and 64b shift one stage. And this time,
The calculation is started for the luminance data of the fourth row, second column of the previous small area and the first row, second column of the current small area. In this way, the calculation is alternately performed on the first column and the second column of the small region, and then the calculation is similarly performed on the third column and the fourth column.

During the calculation, the shift registers 64a, 64b
The last stage and the first stage of are connected to form a ring register, and after calculating the entire small area, the shift clock becomes 2
Once added, the register contents return to the state before the calculation,
When the luminance data of the next small area has been transferred, the data of the fourth row of the current small area is retained in the final stage and the preceding stage.

As described above, since the next data is prepared in the A bus 62a and the B bus 62b or the result is written during the calculation for determining the shift amount, one data is required only in two clocks required for the calculation. Is processed. As a result, even if the initial minimum value HMIN and maximum value HMAX checks are included, all calculations are completed in, for example, 43 clocks, and the minimum value HMI of the city block distance H is set for one small region.
The time required to obtain N and the maximum value HMAX is sufficiently long, and it is possible to add more functions.

When the shift amount δ is determined, the shift amount δ is output as distance distribution information from the output buffer memories 66a and 66b to the dual port memory 90 in step S106, and the processing in the stereo image processing apparatus 20 is performed. finish.

The output buffer memories 66a and 66b
Is the input buffer memories 41a, 41b, 43 described above.
As with a and 43b, for example, there is a capacity of 4 lines,
While writing to one of the sets, the other sends the distance distribution information to the dual port memory 90.

From the distance distribution information written in the dual port memory 90, the three-dimensional position in the XYZ space of the object corresponding to each pixel is determined from the mounting positions of the CCD cameras 11 and 12 and the lens parameters such as the focal length. It can be calculated, and the distance to the object outside the vehicle can be accurately detected without a decrease in the amount of information.

Next, the timing of the stereo image processing apparatus 20 will be described with reference to the timing chart shown in FIG.

First, the left and right CCs which are synchronized
The field signals from the D cameras 11a and 11b are set to 0.1
Every second (one screen out of three screens), the image memory 33a,
Write to 33b.

Next, in response to the capture end signal, block transfer for every four lines starts. In this transfer, three blocks are transferred in the order of the right image, the left image, and the resulting distance distribution image.

During this time, the shift amount δ is calculated for one of the input / output buffer memories. Then, in consideration of the calculation time of the shift amount δ, the transfer is started to the other input / output buffer memory after waiting for a predetermined time.

The calculation of the city block distance H for a small region of 4 × 4 pixels in one right image is 10 for the left image.
The calculation is performed 100 times because the calculation is performed while shifting by 0 pixel.
While the city block distance H of one area is calculated, the shift amount δ of the area before that is output as a distance distribution through each check.

When the number of lines to be processed is 200, the process for 4 lines is repeated 50 times, the processing time for 4 lines for transferring the first data at the start of the calculation, and the final result after the calculation is completed. Processing time for transferring 4 lines to the image recognition unit and a processing time for 8 lines in total are further required.

The time from the start of the transfer of the first input image line to the end of the transfer of the final distance distribution is 0.076 seconds as a result of the actual circuit operation.

Next, regarding the operation of the road / sidewall detecting device 100, the operation of the road detecting unit 110 will be described according to the flowcharts of FIGS. 32 to 35, and the operation of the sidewall detecting unit 120 will be described according to the flowcharts of FIGS. 36 to 38. To do.

The road detection unit 110 first performs road shape estimation processing. That is, when the previous road shape parameter (before Δtsec) is read in step S201 of the program shown in FIG. 32, the process proceeds to step S202, in which the output signal V of the vehicle speed sensor 3 and the output signal η of the steering angle sensor 4 are read. .

Next, in step S203, the movement of the vehicle 1 for Δt seconds is calculated using the output signal of the vehicle speed sensor 3 and the output signal η of the steering angle sensor 4 read in step S202, and in step S204. , Δt seconds later, the road shape viewed from the position of the vehicle 1 is estimated and the road shape parameter is corrected.

When the above road shape estimation processing is completed, the process proceeds to the three-dimensional window generation processing, and in step S205, the parameters (a, b, c, d) of the straight line element Ld on the left side of the first section R1 of the road model. Is read, in step S206,
Three-dimensional window WD3A centered on this linear element Ld
To set.

After that, the process proceeds to step S207, the two-dimensional window WD2A on the two-dimensional image is set from the three-dimensional window WD3A set in step S206, and the process proceeds to the next step S208 and subsequent steps.

Steps S208 to S217 are straight line element detection processing. In step S208, the two-dimensional window W
When the data in D2A is read, the three-dimensional position of each data is calculated in step S209, and in step S210, the data whose distance Z is within the length of the three-dimensional window WD3A is selected.

Then, the process proceeds to step S211, and the previously estimated road-shaped linear element Ld is compared with the three-dimensional position of the object to calculate the positional deviation amounts ΔX and ΔY in the X and Y directions, and in step S212. , Only the data whose deviation amounts ΔX and ΔY are within the width and height of the three-dimensional window WD3A are selected, and the others are excluded.

After that, the process proceeds to step S213, the weighting coefficient of the data is determined according to the deviation amounts ΔX and ΔY in the X and Y directions calculated in step S212, and
Weighting factors corresponding to the deviation amounts ΔX and ΔY are added.

Next, in step S214, linear equations in the horizontal direction (XZ plane) and the vertical direction (YZ plane) are derived using the least squares method, and parameters (a, b, c, d) are obtained.
Is set as a candidate for a new linear element Ld.

Then, in step S215, it is checked whether or not a candidate for the straight line element Ld of the right side line of the road model is obtained. If the result is NO, in step S216, the parameter of the right straight line element Ld is determined. And returns to step S206 described above, and if the result is YES, step S2
Proceed to 17.

In step S217, it is checked whether or not the obtained candidate of the linear element Ld is on the right side of the final section. If it is not the final section, in step S218, the linear element Ld on the left side of the next section is selected. The parameters are read, the process returns to step S206 described above, and the same processing is repeated.

On the other hand, if it is determined in step S217 that the candidates for the straight line element Ld are on the right side of the final section, and the candidates for all the straight line elements Ld forming the road model have been obtained, the above step S217 is performed. From step S219 onward, road shape determination processing is executed.

That is, in step S219, the first section R1
When the parameter of the linear element Ld of is read, step S2
In step 20, the horizontal parallelism of the left and right linear elements Ld is checked to determine its validity, and in step S221, the vertical parallelism of the left and right linear elements Ld is checked to determine its validity.

After that, the process proceeds to step S222, and if the result of the determination in steps S220 and S221 is that the left or right straight line element is not valid, or the white line on the road is broken or obstructed. When there is no candidate for a straight line element on either side because there is no data because it is hidden and cannot be seen, a straight line that is missing by moving the straight line element on the detected side in parallel by the width of the lane The elements are complemented, and the process proceeds to step S223.

When there are no left and right linear elements, the previously estimated linear element of the road shape is used instead.

In step S223, it is checked whether or not it is the final section, and if it is not the final section, in step S224, the parameters of the left and right linear elements Ld of the next section are read, and the process returns to step S220 to return to the final section. If you step
From S223 to S225, the parameters of each linear element Ld are written in the output memory 100e and the program is terminated.

Next, the operation of the side wall detector 120 will be described. Steps S301 to S3 in the program of FIG. 36
07 is a three-dimensional object data extraction process. First, step S3
When the road shape parameter is read in 01, step S302
Then, the first distance data is read from the distance image.

Next, in step S303, the position (X, Z coordinates) and height (Y coordinate) of the subject are calculated, and step
In step S304, the height H (Y coordinate) of the road surface at the distance Z is calculated, and in step S305, it is above the road surface and the vehicle 1
Data below the height of is extracted as three-dimensional object data.

Then, it proceeds to step S306 to check whether or not it is the final data, and if it is not the final data, the next distance data is read in step S307, the process returns to the above-mentioned step S303 and the processing is repeated. Step S306
To Step S308.

Steps S308 to S317 of FIG. 37 are
This is the side wall straight line detection processing, and when the first three-dimensional object data is read in step S308, the position (X, Z coordinates) of the subject is calculated in step S309, and the calculated position (X, Z coordinates) in step S310. Check whether is in the search area.

When the calculated position (X, Z coordinates) is outside the search area, the process jumps from step S310 to step S312. When it is within the search area, the process proceeds from step S310 to step S311 to vote in the parameter space. , Go to step S312.

In step S312, it is checked whether the processed three-dimensional object data is the final data. If it is not the final data, the next three-dimensional object data is read in step S313.
The process from step S309 described above is repeated, and when the data is the final data, the process proceeds to step S314 to detect the local maximum value in the parameter space.

Next, when proceeding to step S315, it is checked whether or not the detected local maximum value is equal to or larger than the judgment value. When it is smaller than the judgment value, it is judged in step S316 that the sidewall does not exist,
When it is equal to or larger than the determination value, it is determined in step S317 that the sidewall exists, and the process proceeds to step S318.

Step S318 and subsequent steps in FIG. 38 are the side wall range detection processing, and in step S318, the parameters corresponding to the grid of the local maximum values detected in step S314, that is, the linear parameters (af indicated by the local maximum points). , Bf)
When is read, the process proceeds to step S319 and a side wall candidate area is set.

Then, in step S320, the first three-dimensional object data in the search area is read, and in step S321 the position (X, Z coordinates) of the subject is calculated.
Data existing in the side wall candidate area is extracted, and step S323 is performed.
Then, it is checked whether the processed data is the final data in the search area.

When it is not the final data in the search area,
After branching from step S323 to step S324, the next three-dimensional object data in the search area is read, and the above step S3 is performed.
Returning to step 21, when it is the final data in the search area, the process proceeds from step S323 to step S325 to create a histogram using the data in the side wall candidate area.

Next, proceeding to step S326, a section whose frequency of the created histogram is equal to or higher than the judgment value is detected, and in step S327, three-dimensional positions at both ends of the section where the frequency of the histogram is equal to or higher than the judgment value, that is, on the side wall. Calculate the front and rear end positions,
In step S328, parameters such as the presence / absence of the side wall, the position, the direction, and the positions of the front and rear ends are written in the output memory 100e, and the program ends. It should be noted that this program is executed for the right side wall after being executed for the left side wall.

As described above, in the present embodiment, since the feature relating to the three-dimensional position is utilized, it is not easily affected by the apparent feature such as the color, shape and material of the side wall, and not only the general guardrail, Various hand rubs, plantings, rows of pylon for road construction, etc. can be detected as side walls, and the side walls as the ones that regulate the running of automobiles can be accurately detected. Since it is converted to a simple data form such as the parameters and coordinates of the front and rear edges, it becomes easy for the user to handle and process the data.
By connecting an external device, more advanced hazard warning and accident avoidance functions can be realized.

39 and 40 relate to the second embodiment of the present invention, and FIG. 39 is an explanatory view showing a search area and a grid section, and FIG.
Reference numeral 0 is a flowchart of the sidewall straight line detection processing.

This embodiment is different from the above-described first embodiment in that the Hough transform process in the sidewall straight line detection is changed. Generally, the Hough transform is a time-consuming process, and various restrictions are set so as to shorten the process time also in the first embodiment described above. Since the processing is performed for each object data, the processing time also changes when the number of three-dimensional object data changes depending on the image condition.

The present embodiment is to improve this point, and is to search the side wall search regions SL and S in the first embodiment.
As shown in FIG. 39, R is divided into grids, each is numbered, and the Hough transform process is executed by the set number of grids. Here, the interval between the sections is, for example, 0.
It is set to about 3 m to 0.6 m, and the Z direction is set in consideration of the accuracy of the distance data and the request from the device side that uses the information of the side wall, but is about 1 m to 5 m, for example.

Below, according to the flow chart of FIG.
The side wall straight line detection processing will be described. Note that this flowchart replaces the flowchart (FIG. 37) of the sidewall straight line detection processing of the above-described first embodiment, and the other processing is the same as that of the above-described first embodiment.

First, for each data extracted in the above-mentioned three-dimensional object data extraction processing, the first three-dimensional object data is read in step S401, and in step S402 the three-dimensional position (X, Z coordinates) of the subject is calculated. And this is the search area S
If it is included in L and SR, the corresponding lattice number is calculated in step S403, and the count of the number of data in the lattice is updated in step S404.

Subsequently, it is checked in step S405 if it is the final data. If it is not the final data, in step S406.
The next three-dimensional object data is read, and the processing from step S402 is repeated in the same manner to obtain the data number for each grid.

When the final data is obtained in step S405, the flow advances from step S405 to step S407 to read the first grid data, and in step S408,
The position (X, Z coordinates) of the center of the lattice is calculated and step S409
Voting for the parameter space is performed and the process proceeds to step S410.

That is, in the above-described first embodiment, the straight line is set and the voting is performed for each data in the search area, but in the present embodiment, the data for each grid is collected at the center position of the grid. Assuming that there are existing data, a straight line is set, and voting is performed collectively for the number of grid data. In this case, the slope afi and the intercept bfi of the linear equation (12) to be set are set.
The range of is the same as that of the first embodiment described above.

The process of obtaining the local maximum value from the voted parameter space and determining the presence / absence of the side wall is the same as that of the first embodiment described above. If not a grid, step S411
Then, the next grid data is read and the processing from step S408 is repeated, and when the final grid is reached, the process proceeds to step S412, and if a local maximum in the parameter space is detected, the detected local maximum is determined in step S413. It is checked whether or not it is the judgment value or more. If it is smaller than the judgment value, it is judged in step S414 that the sidewall does not exist.
At 15, it is determined that the sidewall is present.

In the present embodiment, since the three-dimensional object data is collected for each grid-like space and the Hough transform processing is executed by the set number of grids, the processing time at the side wall detection can be made constant, When the number of three-dimensional object data is large, the processing time can be significantly shortened. Other functions and effects are similar to those of the first embodiment described above.

41 to 45 relate to the third embodiment of the present invention, FIG. 41 is an overall configuration diagram of the vehicle outside monitoring device, FIG. 42 is a circuit block diagram of the vehicle outside monitoring device, and FIG. 43 is a functional block of the side wall detecting device. FIG. 44 is an explanatory diagram showing an example of an image, FIG.
FIG. 4 is an explanatory diagram showing an example of a two-dimensional distribution of a three-dimensional object.

The vehicle exterior monitoring device 200 of this embodiment is shown in FIG.
As shown in FIG. 3, a passage pattern recognition device 210 for recognizing a two-dimensional position distribution of a three-dimensional object is adopted in place of the stereo image processing device 20 of the first embodiment, and the side wall of the passage pattern recognition device 210 is adopted. The detection device 220 is connected, and the side wall is detected by processing the position information of the two-dimensional distribution of the three-dimensional object obtained by the passage pattern recognition device 210 in the same manner as in the first embodiment described above. .

A known device can be applied to the passage pattern recognition device 210 connected to the side wall detection device 220, for example, the Institute of Instrument and Control Engineers Vol. 21,
No. 2 (February 1985) "Passage pattern recognition device" described in "Method of recognizing two-dimensional distribution of obstacles" can be applied.

The passage pattern recognition device 210 has a first
Similar to the stereo image processing device 20 of the embodiment, the image of two cameras is processed to detect the distance distribution to the object, but only the information of the three-dimensional object is output because the data processing method inside the device is different. However, the road shape cannot be detected by the white line as in the first embodiment.

Further, in this embodiment, the stereo optical system 201 for photographing the target scenery outside the vehicle is composed of two CCD cameras 201a and 201b which are attached to the front part of the vehicle 230 at a fixed vertical interval. Further, the passage pattern recognition device 210, as shown in FIG. 42, inputs a pair of upper and lower image signals from the stereo optical system 201 and calculates a two-dimensional distribution of distance and position of a three-dimensional object. 210a, a two-dimensional distribution memory 210b for storing the two-dimensional distribution information from the distance detection circuit 210a, and the like.

The sidewall detecting device 220 reads the two-dimensional distribution information written in the two-dimensional distribution memory 210b and performs various calculation processes on the microprocessor 22.
A read-only memory (ROM) 220b mainly configured to store the control program 0a, a read / write memory (RAM) that stores various parameters during calculation processing
220c, an output memory 220d for storing parameters of the processing result, and the like. The microprocessor 220a inputs the two-dimensional distribution information through the two-dimensional distribution memory 210b, executes the calculation process, and outputs the sidewall parameter as the process result to the output memory 220d.

The side wall detecting device 220 is the side wall detecting unit 1 of the road / side wall detecting device 100 in the first embodiment.
As shown in FIG. 43, only the data in the preset side wall search area is selected from the two-dimensional distribution information of the three-dimensional object stored in the two-dimensional distribution memory 210b as shown in FIG. A side wall straight line detection unit 221 for extracting the presence / absence of a side wall and a straight line expression indicating the position of the side wall by Hough transformation, and a straight line expression indicating the side wall position detected by the side wall straight line detection unit 221. Based on the above, a side wall candidate region in which a side wall is estimated to exist is set, and a side wall range detection unit 222 that detects the positions of the front and rear ends of the side wall based on the distribution state of the three-dimensional object data in the side wall candidate region, It is composed of a side wall parameter storage unit 223 composed of the output memory 220d for storing.

In this embodiment, when an image taken by the stereo optical system 201, for example, an image as shown in FIG. 44 is processed by the passage pattern recognition device 210, a two-dimensional distribution pattern of a three-dimensional object as shown in FIG. 45 is obtained. Is output. this is,
It is similar to the data (see FIG. 21) extracted by the three-dimensional object data extraction processing in the above-described first embodiment, and for this data, the sidewall straight line detection processing similar to that in the above-described first embodiment,
By performing the side wall range detection process, the presence or absence and the position of the left and right side walls can be detected.

As described above, in this embodiment, it is difficult to detect the white line of the road as in the case of using the position information of the two-dimensional distribution of the three-dimensional object output from the passage pattern recognition device, and Even if it is not easy to perform accident risk judgment or data processing in traveling control on the side that uses this position information, the side wall can be reliably detected, and the presence or absence of the side wall and the position of Since the data is converted into a simple data form such as parameters and coordinates of front and rear ends, the user can easily handle and process the data.

46 to 51 relate to the fourth embodiment of the present invention. FIG. 46 is an overall block diagram of the vehicle outside monitoring device, FIG. 47 is a circuit block diagram of the vehicle outside monitoring device, and FIG. 48 is a functional block of the side wall detecting device. 49, FIG. 49 is an explanatory view showing a laser beam scanning method from a side surface, FIG. 50 is an explanatory view showing a laser beam scanning method from an upper surface, and FIG. 51 is a two-dimensional distribution of a three-dimensional object measured by a laser radar range finder. It is an explanatory view showing an example of.

The vehicle outside monitoring system 300 of this embodiment is shown in FIG.
As shown in FIG. 7, a laser radar range finder 310 using a laser beam is adopted in place of the stereo image processing apparatus 20 of the first embodiment, and this laser radar range finder 310 is used.
The side wall detection device 320 is connected to the side wall of the three-dimensional object. The position information of the two-dimensional distribution of the three-dimensional object obtained by the laser radar range finder 310 is processed in the same manner as in the first and second embodiments described above to remove the side wall. It is something to detect.

The laser radar range finder 310 projects a laser beam, receives light reflected by the laser beam upon an object, and measures the distance from the required time to the object. Example vehicle exterior monitoring device 3
A known laser radar range finder can be applied to 00.

That is, the vehicle exterior monitoring device 300 of this embodiment.
Then, a laser projecting unit 301 having a function of projecting / receiving a laser beam and scanning in the left / right direction is attached to the front part of the vehicle 330, and as shown in FIG. A distance detection circuit 310a that calculates the distance to the object from the time required to project and receive the laser beam, and also calculates the two-dimensional position of the object from the scanning direction of the laser beam, the two-dimensional position of the detected object The two-dimensional distribution memory 310b for writing

Further, the side wall detecting device 320 reads the two-dimensional distribution information written in the two-dimensional distribution memory 310b and performs various kinds of calculation processing on the microprocessor 3.
20a, a read-only memory (ROM) 320b for storing a control program, and a read / write memory (RA) for storing various parameters during calculation processing.
M) 320c, an output memory 320d for storing parameters of processing results, and the like. The microprocessor 320a inputs the two-dimensional distribution information through the two-dimensional memory 310b, executes the calculation process, and outputs the sidewall parameter as the process result to the output memory 320d.

The side wall detection device 320 has the same functional configuration as the side wall detection device 220 of the second embodiment described above, and as shown in FIG. 48, the three-dimensional object stored in the two-dimensional distribution memory 310b. Side wall straight line detection that extracts only the data within the preset side wall search area from the two-dimensional information and processes it by Hough transform to detect the presence or absence of the side wall and the linear expression indicating the position of the side wall. Section 321, a side wall candidate area in which a side wall is estimated to exist is set based on a linear expression indicating the position of the side wall detected by the side wall straight line detecting section 321, and the distribution state of three-dimensional object data in this side wall candidate area is set. From the side wall range detection unit 32 that detects the positions of the front and rear ends of the side wall.
2. A sidewall parameter storage unit 323 including the output memory 320d for storing the processing result.

As shown in FIG. 49, the laser beam is projected horizontally from the laser projecting unit 301, and only the three-dimensional object located at a position higher than the road surface is detected. Further, as shown in FIG. 50, the laser beam is scanned in the left-right direction, and the operation of projecting / receiving the laser beam at regular intervals in a predetermined scanning range to detect the distance is repeated, and the two-dimensional object is detected. The dimensional distribution is measured.

For example, when the laser radar range finder 310 measures a situation in which there is a guardrail on the left side and another vehicle on the right front side as shown in FIG. 44, the two-dimensional object as shown in FIG. Distribution information is obtained. this is,
This is the same as the data (see FIG. 21) extracted by the three-dimensional object data extraction processing in the above-described first embodiment. Therefore, the side wall straight line detection processing and the side wall range detection processing similar to those in the first and second embodiments are performed on the two-dimensional distribution of the three-dimensional object which is the output of the laser radar range finder 310 to determine the presence or absence of the left and right side walls. , The position can be detected.

Also in the present embodiment, as in the case of the above-mentioned third embodiment, even when it is difficult to detect the white line of the road, the side wall can be surely detected, and the presence / absence of the side wall, the position, The direction can be detected in the form of data that can be easily processed.

[0223]

As described above, according to the present invention, based on the position information for the object in the set range outside the vehicle, the side wall of the continuous solid body that becomes the boundary of the road such as guardrail, planting, pylon row, etc. Since the presence / absence, a linear expression approximating the position of this side wall, and the range in which this side wall exists are detected, even if there is no white line on the road or if it is difficult to detect the white line, the side wall can be reliably detected. Moreover, since the presence / absence of the side wall, the position, and the direction are detected in a data format that can be easily processed, it is possible to realize more advanced danger warning and accident avoidance functions by using these data. And so on.

[Brief description of drawings]

1 to 38 relate to a first embodiment of the present invention, and FIG. 1 is an overall configuration diagram of a vehicle exterior monitoring device.

[Fig. 2] Front view of the vehicle

FIG. 3 is a circuit block diagram of a vehicle exterior monitoring device.

FIG. 4 is an explanatory diagram showing a relationship between a camera and a subject.

FIG. 5 is a circuit configuration diagram showing a specific example of a stereo image processing device.

FIG. 6 is an explanatory diagram of a city block distance calculation circuit.

FIG. 7 is a block diagram of a minimum value detection circuit.

FIG. 8 is an explanatory diagram showing an example of an image taken by a vehicle-mounted CCD camera.

FIG. 9 is an explanatory diagram showing an example of a range image.

FIG. 10 is a top view of the vehicle

FIG. 11 is a side view of the vehicle.

FIG. 12 is a functional block diagram of a road / sidewall detection device.

FIG. 13 is an explanatory diagram showing an example of a road model.

FIG. 14 is an explanatory diagram showing the shape of a three-dimensional window.

FIG. 15 is an explanatory diagram showing the shape of a two-dimensional window.

FIG. 16 is an explanatory diagram showing a deviation amount between a linear element and data.

FIG. 17 is an explanatory diagram showing a relationship between a shift amount and a weighting coefficient.

FIG. 18 is an explanatory diagram showing an example of a detected road shape.

FIG. 19 is an explanatory diagram showing the shape of a search region in sidewall detection.

FIG. 20 is an explanatory diagram showing an example of an image in side wall detection.

FIG. 21 is an explanatory diagram showing a distribution status of three-dimensional object data.

FIG. 22 is an explanatory diagram showing the assumption of a straight line in the Hough transform.

FIG. 23 is an explanatory diagram showing a voting area in a parameter space.

FIG. 24 is an explanatory diagram showing results of voting on a parameter space.

FIG. 25 is an explanatory diagram showing sidewall candidate regions.

FIG. 26 is an explanatory diagram showing a relationship between a histogram and a side wall existence range.

FIG. 27 is a flowchart showing the operation of the stereo image processing apparatus.

FIG. 28 is an explanatory diagram showing a storage order in the shift register.

FIG. 29 is a timing chart showing the operation of the city block distance calculation circuit.

FIG. 30 is a timing chart showing the operation of the deviation amount determination unit.

FIG. 31 is a timing chart showing the operation of the stereo image processing apparatus.

32 to 35 are flowcharts showing the operation of the road detection unit, and FIG. 32 is a flowchart of road shape estimation processing.

FIG. 33 is a flowchart of a three-dimensional window generation process.

FIG. 34 is a flowchart of linear element detection processing.

FIG. 35 is a flowchart of road shape determination processing.

36 to 38 are flowcharts showing the operation of the sidewall detection unit, and FIG. 36 is a flowchart of solid object data extraction processing.

FIG. 37 is a flowchart of sidewall straight line detection processing.

FIG. 38 is a flowchart of sidewall range detection processing.

39 and 40 relate to the second embodiment of the present invention, and FIG. 39 is an explanatory diagram showing a search area and a grid section.

FIG. 40 is a flowchart of sidewall straight line detection processing.

41 to 45 relate to a third embodiment of the present invention, and FIG. 41 is an overall configuration diagram of a vehicle exterior monitoring device.

FIG. 42 is a circuit block diagram of the vehicle exterior monitoring device.

FIG. 43 is a functional block diagram of a sidewall detection device.

FIG. 44 is an explanatory diagram showing an example of an image

FIG. 45 is an explanatory diagram showing an example of a two-dimensional distribution of a three-dimensional object.

46 to 51 relate to a fourth embodiment of the present invention, and FIG. 46 is an overall configuration diagram of a vehicle exterior monitoring device.

FIG. 47 is a circuit block diagram of the vehicle exterior monitoring device.

FIG. 48 is a functional block diagram of a sidewall detection device.

FIG. 49 is an explanatory view showing a scanning method of a laser beam from a side surface.

FIG. 50 is an explanatory diagram showing a scanning method of a laser beam from above.

FIG. 51 is an explanatory diagram showing an example of a two-dimensional distribution of a three-dimensional object measured by a laser radar range finder.

[Explanation of symbols]

1 vehicle 2 vehicle exterior monitoring device 10 stereo optical system (measurement means) 20 stereo image processing device (measurement means) 110 road detection unit (road detection means) 111 road shape estimation unit (road shape estimation means) 112 three-dimensional window generation unit ( Three-dimensional window generation means) 113 Linear element detection unit (linear element detection means) 114 Road shape determination unit (road shape determination means) 121 Three-dimensional object data extraction unit (data extraction means) 122 Side wall straight line detection unit (side wall straight line detection means) 123 Side Wall Range Detection Unit (Side Wall Range Detection Means)

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁵ Identification code Office reference number FI technical display location // B60R 21/00 C 8812-3D

Claims (10)

[Claims] 1. A measuring means for detecting an object in a set range outside the vehicle and outputting position information for the object, and a side wall as a continuous three-dimensional object which becomes a boundary of a road based on the position information from the measuring means. Side wall straight line detecting means for detecting presence / absence and a straight line expression approximating the position of the side wall, and a space area is set around the straight line expression detected by the side wall straight line detecting means, and only data in this space area is extracted. And a side wall range detecting means for detecting a range where the side wall exists. 2. A measuring means for detecting an object within a set range outside the vehicle and outputting three-dimensional position information for the object, and a road model is determined based on the three-dimensional position information from the measuring means, and the road model is determined. Road detection means for detecting a model as a road shape, data extraction means for extracting only data above the road surface from the three-dimensional position information based on the road model, and extracted by the data extraction means From the data, the presence / absence of a side wall as a continuous three-dimensional object that becomes the boundary of the road, and a side wall straight line detecting means for detecting a straight line expression approximating the position of this side wall, and the periphery of the straight line expression detected by the side wall straight line detecting means And a side wall range detecting means for detecting a range in which the side wall exists by extracting only data in the space area from the outside of the vehicle. Location. 3. The road shape estimating means for estimating the position and shape of a white line of the road based on the three-dimensional position information, and the road detecting means for combining the white line of the road estimated by the road shape estimating means. A three-dimensional window setting means for setting a three-dimensional spatial area as a three-dimensional window, and extracting only data in the three-dimensional window from the three-dimensional position information, and a three-dimensional straight line forming the road model. A straight line element detecting means for detecting an element and the validity of the straight line element detected by the straight line element detecting means are determined, and when the determination criteria are not met, the straight line element is modified or changed to determine the road model. The vehicle exterior monitoring device according to claim 2, further comprising a road shape determination unit. 4. The presence / absence of the side wall is determined by the side wall straight line detecting means by setting a search area for searching for the existence of the side wall and extracting only data in the search area and performing Hough transform. The vehicle exterior monitoring device according to claim 1 or 2, wherein the linear type is detected. 5. The side wall straight line detecting means divides and sets a search area for searching for the existence of the side wall in a grid shape, extracts only data in the search area, and extracts data included in each grid. The number of pieces is obtained, and the Hough transform is performed assuming that the data in each grid exist in the central position of each grid, thereby detecting the presence or absence of the side wall and the linear equation. The vehicle exterior monitoring device according to claim 1 or 2. 6. The side wall range detection means creates a histogram in which the vertical axis represents the number of data in the spatial area and the horizontal axis represents the distance to the vehicle, and the side wall is determined from the frequency of the histogram. The vehicle exterior monitoring device according to claim 1 or 2, which detects an existing range. 7. The presence / absence of the side wall is determined by the side wall straight line detection means by setting a search area for searching for the existence of the side wall, and extracting only data in the search area and performing Hough transform. , The linear equation is detected, the side wall range detection means creates a histogram with the number of data in the spatial region as the vertical axis and the distance to the vehicle as the horizontal axis, and The vehicle exterior monitoring device according to claim 1 or 2, wherein the range in which the side wall is present is detected from the magnitude of the frequency. 8. The side wall straight line detecting means divides and sets a search area for searching for the existence of the side wall in a grid shape, extracts only data in the search area, and extracts data included in each grid. The number of pieces is obtained, and the data in each grid is subjected to a Hough transform assuming that the data exists in the central position of each grid, and the presence or absence of the side wall and the linear expression are detected. The range detecting means creates a histogram with the number of data in the spatial region as the vertical axis and the distance to the vehicle as the horizontal axis, and detects the range where the side wall exists from the magnitude of the histogram frequency. The vehicle exterior monitoring device according to claim 1 or 2, wherein: 9. The vehicle according to claim 1, wherein the measuring means outputs the position information for an object outside the vehicle by imaging a set range outside the vehicle and performing image processing on the set range. Outside vehicle monitoring device. 10. The measuring means outputs the position information for an object outside the vehicle by scanning a set range outside the vehicle and projecting / receiving a laser beam. Vehicle exterior monitoring device.