JPH06215134A - Preprocessing method for on-vehicle picture data to be recognized - Google Patents

Abstract

PURPOSE:To easily adjust the position and inclination of a television camera at the time of fixing the television camera to a vehicle by deciding the setting position or visual field of the television camera so that one part of its own vehicle can be included in a picture, and correcting the coordinate of the picture based on the inclination of a bonnet or the like in the picture. CONSTITUTION:A television camera 6b is set in the neighborhood of the central upper part of a front window in the vehicle, and the setting direction and visual field is decided so that the bonnet of its own vehicle can be sufficiently included in a photographic scene. A CPU 1 controls the diaphragm of the television camera 6b, and the output level of a picture signal or the like, and synchronously controls the input and output of an A/D converter 6c and the writing processing of an image memory 5a or the like. The CPU 1 calculates a role angle thetar and pan moving amounts Xp from the top edge image of the bonnet of the vehicle, and converts the address of the gradation data of each picture element in the input data memory of the image memory 5a by picture correction including picture rotation and parallel movement and interpolation, and writes the data in a corrected picture memory.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an on-vehicle recognition target image data.
Data pretreatment method.

[0002]

2. Description of the Related Art For example, the position of a white line on a road surface or the position of another vehicle traveling ahead can be recognized by processing image information in a range that matches the visual field of a driver driving an automobile. By using the image recognition result, the angle of the steering wheel is controlled so as to keep the vehicle within a predetermined travel lane, or the distance between the vehicle and other vehicles is kept constant. Autopilot such as adjusting the vehicle speed so that it will be possible.

As an image recognition method used for this type of application, for example, a method disclosed in an on-vehicle distance detection device (Japanese Patent Laid-Open No. 1-15605) previously proposed by the present applicant is known. In this recognition method, the positions of the white line displayed on the road surface and the horizontal line at the lower end of the vehicle (preceding vehicle) traveling in front are recognized, and the position of the preceding vehicle is estimated based on the recognition results. It identifies the distance to the vehicle.

[0004]

In the above prior art, in image recognition, the position of the preceding vehicle is estimated from the position of the detected horizontal line under the condition that the horizontal line is always detected at the lower end position of the preceding vehicle. is doing. Therefore, if the horizon is not found, the preceding vehicle cannot be recognized,
If the horizon is detected in a portion other than the preceding vehicle, the position of the preceding vehicle is likely to be erroneously recognized.

By the way, the image data to be image-recognized
The data is usually output from a television camera arranged in the vehicle interior or the like. This type of television camera captures a two-dimensional image of a subject while scanning in a direction along its own horizontal axis and vertical axis, and outputs image data at each scanning position. Therefore, unless each axis of the TV camera is fixed to the vehicle body in a state where the axes of the TV camera are aligned with the horizontal and vertical axes of the vehicle body, the horizontal and vertical directions of the two-dimensional image output from the TV camera are the same. It does not coincide with the horizontal and vertical axes of the subject (leading vehicle, etc.) in the three-dimensional image.
Therefore, if each axis of the TV camera is fixed while being displaced relative to each axial direction on the vehicle body, the horizontal line at the lower end of the preceding vehicle becomes
In the image, it is displayed in a state of being inclined with respect to its horizontal direction. However, the horizontal line detected in the image recognition is a line that coincides with the axis of the television camera in the horizontal scanning direction, and therefore the line inclined in the image is not regarded as the horizontal line. Therefore, the preceding vehicle may not be detected.

Further, for example, it is possible to detect the deviation between the straight traveling direction of the vehicle and the forward traveling lane from the horizontal position of the intersection of the left and right white lines on the road surface at the point of infinity. However, when the direction in which the television camera shoots with respect to the vehicle body changes, the position of the subject reflected in the two-dimensional image output by the television camera shifts, which causes an error in the detected traveling lane direction.

In order to eliminate the above-mentioned inconvenience, when fixing the television camera to the vehicle body, it is necessary to eliminate the inclination of the axis of the television camera, and moreover, to adjust the orientation accurately, which makes the adjustment work difficult. Even if the alignment is performed accurately, there is a high possibility that the television camera may be displaced or tilted due to a change with time due to mechanical vibration or the like.

Therefore, the present invention facilitates the work such as adjustment when fixing the TV camera to the vehicle body, and the image recognition performance even when the TV camera is mounted in a misaligned or tilted state. It is an object of the present invention to prevent the decrease of the error and the occurrence of the detection error.

[0009]

In order to solve the above problems, according to the present invention, an image pickup means fixed to a vehicle body is used.
An image data output by the image pickup means is obtained by taking an image of the outside of the vehicle.
In the pre-processing method of the on-vehicle recognition target image data for processing the image data to recognize the subject in the image, the image pickup means is used in a state where a part of the vehicle body of the vehicle appears in advance in a part of the image data. It is installed, the image data input from the image pickup means is processed, the vehicle body of the own vehicle in the image is recognized, the inclination θr in the image of the recognized vehicle body is detected, and based on the detected inclination θr It is characterized by rotating the image and correcting the coordinates of each position of the image.

According to the second aspect of the invention, an image of the outside of the vehicle is taken by the image pickup means fixed to the vehicle body, and the image data output from the image pickup means is processed to recognize the subject in the image. In the preprocessing method of the on-vehicle recognition target image data:
The image pickup means is installed in advance so that a part of the vehicle body of the vehicle appears in a part of the image data, and the image data input from the image pickup means is processed to display the vehicle body of the vehicle in the image. Recognized, the inclination θr in the recognized image of the vehicle body and the amount of displacement Xp with respect to the reference position are detected, and the detected inclination θ
It is characterized in that the image is rotated and moved in parallel on the basis of r and the displacement amount Xp to correct the coordinates of each position of the image.

[0011]

In the present invention, as preprocessing, the inclination θr of the subject in the image is detected, and the coordinate axes of the image are rotated so that the inclination θr becomes 0 and the coordinates are corrected. Therefore, even when the TV camera is fixed in a state where the scanning axis of the TV camera (imaging means) is tilted with respect to the axis of the vehicle body, an actually horizontal object is not shown on the coordinates of the corrected image. It appears horizontal. Therefore, for example, the horizontal line of the preceding vehicle can be reliably detected as a line oriented in the horizontal direction in the image without performing highly accurate positioning adjustment when mounting the television camera.

There are various subjects such as the road surface, the preceding vehicle, and the sky in the landscape displayed on the TV camera, but the positions in these images change as the vehicle moves (runs), The distance between the own vehicle and the preceding vehicle is also not constant. But,
When the installation position and field of view of the TV camera are set so that a part of the vehicle body, for example, the bonnet (or the upper part of the windshield) is included in the image, the position and tilt of the bonnet in the image indicate the installation position of the TV camera. Also, the position and the inclination do not change as the vehicle moves (runs). In other words, it is possible to detect the deviation and tilt of the mounting position of the TV camera from the position and tilt of the hood in the image, and use the detection result to correct the position and tilt of each subject in the image. You can

In the embodiments described later, the known Hough (Houg
h) The transformation is used to detect a straight line approximating a part of the contour of the bonnet from the information of a plurality of feature points, and the inclination and the positional deviation are detected. That is, a predetermined range of image data input from the image pickup means is vertically scanned at a plurality of horizontal coordinate positions different from each other, and a plurality of characteristic points whose brightness changes at least on a plurality of scanning lines are detected, By polar coordinate conversion, ρ,
Generate a group of coordinate values that form a curve in the θ polar coordinate system,
The polar coordinate values (ρm, θm) on the polar coordinate system where the curves of a plurality of characteristic points pass in a concentrated manner are detected, and the detected polar coordinate values (ρ
The straight line of the X, Y orthogonal coordinate system corresponding to (m, θm) is associated with a part of the outline of the recognition target, and the inclination θr is obtained based on the inclination of the straight line.

[0014]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows the configuration of an embodiment of the present invention. This system is a microcomputer (hereafter called CPU)
1 is mainly configured, and its bus line has a read-only memory (ROM) in which a control program is stored.
2, a read / write memory (RAM) 3 for storing parameters being processed, and input / output ports (I / O) 4, 5, 6 and the like to which various constituent elements are connected.

The television camera 6b, as shown in FIG.
It is installed near the upper center of the front window in the car. The front scene is photographed and 512 x per frame.
It outputs an analog signal of 512 pixels. In addition, in this embodiment, as shown in FIG.
The installation direction and the field of view of the TV camera 6b are taken into consideration so that the hood of the own vehicle is sufficiently included. Since the axis of the television camera 6b is tilted with respect to the vehicle body, the bonnet portion appears slightly tilted in FIG. 5B, but the tilt and position shift of the subject in the image are caused by the processing described later. Is automatically corrected, so it is not necessary to perform precise position adjustment when mounting the TV camera 6b. The analog image signal output by the TV camera 6b is A
It is given to the / D converter 6c and the CRT driver 4a. In the A / D converter 6c, the analog image signal from the television camera 6b is converted into digital data (gradation data of 256 gradations (gradation 0 is black level, gradation 255 is white level)) for each pixel. Data) and gives it to the image memory 5a. The image memory 5a has one screen (512 ×
If the area for storing the gradation data of 512 pixels is one page, the gradation data storage area for several pages and the binary data for storing 1-bit information (binary data) for several screens It has a data storage area.

The CPU 1 is a television camera controller 6
The aperture of the television camera 6b and the output level of the image signal are controlled via a, and the input / output of the A / D converter 6c and the writing process of the image memory 5a are synchronously controlled.
The CRT driver 4a adds C to the image (for example, FIG. 16) represented by the analog image signal supplied from the television camera 6b.
A digital image (digital output image; for example, a straight line approximating the white lines at the left and right ends of the own vehicle running lane, and character and numerical information indicating the distance between the vehicles running ahead) given under the control of PU1 The combined image is displayed on the CRT 4b installed near the center of the inner panel on the vehicle.

FIG. 3 shows an outline of the processing operation of the CPU 1. Generally, the CPU 1 initializes an input / output port, initializes an internal register and an internal counter (initialization A) when the power is turned on, and then determines an analog image signal of one screen. Image memory 5 with digital conversion in cycles
The data is written in the input data memory area of a (image input B). Every time this is performed, the CPU 1 succeeds (successes) in "screen calibration" C, "own vehicle lane detection" D, and own vehicle lane detection success / failure check E, and own vehicle lane detection. In the case, "adjacent lane estimation" F, "curve estimation" G, "own vehicle lane preceding vehicle recognition and distance measurement" H, "right adjacent lane vehicle recognition and distance measurement" I, " Left adjacent lane vehicle recognition and distance measurement "J
And "output" K are executed in this order.

In the "output" K, straight lines representing the white lines at the left and right ends of the own vehicle lane, character information indicating whether or not a preceding vehicle of the own vehicle lane is detected, a square mark surrounding the preceding vehicle, and Distance between the preceding vehicle, character information indicating whether or not a right adjacent lane vehicle is detected, a square mark surrounding the preceding vehicle and a distance between the preceding vehicle, and a left adjacent lane vehicle. The character information indicating the presence / absence of detection, the square mark surrounding the preceding vehicle, and the distance between the preceding vehicle and the preceding vehicle are combined into a captured image and updated and displayed on the CRT 4b, and white lines at the left and right ends of the own vehicle lane are displayed. The data indicating the straight line and the data indicating the inter-vehicle distances are transferred to the host CPU 8 via the communication controller 7 (FIG. 1). The host CPU 8 uses these pieces of information as inter-vehicle distance control (accelerator control corresponding to inter-vehicle distance, overdrive cutoff control and / or wheel brake pressure control), obstacle detection &
It is used for automatic traveling control such as wheel brake pressure control, lane departure warning control, lane copying traveling control (steering control and / or wheel braking pressure control for each wheel). Less than,
The contents of each item of "Screen calibration" C and "Left adjacent lane vehicle recognition and distance measurement" J shown in FIG. 3 will be described in detail. C. "Screen calibration" C (Fig. 4) The contents of this process are shown in Fig. 4. First, the "feature point detection window set" C1 is executed.

C1. “One set of feature point detection window” C1 (FIG. 5) This content is shown in FIG. As shown in FIG. 5 (b), the upper left of the shooting screen of the television camera 6b (the image data is gradation data in the input data memory) is the origin (0,
0), one point (0,350) and another point (51
1, 511) is set as a diagonal corner of an image area for feature point detection (window 1). That is, the X coordinate value 0 of the upper left corner of the area to be determined is set in the register XU.
Write the Y coordinate value 350 to the register YUL, the X coordinate value 511 of the lower right corner of the area to the register XLR, and the Y coordinate value 511 to the register YLR (step 1 in FIG. 5A). ; In the following, in parentheses, the word step is omitted, and only the step number is described). The value indicated by the data of each of these registers will be represented below by the register symbol itself.

"Characteristic point detection (UP)" C2 described below
5 (b) of FIG. 5 by the above-mentioned "feature point detection window set" C1 for the processing of FIG.
The block (feature point detection window 1) shown by the dotted line is
It has been designated as the processing target area. As shown in FIG. 5B, this region sufficiently includes the leading edge image of the hood of the vehicle.

Next, "feature point detection (UP)" C2 is executed. This content is shown in FIG.

C2. “Feature point detection (UP)” C2 (FIG. 6) Here, the lower left corner (0,
511) The pixel (0, 509) two pixels above in the Y direction is scanned in the Y direction from the bottom to the top in which the Y direction coordinate value is sequentially reduced to 350. Pixels to be designated are sequentially designated as target pixels (2 to 11 in FIG. 6A), and for each target pixel (Xs, Ys), a pixel (Xs, Ys + 1) one pixel below and two pixels below Of the luminance represented by the gradation data of the pixel (Xs, Ys + 2), and the pixel (Xs, Ys-1) one pixel above and the pixel (Xs, Ys) two pixels above the target pixel (Xs, Ys). −
The absolute value of the difference between 2) and the sum of the brightness represented by the gradation data in the input data memory, that is, the absolute value of the gradation differential value in the vertical direction (Y direction) is calculated (8). This is defined as the gradation differential value D (Xs, Ys) of the target pixel point (Xs, Ys).
This gradation differential value D (Xs, Ys) is obtained by comparing the image density sum of the upper two pixels and the image density sum of the lower two pixels of the target point (Xs, Ys) as shown in FIG. 6B. Since it is the absolute value of the difference, it represents the change in density in the vertical direction with respect to the point of interest, and has a large value at the contour line extending horizontally of each image on the screen. It is checked whether or not the gradation differential value D (Xs, Ys) is equal to or more than a threshold value Th1 (set value) (9). If it is Th1 or more, it is assigned to a certain area of the image memory 5a. At the address E (Xs, Ys) corresponding to the target pixel point (Xs, Ys) of the value data table,
Write "1" (10). This information "1" means one point on the contour line extending in the horizontal direction of the image. Y
In the direction, such processing is performed for one line (X = 0, Y = 50
9 to 350), the scanning line is moved to the right (6), and the next one line (X = 1, Y = 509 to 35) is moved.
The same process is performed for 0). Similarly, the same processing is performed until the last line (X = 511, Y = 509 to 350).

By the above processing, the binary data table E (Xs, Ys) (Xs = 0 to 51) of the image memory 5a.
1, Ys = 509 to 350), information indicating a large number of points constituting the contour line extending in the horizontal direction of the image in the feature point detection window 1 (b in FIG. 5) is written. Since the feature point detection window 1 is set in a region including the front edge image of the bonnet, the binary data table E
(Xs, Ys) includes information indicating the leading edge of the bonnet. In this embodiment, the leading edge of the bonnet is regarded as a straight line, and the next "bonnet detection" C3 is performed.
The straight line is detected with. This content is shown in FIG.

C3. "Bonnet detection" C3 (Fig. 7) First, the outline is explained here by the known "Houg (Houg
h) Transform ”to detect a straight line. The Hough transform is performed in a rectangular coordinate system (XY coordinate system; for example, the feature point detection window 1
Information (for example, the binary data table E (Xs, Ys)) represented by the pixel distribution in the polar coordinate system (ρ-
This is a method of expressing in the (θ coordinate system), and the conversion of ρ = Xcos (θ) + Ysin (θ) is performed. The coordinates of a certain point (pixel) in the XY coordinate system are represented by a certain curve in the ρ-θ polar coordinate system (the orthogonal coordinate plane that assigns ρ and θ to one and the other of the two orthogonal axes). When each point on a straight line in the XY coordinate system is represented by a curve in the ρ-θ polar coordinate system, these curves intersect at one point. By substituting the polar coordinates (ρm, θm) of this intersection into the conversion formula for the XY coordinate system, a formula representing a straight line in the XY coordinate system can be obtained.

Therefore, the binary data table E
Each coordinate having the feature point information “1” on (Xs, Ys) is converted into a large number of polar coordinate values indicating a curve of the ρ-θ coordinate system, and the intersection point of the curves in the ρ-θ polar coordinate system is obtained. By substituting this intersection into the conversion formula for the XY coordinate system, the straight line connecting the characteristic points can be obtained. A straight line that simply connects two points can be directly obtained in the XY coordinate system, and there is no point in converting to the ρ-θ polar coordinate system. However, the contour line of the bonnet on the screen is that the actual contour line of the tip of the bonnet is curving, or that the bonnet is curving on the screen due to the light hitting the bonnet, etc. Therefore, when the bonnet contour is approximated by a straight line, simply the binary data table E (Xs, Ys)
If a straight line connecting the two points having the above-mentioned feature point information “1” is obtained, the approximation error is too large, and there is a possibility that the contour line of the non-bonnet may be extracted. FIG. 5B
As shown in FIG. 16 and FIG. 16, since the bonnet contour line is distributed over a wide area, the binary data table E (Xs,
There is a lot of feature point information "1" on Ys). Therefore, a large number of straight lines connecting the two points (pixels) having the "1" are obtained. When a straight line descending to the right that represents a large number of straight lines is determined, it is the one that best approximates the right half of the bonnet tip contour line (b in FIG. 7). Similarly, when a straight line to the left that represents a large number of straight lines is determined, it is the one that best approximates the left half of the bonnet tip contour line (b in FIG. 7). However, if such statistical processing is performed in the XY coordinate system, the processing becomes extremely complicated. However, when the coordinates (feature points) having the feature point information “1” on the binary data table E (Xs, Ys) are transformed into the ρ-θ polar coordinate system using the Hough transform, each feature point is represented. The intersections of the curves are concentrated at two points (corresponding to the right half contour line and the left half contour line of the bonnet). Therefore, binary data
Hough transform is performed on the characteristic points of the vertical E (Xs, Ys), and ρ
-At each point on the polar coordinate system, count the number of times the curve passes (the number of curves passing through one point), select the point with the highest frequency and the point with the next highest frequency, and select each selected point with X- By substituting in the conversion formula to the Y coordinate system, respective linear formulas representing the right half and the left half of the bonnet outline can be obtained. In this embodiment, the tip contour line of the bonnet is detected according to such a logic, that is, by the Hough transform and counting the number of times the curve passes through each coordinate point on the ρ-θ polar coordinate system.

By the way, the count of the curve crossing frequency at each point on the ρ-θ polar coordinate system is represented by the binary data table E (Xs,
If it is performed at each point in the entire region of the ρ-θ polar coordinate system having a size corresponding to the size of Ys), the count process of the number of times the curve passes becomes enormous, and the straight line detection process takes a long time. Therefore, in this embodiment, the right and left intermediate points (X = 255, Y = 511) at the lowermost end of the binary data table E (Xs, Ys) are set to ρ-θ.
It is set at the origin (0, 0) of the polar coordinate system, and the binary data table E (Xs, Ys) is divided into left and right, and the right half is converted into binary data by the first Hough transform. -Bull E (Xs,
Ys) characteristic points are converted into a low-density (large sampling pitch) ρ-θ polar coordinate system, and the number of times a curve passes through each point of the coordinate system (the number of curves passing through one point) is counted. Point (first prediction point) is obtained. Next, the second Hough transform is used to set a small range ρ-θ region centered on the first prediction point, and the size of the region is set to the medium density ρ-θ polar coordinate system.
The characteristic point of the table E (Xs, Ys) is converted to
The number of times the curve passes through each point of the −θ polar coordinate system is counted, and the point with the highest frequency (second predicted point) is obtained. Then, in the third Hough transform, a smaller range ρ-θ region centered around the second prediction point is set, and the binary data table E (is set in the high density ρ-θ polar coordinate system of the size of the region. (Xs, Ys) are converted to count the frequency at which the curve passes through each point of this ρ-θ polar coordinate system, and the point with the maximum frequency (third prediction point) is obtained. Then, the straight line represented by the third predicted point is determined as the right half of the bonnet tip contour line (12R to 14R in FIG. 7). Two
The left half of the value data table E (Xs, Ys) is similarly subjected to the first Hough transform, the second Hough transform and the third Hough transform to obtain a straight line representing the left half of the bonnet tip contour line. It is determined (12L to 14L in FIG. 7).

Next, the contents of "bonnet detection" C3 will be described in detail with reference to FIG. First, parameters are set for detecting a straight line that approximates the right half of the bonnet tip contour line (12R). Then, the point (R _m3 , T _m3 ) corresponding to the approximate straight line is detected (13R). R _m3
Is the ρ value in the ρ-θ polar coordinate system, and T _m3 is the θ value. The obtained coordinate value of the point is stored in the registers R _mR and T _mR (14R).
Similar processing is performed to detect a straight line that approximates the left half of the bonnet tip contour line (12L to 14L). "Line fitting" 13R, which detects straight lines by Hough transform,
The contents of 13L are similar to those of the "straight line fitting" 63R shown in FIGS. 17 to 21, which will be described in detail later. The "straight line fitting" 63R detects the approximate straight line of the white line image on the road in the right half area of the upper feature point detection window 2 (FIG. 16) in contact with the feature point detection window 1. "Straight line fit" 1
Since 3R detects a straight line in the right half area of the window 1, the XY coordinate area (start point coordinate value and end point coordinate value) to be processed is different. "Straight line fit" 13L
However, similarly, the XY coordinate area to be processed is different from the "straight line fitting" 63R. The other processing is the same.

[0028] The "line fitting" straight lines obtained in 13R and 13L, the polar coordinate system point (R _mR, T _mR) and (R _mL, T _mL) is represented by, X-Y coordinate system these linear When expressed by, R _mR = Xcos (T _mR ) + Ysin (T _mR ) and R _mL = Xcos (T _mL ) + Ysin (T _mL ). These straight lines are as shown in FIG. 7 (b).

C4. "Straight line intersection calculation" C4 (Fig. 8) Referring again to Fig. 4, when the above "bonnet detection" C3 is completed, the CPU 1 performs the calculation shown in the block adjacent to the "straight line intersection calculation" C4 block. The intersection (Xc, Yc) of the straight line is calculated. In this calculation, X
ch is the X of the area of the binary data table E (Xs, Ys).
The directional center position (Xch = 255), Ych is the lowest end position (Ych = 509), and the point (Xch, Ych) is the polar coordinate origin determined by the above-described straight line detection.

C5. "Calculation of Roll Angle and Pan Movement Amount" C5 (FIG. 8) Here, the roll angle is a straight line that bisects the intersecting angle of the two detected straight lines, as shown in FIG. It is an angle θr formed between the (vertical axis of the vehicle) and the Y-axis of the shooting screen, and the angle formed by the visual field center line of the television camera 6b and the vertical axis of the vehicle when they are projected on a horizontal plane, that is, the television camera 6b It is the angle of deviation of the center line of view from the longitudinal axis of the vehicle in the roll direction. The pan movement amount Xp is a lateral shift amount of the center of the bonnet tip (intersection point of the two straight lines) with respect to a vertical axis that bisects the screen on the shooting screen.

Vehicle ahead of the own vehicle in the traveling direction (leading vehicle)
To calculate the inter-vehicle distance between the vehicle and the preceding vehicle,
First, it is necessary to recognize the preceding vehicle, but it is very difficult to accurately distinguish the preceding vehicle from other image elements.
Usually, there is a shadow of its underside on the preceding vehicle, which constitutes a horizon as a set of pixels in the image that have a relatively darker gray level compared to its surroundings. So, for example,
The shadow can be recognized by detecting the horizontal line from the set of characteristic points (points on the outline of the image) obtained by scanning the vicinity of the center of the screen in the vertical direction, and the preceding vehicle can be accurately recognized based on the position of the shadow. You can. However, if the roll angle θr is large, the shadow of the preceding vehicle shown in the image is inclined with respect to the horizon, which increases the probability that it cannot be recognized. If the pan movement amount Xp is large, the probability that a vehicle or an object in a direction deviating from the traveling direction of the own vehicle is mistaken for a vehicle on the own vehicle lane (the lane in which the own vehicle is traveling) or the like. Becomes higher. In order to reduce this probability, the mounting angle of the camera 6b may be adjusted so that the roll angle θr and the pan movement amount Xp are both zero, but this adjustment is difficult and troublesome. Therefore, to some extent
The tilt angle θr and the pan movement amount Xp cannot be avoided.

Therefore, in this embodiment, "screen calibration" C
The roll angle θr and the pan movement amount Xp are adjusted to zero on the screen by the above. However, in order to calculate the adjustment allowance, the above-mentioned C
The processes of 1 to C4 are performed, and "roll angle and pan movement amount calculation" C5 described here is performed. The calculation formulas for the roll angle θr and the pan movement amount Xp are shown in the block in contact with the block C5 in FIG. The calculated roll angle θr is stored in the roll angle register θr, and the calculated pan movement amount Xp is stored in the pan movement amount register Xp.

C6-8. "Image correction" C6 to 8 (Fig. 4) and the next "corrected image memory initialization" C
6. By "image correction" including "image rotation parallel movement" C7 and "interpolation" C8, the floor of each pixel of the input data memory (one screen of gradation data storage area) of the image memory 5a. The address of the key data (x, y in (b) of FIG. 8) is
The address that has been rotated and translated corresponding to the roll angle θr and the pan movement amount Xp (the address of the screen coordinate system that represents the captured image at θr = 0, Xp = 0; FIG. 8B).
X ', y') of the input data memory and write it in the corrected image memory corresponding to the input data memory (the gradation data storage area for one screen of the image memory 5a) (C6, C7).

C6. "Corrected image memory initialization" C6 (FIG. 9) When such rotation and parallel movement are performed, pixels having no gradation data are generated in the corrected screen. In order to be able to recognize such pixels later, in the “correction image memory initialization” C6, all addresses of the correction image memory are
"-1" that does not exist in photographed image data (gradation data)
Write the data representing the. The details of the processing are shown in FIG.

C7. “Image rotation parallel movement” C7 (FIG. 10) This processing content is shown in FIG. In this case, the input data memory address (x, y); x = 0 to 511, y
= 0 to 511, θr = 0, Xp = 0, the address (x ', y') of the screen coordinate system (correction image memory) representing the captured image; x '= 0 to 511, y' = 0 To 511 (22 to 28, 30 to 33), and the gradation data at the address (x, y) of the input data memory is converted to the address (x, y) of the corrected image memory. Address (x ',
y ') (29). As a result, when the gradation data of the corrected image memory is displayed on the CRT 4b, an image as shown in FIG. 16 in which both the roll angle θr and the pan movement amount Xp shown in FIG. 8A are substantially zero is displayed. To be done. However, due to this "image rotation parallel movement", on the corrected image memory,
An address (pixel = blank pixel) where no gradation data exists may occur, in which case data indicating "-1" remains due to the "corrected image memory initialization" C6 described above. Will be.

C8. "Interpolation" C8 (a in FIG. 11 & FIG. 12) In "interpolation" C8, the average value of the gradation data of the four pixels surrounding the gradation data is assigned to the pixel having no gradation data. If any of the surrounding four pixels is a blank pixel, this allocation is not performed. The contents of this "interpolation" C8 are shown in FIG. 11 (a) and FIG. In this case, the addresses (pixels) of the corrected image memory are sequentially designated, and it is checked whether or not the designated pixel (pixel of interest) is a blank pixel (memory data indicates "-1") ( 36), if the pixel of interest is a blank pixel, as shown in (b) of FIG. -Check for data in this order,
If even one of these four pixels is a blank pixel,
Processing for the pixel of interest ends (leave blank pixel as is)
Then, the pixel of interest is moved to the next (39-42, 42-46, 4
6-50, 50-54). When it is checked whether each of the four pixels has gradation data, if there is gradation data, it is added to the contents of the accumulation register Is, and the obtained sum is updated in the accumulation register Is. Shi (40,44,48,5
2) The content of the frequency register Ns is incremented by 1 (41, 45, 49, 53). When this is completed for all of the four pixels, the average value Ic (x, y) of the gradation data of these pixels is calculated and stored in the target pixel (54, 55).

As described above, the "screen calibration" C (FIG. 3) is completed, and the CPU 1 then executes "own vehicle lane detection" D.

D. "Own vehicle lane detection" D (Fig. 13) This content is shown in Fig. 13. First, the "two sets of feature point detection windows" D1 is executed.

D1. "2 sets of feature point detection windows"
D1 (Fig. 14) This content is the "feature point detection window 1 set" described above.
Similar to C1 (FIG. 5), but the size and position of window 2 is different from window 1 and
The memory to which is applied is a corrected image memory. As shown in FIGS. 14 and 16, window 2 is window 1
Is set above.

D2. “Feature point detection (UP)” D2 When the window 2 is set, the feature point in the window 2 on the corrected image memory is detected. This content is the same as the above-mentioned "feature point detection (UP)" C2, and the "1" information representing the feature point also in the "feature point detection (UP)" D2 is the binary data table E (Xs, Write in Ys). However, the point that the detection area is the window 2 and the processing target data
The difference is that the data is data on the corrected image memory.

D3. "Left and Right White Line Detection" D3 (FIG. 15) Here, similar processing to the above-mentioned "bonnet detection" C3 is performed to detect a straight line approximating the right end white line and a straight line approximating the left end white line of the own vehicle lane. However, since the window 2 is different in size and position from the window 1, the polar coordinate origin is different (FIG. 16). Previous "bonnet detection" C3
Since a detailed description of the contents of the “straight line fitting” 13R and 13L is omitted, here, the “straight line fitting” 6
The contents of 3R will be described in detail.

63R. "Straight line fit" 63R (Figs. 17 & 18) This content is shown in Figs. 17 and 18. Note that the "straight line fit" 63R is applied to the right half of window 2. Here, first, the characteristic points on the binary data table E (Xs, Ys) are converted into a ρ-θ polar coordinate system, and the frequency count value that the curve of each point (r, t) of this coordinate system passes The data of the data table HGn (r, t) (here, n = 1) allocated to a certain area of the image memory 5a to be written is cleared (65 in FIG. 17; details are shown in FIG. 19). Next, Hough transform parameters (transformation area and Hough transform density) are set (66 in FIG. 17), and the first Hough transform (HG1) is performed (67 in FIG. 17; details are shown in FIG. 2).
0).

"Hough Transform (HG1)" 67 (FIG. 20) Referring to FIG. In this case, the right half of the area corresponding to the window 2 in the one screen corresponding to the corrected image memory of the binary data table E (Xs, Ys), that is, the starting point is (150, 255) and the ending point is (511). , 350), there is a feature point (“1” information) (87 to 89, 95 to 98). That is, YWU, YWL and XWC are respectively the Y coordinate (15
0), the bottom Y coordinate (350) and the center X coordinate (25
5).

Each time there is a characteristic point, the origin of the characteristic point is (Xch = 255, Ych = 350) on the XY coordinates, and θ is 0.
To π / 2, and the unit of θ is (π / 2) × (1 /
32) position (r, t) on the polar coordinate ρ-θ, t = 0 to 3
20, r is converted into a value calculated by the arithmetic expression shown in the block of step 91 of FIG.
The data of the address (n = 1 in this case) corresponding to the position (r, t) is incremented by 1 (90 to 94). As described above, when one point on the XY coordinate is converted into the polar coordinate ρ-θ, it becomes a curve on the polar coordinate ρ-θ, and thus the converted position (r, t) is t = 0 to 31. 32 (corresponding to a curve) corresponding to each of the above. Since the data of the addresses assigned to these positions (r, t) in the data table HGn is incremented by 1,
The data at each address represents the frequency at which the characteristic point on the XY coordinates passes through the curve on the ρ-θ coordinates.

In step 91 of FIG. 20, r = {(Xs-Xch) .cos [(t. (Te-Ts) / Td) + Ts] + (Ych-Ys) .sin [(t. (Te-Ts ) / Td) + Ts] −Rs} × Rd / (Re−Rs) (1) is calculated and the calculated value r is stored in the register r. When this equation (1) is modified, it becomes the following equation (1a).

R / Rd / (Re-Rs) + Rs = (Xs-Xch) .cos [(t. (Te-Ts) / Td) + Ts] + (Ych-Ys) .sin [(t. (Te- Ts) / Td) + Ts] (1a) This can be expressed by the following equation (1b).

R / d + c = (Xs−Xch) · cos (a · t + b) + (Ych−Ys) · sin (a · t + b) (1
b) where a = (Te-Ts) / Td, b = Ts, c = Rs, d = R
d / (Re-Rs).

In the equation (1), (Xch, Ych) is the position on the XY coordinate of the polar coordinate origin, and here Xch = 255 and Ych = 350. Xs is the X coordinate value of the feature point on the XY coordinate, Y
s is a Y coordinate value.

Here, if θ = [(t (Te-Ts) / Td) + Ts] is expressed, (Te-Ts) / Td = by the setting in step 62R of FIG. 15 and step 66 of FIG. Since (π / 2) / 32 and Ts = 0, θ = t · (π / 2) / 32 = (π / 64) t a = π / 64, b = 0, c = 0, d = 1 / 8 and t = 0 to 31, so θ = 0, θ = π / 6
4, θ = 2π / 64, θ = 3π / 64, ..., 31π
/ 64 and π / 64 are the minimum units, and the ρ value r of 32 points is calculated. That is, in the first Hough transform, each of the feature points in the right half region of the window 2 has a position (r, t).

Rd / (Re-Rs) in the above equation (1) is Rd / (Re-Rs) = 32/256 according to the settings in step 62R of FIG. 15 and step 66 of FIG. This is because θ = π / 64 is one unit of t, that is, π / 2 is divided into 32, and correspondingly, the range 0 to 256 of ρ is also divided into 32. . That is, one unit of r is 256/32.

In summary, "Hough transform (HG1)" 67
Then, the expression (1) is specifically the following expression (1-1).

R = {(Xs−255) · cos [(t · (π / 64)] + (350−Ys) · sin [(t · (π / 64)]} × 1/8 ( 1-1) By transforming this into the form of equation (1b), r / d ₁ + c ₁ = (Xs−Xch) ・ cos (a ₁・ t + b ₁ ) + (Ych−Ys) ・ sin (a ₁・t + b ₁ ) ... (1-1b), a ₁ = π / 64, b ₁ = 0, c ₁ = 0, d ₁ =
It is 1/8.

As described above, in the first Hough transform, the unit of θ in the ρ-θ polar coordinate is coarse as π / 64, and the unit of ρ is coarse as 256/32, so that one feature point (Xs , Ys) is converted to ρ−θ polar coordinates (the number of times t is switched 32, that is, the number of times the formula (1) is calculated).
Is extremely small, the number of r data bits is small, and each feature point is ρ
-The conversion speed to polar coordinates is fast, and the frequency count processing speed is fast.

Referring to FIG. 17 again, when the CPU 1 completes the first Hough transform “Hough transform (HG1)” 67, it executes “maximum point search (HG1)” 68.
The contents are shown in FIG.

"Maximum point search (HG1)" 68 (FIG. 21) The data (frequency) of each address (r, t) of the above-mentioned data table HGn (here, n = 1) is sequentially read (see FIG. 21). 99, 100, 105-108), the read data is compared with the data in the register Gm, and if the read data is larger, the read data is updated and stored in the register Gm, and at this time, the read data address (r, t )
Of r is updated in the register rm1 and t is updated in the register tm1 (101 to 104). When such processing is completed for all the addresses of the data table HGn, the register Gm has the maximum value of the frequency in the data table HGn, and the registers rm1 and tm1 have the maximum value (rm1, tm1). Will be stored.

Referring again to FIG. 17, the CPU 1
When the "maximum point search (HG1)" 68 is completed, the address (rm1, tm1) expressed in the rt polar coordinate system is changed to X.
-Y coordinate system address (R-θ polar coordinate system address (R
(69) to (m1, Tm1). Here, Rm1 = rm1 · (Re1−Rs1) / Rd1 (2) Tm1 = tm1 · (Te1−Ts1) / Td1 (3) is calculated, and the calculated Rm1 is stored in the register Rm1. The calculated Tm1 is stored in the register Tm1. Step 62 of FIG.
By R and the setting in step 66 of FIG. 17, here, (Re1−Rs1) = 256 Rd1 = 32 (Te1−Ts1) = π / 2 Td1 = 32, and the equation (2) is Rm1 = 8. means rm1 and formula (3) means Tm1 = (π / 64) tm1. Address (Rm
Substituting (1, Tm1) into the conversion formula to the XY coordinate system, a straight line (by the low-density Hough transform 67) in the right half area of the window 2 on the screen displaying the image data of the corrected image memory The detected straight line: hereinafter referred to as the first detection straight line) is obtained.

Next, the CPU 1 assigns a data table HGn (r, t) to a certain area of the image memory 5a,
Here, the data of n = 2 is cleared (7 in FIG. 17).
0). Next, the second Hough transform, "Hough transform (HG
2) ”72 conversion parameters are set (7 in FIG. 17).
1). In the above-mentioned first Hough transform, the range of θ from Ts to Te
Is 0 to π / 2, the unit (Te-Ts) / Td is π / 64, the range Rs to Re of ρ is 0 to 256, and the unit Rd / (R
e-Rs) is set to 1/8, but here, the range of θ and ρ is set to a small range centered on the above (Rm1, Tm1), and the units of θ and ρ are also set small. Specifically, Rs = 8 (rm1-2) Re = 8 (rm1 + 2) Rd = 32 Ts = (π / 64) (tm1-2) Te = (π / 64) (tm1 + 2) Td = 32 are set. . Note that rm1 = Rm1 / 8, tm1 = Tm1 / (π /
64). Then, "Hough transform (HG2)" 72 is performed.

"Hough Transform (HG2)" 72 The content of this "Hough Transform (HG2)" 72 is the same as that of the "Hough Transform (HG1)" 67 described with reference to FIG. , N = 2, and the content of the arithmetic expression in the block 91 (equation (1) above) is different. Step 71
Step 9 of FIG. 20 based on the parameters set in
The arithmetic expression of 1, that is, the expression (1) is specifically the following expression (1-2).

R = {(Xs−255) · cos [t · (π / 512) + π (tm1−2) / 64] + (350−Ys) · sin [t · (π / 512) + π (tm1− 2) / 64] −8 (rm1-2)} × (1/1) ・ ・ ・ (1-2) By transforming this into the form of equation (1b), r / d ₂ + c ₂ = (Xs− Xch) · cos (a ₂ · t + b ₂ ) + (Ych−Ys) · sin (a ₂ · t + b ₂ ) ... (1-2b), the first conversion described above yields a ₁ = (π / 64), b ₁ = 0, c ₁ = 0,
For d ₁ = 1/8, a ₂ = (π / 512), b ₂ = π (tm 1-2) / 64, c ₂ = 8 (rm 1-
2), d ₂ = 1.

Since t = 0 to 31, θ = π (tm1-2) / 64, θ = (π / 512) + π (tm1-2) / 64, θ = 2 (π / 512) + π (tm1 −2) / 64, θ = 3 (π / 512) + π (tm1−2) / 64, ··· θ = 31 (π / 512) + π (tm1−2) / 64, and (π / 512) Ρ corresponding to 32 points θ with the minimum unit
The value r is calculated. That is, in the second Hough transform, θ is (π / 51) for each feature point in the right half of the window 2.
2) is the minimum unit (1 unit of t is π / 512), and θ = π
It is converted to a position (r, t) on the polar coordinate plane in the range of (tm1−2) / 64 or more and θ = 31 (π / 512) + π (tm1−2) / 64 or less. The range of ρ is 8 (rm1-2) or more and 8 (rm1 + 2) or less, and the unit of ρ is to divide this range into 32 parts. That is, one unit of r is 32/32 = (1/1) = 1. Therefore, the second Hough transform (72) transforms the right half feature points of the window 2 into polar coordinates with a narrower range and higher density than the first Hough transform (67). In one polar coordinate conversion of the feature point, the second Hough transform (72) also performs t = 0 to 31, that is, 32 times of calculation, so that the processing time is about the same as that of the first Hough transform (67). The number of operations for converting one feature point (Xs, Ys) to ρ-θ polar coordinates is extremely small, the number of r data bits is small, the speed of converting each feature point to ρ-θ polar coordinates is high, and the frequency is The count processing speed is fast.

Referring to FIG. 17 again, when the CPU 1 finishes the second Hough transform "Hough transform (HG2)" 72, it executes "maximum point search (HG2)" 73.
The content is the same as that of the “maximum point search (HG3)” 68 described above. However, n = 2. When this process ends, the maximum value of the frequency in the data table HGn is stored in the register Gm, and the address (rm2, tm2) having the maximum value is stored in the registers rm2 and tm2.

Referring again to FIG. 17, the CPU 1
When the "maximum point search (HG2)" 73 is completed, the address (rm2, tm2) expressed in the rt polar coordinate system is changed to X.
-Y coordinate system address (R-θ polar coordinate system address (R
m2, Tm2) (74). The calculation formula is Rm2 = rm2 (Re2-Rs2) / Rd2 + Rs2 (4) Tm2 = tm2 (Te2-Ts2) / Td2 + Ts2 (5) The calculated Rm2 is stored in the register Rm2 and the calculated Tm2 is stored.
Is stored in the register Tm2 (74). According to the setting in step 71 of FIG. 17, here, (Re2-Rs2) = 32 Rd2 = 32 (Te2-Ts2) = [pi] / 16 Td2 = 32, and the equation (4) is, specifically, Rm2 = Rm2 + 8rm1-16 (4-1), and the equation (5) is specifically expressed as: Tm2 = tm2 · (π / 512) + (π / 64) · (tm1-2) Means (5-1). By substituting the address (Rm2, Tm2) into the conversion formula to the XY coordinate system, a certain straight line (medium density Hough transform) in the right half area of the window 2 on the screen displaying the image data of the corrected image memory is displayed. The straight line detected by 72: hereinafter referred to as the second detection straight line) is obtained.

Next, the CPU 1 assigns a data table HGn (r, t) to a certain area of the image memory 5a.
The data of (here, n = 3) is cleared (7 in FIG. 18).
5). Next, the third Hough transform, "Hough Transform (HG
3) ”77 conversion parameters are set (7 in FIG. 18).
6). Here, the range of θ and ρ is set to a smaller range centered on the above (Rm2, Tm2) than the case of the “Hough transform (HG2)” 72, and the unit of θ and ρ is also set smaller. (76). Specifically, Rs = rm2 + 8rm1-18 Re = rm2 + 8rm1-14 Rd = 32 Ts = (π / 512) tm2 + (π / 64) tm1-9π / 256 Te = (π / 512) tm2 + (π / 64) tm1 Set -7π / 256 Td = 32. It should be noted that rm2 and tm2 are defined by the above equations (4-1) and (5-1) with respect to Rm2 and Tm2. Then, "Hough transform (HG3)" 77 is performed.

"Hough transform (HG3)" 77 The content of this "Hough transform (HG3)" 77 is the same as that of the above-mentioned "Hough transform (HG2)" 72, but the Hough transform process shown in FIG. The point that n = 3 is different from the content of the arithmetic expression (equation (1) above) in the block 91 in FIG. In the “Hough transform (HG3)” 77, the arithmetic expression in step 91 of FIG. 20, that is, the expression (1) based on the parameters set in step 76 is specifically expressed as the following expression (1-3). Become. r = {(Xs-255) ・ cos [t ・ (π / 4096) ＋ (π / 512) tm2 ＋ (π / 64) tm1-9π / 256] + (350-Ys) ・ sin [t ・ (π / 4096) + (π / 512) tm2 + (π / 64) tm1-9π / 256] −rm2−8rm1 + 18} × 8 ・ ・ ・ (1-3) By transforming this into the form of equation (1b), r / expressed in _{d 3 + c 3 = (Xs} -Xch) · cos (a 3 · t + b 3) + (Ych-Ys) · sin (a 3 · t + b 3) ··· (1-3b), the previously described 1 A ₁ = (π / 64), b ₁ = 0, c ₁ = 0,
d ₁ = 1/8 a ₂ = (π / 512), b ₂ = π (tm1-2) / 64, c ₂ = 8 (rm1−
2), d ₂ = 1 and a ₃ = (π / 4096), b ₃ = (π / 512) tm 2 + (π / 6
_{4) tm1-9π / 256, c 3} = rm2 + 8rm1-18, a d ₃ = 8.

Since t = 0 to 31, θ = (π / 512) tm2 + (π / 64) tm1-9π / 256, θ = (π / 4096) + (π / 512) tm2 + (π / 64) tm1-9π / 256, θ = 2 (π / 4096) + (π / 512) tm2 + (π / 64) tm1-9π / 256, θ = 3 (π / 4096) + (π / 512) tm2 + (π / 64) tm1-9π / 256, ··· θ = 31 (π / 4096) + (π / 512) tm2 + (π / 64) tm1-9π / 256, and (π / 4096) as the minimum unit, 32 points Θ (t = 0-
31) is calculated corresponding to ρ value r. That is, in the third Hough transform, each of the feature points in the right half region of the window 2 has θ of (π / 4096) as the minimum unit (one unit of t is π / 40).
96) and θ = (π / 512) tm2 + (π / 64) tm1-9π / 2
56 or more θ = 31 (π / 4096) ＋ (π / 512) tm2 ＋ (π / 64) tm1
Position (r, t) on the polar coordinate plane in the range of -9π / 256 or less
Is converted to. The range of ρ is Rs = rm2 + 8rm1-18 or more
Re = rm2 + 8rm1−14 or less, and the unit of ρ is to divide this range into 32. That is, one unit of r is 4/3
2 = (1/8). Therefore, the third Hough transform (77)
Is the second Hough transform (7
It is to convert to polar coordinates with a narrower range and higher density than 2). In one polar coordinate transformation of the feature point, t = 0 to 31 in this third Hough transformation (77) as well, so 3
Since the calculation is performed twice, the processing time is the same as that of the second Hough transform (72). Ρ for one feature point (Xs, Ys)
The number of operations for converting to −θ polar coordinates is extremely small, the number of r data bits is small, the speed of converting each feature point to ρ−θ polar coordinates is high, and the frequency count processing speed is high.

Referring to FIG. 18, when the CPU 1 finishes the "Hough transform (HG3)" 77 which is the third Hough transform, it executes the "maximum point search (HG3)" 73. The content is the same as that of the “maximum point search (HG3)” 68 described above. However, n = 3. When this process is completed, the register Gm has the maximum frequency value in the data table HGn, and the registers rm3 and tm3 have the maximum value.
This means that (rm3, tm3) is stored.

Referring to FIG. 18, when the "maximum point search (HG3)" 78 ends, the CPU 1 transfers the address (rm3, tm3) expressed in the rt polar coordinate system to the XY coordinate system screen. Corresponding ρ-θ polar coordinate system address (Rm3, T
m3) (74). The calculation formula is Rm3 = rm3 (Re3-Rs3) / Rd3 + Rs3 (6) Tm3 = Tm3 (Te3-Ts3) / Td3 + Ts3 (7) The calculated Rm3 is stored in the register Rm3 and the calculated Tm3 is stored.
Is stored in the register Tm3 (79). By the setting in step 76 of FIG. 18, here, (Re3-Rs3) = 4 Rd3 = 32 (Te3-Ts3) = π / 128 Td3 = 32, and the equation (6) is concretely Rm3 = (1/8) · rm3 + rm2 + 8rm1−18 (6-1), and the formula (7) is expressed specifically as Tm3 = (π / 128) · tm3 + (π / 512) tm2 + (π / 64 ) tm1-9π / 256 ... (7-1) is meant. By substituting the address (Rm3, Tm3) into the conversion formula to the XY coordinate system, a certain straight line (high density Hough transform) in the right half area of the window 2 on the screen displaying the image data of the corrected image memory is displayed. The straight line detected by 77: hereinafter referred to as the third detection straight line) is obtained. This is the end of the "straight line fitting" 63R (the contents are FIGS. 17 and 18) shown in FIG.

As described above, the straight line (third most representative straight line) in the image in the right half area of the window 2 (third line)
This means that we have obtained the line of detection). Window 2 is set to the most suitable area for detecting the left and right white lines of the vehicle lane. If the right half white line image is in the right half area of window 2, the straight line for the third detection is There is a high probability that it is a straight line that approximates the white line. Therefore, the “straight line fitting” 63R is the detection of the right end white line of the vehicle lane in the right half area of the window 2.

Referring again to FIG. 15, the CPU 1 then stores Rm3 and Tm3 of the data (Rm3, Tm3) representing the straight line of the third detection in the register RmR and the register TmR.
(64R). That is, data representing a straight line representing the right end white line of the vehicle lane is stored in the registers RmR and TmR (6
4R).

Left white line detection 62L to 64L (FIG. 15) Next, the CPU 1 performs the same processing as the processing (62R to 64R) for the right half area of the window 2 described above.
A straight line representing the left end white line of the vehicle lane is detected by applying it to the left half area of the window 2 and data representing it is stored in the registers RmL and TmL (62L to 64L). Thus, the "left and right white line detection" D3 is completed. CPU1 then
Perform "infinity point calculation" D4. This content is shown in FIG.

D4. "Point calculation at infinity" D4 (Fig. 22) First, in "Line intersection calculation" 109, register R
An intersection (Xc, Xc, between a straight line (estimated as the right end white line of the own vehicle lane) represented by mR and TmR data and a straight line (estimated as the left end white line of the own vehicle lane) represented by data of registers RmL and TmL
Yc) is calculated. Next, the calculated intersection (Xc, Yc)
Exists in a region of 60 pixels in the horizontal direction and 60 pixels in the vertical direction centered on the point at infinity (Xv, Yv) obtained by weighting smoothing (averaging) the intersection data calculated in the past in time series. Is checked (110, 111). Within this area, the intersection point (Xc, Yc) obtained this time is highly reliable at the point at infinity, so the infinity point data (Xv, Yv) is set to 1 at the intersection point (Xc, Yc) obtained this time. / 8 weighting,
The infinite point data (Xv, Yv) so far is weighted by 7/8 and updated to a value added (112). Then, the register Nve for counting the number of times the tracking of the point at infinity has failed is cleared (113).

If the intersection point (Xc, Yc) calculated this time is not within the area of 60 pixels in the horizontal direction and 60 pixels in the vertical direction, the tracking of the point at infinity is unsuccessful (the calculation of the intersection point at this time is an error or the infinite point data so far). Assuming that (Xv, Yv) is an error, the content of the register Nve is incremented by 1 (11
4) It is checked whether the content of the register Nve has reached 5 (115). When it is 5, this means that there was an error in the intersection calculation this time and the past four times, a total of 5 times in succession. This is because the infinity point data (Xv, Yv) currently held is an error. Infinite point data (Xv, Y
v) is updated to the intersection (Xc, Yc) calculated this time (1
16).

D5. "White line spacing (WL) calculation" D5
(FIGS. 13 & 23) Referring again to FIG. 13, the CPU 1 next calculates the interval (lane width) WL between the right white line and the left white line of the vehicle lane (D5). In this case, the rightmost white line (RmR, TmR) on the screen at the position where the center line of the field of view of the camera 6b (two-dot chain line in FIG. 23) intersects the road surface (the center point of the screen on the screen)
The X position of is converted to the road position XR, and the X position of the left end white line (RmL, TmL) on the screen is converted to the road position XL.
The interval WL = XR-XL is calculated. Note that Sx and Sy in the operation block D5 of FIG.
Horizontal and vertical scale factors of Hc
Is the height of the center of the lens 9 of the camera 6b from the road surface, as shown in FIG.

Next, the CPU 1 checks whether the calculated lane interval WL on the road surface is correct (success in detecting the own vehicle lane) (D6). That is, it is checked whether the deviation of WL from the reference value WL3D is within the allowable range DwL. In this example, the highway lane width of Japan is 3.5 ± 0.2.
m, the reference value WL3D = 3.5 m, the allowable value DwL
= 0.3m.

If the deviation of WL with respect to the reference value WL3D is within the allowable range DwL, it is determined that the own vehicle lane has been successfully detected, and the information "1" indicating this is written in the register FL (61) to the reference value WL3D. The deviation of WL is the allowable range D
If it is out of wL, it is determined that the vehicle lane detection has failed, and information "0" indicating this is written in the register FL (62). As described above, the “vehicle lane detection” D shown in FIG. 3 is completed, and when the vehicle lane detection is successful, the data in the register FL is “1”.

Referring again to FIG. "Own vehicle lane detection"
When D is finished, the CPU 1 checks the data in the register FL and if it is “1”, “adjacent lane estimation”
Execute F. This content is shown in FIG.

F. “Adjacent lane estimation” F (FIGS. 24 & 25) Referring to FIG. 24, first, the downward angle α of the camera 6b is calculated (117). In the calculation formula of step 117, Yo is the Y coordinate value of the screen center, and Ych is “screen calibration”.
It is the Y coordinate value of the lower edge of the screen defined in “Calculation of straight line intersection” C4. Next, the angle φR formed by the right end white line and the angle φL formed by the left end white line of the vehicle lane with respect to the traveling direction of the vehicle are calculated, and the average value φ thereof is calculated. On the screen (image in the corrected image memory) that has completed the “screen calibration” C, the previous “screen calibration” C indicates that the vehicle traveling direction is aligned with the center line (Y = 255) that bisects the screen left and right. Therefore, φR is the angle at which the right end white line intersects with this line, and φL is the angle at which the left end white line intersects. As shown in FIG. 25, the straight line of the angles of the average values φ is the center line of the own vehicle lane. In the next step 119, formulas for calculating the distance XR of the right end white line and the distance XL of the left end white line to the point where the center line of the field of view of the camera 6b intersects the road surface are shown. XR and XL
Is calculated in "calculation of the space between the left and right white lines (WL)" D5, and the calculated value is used. Next, regardless of the presence or absence of the right end white line of the right side lane (right adjacent lane) of the own vehicle lane right end white line, it is assumed that this exists, and the point of XR is laneed to the right in the horizontal direction. Assuming that the straight line connecting the point having the width WL (calculated in D5) and the point at infinity is the right end white line of the right lane, the value Aex1 of the proportional term coefficient A of the formula Y = AX + B representing the straight line (Ta of the angle TmRR formed with the horizontal line (X axis)
Then, the value Bex1 of the constant term B is calculated (n value) (12)
0). Similarly, the value Aex representing the leftmost white line of the left lane
Calculate 2 and Bex2 (121). Then, these values Aex1, Bex1 and Aex2, Bex2 are converted into ρ-θ polar coordinates (122, 123). As described above, two straight lines defining the left adjacent lane and the right adjacent lane are additionally determined in addition to the white lines (two straight lines) at both ends of the own vehicle traveling lane.

Referring again to FIG. "Adjacent lane estimation"
When F is finished, the CPU 1 executes "curve estimation" G. This content is shown in FIG.

G. "Curve estimation" G (Figs. 26 & 27) On the screen (image of the correction image memory) after "screen calibration" C, the vertical axis of the vehicle is substantially the center line of the screen (X = 2).
55) and the own vehicle lane is a straight line, the point at infinity (Xv, Yv) is substantially on the center line (Xv).
= 255). However, since the point at infinity (Xv, Yv) is the intersection of the extension lines of the left and right white lines in front of the vehicle,
If the center is curved, the point at infinity (Xv, Yv) is to the right (in the case of the right curve) or left (in the case of the left curve) of the center line X = 255 of the screen. Become. Therefore, in the "curve estimation" G, therefore, the outline is as shown in FIG. The distant point of the own vehicle lane (the point where the white lines at the left and right ends of the lane can be seen) is estimated. By the way, since the own vehicle shakes due to driving conditions, road surface conditions, and the like, and the posture of the vehicle body changes, when looking forward from the camera 6b, the front scene causes vertical vibration or rolling,
In addition, since the vehicle traveling direction (vertical axis of the vehicle) may temporarily deviate from the parallel to the own vehicle lane, if the momentary vehicle traveling direction (center line of the screen) is used as the reference line, -There is a possibility that erroneous detection will occur. Therefore, in this embodiment, the curve is estimated with the Y-axis parallel line passing through the time-series average value Xvs of the detected X-coordinate value Xv of the infinite point (Xv, Yv) as the reference line. The details will be described below.

Here, first, the value obtained by subtracting the X coordinate value Xv of the point at infinity obtained in the above "calculation of point at infinity" D4 from the sum of the contents of the deviation accumulation register Sxv and the average value register Xvs Update memory is stored in the accumulation register Sxv (12
4). Next, the content of the count register Nxv is incremented by 1 (125), and it is checked whether the content reaches 3000 (126).
The value obtained by subtracting Sxv / Nxv from the contents of is updated and stored in the register Xvs, and the registers Nxv and Sxv are cleared (128).

By this processing, the "curve estimation" G is set to 30.
Each time it is executed 00 times, the content of the register Xvs is updated to the value obtained by subtracting Sxv / 3000 from the value at that time, and the registers Nxv and Sxv are cleared. As a result, the data of the register Xvs is 3000 times "infinity point calculation" (latest 1 section) which has been backward climbed from the present calculation time point to the past, and 3000 times "infinity" which has been backward climbed immediately before. Far point calculation ”(previous 1 section), Xv average value (previous average value) Xvs is updated to a value obtained by subtracting the average value of deviations of Xv from the average value Xvs during the“ latest 1 section ”. It That is, the register Xvs stores the calculated time-series weighted average value of Xv.

Each time the "curve estimation" G is entered once, the deviation of the latest calculated Xv with respect to Xvs (curve judgment reference value) is calculated, and a coefficient determined based on experiments is calculated. Calculate the value ΔXL by multiplying KL (for example, 4.7) (1
29), the value Xvc obtained by adding Xv to this is stored in the register Xvc (130). As described above, the far point of the curve (the point where the left and right white lines intersect each other) is determined to be (Xvc, Yv).

Referring again to FIG. When the "curve estimation" G is completed, the CPU 1 executes "own vehicle lane preceding vehicle recognition and distance measurement" H. This content is shown in FIG.

H. "Self-vehicle lane preceding vehicle recognition and distance measurement"
H (FIGS. 28 & 29 to 46) Here, "detection 1 of candidate vehicle position 1" H1 is first executed. This content is shown in FIG.

H1. “Detection 1 of vehicle candidate position” H1 (FIG. 29) First, in “feature point memory clear” 131, a multi-gradation data of the number of pixels corresponding to the number of pixels (addresses) in the corrected image memory is displayed.
Data memory D (x, y) and binary data memory E (x,
y) is cleared. Details of the processing are shown in FIG. Next, the "detection of horizontal feature point" 132 is executed. This content is shown in FIG.

[Detection of Horizontal Feature Point] 132 Referring to FIGS. 31 and 27, here, a horizontal line of Y = 350 (X passing Y = 350 on the screen shown in FIG. 27 is displayed.
(Xv, Yv), (Xvc, Y) in a triangle whose base is the (axis parallel line) and whose vertex is the intersection of the left and right white lines of the vehicle lane.
v) and the area enclosed by (Xvc, Yv), a triangle surrounded by three points of the Y-axis parallel line extending in the direction in which the Y-coordinate value increases and the left end white line or the right end white line (Figure) 27
Of the area marked with diagonal lines = the vehicle lane area),
By defining the left half region divided by the Y-axis parallel line where X = Xvc, the inside of the left half region is scanned from the bottom to the top in the Y direction, and this scanning is repeated while updating the X position (see FIG. 31 146L to 150L, 152L to 153L), the Y-direction differential value of the image data on the corrected image memory is calculated, and the point where the differential value is greater than or equal to the threshold value is detected as a feature point (151L).
In other words, "1 line horizontal feature point detection" 151L is executed. This content is shown in FIG. This processing 151L is the same as the "feature point detection (UP)" C2 shown in FIG. 6, and thus detailed description thereof will be omitted.

Next, the right half region of the own vehicle lane region divided by the Y-axis parallel line X = Xvc is defined, and the inside of the right half region (including X = Xvc) is defined as follows. Scan from bottom to top in the Y direction, and repeat this scan while updating the X position (146R to 150R, 152R to 153R in FIG. 31) to calculate the Y direction differential value of the image data on the corrected image memory. ,
Points whose differential value is greater than or equal to the threshold value are detected as feature points (151).
R).

As described above, the characteristic points in the own vehicle lane area are detected, and the binary data memory E (x,
"1" indicating that the point is a feature point is written in the address corresponding to the feature point in y). In addition, in the above-mentioned "feature point memory clear" 131, the binary data memory E
Clear (x, y) in advance (“0” for all addresses)
Writing), so on the binary data memory E (x, y),
"1" (which is a feature point) is written only in the address corresponding to the feature point in the own vehicle lane area.

Next, the CPU 1 clears the histogram memory (133 in FIG. 29). This content is shown in FIG. The CPU 1 then performs “y-direction histogram creation” 13
Do 4. This content is shown in FIG.

“Y-direction histogram creation” 134 (FIG. 34) Here, Y = 0 on the binary data memory E (x, y).
Up to Y = 511 X-axis parallel lines (X scanning lines; a series of pixels in the X direction), the number of "1" (characteristic points) on the line is counted, and the histogram memory H
The count value is written in (y) in correspondence with the scanning line (Y address).

[Detection of maximum point in y-direction] 135 (FIG. 35) From the address of Y = 511 (X-axis scanning line of Y = 511) of the histogram memory H (y) to Y = 1 in sequence, each address is a target line. (Y), the characteristic point count value H (y) of the attention line (y), the characteristic point count value H (y-1) of the scanning line (y-1) whose Y coordinate value is one smaller than that, Also, the feature point count values H (y + 1) of the scanning line (y + 1) whose Y coordinate value is one larger than the line of interest (y) are compared (174 and 175 in FIG. 35), and H (y) is H (y). -1)
And H (y + 1), that is, if the attention line (y) is the peak position of the number of feature points, the left end point of the own vehicle lane at the position (Y address) of the attention line (y). (X
1) and the right end point (X2) are calculated (176, 177; the calculation formula is shown in FIG. 36). Then, the threshold Th2 is set to (X2
If -X1) / 8 is set (178) and the feature point count value H (y) of the attention line (y) is greater than or equal to the threshold Th2, the image contour extending in the horizontal direction on the attention line (y). Assuming that there is a line, the address (Y coordinate value) of the line of interest (y) is written to the address N of the memory My (N) (180). Then, the register N is incremented by 1. When the value of the register N becomes 5 or more (the image contour line is detected by 6 different scanning lines), or when the above-mentioned processing with the last scanning line Y = 1 as the line of interest is completed, the "maximum inspection in the y direction is detected". Out "13
Finish 5

As shown in FIG. 16, when there is a vehicle (leading vehicle) in front of the own vehicle lane, a rear image of the leading vehicle appears in the window 2, and the rear image shows a bumper (the lower edge of It appears as a horizontal line with a particularly high contrast), number
Since there are many image contour lines extending in the lateral (X) direction, such as a plate, a brake lamp panel, a rear window, a roof, etc., it is close to the host vehicle at addresses 0 to 5 of the memory My (N). The positions (Y coordinates) of the six image contour lines (having a large Y coordinate value) are written.

In step 178, the threshold value Th2
Is set to (X2-X1) / 8, the closer to the camera 6b, the higher the contrast of the image contour line and the more feature points are extracted. However, the contrast of the far image contour line is low, and the feature point is extracted. This is because the distance from the camera 6b of the image contour on the line of interest (y) is estimated from (X2-X1) and the threshold value is changed according to the distance.

[Calculation of Center of Gravity in X Direction] 136 (FIG. 37) Next, for each position (Y coordinate value) of the image contour line written in the memory My (N), W1 (setting value) smaller than the position. W2 (setting value) The X coordinate value of the characteristic point on the binary data memory E (x, y) (the characteristic point in the own vehicle lane area shaded in FIG. 27) in the large area is accumulated. The number of additions is counted in the register Nx (187 to 197), the X-coordinate value of the center of gravity (average coordinate value) is calculated, and written in the memory Mx (N) (198).
As a result, the “y-direction maximum point detection” 135 detects
Center-of-gravity position in the X direction of each image contour line extending in the lateral direction (X
The coordinate value) exists in the memory Mx (N). The Y-direction center-of-gravity position exists in the memory My (N).

With the above, the "detection of candidate vehicle position" H1 shown in FIG. 28 (details are shown in FIG. 29) is finished, and the CPU 1 next clears the register NR to be used later (H2),
"Detection of left and right ends of vehicle" H3 is executed. This content is shown in Figure 3.
8 shows.

H3. "Detection of left and right ends of vehicle" H3 (Fig. 38) First, "estimation of left and right ends of vehicle" 201 is executed. The contents are shown in FIG.

"Estimation of vehicle left and right end positions" 201 (FIG. 39) First, the one having the largest Y coordinate value of the image contour line in the memory My (N) My (NR) (address of N = NR = 0) Data), that is, the Y position of the image contour line (horizontal line) of the front object closest to the vehicle, and the horizontal distance (distance on the road surface) L1 from the camera 6b to that position is calculated (2).
08). Next, it is assumed that the front object is a vehicle (the correctness of this assumption is verified by "vehicle verification" H4 described later),
The vehicle width Wv2Dp and vehicle height Hv2Dp on the screen are calculated (20
9). The Wv3D used for this calculation is the data of the register Wv3D (which will be described later with reference to FIG. 46) that stores the vehicle width learning value (the vehicle width of the actual vehicle that is the most typical vehicle that has not detected the preceding vehicle in the past). Hv3D has a vehicle height of 1.2 m, which covers even mini cars. Next, the vehicle width Wv2Dp on the screen
And the barycentric position Mx of the image contour line in the memory Mx (N)
From (NR), the left and right end positions XvLp, X on the screen of the front object
Estimate vRp (210). Then, the upper and lower ends yvHp and yvLp on the screen are estimated and calculated (211).

Next, the multi-gradation data memory D (x, y) and the binary data memory E (x, y) are cleared (202).
L), "Left edge vertical feature point detection" 203L is executed. This content is shown in FIG.

“Left edge vertical feature point detection” 203L (FIG. 40) Corrected image memory in an area of ± Wv2Dp / 4 centered on the left edge position XvLp on the screen between the upper and lower edges yvHp and yvLp on the screen. The differential value of the above image data in the Y direction is calculated and written in the multi-gradation data memory D (x, y), and the multi-gradation data is compared with the threshold value Th3 to be Th3 or more. Then, "1" is written in the binary data memory E (x, y).
This feature point detection processing is the same as the "feature point detection (UP)" C2 processing described above. Next, the CPU 1 clears the histogram memory (H (y) used previously) (204L), and "creates an x-direction histogram" 205L
To execute. This content is shown in FIG.

“X-direction histogram creation” 205L (FIG. 41) The feature points on the Y-axis parallel line (consecution of pixels) in the area where the feature points are detected by the “left-end vertical feature point detection” 203L are displayed in line units. Are counted and written in the histogram memory (herein denoted as H (x)) in correspondence with the line address (X coordinate value). Then, "detection of maximum point in x direction" 206L is executed. The contents are shown in FIG.

"Maximum point detection in x direction" 206L (FIG. 42) This processing is basically the same as the above-described "detection of maximum point in y direction" 135, but with the Y axis replaced with the X axis.
This is performed in the area where the feature point is detected in "left edge vertical feature point detection" 203L. Specifically, since the area is narrow here, the threshold value Th4 for extracting the maximum point is Hv2Dp.
In addition to the fixed value of / 4, the feature point count number of the maximum point is also stored in the maximum value memory Hm every time the maximum point is detected.
If the maximum point detected this time is larger than the data of No. 3, the data is updated and written in the maximum value memory Hm, and the X coordinate value is written in the maximum point register Xm. When this process is completed for all the regions, the X coordinate value of the Y-axis parallel line (left edge of the preceding vehicle) having the most feature points in the region is stored in the register Xm. . This X coordinate value is written in the vehicle left end position register XvL (207
L).

"Characteristic point memory clear" 202L-
A process similar to “maximum point detection in x direction” 206L is performed for a region of ± Wv2Dp / 4 around the right end position XvRp between the upper and lower ends yvHp and yvLp on the screen (20 in FIG. 38).
2R to 206R), the X coordinate value of the Y-axis parallel line (right edge of the preceding vehicle) having the most feature points in the area is written in the vehicle right end position register XvR (207R). FIG. 43 shows the contents of “Detect right vertical feature point” 203R.

This is the end of the "detection of the left and right ends of the vehicle" H3. The CPU 1 then executes "vehicle verification" H3.
The contents are shown in FIG.

H4. "Vehicle verification" H4 (Fig. 44) Here, the register Wv3D (Fig. 4) that stores the vehicle width learning value is stored.
The vehicle width Wv3D of the preceding vehicle held by
Is converted into a vehicle width Wv2D on the screen by using the distance L1 of the front object from the own vehicle calculated in the "vehicle left and right end position estimation" 201 and the scale factor Sx of the camera 6b (247),
Vehicle width W on the screen from the above-mentioned vehicle left and right end positions XvL, XvR
v is calculated (248), and the difference Wv-Wv2D between them is ± Wv
It is checked whether it is within the range of 2D / 4 (249). That is, the vehicle width calculated value Wv on the screen is the vehicle width (actual width) Wv2D of the vehicle width of the register Wv3D storing the vehicle width learning value.
And check if it substantially matches. Information "1" / "0" indicating detection / non-detection of a preceding vehicle when the vehicle substantially matches
Write "1" to the register Fv that stores
a). If they do not match, the register Fv is cleared (250
b).

Next, when the CPU 1 writes "1" in the register Fv, the CPU 1 proceeds to "inter-vehicle distance calculation" H6 described later, but when "0" is written in the register Fv, the register NR is incremented by 1. (H5-H8-H9) or "Detect left and right ends of vehicle" H3. The "detection of the left and right ends of the vehicle" H3 is first executed from the lateral contour line of NR = 0 (closest to the own vehicle). Therefore, in the "detection of the left and right ends of the vehicle" H3 after the second time, NR = 1 , 2, ..., The second, third, ..., And the target lateral contour lines are sequentially changed to those farther from the vehicle. Since the maximum value of NR is 5 (only 5 horizontal contour lines are detected at N = 0 to 4 in FIG. 35), a maximum of 5 times "detection of left and right ends of vehicle" H3 is executed and Fv
If the value does not become "1", the process proceeds to "vehicle width learning calculation" H7 described later, and "vehicle distance calculation" H6 described below is not executed.

H6. "Inter-vehicle distance calculation" H6 (Fig. 45) When "1" is written in the register Fv, the inter-vehicle distance L2
(0) is calculated (251). The vehicle width (actual width) Wv3D of the register Wv3D that stores the vehicle width learning value, the vehicle width calculation value Wv on the screen, the horizontal distance L1 of the front object calculated from the “vehicle left and right end position estimation” 201 from the own vehicle (on the road Using the actual distance) and the scale factor Sx of the camera 6b, the inter-vehicle distance L2 (0) is calculated (251). This time calculated value L2
(0), the previously calculated value L2 (1), the two-previously calculated value L2
(2) and the two-previously calculated value L2 (3) is calculated and the average value is written to the register L3 (252), and the previous calculated value L2 is calculated.
(1), the two-preceding calculation value L2 (2) and the two-preceding calculation value L2 (3) are updated (253). As a result, the inter-vehicle distance (latest time-series average value) from the preceding vehicle in the own vehicle lane is stored in the register L3.

H7. "Vehicle width learning calculation" H7 (Fig. 46) CP is updated when the inter-vehicle distance in register L3 is updated as described above.
U1 converts the vehicle width Wv on the screen to the actual vehicle width Wva, and the data of the register Wv3D (the vehicle width learning value Wv3D held on the time axis until the current inter-vehicle distance calculation) and the current vehicle width The calculated value Wva is added with weighting of 47 to 1, and the value obtained by dividing the sum by 48, that is, the weighted average value is updated and stored in the register Wv3D (254, 255a, 255b). If the preceding vehicle of the own vehicle lane is not detected (Fv = "0") and the process goes to "vehicle width learning calculation" H7, register Wv3
In this embodiment, the vehicle width value 1.7 m of the most typical vehicle (passenger vehicle) is written in D (254, 256).

With the above, the "own vehicle lane preceding vehicle recognition and distance measurement" H is completed. Next, the CPU 1 executes "right adjacent lane preceding vehicle recognition and distance measurement" I. This content is shown in FIG.
Shown in. In the above-mentioned "left and right white line detection" D3, the data (TmR, RmR) representing the right end white line of the own vehicle lane and the data (TmL, RmL) representing the left end white line are stored in the register, and , "Adjacent lane estimation" F, data (TmRR, RmRR) representing the right end white line (estimation) of the right adjacent lane and data (TmRR, RmRR) representing the left end white line (estimation) of the left adjacent lane. Note that TmLL and RmLL) are stored in the register.

I. "Right adjacent lane preceding vehicle recognition and distance measurement" I (Fig. 47, Fig. 48, Fig. 49) Here, the lower side of the window 2 (Y = 350), the vehicle lane right end white line (TmR, RmR) and In the area (right adjacent lane area) surrounded by the white line (TmRR, RmRR) on the right end of the right adjacent lane, the above-mentioned "recognition and distance measurement of preceding vehicle lane preceding vehicle" is performed.
The process of H is similarly applied to detect an object in the right adjacent lane area in front of the own vehicle, and calculate the distance from the own vehicle.

J. "Left adjacent lane preceding vehicle recognition and distance measurement" J (Figs. 50, 51, 52) Also in this case, the lower side of the window 2 (Y = 350),
Vehicle lane Left edge white line (TmL, RmL) and adjacent left rail
In the same way, the above-mentioned "own vehicle lane preceding vehicle recognition and distance measurement" process H is applied to the area (left adjacent lane area) surrounded by the leftmost white line (TmLL, RmLL) of the vehicle. An object in front of the own vehicle in the lane area is detected, and the distance from the own vehicle is calculated.

As described above, the CPU 1 executes one-time "screen calibration" C to "left adjacent lane vehicle" based on the one-screen image data read by "image input" B shown in FIG. Recognition and distance measurement "K has been executed. Then, as described above, the image recognition result is output by "output" K. Then, "image input" B is executed again, and the same processing is performed. That is, “image input” B to “output” K shown in FIG. 3 are repeatedly executed.

In the above embodiment, the case where the front scene seen from the front window of the own vehicle is image-recognized has been shown. However, for example, an image data obtained by photographing the rear scene of the own vehicle is shown. The present invention can also be applied to the processing of data.

[0113]

As described above, in the present invention, as the preprocessing for image recognition, the inclination θr of the subject in the image is detected, and the coordinate axes of the image are rotated so that the inclination θr becomes 0. Fix it. Therefore, even when the TV camera is fixed in a state where the scanning axis of the TV camera (imaging means) is tilted with respect to the axis of the vehicle body, an actually horizontal object is not shown on the coordinates of the corrected image. It appears horizontal. Therefore, for example, the horizontal line of the preceding vehicle can be reliably detected as a line oriented in the horizontal direction in the image without performing highly accurate positioning adjustment when mounting the television camera.

There are various subjects such as the road surface, the preceding vehicle, and the sky in the scenery reflected on the television camera, but the positions in these images change as the vehicle moves (runs), The distance between the own vehicle and the preceding vehicle is also not constant. However, in the present invention, the installation position and the field of view of the TV camera are set so that a part of the vehicle body of the vehicle, for example, the hood (or the upper part of the windshield) is included in the image, and the image is displayed based on the inclination of the hood or the like in the image. The coordinates are corrected. That is, since the vehicle body and the TV camera are integrally fixed, the position and inclination of the vehicle body (bonnet, etc.) in the image do not change even when the vehicle is moving, and at any time,
It is possible to accurately detect the inclination of the mounting of the television camera and correct the image coordinates.

[Brief description of drawings]

FIG. 1 is a block diagram illustrating a front road surface white line detection system on a vehicle embodying the present invention in one aspect.

FIG. 2 is a schematic side view showing an arrangement position on the vehicle of the television camera 6b shown in FIG.

3 is a flowchart showing an outline of processing for detecting a white line on a road surface in front of the own vehicle and a preceding vehicle by a microprocessor (CPU) 1 shown in FIG. 1 based on image data taken by a television camera 6b. -To (main routine).

FIG. 4 is a flowchart (subroutine) showing the contents of “screen calibration” C shown in FIG.

5 (a) is a flow chart showing the contents of "1 set of feature point detection windows" C1 shown in FIG. 4, and FIG. 5 (b) is a plan view showing a shooting screen of the television camera 6b shown in FIG. is there.

6A is a “feature point detection (UP)” shown in FIG.
A flow chart showing the contents of C2, (b) is a target point pixel (x mark) and a reference point pixel (white square) when the differential value of the image data in the Y direction of the photographing screen is calculated. FIG.

7 (a) is a "bonnet detection" C3 shown in FIG.
2B is a plan view showing a straight line approximating the bonnet edge on the photographing screen.

8A is a plan view showing the roll angle and the pan movement amount calculated by “calculation of roll angle and pan movement amount” C5 shown in FIG. 4 on a photographing screen, and FIG. FIG. 9 is a plan view showing a conversion formula for converting a pixel address on a photographing screen by a “image rotation parallel movement” C7 shown in FIG. 4 into a pixel address on a correction screen which is rotated by a roll angle and translated by a pan movement amount. .

9 is a flowchart showing the contents of "corrected image memory initialization" C6 shown in FIG.

FIG. 10 is a flowchart showing the contents of “image rotation parallel movement” C7 shown in FIG.

11 is a flowchart showing a part of the contents of "interpolation" C8 shown in FIG.

12 is a flowchart showing the remaining part of the contents of "interpolation" C8 shown in FIG.

FIG. 13 is a flowchart (subroutine) showing the contents of “own vehicle lane detection” D shown in FIG.

14 (a) is a flowchart showing the contents of "two sets of feature point detection windows" D1 shown in FIG.
FIG. 6B is a plan view showing an area of the window 2 set on the correction screen.

FIG. 15 is a flowchart showing the contents of “left / right white line detection” D3 shown in FIG.

16 is a plan view showing an image of the correction screen and a window 2. FIG.

FIG. 17 is a flowchart showing a part of the content of “straight line fitting” 63R shown in FIG.

FIG. 18 is a flowchart showing the remaining part of the contents of “straight line fitting” 63R shown in FIG.

FIG. 19 shows “Hough grid clear (HG1)” 65 and “Hough grid clear (HG) shown in FIGS. 17 and 18.
2) ”70 and“ Hough Grid Clear (HG3) ”7
5 is a flow chart showing the contents of 5 in a general-purpose format, and n in FIG. 19 is 1, 2, or 3.

FIG. 20 shows the “Hough transform (H
G1) "67," Hough transform (HG2) "72 and" Hough transform (HG3) "77 are flow charts in a general-purpose format, and n in FIG. 20 is 1, 2 or 3 Is.

FIG. 21 is a flow chart showing the contents of “maximum point search (HG1)” 68, “maximum point search (HG2)” 73 and “maximum point search (HG3)” 78 shown in FIGS. 17 and 18 in a general-purpose format. It is a chart, and n in FIG. 21 is 1, 2, or 3.

22 is a flowchart showing the contents of "infinity point calculation" D4 shown in FIG.

FIG. 23 is a schematic side view showing the geometrical relationship between the lens and the image pickup element in the television camera 6b shown in FIG. 1 and a preceding vehicle ahead of the vehicle.

FIG. 24 is a flowchart showing the contents of “adjacent lane estimation” F shown in FIG.

FIG. 25 is a plan view showing the adjacent lanes estimated by the “adjacent lane estimation” F on the correction screen, which are filled with diagonal lines.

FIG. 26 is a flowchart showing the contents of “curve estimation” G shown in FIG.

FIG. 27 is the “vehicle lane detection” on the correction screen.
FIG. 3 is a plan view showing the vehicle lane area estimated by D and “curve estimation” G by hatching.

28 is a flowchart showing the contents of "recognition and distance measurement of preceding vehicle by own vehicle lane" H shown in FIG.

FIG. 29 shows “Detection of candidate vehicle position 1” H shown in FIG. 28.
It is a flowchart showing the contents of 1.

FIG. 30 is a “feature point memory clear” 13 shown in FIG. 29.
It is a flowchart showing the contents of 1.

[FIG. 31] “Horizontal feature point detection 1” 13 shown in FIG. 29
2 is a flowchart showing the contents of 2.

32 (a) is a flow chart showing the contents of "detection of 1-line horizontal feature point" 151L shown in FIG.
(B) is a pixel of interest (x) when calculating the differential value of the image data in the Y direction of the correction screen for the detection of horizontal feature points.
(Mark) and a reference pixel (white square).

FIG. 33 is a flowchart showing the contents of the “histogram memory clear” 133 shown in FIG. 29.

FIG. 34 “y-direction histogram creation” shown in FIG.
It is a flowchart showing the contents of 134.

FIG. 35 is a “y-direction histogram creation” shown in FIG. 29.
It is a flowchart showing the contents of 134.

FIG. 36 is a “lane left end point detection” 176 shown in FIG.
And a flow showing the contents of "lane right end point detection" 177.
It is a chart.

37 is a flowchart showing the contents of "calculation of center of gravity in x direction" 136 shown in FIG.

FIG. 38 is a flowchart showing the contents of “Detect left and right ends of vehicle” H3 shown in FIG. 28.

[FIG. 39] “Vehicle left and right end position estimation” 20 shown in FIG.
It is a flowchart showing the contents of 1.

[FIG. 40] “Left edge vertical feature point detection” 20 shown in FIG. 38
It is a flowchart showing the contents of 3L.

FIG. 41 “x-direction histogram creation” shown in FIG.
It is a flowchart showing the contents of 205L.

42 shows the "maximum point detection in x direction" 206 shown in FIG.
It is a flowchart showing the contents of L.

FIG. 43 shows “right end vertical feature point detection” 20 shown in FIG.
It is a flowchart showing the contents of 3R.

FIG. 44 is a flowchart showing the contents of “Vehicle verification” H4 shown in FIG.

FIG. 45 is a flowchart showing the contents of “vehicle distance calculation” H6 shown in FIG. 28.

FIG. 46 is a flowchart showing the contents of “vehicle width learning calculation” H7 shown in FIG.

47 is a flowchart showing the contents of "right adjacent lane vehicle recognition and distance measurement" I shown in FIG.

[FIG. 48] “Detection of vehicle candidate position 2” shown in FIG. 47
It is a flowchart showing the contents of I1.

[FIG. 49] “Horizontal feature point detection 2” 25 shown in FIG.
8 is a flowchart showing the contents of No. 8.

50 is a flowchart showing the contents of "Left adjacent lane vehicle recognition and distance measurement" J shown in FIG.

51. "Detection position 3 of vehicle candidate" shown in FIG.
It is a flowchart showing the contents of J1.

FIG. 52 shows “Horizontal feature point detection 3” 27 shown in FIG.
7 is a flowchart showing the contents of 7.

[Explanation of symbols]

1: Microprocessor (CPU) 2: ROM 3: RAM 4 to 6: Input / output port 4a: CRT driver 4b: CRT 5a: Image memory 6a: Camera controller 6b: Television camera 6c: A / D converter -Ta 7: Communication controller 8: Host microprocessor (CPU)

Claims (3)

[Claims] 1. An on-vehicle recognition target image data for photographing an image of the outside of the vehicle by an image pickup means fixed to a vehicle body and processing image data output from the image pickup means to recognize a subject in the image. In the pre-processing method of the data: the image pickup means is installed in a state where a part of the vehicle body of the vehicle appears in advance in a part of the image data, and the image data input from the image pickup means is processed, The vehicle body of the own vehicle in the image is recognized, the inclination θr in the image of the recognized vehicle body is detected, the image is rotated based on the detected inclination θr, and the coordinates of each position of the image are corrected. Preprocessing method for on-board recognition target image data. 2. An on-vehicle recognition target image data for photographing an image of the outside of the vehicle by an image pickup means fixed to a vehicle body and processing the image data output from the image pickup means to recognize a subject in the image. In the pre-processing method of the data: the image pickup means is installed in a state where a part of the vehicle body of the vehicle appears in advance in a part of the image data, and the image data input from the image pickup means is processed, The vehicle body of the own vehicle in the image is recognized, the tilt θr in the image of the recognized vehicle body, and the position shift amount Xp with respect to the reference position are detected, and the detected tilt θr and position shift amount Xp are detected.
On-vehicle recognition target image data, which is characterized by rotating and translating the image based on
Pretreatment method 3. A predetermined range of image data input from the image pickup means is vertically scanned at a plurality of horizontal coordinate positions different from each other, and a plurality of characteristic points whose brightness changes at least by a plurality of scanning lines are detected. A coordinate value group that forms a curve in the polar coordinate system is generated from the respective coordinates of the plurality of detected feature points by detecting and polar coordinate conversion, and the curve of the plurality of feature points passes through ρ, θ in a concentrated manner. Polar coordinate values on the polar coordinate system (ρm,
θm) is detected, a straight line in the X, Y orthogonal coordinate system corresponding to the detected polar coordinate value (ρm, θm) is associated with a part of the contour of the recognition target, and the inclination θr is obtained based on the inclination of the straight line. The preprocessing method of the on-vehicle recognition target image data according to claim 1 or 2.