JP3358247B2 - Road vanishing point calculation method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention calculates a road vanishing point, which is a point where a road disappears in an imaged screen, based on image information obtained by photographing the front or rear of a vehicle with a vehicle-mounted camera. About the method.

[0002]

2. Description of the Related Art Various techniques have been proposed for capturing images of the front and rear of a vehicle using a vehicle-mounted camera and recognizing an object in the captured image. For example, if an in-vehicle camera captures an image of the front of a vehicle, and the captured image can be image-processed to recognize another vehicle in front, the recognition result can be used for automatic driving to avoid an accident. That is, when another vehicle abnormally approaches, an accident can be automatically avoided by automatically operating the steering and the brake. If the road sign can be recognized, the information corresponding to the recognized road sign can be used for a so-called route guidance function in the navigation device.

When recognizing an image of an object captured by a vehicle-mounted camera, the position of the object on the screen changes every moment. As described above, when an image of a moving object on a screen is recognized every moment, in order to increase the recognition efficiency, it is necessary to accurately determine a camera posture parameter indicating the posture of the vehicle-mounted camera with respect to the road. That is, the behavior of the object on the screen can be estimated by accurately determining the camera posture parameter. Using this estimation result, it is possible to cut out an image portion including an object from the captured screen and perform image recognition processing on the cut out image.

When calculating the camera posture parameters, the coordinates of the road vanishing point, which is the point at which the road on which the vehicle is running disappears on the screen, are required. Since this road vanishing point corresponds to a point at infinity on the road, it is used as a reference when calculating the camera posture parameter. The road vanishing point can be detected as an intersection of road parallel lines such as a center white line drawn on the road. That is, if two or more straight lines on the screen are extracted and the intersection of the straight lines is obtained, the intersection can be set as a road vanishing point. When a straight line on the screen is extracted, for example, a large change in luminance is used at the edge of a white line. That is, the edge of the white line is detected as a straight line candidate point, and a straight line connecting the straight line candidate points is extracted as a road parallel line.

[0005]

However, it is rare that a road parallel line on a screen continues to a road vanishing point. That is, if there is another vehicle in front of the vehicle, the road center line and the like are hidden by the shadow of the vehicle, and thus are interrupted on the screen. Therefore, it is difficult to obtain a straight line connecting the straight line candidate points by an ordinary method such as the least square method.

Further, even when the road parallel lines are continuous to the road vanishing point on the screen, there is a possibility that an accurate straight line cannot always be found by an ordinary method such as the least square method. Therefore, there is a problem that it is difficult to obtain a road vanishing point that is accurate enough to calculate the camera posture parameter. By the way, conventionally, a navigation processing device is mounted on a vehicle and used for the purpose of supporting traveling by the vehicle. The navigation processing device
This is a device having a basic function of detecting a current position of a vehicle based on a traveling distance and a traveling direction and displaying the current position on a display device together with a road map.

[0007] Some of the data held by this navigation processing device have a considerable effect on the calculation of the road vanishing point. Nevertheless, in the prior art,
Data handled by the navigation processor is not being used effectively. Therefore, the calculation of the road vanishing point has not always been performed efficiently. Therefore, an object of the present invention is to solve the above-described technical problem and to provide a road vanishing point calculation method capable of reliably and accurately finding a road vanishing point.

[0008]

According to a first aspect of the present invention, there is provided a method for calculating a vanishing point of a road, comprising the steps of: A method for calculating a road vanishing point, which is a point where a road disappears in the vehicle, extracting coordinates of a straight line candidate point which is a point forming a straight line portion in a screen captured by the on-vehicle camera, The straight line candidate point belongs to the coordinates of the straight line candidate point.
The first H using the coefficient of the equation of the straight line as the first transformed coordinate
ough transform processing is performed to obtain a coordinate plane of the first transformed coordinates.
Each of the grid cells of a certain size set within the predetermined range of
The count value is obtained, and the case is calculated in the first Hough transform process.
A point at or near a point where the count value of a child cell is a certain value or more
If the total value of the counts of the grid cells within the fixed range is
Using only points and the straight line passing through this used point
The second Hough transform processing using the coefficients of the equation as the second transform coordinates
A predetermined range on the coordinate plane of the second transformed coordinates.
Calculate each count value of a grid cell of a fixed size set in
In the coordinate plane of the second transformed coordinate,
At the point where the count value of the
Find the point where the total value of the counts of the child cells is the maximum, and find this point
The vanishing point of the road is obtained based on the coordinates of

In this method, the road vanishing point is obtained by repeating the Hough transform process twice. Therefore, even when a straight line portion is interrupted in the screen, a road vanishing point can be obtained. In addition, unlike ordinary methods such as the least squares method, the road vanishing point can be obtained extremely accurately .

[0010] When a plurality of straight lines in the screen imaged by the on-vehicle camera intersect at the vanishing point on the road, the point at which the count value of the grid cell has the maximum value is one straight line on the coordinate plane of the first transformed coordinates. Line up. The coefficient of this straight line corresponds to the coordinates of the road vanishing point 1: 1. Therefore, if the second Hough processing is performed on the coordinate plane of the second transformed coordinates for the point where the count value of the grid cell is large, the road disappearance is calculated based on the coordinates of the point where the coefficient value of the grid cell is the maximum. Points can be determined.

According to a second aspect of the present invention, there is provided a method for calculating a vanishing point of a road,
On the coordinate plane of the first transformed coordinates, the point where the count value of the grid cells is equal to or more than a predetermined value, or the point where the total value of the count values of the grid cells in a predetermined range around the grid is equal to or more than the predetermined value, The method is characterized in that points in which the distance of the transformed coordinate on the coordinate plane is equal to or less than a predetermined value are classified as belonging to the same group.

Line candidate points belonging to the same straight line in the screen imaged by the on-vehicle camera gather near a certain point on the coordinate plane of the first transformed coordinates. Therefore, on the coordinate plane of the first transformed coordinates, if the points having a large count value of the grid cells are classified so that adjacent ones belong to the same group, a straight line candidate belonging to one straight line belongs to each group. A point corresponding to the point belongs. In this way, it is possible to easily find points used in the second Hough transform process, and to find points belonging to each group so as to find road parallel lines .

According to a third aspect of the present invention, there is provided a method for calculating a vanishing point of a road,
On the coordinate plane of the first transformed coordinates, a point at which the count value of the grid cells is equal to or more than a certain value, or a point at which the total value of the count values of the grid cells within a predetermined range around the grid is equal to or more than a certain value, Among the points belonging to the same group, the coordinates of a point where a straight line using the coordinates as a coefficient passes closest to the road vanishing point is obtained as a coefficient of a road parallel line.

In this manner, the coefficient of the road parallel line can be obtained using the points belonging to each group. It is possible to determine whether the road parallel line is on the left side or right side of the vehicle based on the sign of the obtained coefficient of the road parallel line related to the inclination of the straight line (claim 4 ). This is because, in the image taken by the onboard camera, the road parallel line on the right side of the vehicle and the road parallel line on the left side of the vehicle are:
Based on the empirical fact that they have different sign slopes.

When the coefficient of the road parallel line is only one of the positive or negative sign among the obtained coefficients of the road parallel line, the obtained road vanishing point and the road parallel line are obtained. Is preferably invalidated (claim 5 ). That is, in the above case, since only the road parallel lines existing on either side of the vehicle are obtained, the road vanishing point may not be obtained with high accuracy.

Further, on the coordinate plane of the first transformed coordinates, a point at which the count value of the grid cells is equal to or more than a predetermined value, or a total value of the count values of the grid cells within a predetermined surrounding area is equal to or more than the predetermined value. For groups that do not have more than a certain number of points,
The points belonging to the group is preferably not used in the second Hough transform process (claim 6). The above situation occurs when the number of straight line candidate points belonging to a certain straight line is small. If the coordinates of such a straight line candidate point are used, there is a possibility that the calculation accuracy of the road vanishing point may deteriorate.

Further, on the coordinate plane of the first transformed coordinates, the point where the count value of the grid cells is equal to or more than a predetermined value, or the total value of the count values of the grid cells in a predetermined surrounding area is equal to or more than the predetermined value. group of points, it is preferable to prohibit the calculation of the road vanishing point when less than two (claim 7). The case where the number of groups is less than two is a case where the number of straight lines to which the straight line candidate points belong is one or less. In this case, since the intersection of the straight lines cannot be obtained, the vanishing point of the road cannot be obtained. Therefore, if the calculation of the road vanishing point is prohibited when the number of groups is less than two, useless calculation processing can be omitted.

In the second Hough transform process,
Of the groups classified on the coordinate plane of the first transformed coordinates, only the point at which the count value of the grid cell is the maximum or the point at which the total value of the count values within a predetermined range around the grid cell is the maximum is used. (Claim 8 ). In the coordinate plane of the second transformed coordinates, when the maximum value of the count values of the grid cells is less than a certain value, or when the maximum value of the total value of the grid cells in a predetermined range is less than a certain value. If not, it is preferable to prohibit the calculation of the vanishing point of the road (claim 9 ).
In such a case, the calculation accuracy of the road vanishing point may be deteriorated.

According to a tenth aspect of the present invention, there is provided a method for calculating a road vanishing point, comprising acquiring at least information from a device having information on linearity of a road on which a vehicle is traveling, and determining whether or not the traveling road has a predetermined value or more. It is characterized in that it is determined whether or not the road has a linearity, and if the running road does not have a linearity equal to or more than a certain value, the calculation of the road vanishing point is prohibited.

In this method, information is acquired from a device having information on the linearity of the road on which the vehicle is traveling. A typical example of such a device is a navigation processing device. Based on the acquired information, it is determined whether or not the traveling road has a linearity equal to or more than a certain value. If the road does not have linearity equal to or more than a certain value, the calculation of the road vanishing point is prohibited. . If the vehicle is not traveling on a road having a certain degree of linearity, there is no road vanishing point on the screen. Therefore, according to the above method, it is possible to prevent the road vanishing point from being calculated needlessly.

According to a eleventh aspect of the present invention, there is provided a method for calculating a road vanishing point, wherein at least information obtained from a device having information on attributes of a road on which a vehicle is traveling is obtained, and the attribute of the road on which the vehicle is traveling is constant. It is characterized in that it is determined whether or not a criterion is satisfied, and when a running road does not satisfy a certain criterion, calculation of a road vanishing point is prohibited. In this method, information is acquired from a device that has information on the attributes of the road on which the vehicle is traveling. A typical example of such a device is a navigation processing device. Based on the acquired information, the calculation of the road vanishing point is performed only when the traveling road satisfies a certain standard.
This allows, for example, when traveling on narrow streets,
It is unlikely that you can see ahead or behind,
Therefore, when it is unlikely that the road vanishing point can be calculated, the calculation of the road vanishing point can be prevented.

According to a twelfth aspect of the present invention, there is provided a method for calculating a vanishing point of a road, comprising acquiring at least information from a device having information on a current position of the vehicle, a traveling direction of the vehicle, and a direction of a road on which the vehicle is traveling. It is determined whether or not the difference between the traveling direction of the vehicle and the direction of the road is greater than a certain value.If the difference between the traveling direction of the vehicle and the direction of the road is greater than a certain value, calculation of the road vanishing point is performed. It is characterized by prohibition.

According to this method, information is acquired from a device having information on the current position of the vehicle, the traveling direction of the vehicle, and the direction of the road on which the vehicle is traveling. A typical example of such a device is a navigation processing device. Based on the acquired information, when it is determined that the difference between the traveling direction of the vehicle and the direction of the road has a certain value or more, the calculation of the road vanishing point is prohibited. This is because when the vehicle is not traveling parallel to the road, there is a high possibility that the road vanishing point does not exist in the screen.

[0024]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a vehicle image recognition system to which an embodiment of the present invention is applied will be described in detail. 1. Overview of Vehicle Image Recognition Processing System A vehicle image recognition processing system described below captures an image of the periphery of a vehicle with a color camera mounted on the vehicle, and recognizes an object around a road based on the captured image. It is a device for. Recognized objects are, for example, traffic lights, road signs, road markings drawn on the surface of the road, road facilities such as elevated roads that cross over the running road, and other vehicles.

The vehicle image recognition processing system has the following:
By executing the four processes, an object such as a facility related to a road is recognized from an image captured by the in-vehicle color camera. It should be noted that a color camera does not necessarily need to be used for the and processes. Straight line portion extraction process Road vanishing point calculation process Camera posture parameter calculation process Object recognition process The straight line portion extraction process is a process of extracting a straight line portion along an advancing direction of a vehicle in an image captured by an in-vehicle color camera. The straight portion includes both sides of the road, a white line and a yellow line on the road, a median strip, a road side zone, and a contour line of a vehicle ahead.

The road vanishing point calculation process is a process of calculating the point at which the road on which the vehicle is traveling disappears on the screen. Specifically, a point at which the straight lines extracted by the straight line portion extraction processing intersect is calculated as a road vanishing point. The camera posture parameter calculation process is a process for obtaining the posture of the vehicle-mounted camera with respect to the road. For this processing, the road vanishing point obtained by the road vanishing point calculation processing is used.

The object recognition process is a process for recognizing an object in an image picked up by a vehicle-mounted camera. In the object recognition process, when recognizing a specific object, a process of cutting out an image region including the object from the screen is performed. In this case, the position of the object on the screen changes as the vehicle travels. Therefore, the displacement of the object on the screen is estimated based on the above-described camera posture parameters and the like, and an image is cut out based on the estimated position.

According to such an in-vehicle image recognition processing system, objects around the own vehicle can be recognized.
Therefore, by utilizing the recognition result for the automatic control of the steering and the brake of the vehicle, the way to the automatic driving of the vehicle is opened, which can contribute to the safe operation of the vehicle. Hereinafter, the configuration of the in-vehicle image recognition processing system will be described first. Then, the above-described linear portion extraction processing, road vanishing point calculation processing, camera posture parameter calculation processing, and object recognition processing will be sequentially described in detail. 2. Configuration of Vehicle Image Recognition Processing System FIG. 1 is a block diagram showing an electrical configuration of the vehicle image recognition processing system. This vehicle image recognition processing system includes an image recognition processing device 1 for recognizing an image of an object around a vehicle. This image recognition processing device 1 includes:
A navigation processing device 2 for detecting the current position of the vehicle and the traveling direction of the vehicle and displaying the current position and the traveling direction of the vehicle on a display device together with a road map; and an anti-lock brake for preventing the wheels from being locked during a sudden braking operation or the like. The device 3 is connected. In the processing in the image recognition processing device 1, each internal information of the navigation processing device 2 and the anti-lock brake device 3 is used as support information.

The image recognition processing device 1 includes an in-vehicle color camera 11 mounted in, for example, a front portion of a vehicle or a vehicle interior. The in-vehicle color camera 11 can capture an image of the front of the vehicle. In addition to or instead of the in-vehicle color camera 11,
Another vehicle-mounted camera that can image the rear of the vehicle or the side of the vehicle may be provided.

The in-vehicle color camera 11 outputs an analog electric signal representing each point of the captured image in color. The analog signal is subjected to processing such as analog / digital conversion in the image processing circuit 13 and is converted into image data. This image data is input to a recognition processing unit 15 including a microcomputer and the like. Recognition processing unit 15
Is connected to a storage unit 17 including a RAM (random access memory) and the like. In addition, the recognition processing unit 15
Is provided with support information from the navigation processing device 2 and the anti-lock brake device 3.

The navigation processor 2 has a distance sensor 21 for detecting the traveling distance of the vehicle and a direction sensor 22 for detecting the traveling direction of the vehicle. Outputs of these sensors 21 and 22 are processed in a sensor processing circuit 23 to be converted into travel distance data and current direction data. These data
The data is input to the position detection processing unit 25 including a microcomputer and the like. The position detection processing unit 25 calculates the current position data of the vehicle based on the data input from the sensor processing circuit 23.

The position detection processing unit 25 includes a road map memory 27 storing a road map and a storage unit 2 including a RAM and the like.
8 and a display unit 29 composed of a CRT (cathode ray tube) or a liquid crystal display panel. The road map memory 27
For example, it is constituted by a CD-ROM. The position detector 25 searches the road map memory 27 based on the calculated current position data, and reads a road map around the current position. This road map is displayed on the display unit 29. At this time, a mark indicating the current position of the vehicle is displayed over the road map.

The position detection processing unit 25 gives the road map data, the current azimuth data, the current position data and the traveling distance data to the recognition processing unit 15 of the image recognition processing device 1 as support information. In addition to these data, azimuth change data representing the amount of change in the traveling azimuth per unit time or per unit traveling distance may be provided to the recognition processing unit 15 as support information.

The anti-lock brake device 3 includes a wheel speed sensor 31 for detecting the rotational speed of the wheels and a g sensor 32 for detecting the acceleration (including deceleration) of the vehicle body. Outputs of these sensors 31 and 32 are converted into wheel speed data and g sensor data by being processed by a sensor processing circuit 33. These data are input to a braking processing unit 35 including a microcomputer. A storage unit 37 including a RAM and the like is connected to the braking processing unit 35. Note that the distance sensor 21 and the wheel speed sensor 31 can be shared, and only one of the sensors is provided, and the output of the one sensor is commonly used in the navigation processing device 2 and the antilock brake device 3. Is also good.

When the brake pedal is operated, the brake processing section 35 creates brake data based on the wheel speed data and the g sensor data. Then, an electromagnetic solenoid (not shown) is controlled based on the brake data. This controls the brake pressure on each wheel,
Locking of the wheels is prevented. Braking processing unit 35
Supplies the g sensor data and the brake data to the recognition processing unit 15 of the image recognition processing device 1 as support information. 3. Next, a straight line portion extraction process will be described.

FIG. 2 is a diagram showing an example of an image picked up by the in-vehicle color camera 11. The image taken by the in-vehicle color camera 11 directed in front of the vehicle includes a traveling road 41. A center white line 42 is formed on the road 41, and a roadside white line 43 is formed near the roadside. A roadside belt 44 is provided on the side of the road 41. In the screen, the point where the traveling road 41 disappears is the road vanishing point NP. In addition, 46, 47, and 48 are other vehicles running ahead.

FIG. 3 is a diagram illustrating a straight line portion extracted from an image. That is, both sides of the road 41, the center white line 42, the road side white line 43, the side portions of the road side belt 44, and the other vehicles 46, 4
7, 48 contours are extracted as straight line portions. As is apparent from FIG. 3, the road vanishing point NP is determined by the straight lines L1, L2,... Along the traveling direction 50 of the vehicle (see FIG. 2).
··· Can be obtained as the intersection of

In the straight line portion extraction processing, the in-vehicle color camera 1
The image captured in 1 is scanned along the horizontal scanning direction DH. This scanning is performed from the upper end to the lower end of the screen. On each scanning line along the horizontal direction DH, a straight line L
1, L2,..., The upper points are straight line candidate points P ₁₁ , P ₁₂ ,.
.. Are detected as P ₂₁ , P ₂₂ ,. The detection processing of this straight line candidate point is performed by the in-vehicle color camera 11 and the recognition processing unit 15.
The process is executed by sequentially reading out the image data stored in the storage unit 17 through the storage unit 17.

The straight line candidate points are, when each pixel constituting the screen is scanned along the horizontal scanning direction DH, a stable state in which chromaticity and / or luminance are stable, and a chromaticity and / or luminance when both are stable. Is detected on the basis of a transition to an unstable state that greatly changes. The stable state and the unstable state are respectively detected as follows.

For example, it is assumed that an in-vehicle color camera outputs three primary color signals corresponding to red (R), green (G), and blue (B). These three primary color signals are signals representing a color tone. The color tone is a quantity expressed by combining chromaticity and luminance. In this case, the storage unit 17 stores the RGB primary color image data. R, G, B at an arbitrary point on a scanning line along the horizontal scanning direction DH
Are image data r (t), g (t) and b (t). t
Represents a processing sequence, and corresponds to one point on a scanning line along the horizontal scanning direction DH.

In this case, the unstable state is detected based on the following equation (2) being satisfied with respect to the judgment value P (t) defined by the following equation (1). m1 is a constant.
Further, j ₁ , j ₂ and j ₃ are constants for weighting. For example, the constant j ₁ has a relatively large value because the R data has a large variation with respect to the change in brightness, and the constant j ₃ has a relatively small value since the B data has a small variation with respect to the change in brightness.

P (t) = j ₁ | r (t) −r (t−1) | + j ₂ | g (t) −g (t−1) | + j ₃ | b (t) −b (t− (1) P (t)> m1 (2) That is, the linear sum of the absolute values of the color tone changes of two processing target points adjacent along the horizontal scanning direction DH is Is larger than the predetermined constant m1, it is detected that the color tone shows a large change. The adjacent processing target points are not necessarily two adjacent pixels, and the processing target points may be set at a certain predetermined number of pixel intervals.

Note that, instead of the above determination value P (t),
Judgment value P defined by equation (3)₁(t)
Fixed value P₁Unstable state when (t) satisfies the following equation (4)
May be determined. Here, m2 is a constant.
You. Also, k₁, K_TwoAnd k _ThreeIs a constant for weighting
Is a number. P₁(t) = k₁｛R (t) −r (t-1)｝^Two + K_Two｛G (t) -g (t-1)｝^Two+ K_Three｛B (t) −b (t-1)｝^Two ・ ・ ・ ・ (3) P₁(t)> m2 (4) In this case, two adjacent pixels along the horizontal scanning direction DH
The linear sum of the squares of the change in the color tone of the processing target point
If it is larger than several m2, it is detected that the state is unstable.
Will be issued.

On the other hand, the stable state of the color tone is determined by the following equation (5) with respect to the judgment value P (t) of the above equation (1), where a predetermined number (for example, 10) or more consecutive processing target points are obtained. Are detected based on the fact that n1 is a constant (where n1 <m1). P (t) <n1 (5) That is, a state where the linear sum of the absolute values of the color tone changes of two processing target points adjacent in the horizontal scanning direction DH is smaller than a predetermined constant n1 is constant. If the number of processing target points continues, it is detected that the color tone is stable.

It is to be noted that, using the judgment value P ₁ (t) of the above equation (3), this judgment value P ₁ (t) is obtained by calculating the following equation (6) by a predetermined number (for example, 10) or more. When the processing target point is satisfied, it may be determined that the state is a stable state.
Note that n2 is a constant (where n2 <m2). P ₁ (t) <n2 (6) In this case, the sum of the squares of the change in the tone of the two processing target points adjacent in the horizontal scanning direction DH is smaller than a predetermined constant n2. If the state continues for a certain number of processing target points, it is detected that the state is stable.

For the detection of a stable state, a method using exponential smoothing can be applied in addition to the above method. For example, using the determination value P (t) of the above equation (1),
A new determination value Q (t) of the equation (7) is obtained, and this determination value Q (t) is obtained.
May detect a stable state based on satisfying the following expression (8). In this case, there is no need to continuously monitor the determination results for a fixed number of consecutive processing target points.
In addition, the technique using the exponential smoothing has an advantage that the influence of noise in the image is eliminated and the stable state can be detected satisfactorily. Note that α in the following equation (7) and
N3 in the equation (8) is a constant.

Q (t) = α · P (t) + (1−α) · Q (t−1) (7) Q (t) <n3 (8) Similarly, Judgment value P of the above equation (3)₁(t)
(9) New judgment value Q of equation ₁(t), and this determination value Q₁(t)
Detects a stable state based on satisfying the following equation (10)
You can also. n4 is a constant.

Q ₁ (t) = α · P ₁ (t) + (1−α) · Q ₁ (t−1) (9) Q ₁ (t) <n4 (10) In addition, not only when using RGB digital signals,
The same applies to the case of using HCV (Hue (saturation), Contrast (brightness), Value (brightness)) digital signals and YUV digital signals (signals composed of one piece of luminance information and two pieces of color information). A stable state and an unstable state can be respectively detected. Alternatively, the stable state and the unstable state may be detected using only one of the luminance and the chromaticity. Note that saturation and lightness correspond to chromaticity.

The RGB digital signal and the HCV digital signal or the YUV digital signal can be converted into each other. For example, an RGB digital signal and a YUV digital signal can be mutually converted by the following equations (11) to (16) ("Interface" 1991).
December issue). Y = 0.2990 · R + 0.5870 · G + 0.1140 · B · · · (11) U = -0.1684 · R-0.3316 · G + 0.5000 · B · · · (12) V = 0.5000 · R-0.4187 · G −0.0813 · B (13) R = Y + 1.4020 · V (14) G = Y−0.3441 · U−0.7139 · V (15) B = Y + 1.7718 · U−0.0012 · V (16) FIG. 4 is a diagram for explaining a process of detecting a straight line candidate point. FIG. 4A shows an example of an image taken by the in-vehicle color camera 11, and FIG. 4B shows changes in R, G, and B digital data on a certain scanning line SHL. R, G,
The change of each data of B is represented by curves LR, LG, and LB, respectively.
Indicated by From FIG. 4, it is understood that the color tone changes drastically in the vicinity of the white lines 52, 53, 54 formed on the road 51 and the median strip 55. Further, it is understood that the color tone is stable for the image portion corresponding to the road surface where the white line or the like is not formed.

FIG. 5 is a flowchart for explaining the processing executed by the recognition processing unit 15 to detect a straight line candidate point. In step S <b> 1, data of one color image captured by the in-vehicle color camera 11 is stored in the storage unit 17. Then, processing is performed on the pixel to be processed while scanning the pixel in the horizontal direction from the upper end of the image (step S2). When the processing on all the pixels to be processed on one scanning line is completed (step S3), the processing is executed by moving the object to be scanned in the vertical direction (step S4). When the processing in the vertical direction is also completed (step S5), the processing of detecting a straight line candidate point for one color image is completed.

Before the processing for all the pixels to be processed on the scanning line along the horizontal scanning direction is completed, the processing shifts from step S3 to step S6. In step S6, data necessary for detecting whether a certain pixel to be processed is in a stable state or an unstable state is read from the storage unit 17. Then, in step S7, it is determined whether or not the vehicle is in an unstable state. That is, the above (2)
It is determined whether the expression or the above expression (4) is satisfied. When it is determined that the vehicle is in the unstable state, it is determined in step S8 whether or not it has been determined that the vehicle is in the stable state before that. If so, the current pixel to be processed is the pixel at the transition point from the stable state to the unstable state. Therefore, in step S11, the coordinates of the processing target pixel (coordinates on the screen) are stored in the storage unit 17 as the coordinates of the straight line candidate point.
Is stored in If it is determined that the state is unstable before the state is determined to be unstable in step S7,
Since the unstable state only continues, it is determined that no straight line candidate point has been detected, and the process returns to step S2.

If it is determined in step S7 that the vehicle is not in an unstable state, the process proceeds to step S9. Step S
In step 9, it is checked whether or not a change in color tone is small with respect to a predetermined number (N) of processing target pixels processed before that. That is, it is determined whether or not the processing target points satisfying Expression (5) or Expression (6) are continuous for a certain number (N) or more. If the processing target points satisfying the above formula (5) or (6) are continuous for a certain number (N) or more, in step S10, the unstable state is further set at the (N + 1) -th preceding processing target point. Is detected. If an unstable state has been detected, the processing target point where the unstable state has been detected is detected as a straight line candidate point, and its coordinates are stored in the storage unit 17 (step S1).
1).

On the other hand, if it is determined in step S9 that the state in which the change in color tone is not small for a certain number (N) of processing target points, the process returns to step S2. Also, in step S10, (N
Even when an unstable state is not detected at the (+1) -th previous processing target point, the process returns to step S2. In this manner, the image captured by the in-vehicle color camera 11 is scanned along the horizontal scanning direction, and the degree of change in color tone on the scanning line is checked. Then, the processing target point at which the stable state in which the color tone is stable and the unstable state in which the color tone is unstable is detected as a straight line candidate point.

In order to detect a straight line, for example, it is conceivable that only a change point of the intensity of the luminance is captured and used as a straight line candidate point. However, in this case, there is a possibility that only a straight line with a clear contrast can be detected, such as a white line on a road. On the other hand, in the above-described configuration in which the straight line candidate point is detected based on the switching between the stable state and the unstable state of the chromaticity and / or the luminance, the roadside zone, the central divider, and the outline of the vehicle are detected. Can be extracted as straight line candidate points. Therefore, since a large number of straight lines included in the image can be extracted, the road vanishing point calculation processing described below can be favorably performed.

In order to eliminate the influence of noise included in an image, the detection of an unstable state may be performed under the condition that a state in which a change in color tone or the like is large continues for a certain number of processing target points. Good. In addition, not only data on one scanning line but also data on a neighboring scanning line may be used to detect a straight line candidate point. For example, when a certain processing target point is considered to be a straight line candidate point, the processing target point is determined as a straight line candidate point on the condition that a point adjacent in the vertical direction is a straight line candidate point. Is also good. In this way, it is possible to eliminate the influence of noise included in the image and detect the line candidate points favorably. 4. Road vanishing point calculation process The road vanishing point calculation process is a process of calculating a road vanishing point NP shown in FIG. 2 using the straight line candidate points obtained by the straight line portion extraction process. As apparent from FIG. 3, the process for obtaining the road vanishing point, line candidate point _{P 11, P 12, ····;} P 21,
P _22, none other than the process of obtaining an intersection of a straight line connecting ....

The road vanishing point calculation process is performed by repeating the Hough transform process twice on the coordinate sequence of the straight line candidate point. Therefore, first, the Hough transform will be outlined. FIGS. 6A and 6B are diagrams for explaining the Hough transform. As shown in FIG. 6A, if a plurality of points (x _i , y _i ) (i = 1, 2, 3,...) Exist on a straight line x = ay + b, For any i, x _i = ay _i + b holds. This equation is (a, b)
Given in the ab coordinates plane is regarded as a variable, wherein the straight line in the coordinate plane, a _{b = -y i a + x i} . The graph on the ab plane for all i is shown in FIG.
Is shown in That is, a plurality of straight line groups corresponding to a plurality of i pass a certain point (a ₀ , b ₀ ). This is a natural consequence of the fact that the plurality of points (x _i , y _i ) all exist on one straight line.

Therefore, the ab coordinate plane is divided into sufficiently fine grid cells, and when the straight line corresponding to (x _i , y _i ) passes through a certain grid cell, the count of the grid cell is counted. Increase by one. The operation of performing this operation for all (x _i , y _i ) is Hough transform. In the above case,
The count value of the grid cell at the point (a ₀ , b ₀ ) should be the maximum. Therefore, if the grid cell with the largest count value is obtained on the ab coordinate plane, (a ₀ , b ₀ ) is obtained. Therefore, the equation of a straight line passing through a plurality of points (x _i , y _i ) is x
= A ₀ y + b ₀ . Thus, H
The ough transformation is used in the field of image processing when obtaining a straight line passing through a plurality of points (x _i , y _i ).

FIG. 7 is a diagram for explaining a process for obtaining the road vanishing point by repeating the Hough transform twice. FIG. 7A shows an xy coordinate plane which is a coordinate plane corresponding to the screen imaged by the on-board camera 11, and FIG. 7B shows the transformed coordinates (first coordinate) in the first Hough transform.
Ab coordinate plane, which is a plane of (transformed coordinates)
FIG. 7C shows an mn coordinate plane which is a plane of the transformed coordinates (second transformed coordinates) in the second Hough transform.

As shown in FIG. 7A, the straight line candidate points P ₁₁ ,
_{P 12, ····; P 21,} P 22,; P 31, P 32, the straight line L1, L2, L3 which ... belongs respectively, the road vanishing point (x _0,
y ₀ ). The equation of a straight line passing through the coordinates (x ₀ , y ₀ ) is as shown in the following equation (17). Note that C is a constant. x = C (y−y ₀ ) + x ₀ = Cy + (x ₀ −Cy ₀ ) (17) Therefore, if a = C and b = x ₀ −Cy ₀ , the conversion equation x
= Ay + b, and the relationship between a and b is expressed by the following equation (18).

B = −ay ₀ + x ₀ (18) Straight line candidate points P ₁₁ , P ₁₂ ,...; P ₂₁ , P ₂₂ ,; P ₃₁ , P
_When Hough transform is applied to the coordinates of ₃₂ ,.
On the coordinate plane, a plurality of grid cells whose count values take the maximum value should be obtained corresponding to the plurality of straight lines L1, L2, and L3. However, since the straight lines L1, L2, and L3 intersect at one point (x ₀ , y ₀ ), the grid cells D ₁ , D ₂ , and D ₃ having the maximum value must be on the straight line of the above equation (18). (See Fig. 7 (b)).

Therefore, the grid cells D ₁ ,
The _second Hough transform is performed on the mn coordinate plane for the coordinates of D ₂ and D ₃ using the conversion formula of the following expression (19). b = ma + n (19) Since the grid cells D ₁ , D ₂ , and D ₃ at which the count value becomes maximum on the ab coordinate plane are on the straight line of the expression (18), mn
On the coordinate plane, the count value of the grid cell corresponding to m = −y ₀ and n = x ₀ is the largest. Thereby, the coordinates (x ₀ , y ₀ ) of the road vanishing point nP on the xy coordinate plane can be obtained.

FIGS. 8, 9, 10 and 14 are flowcharts for explaining the processing executed in the recognition processing section 15 (see FIG. 1) to calculate the road vanishing point. First, the support information provided from the navigation processing device 2 is referred to (step S21). Based on this information, whether the road ahead of the vehicle is a straight road (step S22), whether the attribute of the running road (such as whether it is a main road) satisfies a predetermined criterion (step S22).
23), and whether the vehicle is traveling substantially parallel to the road (step S24). If any of the conditions is not satisfied, the calculation of the road vanishing point is prohibited, and the process ends. If all of these conditions are not satisfied, there is a high possibility that the road vanishing point cannot be calculated. For example, if the vehicle is traveling on a curve, there is a high possibility that there is no road vanishing point in the screen imaged by the on-vehicle camera 11, and it is difficult to calculate the road vanishing point by the Hough transform because a straight line cannot be extracted. . In addition, if the road on which the vehicle is traveling is a narrow road other than the main road, a long straight section often does not continue, so that there is a high possibility that a road vanishing point cannot be found.
Furthermore, when the vehicle is not traveling parallel to the road, such as during a lane change, there is a possibility that the road vanishing point does not exist in the captured image.

It is to be noted that the determination as to whether the running road is a straight road or whether the road is a road having a predetermined attribute is made based on road map data provided from the navigation processing device 2. The position detection processing unit 25 of the navigation processing device 2 reads data on the road on which the vehicle is traveling from the road map memory 27,
This is given to the recognition processing unit 15. The determination as to whether the vehicle is traveling parallel to the road is made based on the azimuth of the traveling road (given from the navigation processing device 2 as road map data) and the current azimuth data corresponding to the traveling azimuth of the vehicle. This is done by matching

The above steps S22, S23 and S2
If the conditions of step 4 are satisfied, step S25
Proceed to. In step S25, the coordinates (x, y) of the line candidate point are stored in the storage unit 17 for the first Hough transform process.
Is read. The conversion equation is x = ay + b. On the screen, x and y are respectively 1 ≦ x ≦ 256, 1
It is assumed that the value is within a range of ≦ y ≦ 256. Note that a
On the b-coordinate plane, a lattice cell having a length Δa along the a-axis direction of “0.01” and a length Δb along the b-axis direction of “1” is set.

In step S26, the value of the coefficient a corresponding to the slope of the straight line is set to any value within the range of the following equation (20). The range of Expression (20) is a range that can cover the inclination of all the straight lines in the screen up to the road vanishing point when the vehicle is traveling substantially along the road. k1 ≦ a ≦ k2 (for example, k1 = −2, k2 = 2) (20) Then, in step S27, according to the following equation (21),
The value of the coefficient b is determined.

B = −ay + x (21) In step S 28, it is checked whether or not the obtained b is a value within the range of the following equation (22). This range is a necessary and sufficient range to cover a straight line appearing on the screen when the vehicle is running parallel to the road and reaching the road vanishing point.

K3 ≦ b ≦ k4 (for example, k3 = −511, k4 = 768) (22) If the value of b is not within the above range (step S29),
Returning to step S26, the value of a is changed (0.01
Change in steps. ) Similar processing is performed. If the value of b is within the above range, the count value of the corresponding grid cell is counted up (step S30). Step S31
In, the maximum value p of the count values of the grid cells in the range of a and b is stored in the storage unit 17.

Such a process is performed for the range of a in the above equation (20) (step S32). Then, when the processing for a in the entire range of Expression (20) is completed, the coordinates (x, y) of the next straight line candidate point are read from the storage unit 17 and the same processing is performed (step S33). ). In this way, step S25 is performed for all the straight line candidate points.
To S33, the first Hough
A conversion process is achieved.

When the first Hough transform process is completed, the count values of the grid cells are sequentially referred to one by one (step S34 in FIG. 9). In step S35, the counted value referred to is a constant value k5 · p (k5 is a constant, for example, 0.5
It is said. At this time, k · p is a half of the maximum count value p.
Becomes ) Is checked. If the count value is less than the fixed value k5 · p, the process returns to step S34, and the same process is performed on the count value of the next grid cell. When the count value is equal to or greater than k5 · p, the constant value k is further increased.
It is checked whether the number is 6 (for example, 3) or more (step S36). This takes into account the case where the maximum count value p is small. If the count value does not reach the certain value k6, the process returns to step S34.

Next, the coordinates of the point at which the count value of the grid cell is determined to be sufficiently large are compared with the grid cell representative value registered in the storage unit 17 (step S37). The grid cell representative value is a coordinate of a point whose count value of the grid cell is sufficiently large and represents a group of points within a predetermined distance range (for example, 50 grid cells). That is,
Points with a large count value in the grid cells are grouped so that nearby ones belong to the same group. The lattice cell representative value is the coordinates of a point of one lattice cell in the group. For example, the coordinates of the last registered point in the group are set as lattice grid representative values. Each group is assigned a group number.

If there is no lattice cell representative value nearby (step S38), the coordinates of points of the lattice cell to be processed are registered as lattice cell representative values, and a group number is assigned to this lattice cell representative value. Is performed (step S39). If there is a lattice cell representative value nearby, this processing is omitted. next,
A group number corresponding to the lattice cell representative value is assigned to the lattice cell point to be processed (step S40). In this manner, any group number is assigned to a point having a large count value of a grid cell, whereby classification of the grid cell is achieved.

The above processing is executed for all the grid cells within the range of a and b in the above-mentioned equations (20) and (22) (step S41). In step S42, it is determined whether or not there are two or more groups in which the grid cell representative values are registered. If there are no more than two groups, the calculation of the road vanishing point is prohibited, and the process ends (step S42). In this case, since only one straight line is found, it is not possible to calculate the road vanishing point. When two or more groups are registered, the second Hough transform process from step S43 in FIG. 10 is executed.

In the second Hough transform process, only points to which group numbers are assigned, that is, only points having a large count value of a grid cell are used. In the above processing, the group numbers are assigned to points whose lattice cells are equal to or more than a certain value. Instead, the lattice cells within a predetermined range (for example, within the range of five lattice cells) in the surrounding area are assigned instead. A total value of the count values may be obtained, a group number may be assigned only to a point where the total value is equal to or more than a certain value, and only the point to which the group number is assigned may be used in the second Hough transform process. In addition, only the point where the count value of the grid cells in each group is the largest is used for the second Hough transform processing, or the point where the total count value of the grid cells in the surrounding predetermined range is the largest in each group. Only the second Hough transform may be used for the second Hough transform, or the second Hough transform may be reduced.

Further, a certain number (for example, 5) of points where the count value of the grid cells is equal to or more than a certain value or a point where the total value of the count values of the grid cells in a predetermined range around the grid is equal to or more than a certain value.
For a group that does not exist, points belonging to the group may not be used in the second Hough transform process.
In this way, for example, points corresponding to straight line candidate points corresponding to the outline of the vehicle, such as the straight lines L5 to L10 in FIG.
Can be used for gh conversion processing. Therefore, the calculation accuracy of the road vanishing point can be improved.

Referring to FIG. 10, the second Hough transform processing will be described. In step S43, the storage unit 17
(A, b) of the point on the ab coordinate plane stored in
Are sequentially referred to. Then, the Hough conversion process is performed only on the point to which the group number is assigned (step S44). The conversion equation in this case is b = ma + n. In the mn coordinate plane, the length Δm along the m-axis direction
Is set to “1”, and a grid cell whose length Δn along the n-axis direction is “1” is set.

In step S45, the value of m is set to the following (23)
Set to any value within the range of the expression. The range of Expression (23) is a range that can cover all the straight lines in the screen up to the road vanishing point when the vehicle is traveling substantially along the road. k7 ≦ m ≦ k8 (for example, k7 = −256, k8 = −1) (23) Subsequently, in step S46, according to the following equation (24),
The value of n is determined.

N = −ma + b (24) In step S 47, it is checked whether or not the obtained n is within a range of the following equation (25). This range is a necessary and sufficient range to cover a straight line appearing on the screen when the vehicle is running parallel to the road and reaching the road vanishing point.

K9 ≦ n ≦ k10 (for example, k9 = 1, k10 = 256) (25) If the value of n is not within the above range (step S48),
Returning to step S45, the same processing is performed by changing the value of m. If the value of n is within the above range, the count value of the corresponding grid cell is counted up (step S).
49). In step S50, the maximum value q of the count values of the grid cells in the range of m and n is stored in the storage unit 17.

Such processing is performed for the range of m in the above equation (23) (step S51). Then, when the processing for m in the entire range of the above equation (23) is completed, the coordinates (a, b) of the point of the next grid cell are read from the storage unit 17 and the same processing is performed (step S23). S52). In this way, by performing the processing for all the grid points, the second Hough transform processing is achieved.

When the second Hough transform processing is completed, m
The count values of the grid cells on the n-coordinate plane are sequentially referred to one by one (step S53). In step S54, the counted value referred to is equal to or greater than a fixed value k11 · q (k11 is a constant, for example, 0.5. At this time, k11 · q is one half of the maximum count value q). It is checked whether it is (step S54). If the count value is less than the fixed value k11 · q, the process returns to step S53, and the same processing is performed on the count value of the next grid cell. If the count value is equal to or larger than k11 · q, the total value of the count values of the grid cells within a predetermined range around the point is calculated (step S55). For example, the total value of ± 5 grid cells in the m-axis direction and the n-axis direction is calculated.

In step S56, the maximum value r of the total value and the coordinates (m ₀ ,
n ₀ ) are stored in the storage unit 17. Such processing is executed for all the grid cells within the range of m and n in the above formulas (23) and (25) (step S57). As a result, the coordinates (m ₀ , n ₀ ) are the coordinates of the point in the range of m and n where the sum of the count values of the lattice cells in a predetermined range around the coordinate is the largest.

In step S58, it is determined whether the maximum value r of the total value is equal to or larger than a fixed value k12 (for example, 50). If the maximum value r is less than the constant value k12, the confidence is low and the calculation of the road vanishing point is prohibited, and the process is terminated. When the maximum value r is equal to or larger than the constant value k12, (n ₀ , −m ₀ ) is determined as a road vanishing point.
In addition, instead of finding the point at which the count value of the grid cells in the surrounding predetermined range is the largest, finding the point at which the count value of the grid cells is the largest, and finding the road vanishing point based on the coordinates of this point. Is also good. In this case, if the maximum count value of the grid cells is less than a certain value, the calculation of the road vanishing point may be prohibited.

The processing from step S60 is processing for obtaining a road parallel line that is a straight line parallel to the road. First, in step S60, the contents stored in the storage unit 17 are searched, and a specific group number among the points of the grid cell on the ab coordinate plane is referred to. In step S61, b
The absolute value of-(am ₀ + n ₀ ) is calculated as an error.
If b = am ₀ + n ₀ holds, then (a, b) at that time
Gives the intercept and slope of a straight line passing through the road vanishing point (x ₀ , y ₀ ). Therefore, by finding (a, b) where the absolute value of b− (am ₀ + n ₀ ) takes the minimum value, the road parallel line is obtained. That is, among the points belonging to the same group, the coordinates of the point at which the straight line on the xy coordinate plane using the coordinates as the coefficient passes closest to the road vanishing point are obtained as the coefficient of the road parallel line.

In step S62, it is determined whether or not the absolute value of b− (am ₀ + n ₀ ) obtained as an error in step S61 is within a fixed error k13 (for example, 5). If the error is within the fixed error k1, it is determined whether the error is the minimum among the errors corresponding to the coordinates of the point to which the group number is assigned and the processing has already been completed (step S63). If it is the minimum value (step S
64), the error obtained with respect to the coordinates of the points of the grid cell is determined as the minimum error in the group to which the group number is assigned (step S65).

In step S66, it is checked whether or not the processing has been completed for all the grid cell points to which a certain group number has been assigned. When the processing for all points to which the same group number has been given is completed, it is further checked whether or not the processing has been completed for all group numbers (step S67). When the above processing is completed for all the group numbers, the grid cell having the minimum error of each group is referred to (Step S68). Then, for the grid cell group in which the minimum error is stored in the storage unit 17, the coordinates (a, b) of the grid cell corresponding to the minimum error are set as the coefficients (slope and intercept) of the road parallel line (step). S69). When the coefficient a is positive, the road parallel line is set as the road parallel line on the left side of the vehicle (step S70). When the coefficient a is negative, the road parallel line is set as the road parallel line on the right side of the vehicle. You.

In the case of FIG. 3, the road parallel lines L1, L2,
Coefficients a and b are obtained for L3 and L4. In this case, the equation of each road parallel line L1, L2, L3, L4 is:
Equations (26) to (29) below, and the road vanishing point N
The coordinates of P are (120, 170). L1: x = 1.333y−106.7 (26) L2: x = 0.63y−26.8 (27) L3: x = −0.992y + 288.7 (28) L4: x = − 1.844y + 433.5 (29) If the coefficient a obtained for each group number is either a positive value or a negative value, the calculated road is used. It is preferable to disable the vanishing point and the road parallel line. In such a case, since only the road parallel lines located on either the left side or the right side of the vehicle are detected, the calculation accuracy of the road vanishing point may be low.

The inventor experimentally executed the above-described road vanishing point calculation process in the following four cases. The coordinate data of the straight line candidate points on the straight lines L2 and L3 in FIG. 3 are used. The coordinate data of the straight line candidate points on the straight lines L2, L3, L5 to L10 in FIG. 3 are used. The coordinate data of the straight line candidate points on the straight lines L1 to L3 and L5 to L10 in FIG. 3 are used.

The coordinate data of the straight line candidate points on the straight lines L1 to L10 in FIG. 3 is used. In each of these cases, the coordinates of the road vanishing point could be obtained correctly. In particular, in the case of ,, the case where a straight line candidate point corresponding to the contour of the other vehicle in front is included, it is confirmed that the coordinates of the road vanishing point can be correctly obtained also in such a case. Was. In addition, the road parallel lines could be obtained almost without errors. However, in the case of, the straight line L4 could not be obtained as a road parallel line. It is considered that this is because the sample data of the straight line candidate points is small. Therefore, by increasing the sample data, the straight line L4
It is considered that the road parallel line can also be obtained for.

The road vanishing point and the road parallel line obtained in this way are used in a camera posture parameter calculation process described below. 5. Camera Posture Parameter Calculation Process In this process, a posture parameter representing the posture of the vehicle-mounted camera 11 with respect to the road is obtained. Attitude parameters include
The yaw angle which is the rotation angle around the vertical axis, the roll angle which is the rotation angle around the traveling direction of the vehicle, the pitch angle which is the rotation angle around the direction which is along the horizontal plane and orthogonal to the traveling direction, and parallel to the road A lateral displacement of the camera from a predetermined reference line (lateral relative position to the road) is included.

Although the on-vehicle camera 11 is accurately mounted on the vehicle in a predetermined posture, occurrence of mounting errors cannot be avoided.
Therefore, in the camera posture parameter calculation process, the mounting posture of the vehicle-mounted camera 11 with respect to the vehicle is also calculated.
First, the coordinate system will be described. A road coordinate system XYZ,
A camera coordinate system X'Y'Z 'is defined. It is assumed that the vehicle is at the origin of the road coordinate system irrespective of the movement of the vehicle, and the Y axis is set in the traveling direction of the vehicle. The X axis is taken in the right direction with respect to the traveling direction, and the Z axis is taken in the vertical direction. The camera coordinate system and the road coordinate system share the origin. It is assumed that the imaging surface of the vehicle-mounted camera 11 is parallel to the XZ plane and at a distance F (F is the focal length of the lens of the camera 11) from the origin.

The rotation angles about the X, Y, and Z axes are a pitch angle θ, a roll angle φ, and a yaw angle そ れ ぞ れ, respectively, and the direction of the right-hand thread is the positive direction. At this time, the conversion formula of the camera's coordinate system accompanying the camera mounting error or the rotation of the vehicle is:
It is given by the following equation (30). However, it is assumed that the Y 'axis is set in the main axis direction of the lens of the vehicle-mounted camera 11, and the X axis and the Z' axis are set in parallel with the imaging surface.

[0092]

(Equation 1)

If each rotation angle is small, the above equation (30) can be transformed into an approximate equation of the following equation (31).

[0094]

(Equation 2)

The point P (X, Y, Z) is a point p 'on the imaging surface.
When projected on (x ', y'), the following equation holds. Here, the coordinates (x ′, y ′) are two-dimensional coordinates on the imaging surface. The x 'axis is in the X' axis direction of the camera coordinate system, and the y 'axis is in the Z' axis direction of the camera coordinate system. x '= F.X' / Y '(32) y' = F.Z '/ Y' (33) The posture parameters of the camera are pitch angle θ, roll angle φ, and yaw angle. ψ is divided as follows.

Pitch angle θ: (Pitch angle θ ₀ of vehicle with respect to road) + (Pitch angle θ ₁ of mounting vehicle-mounted camera 11 with respect to vehicle) Roll angle φ: (Roll angle φ _{0 of} vehicle with respect to road) +
Yaw angle [psi ₀ of the vehicle with respect to the yaw angle [psi :( road (attachment roll angle phi ₁ of the on-vehicle camera 11 with respect to the vehicle)) + (the yaw angle [psi ₀ of the vehicle with respect to the mounting yaw angle [psi ₁₎ Road vehicle camera 11 with respect to the vehicle Can be obtained from the road map data and the current direction data provided from the navigation processing device 2. That is, the direction of the road on which the vehicle is traveling is known from the road map data, and the actual traveling direction of the vehicle is known from the current direction data. for that reason,
The difference between the actual traveling direction and the direction of the road may be defined as the yaw angle ψ _{0 of the} vehicle with respect to the road.

The roll angle φ ₀ and the pitch angle θ ₀ of the vehicle with respect to the road cannot be detected by themselves, but can be grasped as noise that varies with the average value “0”. That is,
If the average values of the roll angle φ and the pitch angle θ over a sufficiently long time are taken, it may be considered that the average value does not include the roll angle φ ₀ and the pitch angle θ ₀ of the vehicle with respect to the road.

[0098] the vehicle is assumed to be traveling in a direction forming an angle of [psi ₀ relative to the road, the road is assumed to be within the linear a and vehicle plane (XY plane) sufficiently far. Also,
It is assumed that left and right inclinations of road banks and vehicles can be ignored. Then, the coordinates of the road vanishing point in the road coordinate system XYZ are set to (X ₀ , Y ₀ , Z ₀ ) (where Y ₀ = 、), and the coordinates of the mapping point of this road vanishing point on the imaging surface are (X ₀ , y ₀ ). The coordinates of the mapping point are nothing but the road vanishing point determined by the road vanishing point calculation process.

Assuming that the posture parameters other than 微小₀ are minute, ψ → ψ ₀ + ψ ₁ and θ, φ and ψ ₁ are minute. Therefore, the above equation (30) and the above (32) From the equation and the equation (33), the following equations (34) and (35) are obtained. _{x 0 = R 12 F ≒ sin} (ψ 0 + ψ 1) F = (sinψ 0 + ψ 1 cosψ 0) F ···· (34) y 0 = R 32 F ≒ {φsin (ψ 0 + ψ 1) -θcos ( ψ ₀ + ψ ₁ )｝ F (35) Further, if 微小₀ is minute, the following equation (36) and (3
7) Obtain the formula.

X ₀ = (ψ ₀ + ψ ₁ ) F (36) y ₀ = −θF (37) Particularly, when the vehicle is running parallel to the road, ψ ₀
Since = 0, the following equations (38) and (39) are obtained. x ₀ = R ₁₂ F ≒ ψF = ψ ₁ F (38) y ₀ = R ₃₂ F ≒ −θF (39) On the other hand, if each of the above parameters is minute,
From the expression (31) and the above expressions (32) and (33), the following expressions (41) and (42) are obtained.

[0102] x '= F (X + ψ 1 Y-φZ) / (- ψ 1 X + Y + θZ) ···· (41) y' = F (φX-θY + Z) / (- ψ 1 X + Y + θZ) ···· (42 ) The height Z of the road parallel line is -h when the height from the road to the vehicle-mounted camera 11 is h. Further, when the displacement in the X direction of the on-vehicle camera 11 and A for a reference line parallel to the road, because the vehicle is traveling in the direction of the road and [psi _0,
The following equation (43) holds.

[0102] _{X = A / cosψ 0 + Y} tanψ 0 ≒ A + Yψ 0 ···· (43) Therefore, the first (41) below and the (42) equation, the following first (4
Equation (4) and equation (45) can be rewritten respectively.

[0103]

(Equation 3)

From this, when Y is eliminated, the following equation (46) is obtained.

[0105]

(Equation 4)

If two road parallel lines are obtained and the distance B between them is known, the following equations (47), (48) and (49) similar to equation (46) are obtained. can get. However, the suffixes “1” and “2” added to the coefficients a, b, and A indicate that the assigned coefficient corresponds to each of the two road parallel lines.

[0107]

(Equation 5)

Thus, the vanishing point of the road (x ₀ ,
y ₀ ), and if the coefficient a or b of the road parallel line and the interval B of the road parallel line are obtained, the above equation (36) and (3
7) or the second (38) below and a (39) Motomari is [psi ₁ and θ from the equation above the expression (47), the equation (48) and a (49) and φ from equation A ₁ and A ₂ Is found. Of the above equations (47) and (48), solving using the equations relating to a ₁ and a ₂ gives the following equations (50), (51) and (52):
A ₁ , A ₂ and φ are obtained respectively.

[0109]

(Equation 6)

The interval B may be, for example, the width of a road or the interval between white lines of a road. These intervals can be obtained from the road map data. For example, since the road width of the expressway is determined, the above-described camera posture parameter calculation process may be performed only when the navigation processing device 2 detects that the vehicle is traveling on the expressway.

The above-described processing may be performed every time the vehicle travels a certain distance, and the averaging processing may be performed on θ and φ of the obtained attitude parameters. In this case, since the average value of θ ₀ and φ ₀ is considered to be zero as described above, the mounting pitch angle θ ₁ and the mounting roll angle φ ₁ of the vehicle-mounted camera 11 with respect to the vehicle can be obtained. Therefore, the above process can also be executed as an initial setting process for calculating the mounting posture of the vehicle-mounted camera 11 with respect to the vehicle. Therefore, θ ₀ , φ
_{If 0} is negligibly small, it is possible to substitute only the result of the initial setting process in the normal periodic process.

If φ ₀ is a value small enough to be ignored, in the periodic processing, if φ is set to φ ₁ obtained by the initial setting processing in the above equation (46), the road interval B between the road parallel lines becomes Even if unknown, the lateral deviation A can be obtained. FIG. 12 is a flowchart illustrating a process executed by the recognition processing unit 15 for the above-described camera posture parameter calculation process. In step S81, the image data of one image picked up by the vehicle-mounted camera 11 is stored in the storage unit 17. In step S82, support data from the navigation processing device 2 is fetched,
In step S83, support data from the anti-lock brake device 3 is fetched. In step S84, the yaw angle ψ _{0 of the} vehicle with respect to the road is calculated based on the data provided from the navigation processing device 2.

In step S85, it is determined whether the calculation of the vanishing point of the road is possible. If the calculation is impossible, the process is terminated. If the calculation is possible, the above-described road vanishing point calculation process is executed to calculate the road vanishing point (x ₀ , y ₀ ) (step S86). Further, in step S87, coefficients a and b of the road parallel line are calculated. The determination as to whether the road vanishing point can be calculated is made, for example, by determining the yaw angle of the vehicle with respect to the road.
_This is performed based on whether ₀ is equal to or less than a predetermined value. In addition, as clarified in the explanation related to the road vanishing point calculation process, conditions such as whether the traveling road is a straight road, and whether the attribute of the traveling road satisfies a certain standard are also determined. You.

When the coefficients of the road vanishing point and the road parallel line are calculated, it is next determined whether or not the posture parameter can be calculated (step S88). The judgment criteria at this time are as follows. When the angle between the traveling direction of the vehicle and the road parallel line can be regarded as zero, it is assumed that the posture parameter can be calculated. Imposing such conditions is
When the yaw angle [psi ₀ of the vehicle is large with respect to the road, there is a possibility that the error is large. In this case, for example,
An error between a change in the traveling azimuth of the vehicle before and after traveling a certain distance and a difference in the road azimuth (obtained from road map data) before and after the constant azimuth direction is within a certain value (for example,
(Within 0.1 °), it may be determined that the vehicle is traveling parallel to the road and the angle between the traveling direction and the road parallel line is zero. Further, the traveling direction of the vehicle is continuously smaller than a predetermined value (for example, 0.1
° or less), it is determined that the angle between the traveling direction and the road parallel is zero, assuming that the traveling road is straight and the vehicle is traveling parallel to the road. You may.

When the speed of the vehicle is within a certain range (for example,
(Within 100 km / h), it is assumed that the posture parameter can be calculated. The speed of the vehicle can be estimated from the current position data or the traveling distance data. If the speed of the vehicle is not within the above-mentioned predetermined range, vibration of the vehicle body may occur, and the posture parameter of the vehicle with respect to the road may increase, and the calculation of the posture parameter may be poor.

When the acceleration (including deceleration) of the vehicle is less than a certain value (for example, 0.1 g), it is assumed that the posture parameter can be calculated. The acceleration can be estimated from the current position data or traveling distance data, or can be obtained from g sensor data or brake data. Further, when the operation of the brake is detected from the brake data, the calculation of the posture parameter may not be performed. When the deceleration of the vehicle is equal to or more than the above-mentioned fixed value, the pitch angle θ _{0 of the} vehicle with respect to the road may become abnormally large, and the calculation of the posture parameter may become poor.

It is assumed that the posture parameter can be calculated when the change in the traveling direction of the vehicle before and after traveling a certain distance is less than a certain value (for example, 0.1 °). If the change in heading before and after traveling a certain distance is equal to or greater than the certain value,
Vehicle is considered to be traveling on a curve, in this case there is a risk that the roll angle phi ₀ of the vehicle with respect to the road becomes abnormally large due to the centrifugal force.

[0118] When the calculation of the orientation parameters are determined to be in step S89, in accordance with said first (36) below or the (38) equation, mounting the yaw angle [psi ₁ is calculated for vehicle-vehicle camera 11. Further, the pitch angle θ is obtained by the above equation (37) (step S90).
The roll angle φ is obtained by the equation (46) or the equation (52) (step S91). Then, the above equation (46) or (5
The lateral shift A is calculated by the equations (0) and (51) (step S92).

In step S93, the average values of the mounting yaw angle ψ ₁ , the pitch angle θ, and the roll angle φ obtained so far are obtained. In this case, if the posture parameter calculation processing shown in FIG. 12 is performed for each constant traveling distance (for example, 100 km), an average value of the parameters obtained for each constant traveling distance is obtained. When the posture parameter calculation process is performed at regular time intervals (for example, two hours), an average value of the parameters determined at regular time intervals is obtained. By the process of calculating the average value, a plausible mounting yaw angle ψ ₁ , pitch angle θ, and roll angle φ are determined.

Further, the average value of the pitch angle θ is set to the mounting pitch angle θ ₁ of the vehicle-mounted camera 11 with respect to the vehicle (step S94), and the average value of the roll angle φ is set to the mounting roll angle φ ₁ of the vehicle-mounted camera 11 with respect to the vehicle. (Step S9
5). The relevance of these processes will be apparent in light of the fact that the pitch angle θ ₀ and the roll angle φ ₀ of the vehicle relative to the road can be made zero by averaging the time over a sufficiently long period.

By the above processing, the on-vehicle camera 11
Are obtained. These posture parameters are used in the object recognition processing described below. 6. Object Recognition Processing The object recognition processing is processing for recognizing a certain type of object in an image captured by the on-vehicle camera 11. That is, the image of a road sign, a display on a road, another vehicle, or the like is recognized. In this image recognition, camera posture parameters and support information given from the navigation processing device 2 and the antilock brake device 3 are used.

In the following, first, the following recognition of a stationary object such as a road sign or a display on a road will be described, and then the following recognition of a moving object such as another vehicle will be described. 6-1. Tracking Recognition of Still Object In the above equations (41) and (42) used in the description of the camera posture parameter calculation processing, X and Y are variables,
Solving assuming that the point (x ′, y ′) on the imaging surface, the camera attitude parameter, and the variable Z are known, the following equation is obtained.

[0123]

(Equation 7)

Incidentally, each variable or parameter is a variable of the time t, and changes with the passage of time. At time t, the position of an object on the imaging plane is (x _t ′,
y _t '), and if the camera orientation parameters and the position Z (constant value) in the height direction of the object viewed from the vehicle are known,
The position (X _t , Y _t ) of the object as seen from the vehicle is determined by the above (53)
This is given by the following equation (54), which expresses the time t explicitly in the equation.

[0125]

(Equation 8)

Further, it is assumed that the vehicle travels and time t + 1 is reached. Time L _{t + 1} the travel distance data of the vehicle between the time t + 1 from t, when the direction change data from the azimuth change data (navigation processing unit 2 which is obtained from the current heading data is given, can also be used as such .) Is Δψ _{1t + 1} , the position of the object at time t + 1 Λ
_{t + 1} (X _{t + 1} , Y _{t + 1} ) is given by the following expression, expressing the time explicitly.

[0127]

(Equation 9)

The expression (55) is obtained by dividing the expression (41) and the expression (4).
Equation (56), which expresses the time explicitly in equation (2), and
By substituting into the expression (57), the position (x _{t + 1} ′, y _{t + 1} ′) of the object on the screen at the time t + 1 can be estimated.

[0129]

(Equation 10)

That is, if the position of the object at time t is known, the position on the screen at which the object moves at time t + 1 is estimated based on the camera posture parameters and the like. Therefore, when recognizing a certain object from a screen imaged momentarily, it is sufficient to cut out an image of a region around the estimated position and perform the recognition process, so that the efficiency of the recognition process can be improved.

The image cut-out range around the estimated position is determined by monitoring the error between the estimated value that is the estimated position on the screen and the actual value that is the position of the actually recognized object. You. This will be specifically described. First, the estimated value at time t and _{^{(E x t ', E y}} t'), representing the actual value _{^{(M x t ', M y}} t'). Further, when the averaging operation is represented by ave, the variance operation is represented by var, and the square root operation is represented by sqrt, the following expressions (58) to (63) are obtained.

[0132] var (x) = ave (x -ave (x)) 2 ···· (58) std (x) = sqrt (var (x)) ···· (59) x = E x t ' ^−M _xt ′ (60) var (y) = ave (y−ave (y)) ² (61) std (y) = sqrt (var (y)) _{^{(62) y = E y t}} '- M y t' ···· (63) Therefore, the error of the x-axis and y-axis direction position on the screen, for example, k · std (x), k · std (y) (k is a constant). Therefore,
If the range of the image including the error range is set as the image cutout range, an image including the object to be recognized can be cut out.

If the constant k is variably set based on the running speed of the vehicle, the change in azimuth, the reliability of the posture parameter, and the like, the error range can be set more appropriately.
Specifically, the traveling speed is a constant value (for example, 100 km / h)
In the above case, the constant k may be set so as to be substantially proportional to the traveling speed of the vehicle. When the amount of change in the heading of the vehicle per unit time or per unit traveling distance is equal to or greater than a fixed value (for example, 0.1 °), the value of the constant k is set so as to be substantially proportional to the amount of change in heading. May be.

The error range may be changed for each type of object to be recognized, or a common error range may be used regardless of the type of the object. When the object is recognized, based on the newly obtained position (x _{t + 1} ′, y _{t + 1} ′) on the screen, the above equation (54) (where t is replaced by t + 1) is used. , The actual position of the object in space (X
_{t + 1} , _{Yt + 1} , _{Zt + 1} ). Using this, the position of the object at the next time is estimated, and a new image cutout range is obtained.

Assuming that the equations (54) and (55) include errors, using a Kalman filter or the like, the mileage, the azimuth change error, the attitude parameter estimation error,
From the difference between the estimated value and the actual value of the position (x ', y') on the screen, the position (X, Y, Z) of the object is obtained by filtering.
Can be estimated every moment. In this way, for example, when recognizing a road sign or a traffic signal by following it moment by moment, assuming that the height of the object to be recognized is almost constant, the central portion of the traffic signal is to be tracked, or If the height of the ground contact surface is considered to be lower than the height of the camera and the object is to be followed, the object can be easily followed and recognized by limiting the search range by the above-described method. 6-2. Tracking Recognition of Moving Object When a moving object is tracked and recognized, the conversion formula of the above-mentioned formula (55) cannot be used. This is because the motion characteristics of the moving object are unknown. Therefore, the following equation (64) considering the temporal change of the position of the moving object is used instead of the above equation (55).

[0136]

[Equation 11]

As is clear from the above equation, Λ _t-1 is not defined in the next processing cycle after the first recognition of the object.
Therefore, at this time, it is necessary to make the error range sufficiently large. When the moving object to be recognized is another vehicle ahead, for example, when following only the rear part of this other vehicle, if the left and right lowermost parts are to be followed,
Its height can be almost specified.

It should be noted that the object to be recognized can be changed according to the type of road on which the vehicle is traveling, and recognition can be performed according to the purpose. For example, when driving on a highway, only vehicles are recognized for the purpose of safe operation of vehicles.When driving on other highways, only road signs and traffic lights are recognized. There is a method such as doing. The recognition result of the other vehicle ahead can be used for issuing an alarm for preventing a collision or for automatically controlling the brake. Further, for example, when route guidance information is displayed on the display unit 29 of the navigation processing device 2 to perform route guidance to a destination, by recognizing a road sign, route guidance that violates traffic regulations is performed. Can be prevented in advance. Further, by recognizing the traffic light, it is possible to determine whether the traveling road is a general road or an expressway. 6-3. Object Recognition This section outlines how to recognize objects from data on the screen. Various methods for recognizing an object based on image data have been conventionally devised. For example, in the case of data from a black-and-white camera (only luminance data), an edge point at which the density greatly changes is detected, and an outline of the object is extracted by connecting a plurality of edge points. The object on the screen is recognized by checking the shape of the contour or performing pattern matching with a standard pattern of the target object registered in advance. When color data is used, not only density but also chromaticity data can be used, so that the recognition probability can be increased.

An example of a feature for each recognition target extracted from the image data is shown below. For example, the storage unit 17 of the image recognition processing device 1 stores standard values for these features. The recognition processing unit 15 performs the object recognition by extracting the following features from the image data and comparing the extracted features with standard features stored in the storage unit 17.

(1) Road sign Circle radius of circle Circle chromaticity Data in circle (2) Traffic light Rectangle Vertical and horizontal size Three circles Radius of circle Light color (3) Car front and rear Shape Shape of Horizontal Part Vehicle Height, Vehicle Width, Vehicle Length 6-4. Processing of Recognition Processing Unit Next, with reference to the flowcharts of FIGS. 13 and 14, the recognition processing unit 15 executes for recognition of an object. The processing will be described. This process is executed at a constant cycle.

First, in step S 101, one image picked up by the vehicle-mounted camera 11 is loaded into the storage unit 17. Then, in step S102, the reliability of the posture parameter of the camera is calculated. This reliability is calculated based on how the posture parameters were used. For example, when the posture parameter cannot be calculated in the cycle and the average value of the previously obtained posture parameters is used as the posture parameter in the cycle, the reliability is low. When the acceleration (including deceleration) of the vehicle is equal to or more than a certain value (for example, 0.1 g), the reliability of the posture parameter is low. The acceleration is
It is obtained from the current position data, travel distance data, brake data or g sensor data. Further, when the brake data is detected to be operated by the brake data, the reliability is low.

When the reliability of the posture parameter is calculated,
Next, the attributes of the road on which the vehicle is traveling (expressways, general roads, etc.) are referenced (step S103). Road attributes are included in the road map data. Then, the type of the object to be recognized is determined based on the attribute of the traveling road (step S104). That is, for example, if the vehicle is traveling on a highway, another vehicle ahead is regarded as a recognition target object, and if the vehicle is traveling on a general road, a traffic signal or a road sign is regarded as a recognition target object.

Next, it is checked whether or not the attribute of the road in the previous cycle has changed (step S105). Then, if the attribute has not changed from the previous cycle, it is determined whether at least one recognized object is registered in the storage unit 17 (step S106). If the road attribute is different from the previous cycle, in step S130, all the recognized objects registered in the storage unit 17 before that are cleared.

If it is determined in step S106 that no recognized objects have been registered, and if all previously registered recognized objects have been cleared in step S130, the flow advances to step S131 to change the entire area of the screen. Initial recognition as a recognition target area is performed. Then, the position (X, Y, Z) of the recognized object is calculated (step S132), and the position of the recognized object (X, Y, Z) is calculated.
Is registered in the storage unit 17 (step S13).
3), the processing of the cycle ends.

In step S107, the processing relating to the type of the object to be recognized is sequentially performed. That is, the storage unit 17
, The recognition target objects are classified and stored for each type, and information for each type is referred to. If there is an unprocessed type among the types of the recognition target objects (step S1)
08), it is checked whether or not there is an object recognized in the previous cycle among the objects to be recognized of the type (step S1).
09). If not, the process returns to step S107. If there is an object recognized in the previous cycle, it is checked whether the type of the object is a car (step S110). If so, it is checked whether or not the object (car) has been recognized even in the last two cycles (step S111).
If the object has not been recognized in the last two cycles, the error range for determining the image cutout area is set to a sufficiently large constant value in step S112. This is because, as described above with respect to the following recognition of a moving object, in order to accurately estimate the position of the moving object on the screen in the main cycle, the positions of the moving object in the previous cycle and the two cycles before the previous cycle are necessary. Because. Therefore, when the moving object is not recognized in the last two cycles, by setting a sufficiently large error range, it is possible to reliably cut out the image area including the moving object.

In step S113, the position (X, Y, Z) of the object in the previous cycle is referred to, and further in step S114
Then, the travel distance L between the previous cycle and the main cycle, and the azimuth change data Δψ ₀ are obtained based on the support information provided from the navigation processing device 2. Then, FIG.
The process moves to step S115 of No. 4. Step S115
Then, the position (X, Y, Z) of the object in this cycle is calculated according to the above equation (55). Based on this, the position (x ′, y ′) of the object on the screen in this cycle is
It is estimated according to the equations (56) and (57) (step S
116).

In step S117, the running speed, the change in the traveling direction, and the reliability of the posture parameter are referred to. Then, the size of the error range of the estimated position is corrected based on these (step S118). That is, the value of the constant k for determining the error range is updated.
In step S119, an image in the image cut-out range determined based on the error range is cut out from the screen in the main cycle, and an object recognition process is performed based on the cut-out image. If the object to be processed cannot be recognized normally by this process (step S120), the registration of the object is deleted (step S121), and the process proceeds to step S121 in FIG.
Return to 107. On the other hand, when the object to be processed is normally recognized, the exact position of the object on the screen (x ', y')
Are obtained (step S122), and based on the obtained values, the accurate position (X, Y, Z) in the space of the object is obtained according to the above equation (54) (step S123). Thereafter, an error range is determined from the estimated value and the actual value obtained so far (step S124), and the process returns to step S107 in FIG.

In this way, the position of the object recognized in a certain processing cycle on the screen in the next processing cycle is estimated,
An image cut-out range is obtained based on the estimated position and the error range. Then, the object recognition processing is performed only on the extracted image portion. Therefore, it is not necessary to perform the object recognition processing on all image areas in each cycle. Thereby, the object recognition processing can be performed very efficiently and at high speed. This means that the onboard camera 11
This is extremely useful in recognizing an object that is displaced within the imaging plane every moment. 7. Conclusion As described above, according to this in-vehicle image recognition processing system,
In the straight line detection processing, road parallel lines other than the white line can also be calculated, so that the vanishing point of the road can be calculated based on this. Then, a camera posture parameter is obtained based on the calculated road vanishing point, and an object moving every moment on the screen can be satisfactorily recognized based on the camera posture parameter. In each process, support information from the navigation processing device 2 and the anti-lock brake device 3 is used as needed.

With such a configuration, an object ahead of the vehicle can be reliably recognized. Therefore, the vehicle can be automatically driven using the recognition result, or the navigation processing device 2 can be used.
, And support information for route guidance can be created. In the above example, recognition of an object in front of the vehicle has been mainly described. However, a camera that captures an image of the rear of the vehicle may be provided to perform recognition of an object behind.

In the above example, the image recognition processing device 1, the navigation processing device 2, and the anti-lock brake device 3 are individually configured, but any two or all of them may be integrated. . In addition, various design changes can be made without changing the gist of the present invention.

[0151]

As described above, according to the present invention, the road vanishing point is obtained by repeating the Hough transform process twice, so that even if a straight line portion is interrupted in the screen, the road disappears. A vanishing point can be determined. In addition, unlike ordinary methods such as the least squares method, the road vanishing point can be determined very accurately. By using the road vanishing point calculated in this way, it is possible to efficiently recognize an object in an image captured by the on-vehicle camera.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image recognition processing system to which an embodiment of the present invention has been applied.

FIG. 2 is a diagram illustrating an example of an image captured by a vehicle-mounted camera.

FIG. 3 is a diagram showing an example of extracting a straight line portion from the image of FIG. 2;

FIG. 4 is a diagram for explaining a straight line portion extraction process;
(a) shows an example of an image taken by the onboard camera,
(b) shows the fluctuation of the three primary color image data on a certain scanning line.

FIG. 5 is a flowchart for explaining a straight line portion extraction process.

FIG. 6 is a diagram for explaining Hough transform.

FIG. 7 is a diagram illustrating a process for obtaining a road vanishing point by repeating a Hough transform process twice.

FIG. 8 is a flowchart illustrating a road vanishing point calculation process.

FIG. 9 is a flowchart illustrating a road vanishing point calculation process.

FIG. 10 is a flowchart illustrating a road vanishing point calculation process.

FIG. 11 is a flowchart illustrating a road vanishing point calculation process.

FIG. 12 is a flowchart illustrating a camera posture parameter calculation process.

FIG. 13 is a flowchart illustrating an object recognition process.

FIG. 14 is a flowchart illustrating an object recognition process.

[Explanation of symbols]

 Reference Signs List 1 image recognition processing device 2 navigation processing device 3 anti-lock brake device 11 in-vehicle color camera 15 recognition processing unit 17 storage unit 21 distance sensor 22 azimuth sensor 25 position detection processing unit 27 road map memory 28 storage unit 29 display unit 31 wheel speed sensor 32 g sensor 35 braking processing unit 37 storage unit

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-63-142478 (JP, A) JP-A-4-295973 (JP, A) JP-A-60-118987 (JP, A) JP-A-2- 90379 (JP, A) JP-A-5-151341 (JP, A) JP-A-5-149756 (JP, A) JP-A-3-52083 (JP, A) JP-A-4-205320 (JP, A) (58) Field surveyed (Int.Cl. ⁷ , DB name) G06T 1/00 330 B60R 21/00 624 G06T 7/60 200

Claims (12)

(57) [Claims] 1. A method for calculating a road vanishing point, which is a point where a road disappears in a screen imaged by a vehicle-mounted camera capable of imaging the front or rear of a vehicle, comprising: extracting coordinates of the line candidate point is a point to form a linear portion in the screen, with respect to the coordinates of the extracted line candidate points, the straight line candidate point
Where the coefficient of the equation of the straight line to which
The second Hough transform process is performed, and the first transform coordinates are
A fixed-size grid set within a predetermined range of the coordinate plane
The respective count values of the grid cells are obtained, and the count values of the grid cells are calculated in the first Hough transform process.
Is a certain value or more, or a grid within a certain range around
Use only points where the sum of the cell counts is above a certain value
And the coefficients of the equation of a straight line passing through this used point
Perform the second Hough transform process with the transform coordinates of
The second transformation coordinates are set within a predetermined range of a coordinate plane.
Each count value of a grid cell of a fixed size is obtained, and the total value of the grid cell is calculated on the coordinate plane of the second transformed coordinates.
The point where the numerical value is the maximum, or a grid cell within a predetermined range around it
Find the point where the sum of the eye counts is the largest,
A road vanishing point calculation method, wherein a road vanishing point is obtained based on a mark . 2. In the coordinate plane of the first transformed coordinates,
A point at which the count value of the grid cell is equal to or greater than a predetermined value, or a point at which the sum of the count values of the grid cells within a predetermined range in the surrounding area is equal to or greater than a predetermined value, is determined by determining that the distance on the coordinate plane of the first transformed coordinates is equal to the method of calculating the road vanishing point according to claim 1, wherein the point is a value less are classified as belonging to the same group. 3. In the coordinate plane of the first transformed coordinates,
A point at which the count value of the grid cells is equal to or more than a certain value, or a point at which the total value of the count values of the grid cells within a predetermined range around the certain value is equal to or more than a certain value, and among the points belonging to the same group, 3. The method of calculating a road vanishing point according to claim 2 , wherein coordinates of a point at which a straight line used as a coefficient passes closest to the road vanishing point are obtained as coefficients of a road parallel line. 4. A method according to claim 1, wherein said road parallel line is located on the left side or on the right side of the vehicle based on the sign of the obtained coefficient of the road parallel line relating to the inclination of the straight line. Item 3. The method for calculating a road vanishing point according to item 3 . 5. A road vanishing point and a road parallel line obtained when the coefficient relating to the inclination of the road parallel line among the obtained road parallel line coefficients is only one of positive or negative sign. Claims characterized by invalidating
3. The method of calculating a road vanishing point according to 3 or 4 . 6. In the coordinate plane of the first transformed coordinates,
For a group in which the count value of a grid cell is equal to or more than a certain value, or a group in which the total number of count values of grid cells in a surrounding predetermined range is not less than a certain value does not exist for a certain number or more, points belonging to the group the method for calculating the road vanishing point according to any one of claims 2 to 5, characterized in that not used in the second Hough transform process. 7. In the coordinate plane of the first transformed coordinates,
Calculation of a road vanishing point when there are less than two groups of points where the count value of the grid cells is greater than or equal to a predetermined value, or where the sum of the count values of the grid cells in a surrounding predetermined range is greater than or equal to a predetermined value. The method of calculating a road vanishing point according to any one of claims 2 to 6 , wherein the method is prohibited. 8. In the second Hough transform process, a point within a predetermined range around a point where the count value of a grid cell is maximum or a predetermined range around each of the groups classified on the coordinate plane of the first transformed coordinates. The method for calculating a road vanishing point according to any one of claims 2 to 7 , wherein only a point at which a total value of the numerical values is maximum is used. 9. In the coordinate plane of the second transformed coordinates,
If the maximum value of the count values of the grid cells is less than a certain value, or if the maximum value of the sum of the count values of the grid cells within a predetermined range is less than a certain value, the calculation of the road vanishing point is prohibited. The method for calculating a road vanishing point according to any one of claims 1 to 8 , wherein: 10. Obtaining at least information from a device having information on linearity of a road on which a vehicle is traveling, and determining whether or not the traveling road has linearity equal to or more than a predetermined value. The calculation of a road vanishing point according to any one of claims 1 to 9 , wherein the calculation of the road vanishing point is prohibited when the running road does not have a linearity equal to or more than a certain value. Method. 11. A method for acquiring information from at least an apparatus having information on attributes of a road on which a vehicle is traveling, determining whether or not the attributes of the traveling road satisfy a predetermined criterion. The method according to any one of claims 1 to 10 , wherein calculation of the vanishing point is prohibited when a middle road does not satisfy a certain standard. 12. At least information obtained from a device having information on the current position of the vehicle, the traveling direction of the vehicle, and the direction of the road on which the vehicle is traveling is obtained, and the information is fixed to the traveling direction of the vehicle and the direction of the road. Determining whether there is a difference equal to or greater than a value, and prohibiting calculation of a road vanishing point if the difference between the traveling direction of the vehicle and the direction of the road is greater than a certain value. 12. The method for calculating a road vanishing point according to any one of 11 .