JPH08241498A - Vehicle peripheral monitor device - Google Patents

Abstract

PURPOSE: To calculate the relative position of a vehicle to an object with a high precision by providing the means which calculates correction information related to be set state of an image pickup means and preventing the erroneous recognition of a road surface image based on the set state to accurately extract the object image. CONSTITUTION: A correction information acquisition means 41a calculates correction information based on the set state of an image pickup means 1, and an object extraction means 41b eliminates the background image at height '0' based on right and left image information picked up by the image pickup means 1 and correction information to extract the image information of the object. An object edge detection means 41c extracts the edge image of the object based on differential information of extracted object image information and image information of the object. The object existing in the periphery of the vehicle is recognized in accordance with object information and object edge information. As the result, erroneous recognition of the road surface image based on the set state is prevented to accurately extract the object image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a vehicle periphery monitoring device which is effectively applied to monitor the periphery of a vehicle such as an automobile and assist the driver's safety confirmation in driving the vehicle.
More specifically, the present invention relates to a vehicle periphery monitoring device that recognizes an object such as an obstacle based on image information captured by two cameras (imaging means) installed in a vehicle.

[0002]

2. Description of the Related Art Conventionally, as a vehicle periphery monitoring device of this type, for example, Japanese Patent Laid-Open No. 4-261000 and Japanese Patent Laid-Open No. 4-261000.
There is a device disclosed in Japanese Patent Laid-Open No. 301513 (hereinafter, these are collectively referred to as a conventional device). This conventional device includes a right-side camera and a left-side camera, which are horizontally spaced apart from each other by a predetermined distance at a predetermined position of a vehicle, and a right-side frame memory and a left-side frame memory that temporarily store image signals from these cameras. have. These frame memories are composed of a plurality of pixels, and the image signal stored in one of the right side frame memory and the left side frame memory is referenced to detect the edge point of the object based on the brightness difference between adjacent pixels.

With the image signal of one of the frame memories as a reference, the image signal of the other frame memory is superimposed on the image signal of the other frame memory while sequentially shifting its position and the superimposed state. The difference between both image data in is calculated.
Then, the shift amount that minimizes this difference is acquired as the parallax between both cameras, and the relative distance between the object and the camera is calculated based on this parallax.

In such a conventional apparatus, the image signal to be superimposed contains an image of height "0" (on the road surface), for example, a white line, a character or a pattern drawn on the road surface. An image on such a road surface is also a calculation target. However, since the image on the road surface does not hinder the traveling of the vehicle, the detection operation for the image on the road surface is unnecessary processing. There is a problem that the processing time becomes long due to the unnecessary processing.

In order to solve this problem,
The applicant of the present application, in Japanese Patent Application No. 6-42433 (hereinafter referred to as prior application device), extracts only the object image by removing the image on the road surface and detects the edge of the extracted object image. Then, we proposed a vehicle periphery monitoring device that recognizes the position of an object based on this edge. The prior application device will be described below.

This prior application device has a device configuration shown in FIG. In the figure, 1 is an image pickup unit as an image pickup unit, 2 is a steering angle detection unit that outputs turning information of a vehicle, 3 is a storage unit that stores image information and the like, 4 is a data processing unit that executes various calculation processes, 5 Is a display unit for displaying the position of the obstacle or a message, and 6 is a voice instruction unit for generating an alarm sound or voice guidance.

The image pickup section 1 is a right CCD camera 11R.
And the left CCD camera 11L and the right CCD camera 11
It has a right side image plane 12R for projecting the image information picked up by the R and a left side image plane 12L for projecting the image information picked up by the left CCD camera 11L. Then, as shown in FIG. 11, the image pickup unit 1 is attached to the rear of the vehicle at a position of height H at the rear center of the vehicle 100. The pair of CCD cameras 11R and 11L forming the image pickup unit 1 are attached at a predetermined distance in parallel with each other at a depression angle θ _S. As a result, the image pickup unit 1 is provided in the monitoring area 1 behind the vehicle.
00a [100a (11R) and 100a (11L)]
Is imaged.

The steering angle detector 2 includes a steering wheel angle sensor for detecting the amount and direction of rotation of the steering wheel, a steering angle sensor for detecting the steering angle of the steered wheels (generally the front wheels) (neither is shown), and the like. The turning direction of the vehicle is output as turning information based on detection signals from these sensors. The storage unit 3 includes the right image plane 12 of the imaging unit 1.
A right frame memory 31R that temporarily stores the image information projected on R as a right image signal, a left frame memory 31L that also temporarily stores the image information projected on the left image plane 12L as a left image signal, and a right frame memory 31L. A right-side corrected image memory 32R and a left-side corrected image memory 32L that hold corrected image information obtained by performing a predetermined correction process on each of the image signals stored in the left-side frame memories 31R and 31L, and a difference image, a differential image and Difference image memory 3 that holds each edge image
3, a differential image memory 34 and an edge image memory 35. Each of these memories 31 to 35 has a structure of m rows and n columns, and is configured as a memory of 512 pixels (m) × 512 pixels (n), for example. Then, in each pixel forming this memory, for example, luminance value data of 256 gradations is stored.

The data processing unit 4 has a ROM 42 in which an operation program is stored, a CPU 41 which operates according to this operation program, and a RAM 43 which temporarily stores information required when the CPU 41 operates. The display unit 5 has a display 51, and the display 51
Is based on the display image signal output from the CPU 41,
It displays the relative position of the object to the vehicle or displays a message to the driver. The voice instruction unit 6 is
Based on the buzzer sound signal or the voice guidance signal output from the CPU 41, the speaker 61 outputs the buzzer sound or the voice guidance.

In the prior application device having such a configuration, an object (obstacle) is detected and the relative position between the object and the vehicle is calculated based on the video signal from the image pickup section 1. First, the image capturing process by the image capturing unit 1 will be described.
The video signal picked up by the image pickup unit 1 is once projected onto the image planes 12R and 12L, and then stored in the right frame memory 31R and the left frame memory 31L as image information, that is, as brightness value data for each pixel. The image information stored in the right frame memory 31R and the left frame memory 31L indicates the distortion aberration of the lens of the right CCD camera 11R and the left CCD camera 11L (hereinafter, simply referred to as the CCD camera 11). It is an included image. This distortion aberration causes an error in detecting the position of the object, which will be described later.

That is, considering a case where a lattice pattern is imaged, when the lens of the CCD camera 11 has no aberration, the lattice pattern is stored in the frame memory 31 as it is, as shown in FIG. It Also,
When the lens has an aberration, as shown in FIG. 10 (b), an image in the center of the optical axis is distorted into a barrel shape bulging outward, or as shown in FIG. 10 (c). An image in which the image at the center of the optical axis is distorted into a pincushion curved inward is stored in the frame memory 31.

Then, the data processing unit 4 operates as shown in FIG.
(B) or corrected image information is generated by performing a correction process on a distorted image as shown in FIG.
This corrected image information is stored in the right side corrected image memory 32R and the left side corrected image memory 32L. This corrected image information is also stored as brightness value data for each pixel, as in the above image information. The process of correcting the distortion of the image will be described below.

As shown in FIG. 10D, the point P (x ₀ , y ₀ ) when there is no distortion is point P'due to the distortion.
Assuming that an image is formed at (x, y), the aberration amount D can be expressed by the following equation (1). _{D = [(x 0 -x)} 2 + (y 0 -y) 2] 0.5 ··· (1)

Then, FIG. 10B or FIG.
In the distorted image as shown in (c), the aberration amount D increases in proportion to the cube of the distance from the optical axis. That is, since it is proportional to the cube of the distance from the center point of the lens to the point P, the right side of the above equation (1) can be expressed by the following equation (2). [(X ₀ −x) ² + (y ₀ −y) ² ] ^0.5 = k ₁ [(x ₀ ² −y ₀ ² ) ^0.5 ] ³ (2) In the above formula, k ₁ is determined by the lens. Proportional constant

Therefore, when the lens has distortion, the distortion can be corrected by, for example, performing the calculation of the following equation (3). x ₀ ≈x (1-k ₁ (x ² + y ² )) y ₀ ≈y (1-k ₁ (x ² + y ² )) (3)

By the way, when the distortion aberration of the image is corrected by the calculation of the equation (3), the distance between the adjacent pixels becomes large and the image may be omitted. The data processing unit 4 interpolates this missing image based on the information of the adjacent pixels. Then, the image information that has undergone such correction processing and interpolation processing is corrected image information as the right corrected image memory 32R and the left corrected image memory 3
It is stored in each 2L. Thus, the image capturing process by the image capturing unit 1 is completed.

Next, the data processing unit 4 calculates the position in the captured image based on the corrected image information,
The background image (image of height “0”) is removed, and the edge image of the object is generated. Hereinafter, these processes will be described.

First, the process of calculating the position based on the captured image will be described. As described with reference to FIG. 11, the image pickup unit 1, that is, the CCD camera 11 is attached at a predetermined position behind the vehicle at an installation depression angle θ _S and picks up an image of the monitoring area 100a. Therefore, the image captured by the image capturing unit 1 is an image based on this predetermined position, and the position calculated from this captured image is also the position based on this predetermined position. Therefore, in order to facilitate the following description, as shown in FIG. 12, the coordinates based on this predetermined position, that is, the camera installation position are X ', Y', Z '.
The coordinates with respect to the road surface are represented by X, Y, Z.

In the X ', Y', Z'coordinate system with the camera installation position as a reference, the Z'axis is defined as the lens optical axis of the CCD camera 11, as shown in FIG. , The X'axis is defined as an axis parallel to the road surface, and the Y'axis is defined as an axis orthogonal to both the Z'axis and the X'axis. Therefore, the right CCD camera 1
The 1R and the left CCD camera 11L are arranged such that their lens optical axes coincide with the Z ′ axis. Also,
With respect to the X'-axis, the center points of the respective lenses are arranged on the X'-axis and the distances dxa from each other.
Are spaced apart. In order to facilitate the following description, the origin O of the X ′, Y ′ and Z ′ axes will be defined as the center of the lens of the left CCD camera 11R.

CCD camera 11 installed as described above
Point P (X _P ′, Y _P ′, Z _P ′) imaged by
Is held as P _R (x _RP , y _RP ) in the right side correction information memory 32R, and P is stored in the left side correction information memory 32L.
It is held as _L (x _LP , y _LP ).

The Z ′ coordinate Z _P ′ of this point P is shown in FIG.
As shown in the schematic view in the X′Z ′ plane of 3 (b),
It can be determined by the similarity of triangles. That is, this distance Z _P ′ can be expressed by the following equation (4). Z _P ′ = dxa · f / (x _LP −x _RP ) ... (4) In the above equation, dxa is the distance between both lenses and f is the focal length of the lens.

Similarly, for the X ′ coordinate X _P ′ of the point P,
As shown in the schematic view of the X′Z ′ plane of FIG. 13B, it can be obtained by the similarity of triangles. And Y'of point P
The coordinates Y _P ′ can also be obtained in the same manner by assuming a Y′Z ′ plane (not shown). In other words, these points P X 'coordinate X _P' and Y 'coordinate Y _P'
Can be calculated by the following equations (5) and (6), respectively. _{X P '= Z P' x} LP / f = dxa · x LP / (x LP -x RP) ··· (5) Y P '= Z P' y LP / f = dxa · y LP / (x LP -X _RP ) ... (6) Regarding the coordinate X _P ′, the reference coordinate is the right CC
When the distance is between the D camera 11R and the left CCD camera 11L, the difference of 1/2 of the coordinate X _P ′ calculated by the above equation (5) and the distance dxa between both lenses may be taken.

A point P (X _P ′, Y _P ′, Z in the X′Y′Z ′ coordinate system calculated by the above equations (4) to (6).
_{Since P} ') is the coordinate value of the coordinate system with the camera installation position as the reference as described above, it is necessary to convert this coordinate value into the coordinate value in the XYZ coordinate system with the road surface as the reference. In this case, if the depression angle of the image pickup unit 1 (CCD camera 11) is θ _S , the coordinates (X _P ′, Y _P ′, Z _P ′) of the point P at the X′Y′Z ′ coordinates and the road surface. XYZ based on
The relationship with the coordinates is as shown in FIG.

Therefore, the coordinates (Z _P ′, Y _P ′, _XP ′) calculated by the above equations (4) to (6) are converted into XYZ coordinates by executing the following equations (7) to (9). It can be converted into the coordinates (X _P , Y _P , Z _P ) of the point P in the system. _{X P = X P '··· (} 7) Y P = H-Z P' cos θ S + Y P 'sin θ S ··· (8) Z P = Z P' sin θ S + Y P 'cos θ S ... (9)

Next, the object recognition processing by the data processing unit 4 will be described. First, the removal process of the background image (image of height “0”) will be described. 14
(A) shows the right-side corrected image stored in the right-side corrected image memory 32R after being subjected to the above-described distortion aberration correction after being picked up by the right-side CCD camera 11R. In the figure, 300 is a white line drawn on the road surface, and 240 is a pole-shaped object. Here, it is assumed that all the right-side corrected images stored in the right-side corrected image memory 32R are background images having a height of "0", that is, images drawn on the road surface. Then, based on the right-side corrected image assumed in this way, conversion processing is performed so that this right-side corrected image becomes an image (projection image) taken at the image pickup position of the left CCD camera 11L [FIG. 14 (b)].

The conversion process of the projected image will be described below. Here, the point of the projection image corresponding to the point P _R (x _RP , y _RP ) of the right image is P _L ′ (x _LP ′, y _LP ′).
As shown in FIG. 12, the X ′ axis of the camera coordinates and the X axis of the road surface coordinates are parallel, and the x axes (x _L axis and x _R axis in FIG. 13) of the scanning lines imaged by the camera are also parallel. , Y of the captured image when the same object is captured.
_{The L} and y _R values match. If all the images are on the road surface, the value of Y _P shown in the above equation (8) is zero.
From the above, the following equations (10) and (11) can be derived. Then, by substituting equation (11) Z _P 'Y _P and of the formula (6)' of the formula (4) Z _P 'and Y _P' of, as shown in equation (12), x _LP ' Can be asked.

Y _LP ′ = y _RP (10) 0 = H _P −Z _P ′ cos θ _S + Y _P ′ sin θ _S (11) x _LP ′ = (dxa · f cos θ _S −dxa · Y _RP sin θ _S ) / H + x _RP (12) The data processing unit 4 creates a projection image [FIG. 14 (b)] by performing the calculations of the above formulas (10) and (12). To do.

Then, when the projection image thus created is superimposed on the left side corrected image, it becomes as shown in FIG. 14 (c). That is, when the image captured by the right CCD camera 11R is projected, the pattern such as a white line drawn on the road surface matches the pattern captured by the left CCD camera 11L in position and brightness, and the object is higher than the road surface. The greater the difference, the greater the difference. Therefore, by taking the difference between the left image data and the projection image data, the brightness value of the pixels forming the road surface other than the pixels forming the object having a height becomes a value "0" or a value close to "0". . If the value equal to or smaller than the predetermined threshold value is set to "0", all values will be "0". In this way, by taking the difference between the left image data and the projection image data, the road surface image (background image of height “0”) is removed from the difference image, as shown in FIG. 14D. Only the portion with the depth is extracted as a value other than the value “0”. Then, this difference image is stored in the difference image memory 33 of the storage unit 3.

By the way, the projected image contains an error.
The difference image [Fig. 14 (d)]
In some cases, the road surface image may remain. this
Therefore, a difference image between the projected image and the left-side corrected image is generated.
In that case, the following processing is performed. That is, FIG.
As shown in (a) and (b), the brightness of each pixel of the projected image
Degree value I₁~ I₇The brightness of each pixel of the left-side corrected image corresponding to
Value I₁'~ I ₇′ Is calculated sequentially, and the difference value is predetermined
Value I₀The above pixels [pixel I_FiveEquivalent to]
Is the pixel I as shown in FIG._FiveAdjacent to ′
Pixel I_Four'And I₆Find the difference with ′. And that
Minimum value of difference (pixel I in FIG._FourThe difference value from ′)
The difference value with respect to the prime.

In this way, the background image of height "0" in the captured image is removed, and only the image with height, that is, the image of the object is extracted. Following this, edge extraction processing is performed on the extracted object image. Next, this object edge extraction processing will be described.

This object edge extraction processing is performed based on the image information stored in the left side corrected image memory 32L. That is, with respect to the left-side corrected image stored in the left-side corrected image memory 32L, the brightness value I _{m, n} of the image data in the m-th row and the n-th column is scanned in the horizontal direction, that is, in the X′-axis direction in FIG. A differential image is generated by the operation of 13). | I _{m, n + 1} −I _{m, n} | ≧ E ₀ , E _{m, n} = 1 | I _{m, n + 1} −I _{m, n} | <E ₀ , E _{m, n} = 0. (13) In the above equation, E ₀ is the threshold value

In this differential image, as shown in FIG. 16B, an image in which the vertical edge portion of an object or a character drawn on the road surface is "1" and the other portions are "0". Becomes Then, this differential image is stored in the differential image memory 34 of the storage unit 3. The differential image obtained in this way [FIG. 16 (b)] and the differential image generated by the above-described road surface image removal processing and held in the differential image memory 33 [FIG.
When (d)] is superimposed and AND is taken, an image of the object edge in which only the edge part of the object shown in FIG. 16C is extracted is generated. This object edge image is stored in the edge image memory 35.

Then, the data processing unit 4 recognizes the object based on the object edge image stored in the edge image memory 35 and calculates the relative position between the object edge and the vehicle. Further, it predicts the course of the vehicle from the turning information of the vehicle, predicts a collision between the object recognized based on the predicted course and the vehicle, and issues a warning or the like to warn the driver when necessary.

[0034]

In such a prior application device, the background image having the height of "0", that is, the road surface image is removed based on the video signal picked up by the image pickup unit 1 to remove the object image. Since it is configured to generate an edge image and calculate the relative position between the object and the vehicle based on the generated edge image, there is an advantage that unnecessary calculation processing can be omitted and the processing can be speeded up, which is a good method. .

By the way, in this prior application device, the right side CC is used as a parameter for performing the various calculations described above.
Center of lens of D camera 11R and left CCD camera 11L
The distance dxa between the lens centers, the focal length f of each lens, the installation height H of the CCD cameras 11R and 11L, the installation depression angle θ _S of the CCD cameras 11R and 11L, and the like are used. However, it has been difficult to make these parameters (hereinafter referred to as installation parameters) constant depending on the mounting state of the imaging unit 1 on the vehicle.

The deviation of the installation parameter appears as a difference between the coordinate value calculated by calculation and the actual coordinate value and becomes a detection error, and the background image of the height "0" which should originally be recognized as the "road surface image". Is recognized as a tall object, that is, an "obstacle". Therefore, in order to set the installation parameter to a constant value, it is necessary to accurately mount the image pickup unit 1 on the vehicle 100, which requires a long time for mounting. In addition, the image pickup unit 1 is connected to the vehicle
Even if it is correctly attached to the predetermined position, since this attachment position will shift with the passage of time, it was difficult to maintain the installation parameter at a constant value.

As described above, in this prior application device,
In the above points, there was room for further improvement. Further, the calculated position of the obstacle may be different from the actual position due to the variation of the installation parameters, and there is room for further improvement in this respect as well. Therefore, in view of such improvements, the present invention provides a vehicle periphery monitoring device capable of preventing an erroneous recognition of a road surface image due to the variation of the installation parameters, and accurately extracting an object, and a position of an obstacle. An object of the present invention is to provide a vehicle surroundings monitoring device that can be accurately calculated.

[0038]

A vehicle periphery monitoring device made according to the present invention to solve the above problems will be described with reference to the basic configuration diagram of FIG. That is, the vehicle periphery monitoring device of the present invention is based on the image information from the image pickup means 1 including the right side image pickup means 11R and the left side image pickup means 11L which are arranged at predetermined positions of the vehicle and are separated from each other by a predetermined distance. In the vehicle periphery monitoring device for monitoring the periphery of the vehicle by means of the right side image holding means 31R for holding the right side image information from the right side image pickup means 11R, and the left side image pickup means 1
Left side image holding means 3 for holding left side image information from 1L
1L, an image holding unit 31, a correction information obtaining unit 41a that obtains correction information for correcting a detection error based on the installation state of the image pickup unit 1, and based on the right and left image information and the correction information. , The object image extracting unit 41b which extracts the image information of the object by removing the background image of the height “0”, and outputs the extracted image information of the object as the object information, and the image storing unit 31 which holds the image information. Calculate the differential value in the horizontal direction for the image information
An object edge detection unit 41c that detects an edge of an object based on the differential information and the object information and outputs the detection result as the object edge information, and detects the vehicle periphery from the object information and the object edge information. It is characterized by recognizing objects existing in.

Further, the correction information acquisition means 41a calculates the actual parallax of the image signal picked up by the image pickup means 1 based on the right side image information and the left side image information.
1e, a theoretical parallax calculating unit 41f that calculates a theoretical theoretical parallax based on the installation state of the image pickup unit 1, and a correction information calculating unit 41g that calculates correction information based on these actual parallax and theoretical parallax. Is characterized by.

The image pickup means 1 is placed at an arbitrary position on the vehicle.
A reference image projecting unit 7 for projecting a reference image serving as a reference for measurement is provided in the image pickup area of the real parallax calculating unit 41e.
Is imaged by the imaging unit 1 based on the right side reference image acquired by the right side imaging unit 11R capturing the reference image and the left side reference image acquired by the left side imaging unit 11L capturing the reference image. The feature is that the actual parallax of the image signal is calculated.

Also, an object position calculating means 41d for calculating the position of the object based on the image information held in the image holding means 31 using the object edge detected by the object edge detecting means 41c, Display means 5 for displaying the relative position of the vehicle and the object is provided, and the driver is informed of the relative position of the object of the vehicle based on the position information of the object output from the object position calculation means 41d. It has a feature.

Also, an object position calculating means 41d for calculating the position of the object based on the image information held in the image holding means 31, utilizing the object edge detected by the object edge detecting means 41c, Route prediction means 41h for predicting the route of the vehicle based on the turning information of the vehicle
And an alarm means 6 for issuing an alarm such as an alarm sound,
When the collision between the vehicle and the object is predicted based on the position information of the object output from the object position calculation means 41d and the predicted course information from the course prediction means 41h, and the "collision" is predicted, the alarm is issued. An alarm is generated by at least one of the means 6 and the display means 5.

[0043]

In the above structure, the correction information acquisition means 41a calculates the correction information based on the installation state of the image pickup means 1, and the object extraction means 41b the right and left image information picked up by the image pickup means 1 and the correction information. The image information of the object is extracted by removing the background image of the height “0” based on and. The object edge detecting means 41c extracts the edge image of the object based on the differential information of the extracted object image information and the image information of the object. Then, the object existing around the vehicle is recognized from the object information and the object edge information. That is, since the means for calculating the correction information regarding the installation state of the image pickup means 1 is provided, it is possible to eliminate the erroneous recognition of the road surface image based on the installation state, and to accurately extract the object image. Further, the relative position between the vehicle and the object can be calculated with high accuracy.

The actual parallax calculation means 41e is the right side image pickup means 11
The actual parallax of the image pickup means 1 is calculated based on the right side image information from R and the left side image information from the left side image pickup means 11L, and the theoretical parallax calculation means 41f calculates the theoretical parallax based on the installation state of the image pickup means 1. That is, since the correction information is calculated based on the right-side image information and the left-side image information, the device can be easily configured.

The reference image projecting means 7 projects a reference image serving as a measurement reference into the image pickup area of the image pickup means 1, and the right side image pickup means 11R and the left side image pickup means 11L respectively pick up the reference image. The actual parallax calculating means 41e is the right imaging means 1
The actual parallax is calculated from the right side reference image captured by 1R and the left side reference image captured by the left side imaging means 11L. That is, since the means for projecting the reference image for calculating the actual parallax is provided, the correction information can be calculated regardless of the road surface condition.

The object position calculating means 41d calculates the relative position between the object and the vehicle based on the object edge information from the object edge detecting means 41c and the image information held in the image holding means 31. The display means 5 informs the driver of the relative position of the object detected based on the relative position information of the vehicle and the object. That is, since the means for calculating the relative position of the detected object with respect to the vehicle and the means for displaying this object are provided, it is possible to inform the driver of the relative position of the detected object with respect to the vehicle.

The route predicting means 41h predicts the course of the vehicle based on the turning information of the vehicle, and the display means 5 or the warning means 6 predicts that the predicted vehicle course and the object "collide". Generate an alarm. Therefore, an accident such as a collision with an object can be prevented.

[0048]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A vehicle periphery monitoring device of the present invention will be described below with reference to the drawings. The vehicle periphery monitoring device according to the present invention has substantially the same configuration as the prior application device described with reference to FIG. That is, as shown in FIG.
An image pickup unit 1 having a D camera 11R, a left CCD camera 11L, and the like, and a steering angle detection unit 2 configured as a steering wheel steering angle sensor or a steering angle sensor to output vehicle turning information.
Right frame memory 31R, left frame memory 31
L, right side corrected image memory 32R, left side corrected image memory 3
2L, a difference image memory 33, a differential image memory 34, and an edge image memory 35, a storage unit 3, a CPU 41 that operates according to an operation program, a ROM 42 that stores the operation program, and a RAM that temporarily stores necessary information.
A data processing unit 4 composed of 43 and a display unit 5 for displaying a relative position between an object and a vehicle, a message, etc. to a driver.
And a voice instruction unit 6 that outputs a buzzer sound or voice guidance to the driver, and in addition to this, an image projection unit 7 that projects a reference image is provided.

The image projection unit 7 is incorporated in the image pickup unit 1 as shown in FIG. 6A, for example, and more specifically, it is arranged at a substantially central position between the right CCD camera 11R and the left CCD camera 11L. ing. Then, as shown in FIG. 6B, the image projecting unit 7 uses the light bulb 71 as a light source.
a and the vertical direction in the central part (vertical direction in the figure)
The mask pattern 71 in which the slit extended to the
b and the lens 71c, the light from the electric bulb 71a is made into linear light by the mask pattern 71b, and this linear light is projected onto the road surface via the lens 71c. Further, as shown in FIG. 6C, the image projection unit 7 is replaced with a laser light source 72.
The laser light emitted from the laser light source 72a after being appropriately scanned may be converted into linear light via the cylindrical lens 72b and projected onto the road surface.

Such an image projection unit 7 is shown in FIG.
As shown in (a), the imaging area 100a by the imaging unit 1
At, the reference image 400 is projected as linear light extending from the front side to the back side of the central portion. The image projection unit 7 may be configured as a single unit, and in this case, the image projection unit 7 can be arranged at an arbitrary position different from the image pickup unit 1, as indicated by the symbol A in FIG. In addition, the slits of the mask pattern 71b described in FIG. 6B are a plurality of rectangular slit patterns linearly arranged, and
As shown in (b), a plurality of reference points 400 arranged in a straight line from the front side to the back side of the central portion of the imaging region 100a.
It may be configured to project a.

In the present invention, the ROM 42 is used.
The operation program stored in the above, that is, the processing operation by the data processing unit 4 is also different. Hereinafter, this processing operation will be described with reference to the flowcharts of FIGS. In this processing operation, first, in step S310 in the flowchart of FIG. 3, a “0” correction value acquisition operation is performed. The "0" correction value acquisition operation in step S310 is specifically performed by the flowchart of FIG.

That is, first in step S510,
The reference image is captured by the right CCD camera 11R and the left CCD camera 11L. That is, in this step S510, first, the image projection unit 7 (see FIG. 6).
The reference image 400 is projected (see FIG. 7) in the monitoring area 100a by the. Then, the right side C of the image pickup unit 1
The reference image 400 projected by the CD camera 11R and the left CCD camera 11L is captured. As a result, the right reference image information and the left reference image information are stored in the frame memory 31R and the frame memory 31L, respectively. Then, the frame memories 31R and 3R
With respect to the image information stored in 1L, the distortion aberration correction process is performed in FIG. 10, and the right reference image shown in FIG. 8A and the left reference image shown in FIG. 8B are generated. And
These reference images are stored in the right side correction memory 32R and the left side correction memory 32L, respectively.

In step S510, if there is a straight line image such as a center line drawn on the road surface in the monitoring area 100a of the image pickup section 1, this may be taken in as a reference image. And this step S510
Is completed, the process proceeds to step S520.

In step S520, the above step S5 is performed.
Based on the left and right reference images acquired in 10, the coordinate values of the point of interest in the reference image are acquired for each of the left and right reference images. More specifically, the intersection coordinates of the left edge of the right reference image 400R in FIG. 8A and the scanning line 41n are acquired as P _RO (x _RPO , y _RPO ) and the left reference image 400L in FIG. The intersection coordinates of the left edge and the scanning line 41n are acquired as P _LO (x _LPO , y _LPO ). Then, such a coordinate acquisition operation is sequentially performed from the scanning line 41a at the upper end to the scanning line 41b at the lower end to acquire a group of intersection coordinates, and step S5
Move to 30.

In step S520, a plurality of scanning lines set at arbitrary intervals are targeted for calculation, whereby the subsequent calculation processing can be speeded up. For example, as shown in FIG. 8C, a plurality of scanning lines 411 to 419 are set at predetermined intervals, and the coordinates of the intersections of the scanning lines 411 to 419 and the edges of the reference image are acquired. By this acquisition operation, the scanning line 41
4, the intersection point coordinate P _R4 with the right side reference image 400R and the intersection point coordinate P _L4 with the left side reference image 400L are acquired,
Similarly, for the scan line 418, the intersection coordinates P _R8 and P _L8
And are obtained.

In step S530, the above step S5 is performed.
The right CCD camera 11R is calculated by calculating the following equation (14) for each intersection coordinate of the intersection coordinate group acquired in 20.
And the parallax (actual parallax) D _ac of the image captured by the left CCD camera 11L is calculated. D _ac = x _LPO −x _RPO (14) For example, regarding the scanning lines 414 and 418 of FIG. 8C, the actual parallax 424 (scanning line 414) and the actual parallax 418 ( The scan line 418) is calculated.

In step S540, the theoretical parallax D _th based on the installation parameters of the image pickup section 1 is calculated. This theoretical parallax D _th, that is, x _LP ′ −x _RP can be calculated by the calculation of the following expression (15) derived based on the above expression (12). D _th = x _LP ′ −x _RP = (dxa · fcos θ _S −dxa · y _RP sin θ _S ) / H (15) When the theoretical parallax is calculated in step S540, the process proceeds to step S550. To do.

In step S550, the above step S5 is performed.
The “0” correction value k ₂ is calculated from the difference between the actual parallax D _ac calculated in step S30 and the theoretical parallax D _th calculated in step S540, that is, the following equation (16). k ₂ = D _ac −D _th (16) Then, the calculated “0” correction value k ₂ is stored in the RAM 43 of the data processing unit 4, and a series of “0” correction value acquisition operation is completed. The process moves to step S320 in the flowchart of FIG.

In step S320, the right side C of the image pickup unit 1
Video information from the CD camera 11R and the left CCD camera 11L is used as right image information and left image information, respectively, the right frame memory 31R and the left frame memory 31.
Store in L and move to step S330. The image information acquired in step S320 is the image information in the monitoring area 100a.
This is for obtaining obstacles in 00a. Therefore, the above step S320 (flow chart of FIG. 5)
It is an image different from the reference image acquired in. Then, in step S330, the image information held in the right frame memory 31R and the left frame memory 31L is read, and the distortion aberration correction processing described in FIG. 10 is performed on the read image information. Then, the corrected image information that has been subjected to this correction processing is stored in the right side corrected image memory 32
After storing in the R and left side corrected image memory 32L, the process proceeds to step S340.

In step S340, with respect to the left side corrected image stored in the left side corrected image memory 32L, the above equation (1
The differential image [FIG. 16 (b)] for the left side corrected image is generated by performing the calculation process of 3), and the generated differential image is stored in the differential image memory 34. Then, the process proceeds to step S350. In step S350, the projection image creation process described in FIG. 14, that is, it is assumed that all the heights of the right side corrected images stored in the right side corrected image memory 32R are “0”, and the coordinate conversion process is performed on the right side corrected image. And left side CCD camera 11L
A projection image [Fig. 14 (b)] imaged in step 1 is created.

In step S350, the projection image in which the variation in the installation parameters of the image pickup unit 1 is corrected is created by using the "0" correction value k ₂ acquired in the flowchart of FIG. That is, this step S3
At 50, the following equations (17) and (18) are executed to create a projected image. y _LP ′ = y _RP (17) x _LP ′ = (dxa · fcos θ _S −dxa · y _RP sin θ _S ) / H + x _RP + k ₂ (18) Then, when this step S350 ends Step S
Move to 360. In step S360, step S
Superimpose the projection image created in 350 on the left correction image [Fig.
4 (c)] and the difference between the two is taken to create a difference image [FIG. 14 (d)], and the created difference image is stored in the difference image memory 33. And step S
Move to 370.

In step S370, object edge image creation processing is performed. That is, the differential image stored in the differential image memory 34 in step S340 and the differential image stored in the differential image memory 33 in step S350 are overlapped and the AND is performed to obtain the object edge image [FIG. )] Is generated. Then, the generated object edge image is stored in the edge image memory 35, and step S
Move to 380. In step S380, for the edge portion of the object edge image data created in step S370, the corresponding points of the left and right images of the object to be measured are corrected to the right corrected image memory 32R and the left corrected image memory 32L.
More extraction is performed, and the process proceeds to step S390.

In step S390, the above step S3 is performed.
For each coordinate point of the object edge extracted in 70, FIG.
Also, the three-dimensional position coordinate calculation operation described with reference to FIG. 13 is performed to calculate the three-dimensional coordinate values of this edge image. Then, in the coordinate position calculating operation in step S390, the “0” correction value k ₂ described in the flowchart of FIG.
Coordinates are calculated using. That is, this step S
In 390, the coordinate values Z _P ′, X _P ′, and Y _P ′ are calculated by executing the operations of the following equations (19) to (21), and the calculated coordinate values are used to calculate the above equation (7). ) -Equation (9) is executed to obtain the coordinate values Z _P , X _P and Y _P, that is, the position information of the obstacle in the monitoring area 100a. _{Z P '= dxa · f /} (x LP -x RP + k 2) ··· (19) X P' = dxa · x LP / (x LP -x RP + k 2) ··· (20) Y P ' = Dxa · y _LP / (x _LP −x _RP + k ₂ ) ... (21) Then, when this step S390 ends, the process moves to step S410 shown in the flowchart of FIG.

In step S410, the data processing unit 4
Reads the turning information from the steering angle detector 2, for example, the steering angle information of the steering wheel. Then, the process proceeds to step S420. In step S420, the course of the vehicle is predicted based on the turning information read in step S410, and the predicted trajectory of the vehicle is calculated by adding the shape information of the vehicle to the predicted course. Then, the calculated predicted trajectory is stored in the RAM 43.
In step S430.

In step S430, the above-mentioned step S3 is performed.
Based on the obstacle information acquired at 90, the monitoring area 100
It is determined whether or not there is an obstacle in a. If it is determined in step 430 that “exists”, that is, “exists”, the process proceeds to subsequent step S440, while if it is determined that “does not exist”, that is, “none”, proceeds to step S300 in the flowchart of FIG. Transition.

In step S440, the above-mentioned step S4
Based on the predicted trajectory calculated in 20 and the obstacle position information calculated in step S430, it is determined whether there is a possibility of collision between the edge area of the obstacle and the vehicle. If it is determined that there is a possibility of collision, the process proceeds to step S441, and if it is determined that there is no possibility of collision, the process proceeds to step S442.

In step S441, the voice instruction unit 6 issues a buzzer sound or a voice guidance alarm to the driver, and the display unit 5 displays a predicted collision position between the vehicle and the obstacle and a message. Also,
In step S442, the relative position between the obstacle and the vehicle is displayed on the display unit 5. Then, these steps S4
41 and step S442, the series of processes is terminated,
The process moves to step S300 in the flowchart of FIG.
In step S300, it is determined whether or not a “0” correction value in the next processing operation is to be acquired. If it is determined to “acquire”, that is, “Y”, the process proceeds to step 310 and the “0” correction is performed. Update the value k ₂ . Also "do not get"
That is, when it is determined to be "N", the process proceeds to step S320 and the series of processes is performed again. This "0" correction value k ₂ of the case, the use of "0" correction value k ₂ up to the last time.

As is clear from the above description, the basic configuration of the present invention and the embodiments have the following correspondence. That is, the image pickup unit 1 is the same as the image pickup unit 1 in the embodiment, and the right side image pickup unit 11R and the left side image pickup unit 11L are each on the right side C.
The CD camera 11R and the left CCD camera 11L, the image holding means 31 in the frame memories 31R and 31L, the correction information acquisition means 41a in steps S510 to S550 in the flowchart of FIG. 5, and the object extraction means 41b in the flowchart of FIG. S340-S3
In step 60, the object edge detection means 41c similarly performs step S
In 370, the object position calculation means 41d similarly performs step S
It corresponds to 390.

The actual parallax calculating means 41e corresponds to steps S510 to S530 in the flowchart of FIG. 5, the theoretical parallax calculating means 41f corresponds to step S540, and the correction information calculating means 41g corresponds to step S550. Further, the reference image projection means 7 is the image projection unit 7
The route prediction means 41i corresponds to steps S410 and S420 in the flowchart of FIG. 4, the display means 5 corresponds to the display unit 5, and the alarm means 6 corresponds to the voice instruction unit 6.

[0070]

As described above, according to the vehicle periphery monitoring apparatus of the present invention, since the means for calculating the correction information regarding the installation state of the image pickup means is provided, the erroneous recognition of the road surface image based on the installation state is eliminated. Therefore, the object image can be accurately extracted. Further, the relative position between the vehicle and the object can be calculated with high accuracy.

Further, since the correction information is calculated on the basis of the right side image information and the left side image information, the apparatus can be simply constructed.

Further, since the means for projecting the reference image for calculating the actual parallax is provided, the correction information can be calculated regardless of the condition of the road surface.

Since the means for calculating the relative position of the detected object with respect to the vehicle and the means for displaying this object are provided, it is possible to inform the driver of the relative position of the detected object with respect to the vehicle. it can.

Further, means for predicting the course of the vehicle based on the turning information of the vehicle and means for issuing an alarm when the predicted vehicle course and an object are predicted to "collide" are provided. Therefore, an accident such as a collision with an object can be prevented.

[Brief description of drawings]

FIG. 1 is a basic configuration diagram of the present invention.

FIG. 2 is a configuration diagram of an embodiment of the present invention.

FIG. 3 is an operation flowchart of the same embodiment.

FIG. 4 is an operation flowchart of the embodiment.

FIG. 5 is an operation flowchart of the embodiment.

FIG. 6 is a diagram illustrating an image projection unit 7.

7 is a diagram illustrating a reference image 400. FIG.

FIG. 8 is a diagram illustrating an operation of acquiring a “0” correction value.

FIG. 9 is a block diagram of a prior application device.

FIG. 10 is an explanatory diagram of lens aberration correction in the example and the prior application device.

FIG. 11 is an explanatory diagram of a manner of mounting the image pickup unit.

FIG. 12 is an explanatory diagram of camera depression angle correction.

FIG. 13 is also an explanatory diagram of three-dimensional position measurement.

FIG. 14 is also an explanatory diagram of road surface image removal.

FIG. 15 is an explanatory diagram of difference image creation.

FIG. 16 is also an explanatory diagram of object edge detection.

[Explanation of symbols]

 1 Image pickup unit 11R, 11L CCD camera 2 Steering angle detection unit 3 Storage unit 31R, 31L Frame memory 35 Edge image memory 4 Data processing unit 41 CPU 42 ROM 43 RAM 5 Display unit 6 Voice instruction unit 7 Image projection unit 100 Vehicle 200 Obstacle Object 300 White line (Road surface image) 400 Reference image

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Kazuyuki Sasaki 1500 Onjuku, Susono-shi, Shizuoka Yazaki Corporation

Claims (5)

[Claims] 1. A vehicle periphery for monitoring the periphery of the vehicle on the basis of image information from an image pickup means provided with a right side image pickup means and a left side image pickup means arranged at a predetermined position of the vehicle and separated from each other by a predetermined distance. In the monitoring device, an image holding unit having a right side image holding unit holding the right side image information from the right side imaging unit and a left side image holding unit holding the left side image information from the left side imaging unit; A correction information acquisition unit that acquires correction information for correcting a detection error based on the installation state, and a background image of height “0” is removed based on the right and left image information and the correction information. Object extracting means for extracting image information and outputting the image information of the extracted object as object information, and image information held by the image holding means A differential value in the horizontal direction is calculated, and based on the differential information and the object information, an object edge detection unit that detects the edge of the object and outputs the detection result as object edge information is provided, and the object information is A vehicle periphery monitoring device, which recognizes an object existing around the vehicle from the object edge information. 2. The correction information acquisition means, based on the right side image information and the left side image information, an actual parallax calculation means for calculating an actual parallax of an image signal imaged by the imaging means, and an installation state of the imaging means. The vehicle periphery monitoring device according to claim 1, further comprising a theoretical parallax calculating unit that calculates a theoretical parallax and a correction information calculating unit that calculates correction information based on the actual parallax and the theoretical parallax. 3. A reference image projecting means for projecting a reference image serving as a measurement reference in an image pickup area of the image pickup means is provided at an arbitrary position of a vehicle, and the actual parallax calculation means picks up the reference image by the right side image pickup. Characterized in that the actual parallax of the image signal captured by the image capturing means is calculated based on the right side reference image acquired by the image capturing by the means and the left side reference image acquired by capturing the reference image by the left side image capturing means. The vehicle periphery monitoring device according to claim 2. 4. An object position calculating means for calculating the position of the object based on the image information held in the image holding means using the object edge detected by the object edge detecting means, the vehicle and the object. A display means for displaying the relative position of the object is provided, and the driver is informed of the relative position of the object and the vehicle based on the position information of the object output from the object position calculation means. The vehicle periphery monitoring device according to 1, 2, or 3. 5. An object position calculating means for calculating the position of the object based on the image information held in the image holding means using the object edge detected by the object edge detecting means, and the turning of the vehicle. A route prediction means for predicting the route of the vehicle based on the information, and an alarm means for issuing an alarm such as an alarm sound are provided, and the position information of the object output from the object position calculation means and the route prediction means are provided. 2. A collision is predicted between a vehicle and an object on the basis of predicted route information, and when a "collision" is predicted, an alarm is issued by at least one of the alarm means and the display means. The vehicle periphery monitoring device according to 2, 3, or 4.