JP2004289305A - Vehicle-mounted imaging system and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make one imaging apparatus able to image both a distance calculating image and a displaying image. <P>SOLUTION: In the imaging apparatus 2, a left and right mirrors 7A, 7B inclined with specified angles are arranged on the left and right sides of a wide-angle lens 6, and an imaging pixel part 8 is provided facing the wide-angle lens 6. The imaging pixel part 8 is provided with a direct zone 8c at the center for directly inputting an image passed through the wide-angle lens 6 and a left and right reflecting zones 8a, 8b at the left and right for inputting images reflected from the left and right mirrors 7A, 7B. The image taken by the image pixel part 8 is sent to an image processor ECU connected through a network cable 5a, and the processor ECU converts it to a displaying image and processes the images taken on the left and right reflecting zones 8a, 8b to calculate the distance. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an in-vehicle image pickup system and an image pickup apparatus in which a single image pickup apparatus obtains a picked-up image and information relating to calculation of a distance to an image pickup target.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, there is a system in which a camera for capturing an image around a vehicle is installed at a front portion, a rear portion, or the like of a vehicle, and an object existing around the vehicle is detected using a captured image and a distance to the detected object is calculated. . Such a system detects an object by image processing of a captured image, image conversion, and the like, and calculates a distance by using a type using an image captured by one monocular camera and an image captured by two monocular cameras. There are types to use.
[0003]
In the type using an image captured by one single-lens camera, the distance is calculated based on images captured in the same direction, so that the error between the calculated distance and the actual distance is large. On the other hand, in the type that uses images captured by two monocular cameras, a configuration similar to a stereo camera is realized by arranging each camera at an interval, and highly accurate distance calculation is performed based on the principle of triangulation. I was
It should be noted that imaging of the surroundings of the vehicle is also disclosed in Patent Document 1 below.
[0004]
[Patent Document 1]
JP 2001-219784 A
[Problems to be solved by the invention]
The above-described stereo camera-like configuration can achieve high-accuracy distance calculation. However, since two cameras are used, the time and effort required for arranging and connecting the cameras increases, and the cost is reduced to a system using one camera. There was a problem that it increased. In particular, the arrangement of each camera had to be strictly matched with the imaging direction in order to make a configuration like a stereo camera.
[0006]
It is also conceivable to realize a configuration like a stereo camera using one camera and a plurality of mirrors. For example, by arranging two or more mirrors like a stereo camera and setting the angle of each mirror so that the reflected light of each mirror is incident on the lens of one camera, the reflected light of each incident High-precision distance calculation can be performed on a captured image using the principle of triangulation.
[0007]
However, in this case, since the image to be captured is all due to the reflected light, the captured image becomes a mirror image and the illusion due to the influence of each mirror, lens, and the like appears in the captured image. There is a problem that cannot be displayed in the state of. In addition, when an object is detected using such a captured image, there is a problem that the accuracy of the detection processing is deteriorated and a required detection capability cannot be secured.
[0008]
Therefore, in the case of using one camera and a plurality of mirrors as described above, in order to display the situation around the vehicle in a natural state on the monitor and to maintain the detection accuracy of the object, it is necessary to directly It is necessary to separately prepare a camera into which light is incident, which complicates the configuration of the entire system, and there is a concern that the arrangement of mirrors, the connection of cameras, and the like require time and effort.
[0009]
The present invention has been made in view of such circumstances, and provides an imaging pixel portion of one camera with an area where an image reflected by a mirror is incident and an area where an image is directly incident. Accordingly, it is an object of the present invention to provide an in-vehicle imaging system and an imaging apparatus that enable a single camera to capture a reflected image and a direct image.
[0010]
It is another object of the present invention to provide an in-vehicle imaging system in which a plurality of regions corresponding to a plurality of mirrors are provided in an imaging pixel unit, so that the calculation accuracy of a distance to a captured image is improved.
Still another object of the present invention is to provide an in-vehicle imaging system capable of displaying an image in which the influence of a lens has been removed by performing distortion correction on a captured image in an area where an image is directly incident. I do.
[0011]
[Means for Solving the Problems]
A vehicle-mounted imaging system according to a first invention is a vehicle-mounted imaging system that includes a vehicle-mounted imaging device that outputs a captured image and an image processing device that performs processing on an input image, wherein the imaging device includes a mirror, An imaging pixel unit having a reflection area on which an image reflected by the mirror is incident and a direct area on which the image is directly incident, wherein the image processing apparatus is imaged with respect to an image captured by the reflection area. A distance calculation unit that performs a process for calculating a distance to the image, and a conversion unit that performs a process of converting a captured image of the direct area into a display image.
[0012]
An in-vehicle imaging system according to a second aspect is characterized in that the plurality of mirrors are provided, and the imaging pixel section has a plurality of reflection areas into which images reflected by the plurality of mirrors are respectively incident.
[0013]
In the vehicle-mounted imaging system according to a third aspect, the imaging apparatus further includes a curved lens facing the imaging pixel unit, and the conversion unit includes a correction unit configured to correct image distortion caused by the curved lens. Features.
In the vehicle-mounted imaging system according to a fourth aspect, the image processing apparatus further includes a distortion correction table having distortion correction information corresponding to each pixel of the image, and the correction unit corrects image distortion based on the distortion correction table. Is performed.
[0014]
An imaging device according to a fifth aspect of the present invention is the imaging device for mounting on a vehicle, wherein the imaging pixel includes a mirror, a reflection region where an image reflected by the mirror is incident, and a direct region where the image is directly incident. And a unit.
[0015]
In the first invention and the fifth invention, the imaging device includes the mirror and the reflection region and the direct region are provided in the imaging pixel portion, so that both the reflected light and the direct light of the mirror can be performed by one imaging device. Can be imaged. As a result, it is possible to calculate the distance to the imaging target and display the surroundings of the vehicle with a simple configuration.
[0016]
According to the second aspect of the present invention, by providing a plurality of mirrors and a plurality of reflection areas, each mirror can be arranged as a stereo camera, which can contribute to an improvement in distance calculation, and more than two mirrors Is used, more complicated three-dimensional image information can be obtained, and the range of the processing target can be expanded.
[0017]
According to the third aspect of the invention, even if a curved lens is used in the imaging device, the image distortion due to the curved lens is corrected, so that a natural display image with no distortion can be obtained. Therefore, by using a wide-angle lens having a wide imaging range as the curved lens, a wide range around the vehicle can be displayed on the monitor in a natural state.
According to the fourth aspect, by performing the image distortion correction processing based on the distortion correction table, the processing load on the correction unit can be reduced, and the distortion correction processing can be made more efficient.
[0018]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, the present invention will be described with reference to the drawings showing the embodiments.
FIG. 1 shows an overall configuration of an in-vehicle imaging system 1 according to an embodiment of the present invention. The in-vehicle imaging system 1 includes an imaging device 2 mounted at the front center of the vehicle S, an image processing ECU (Electric Control Unit) 3 corresponding to an image processing device mounted inside the vehicle S, and a monitor installed in the vehicle interior. 4 are connected by a network cable 5a and a connection cable 5b, respectively.
[0019]
The imaging device 2 captures an image of the front of the vehicle, the image processing ECU 3 performs a process of converting the captured image into a display image, and the monitor 4 displays the display image. Therefore, by checking the display screen of the monitor 4, the occupant of the vehicle S can check the situation in front of the vehicle. The image processing ECU 3 also calculates the distance from the image captured by the imaging device 2 to the object T to be imaged, the speed of the object T, and the like, and displays the calculation result on the monitor 4.
[0020]
FIG. 2 shows the internal structure of the imaging device 2. The imaging device 2 is provided with an opening 2b for taking in light from the outside on one surface side in front of a box-shaped housing 2a, and arranges a left mirror 7A and a right mirror 7B at both ends of the opening 2b in a state of being inclined at a required angle. In addition, a curved wide-angle lens 6 having an imaging range of about 120 degrees is arranged near the center.
[0021]
Further, the imaging device 2 has a flat imaging pixel unit 8 that is arranged to face the wide-angle lens 6 and captures 30 frames per second in the housing 2. Note that the imaging device 2 has a transparent plate attached to the opening 2b to prevent rain, dust, and the like from entering the inside of the housing 2a, and a communication interface 9 connected to the imaging pixel unit 8 and the internal bus 10 as well. Have.
[0022]
The imaging pixel section 8 is configured by a CCD (Charge Coupled Device), and the surface facing the wide-angle lens 6 is divided into three areas, the left area is a left reflection area 8a, the right area is a right reflection area 8b, and the center is The area is directly set to the area 8c. The image reflected by the left mirror 7A enters the left reflection area 8a, the image reflected by the right mirror 7B enters the right reflection area 8b, and the image passing through the wide-angle lens 6a directly enters the direct area 8c. Incident. The arrangement angles of the left mirror 7A and the right mirror 7B are set so that the reflection images are input to the left reflection area 8a and the right reflection area 8b, respectively.
[0023]
When the columnar object T exists in front of the vehicle S as shown in FIGS. 1 and 2, the imaging device 2 captures an image 11 as shown in FIG. The image 11 is a combination of three types of imaging contents corresponding to the three regions 8a to 8c of the captured image unit 8, and the left end left image unit 11a is a left reflection region 8a of the imaging pixel unit 8. Hereinafter, the right image portion 11b at the right end is the right reflection region 8b of the imaging pixel portion 8, and the central image portion 11c at the center is the direct image 8c of the imaging pixel portion 8. .
[0024]
The left imaged object T1 shown in the left image portion 11a is an image of the actual object T, and the distance x1 from the left boundary line 11d to the center image portion 11c to the center line T1a of the left imaged object T1 is shown in FIG. 2 corresponds to the angle A between the left mirror 7A and the object T of the imaging device 2 shown in FIG. That is, if the distance x1 in the left image section 11a can be specified, the angle A can also be specified.
[0025]
Also, the right imaged object T2 reflected in the right image portion 11b is an image of the object T, and the distance x2 from the right boundary line 11e to the center image portion 11c to the center line T2a of the right imaged object T2 as described above. Corresponds to the angle B between the right mirror 7B and the object T shown in FIG. Further, the central imaging object T3 reflected in the central image portion 11c has a shape in which the image is distorted due to the influence of the wide-angle lens 6.
[0026]
The image captured by the imaging pixel unit 8 is sent to the communication interface 9 via the internal bus 10. The communication interface 9 is connected to the network cable 5a, and outputs an image 11 as shown in FIG. 4A sent from the imaging pixel unit 8 to the image processing ECU 3 through the network cable 5a.
[0027]
FIG. 3 shows an internal configuration of the image processing ECU 3. The image processing ECU 3 receives a communication interface 3a for connection to the network cable 5a, a CPU 3b for overall control of the image processing ECU 3, and an image processing ASIC (Application Specific IC) 3c which is a custom IC dedicated to image processing. A frame memory 3d for temporarily storing an image, a conversion table memory 3e for storing a conversion table for conversion processing, and a monitor interface 3f for outputting a processed image to the monitor 4 are provided.
[0028]
The communication interface 3a performs a process of receiving an image output from the imaging device 2 via the network cable 5a and sending the image to the image processing ASIC 3c. The frame memory 3d is composed of an SDRAM, and stores an image and reads the stored image according to an instruction from the image processing ASIC 3c. The conversion table memory 3e includes a masking table for hiding unnecessary portions in an image to be processed, a distortion correction table having distortion correction information corresponding to each pixel of the image to be processed, and a process as a conversion table for image processing. It stores a detection table for detecting whether or not an object in the target image is shown.
[0029]
On the other hand, the image processing ASIC 3c determines whether to store the image sent from the communication interface 3a in the frame memory 3d according to the processing status or to immediately perform the processing, and displays the image to be processed as a display image. , An object in the image, a distance to the detected object, a relative speed of the detected object, and the like. Note that the image processing ASIC 3c stores the distance between the left mirror 7A and the right mirror 7B shown in FIG. 2 for the distance calculation.
[0030]
The image processing ASIC 3c first performs a masking process as a process of converting a display image. For example, when masking the image 11 shown in FIG. 4A, a masking table is read from the conversion table memory 3e, and the left and right left image portions 11a and right image portions 11b are masked based on the masking table and the center is read. A process for leaving only the image portion 11c is performed. The masking table has information on each pixel constituting the image 11, and all the pixels corresponding to the left image portion 11a and the right image portion 11b in the image 11 in FIG. The center image portion 11c defines the contents to be left as it is.
[0031]
Next, the image processing ASIC 3c performs image distortion correction processing on the central image section 11c based on the distortion correction table read from the conversion table memory 3e. By performing the distortion correction process in this manner, the display image 12 shown in FIG. 4B is generated, and the display object T10 in the display image 12 is displayed in the central image portion 11c in FIG. The barrel-shaped distorted central imaging object T3 has a corrected shape.
[0032]
It should be noted that the correction table stipulates, in accordance with the curvature of the wide-angle lens 6, the position of the destination of each pixel forming an image before distortion correction so that distortion is corrected. For example, when the horizontal direction is the X coordinate and the vertical direction is the Y coordinate as shown in FIG. 4A, the coordinates (X, Y) of the pixel G in the image 11 before distortion correction = (350, 310). Is moved to (X, Y) = (370, 310) based on the correction table, and the distortion is corrected.
[0033]
Further, the image processing ASIC 3c detects the presence or absence of an object by binarizing each pixel of the display image 12 with the distortion corrected based on the object detection table read from the conversion table memory 3e. ing. Note that the image processing ASIC 3c performs the image processing based on the various tables as described above, thereby reducing its own processing load and enabling efficient processing.
[0034]
Further, when the image processing ASIC 3c detects an object by the above processing, the image processing ASIC 3c performs processing related to calculation of the distance to the object T and the relative speed of the object T. Regarding the calculation of the distance to the object, the image processing ASIC 3c specifies the distances x1 and x2 in the left image portion 11a and the right image portion 11b to be masked, and based on the specified distances x1 and x2, the angle shown in FIG. A and B are specified.
[0035]
Since the image processing ASIC 3c stores in advance the distance between the mirrors 7A and 7B, the distance L to the object T is assumed (see FIG. 2) assuming a triangle having the object T and the mirrors 7A and 7B as vertices. ) Is calculated based on the following equation.
L = (ktanAtanB) / (tanA + tanB)
However, in the above equation, k is the distance between the mirrors 7A and 7B.
Note that the image processing ASIC 3c adds the value of the distance L calculated in this way to the right corner of the display image 12, as shown in FIG. 4B.
[0036]
Further, the image processing ASIC 3c calculates the change in the distance L per unit time by calculating the distance L described above for each frame of the image pickup device 2 imaging at 30 frames / sec. The relative speed of is specified. The image processing ASIC 3c also adds the calculated relative speed to the right corner of the display image 12, as shown in FIG. 4B. Further, the image processing ASIC 3c sends the display image 12 created through various processes to the monitor interface 3f.
[0037]
The monitor interface 3f is connected to the connection cable 5b, and performs a process of outputting the display image 12 processed by the image processing ASIC 3c to the monitor 4 through the connection cable 5b. The monitor 4 receives the output display image 12 and displays it on the screen. Therefore, the in-vehicle imaging system 1 according to the present invention can display a natural image with only one imaging device 2 and calculate the distance to the object, and the driver only needs to look at the screen of the monitor 4. The situation around the vehicle S can be confirmed.
[0038]
The in-vehicle imaging system 1 is not limited to the above-described embodiment, and various modifications can be applied. For example, the image processing ECU 3 performs image processing based on various tables, but may perform image processing based on a conversion formula and a conversion program without using a table. Further, the image processing ECU 3 can omit the processing related to the detection of the object. In this case, the image processing ECU 3 always calculates the distance of the object shown in the captured image, or the image processing ECU 3 May be provided so that distance calculation is performed only when the switch is turned on. Further, the image processing ECU 3 can omit the process related to the calculation of the relative speed.
[0039]
Further, as the imaging device 2, it is also possible to apply the imaging pixel unit 8 formed of a CMOS (Complementary Metal-Oxide Semiconductor). Further, the imaging device 2 may be configured to use only one mirror and provide one reflection region and one direct region in the imaging pixel unit 8. In this case, the captured image in the direct region is also provided. It is preferable to perform the processing related to the distance calculation using the information.
[0040]
Furthermore, the imaging device 2 may capture an object three-dimensionally using three or more mirrors. For example, as in the imaging device 20 shown in FIG. , The left mirror 27A and the right mirror 27B may be arranged, and the lower mirror 27C may be arranged below the wide-angle lens 26 at a required angle. In this case, the imaging pixel unit 28 is divided into a left reflection area 28a, a right reflection area 28b, and a direct area 28c, and the lower part is divided into a lower reflection area 28d, and an image reflected by the lower mirror 27C is made incident. By inputting the reflected images from three directions to the imaging pixel unit 28 in this manner, even when the shape of the object to be imaged is complicated, it is possible to reliably grasp the shape of the object, the distance to the object, and the like. .
[0041]
In addition, the imaging device 2 (or 20) as described above is attached not only in front of the vehicle S but also in the rear and side of the vehicle S and the like, and one image processing ECU 3 or an image processing ECU 3 for each imaging device. May be provided to perform imaging of an object, distance calculation, and the like for each surrounding of the vehicle S. When a plurality of imaging devices 2 (or 20) are attached in this way, the image processing ECU 3 switches the display image so that the image displayed on the monitor 4 preferentially displays the image in which the object is reflected. In this case, or by providing an image changeover switch, the image is changed over by the operation of the occupant of the vehicle S.
[0042]
Further, the above-described in-vehicle imaging system 1 according to the present invention can be suitably used for the in-vehicle network system 90 constructed in the vehicle S shown in FIG. The in-vehicle network system 90 has a configuration in which an information network 60, a vehicle body network 70, and a control network 80 are connected by a gateway device 50, and the information network 60 includes the above-described in-vehicle imaging system 1.
[0043]
The vehicle body system network 70 has a steering angle sensor 71 and an axle pulse sensor 72 for detecting a vehicle speed in the steering device, and the control system network 80 has a brake ECU 81 for controlling a brake device and an accelerator ECU 82 for controlling an accelerator device 82. have.
[0044]
The vehicle system network 70 transmits a sensor signal relating to detection by each of the sensors 71 and 72 to the image processing ECU 3 of the information system network 60 via the gateway device 50. The CPU 3b of the image processing ECU 3 determines the current situation of the vehicle S based on the above-described sensor signal and information on the image captured by the imaging device 2. Further, the CPU 3b generates a control signal of the vehicle S corresponding to the determined situation, and transmits the control signal to the brake ECU 81 and the accelerator ECU 82 via the gateway device 50.
[0045]
Therefore, the brake ECU 81 and the accelerator ECU 82 control the brake device and the accelerator device based on the received control signals, and can appropriately control the traveling of the vehicle S with respect to the current situation, thereby realizing the driver's operation support. .
[0046]
【The invention's effect】
As described in detail above, in the first invention and the fifth invention, the imaging pixel portion is provided with a reflection region for making an image reflected by a mirror provided in the imaging device and a direct region for making the image directly enter. Thus, one image capturing apparatus can capture both an image required for distance calculation and an image required for display.
In the second aspect, by providing a plurality of mirrors and a plurality of reflection areas, it is possible to improve processing accuracy related to distance calculation.
[0047]
According to the third aspect, a wide imaging range can be ensured by using the curved lens, and image distortion due to the curved lens is corrected, so that a natural display image with distortion removed can be obtained.
According to the fourth aspect, by performing the image distortion correction processing based on the distortion correction table, it is possible to reduce the processing load related to the distortion correction of the image processing apparatus.
[Brief description of the drawings]
FIG. 1 is an overall configuration diagram of an in-vehicle imaging system according to an embodiment of the present invention.
FIG. 2 is a schematic diagram of an imaging device.
FIG. 3 is a block diagram illustrating an internal configuration of an image processing ECU.
FIG. 4A is a schematic diagram of a captured image, and FIG. 4B is a schematic diagram of a display image on which image processing has been performed.
FIG. 5 is a schematic perspective view of an imaging device according to a modified example.
FIG. 6 is a schematic diagram of a vehicle network system to which the vehicle-mounted imaging system according to the present invention is applied.
[Explanation of symbols]
1 in-vehicle imaging system 2 imaging device 3 image processing ECU
3c Image processing ASIC
3e Conversion table memory 4 Monitor 6 Wide-angle lens 7A, 7B Left mirror, right mirror 8 Image pickup pixel section 8a Left reflection area 8b Right reflection area 8c Direct area 11 Image 12 Display image S Vehicle

Claims (5)

In a vehicle-mounted imaging system including an in-vehicle imaging device that outputs a captured image and an image processing device that performs processing on the input image,
The imaging device,
With a mirror,
An imaging pixel unit having a reflection area where the image reflected by the mirror is incident and a direct area where the image is directly incident,
The image processing device,
A distance calculation unit that performs a process related to calculating a distance to an image captured with respect to the image captured by the reflection area,
A vehicle-mounted imaging system, comprising: a conversion unit configured to perform a conversion process on a captured image of the direct area into a display image. A plurality of said mirrors,
The in-vehicle imaging system according to claim 1, wherein the imaging pixel unit has a plurality of reflection areas into which the images reflected by the plurality of mirrors are respectively incident. The imaging device further comprises:
A curved lens is provided to face the imaging pixel unit,
The conversion means,
The vehicle-mounted imaging system according to claim 1, further comprising a correction unit configured to correct image distortion caused by the curved lens. The image processing apparatus further includes:
A distortion correction table having distortion correction information corresponding to each pixel of the image,
The correction means,
The vehicle-mounted imaging system according to claim 3, wherein the image distortion is corrected based on the distortion correction table. In an imaging device mounted on a vehicle,
With a mirror,
An image pickup apparatus comprising: a reflection area on which an image reflected by the mirror is incident; and an imaging pixel section having a direct area on which an image is directly incident.

Images