JP2010146082A - Image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor allowing suitable suppression of influence of road surface reflection light in a pickup image. <P>SOLUTION: Of a plurality of imaging devices imaging an identical road surface area from different positions, one imaging device is set as a reference imaging device, and the other imaging devices are set as target imaging devices, the respective pickup images are obtained from the imaging devices. Of the respective obtained pickup images, the image by the reference imaging device is set as a reference image, the images by the target imaging devices are set as target images, each target image is performed with viewpoint conversion such that the road surface area in the target image (a target road surface area) accords with the road surface area in the reference image (a reference road surface area), and the target images after the conversion and the reference image are composed. The road surface area in the composite image (a composite road surface area) is extracted, and the reference road surface area of the reference image is replaced with the composite road surface area. Accordingly, a luminance of an object (the road surface reflection light) having height (minus height) with the road surface area as a reference plane is reduced inside the image road surface area in the replaced image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that processes an image mounted on a vehicle and picked up by an image pickup apparatus.

  Conventionally, on-vehicle display system that displays an image captured by an in-vehicle camera mounted on a vehicle on a display inside the vehicle, and a situation such as a traffic jam or an accident based on an image captured by a roadside camera installed on the roadside There is known an information providing system that generates various types of traffic information and provides the traffic information to each vehicle on a traveling road. In general, in an information providing system, an image (captured image) obtained in real time by imaging a predetermined area including a traveling road (road surface) with a roadside camera, and when a vehicle is not passing on the road surface A vehicle detection process for extracting (detecting) a vehicle on the road surface is performed by comparing the obtained image (road image).

  In this type of in-vehicle display system and information provision system, the visibility of the display and the accuracy of the vehicle detection process decrease due to the influence of the reflected image (road surface reflected light) reflected on the road surface at night or in dark places (for example, tunnels). It is regarded as a problem.

For this reason, for example, in an information providing system, in order to extract only a vehicle light from a captured image and detect a vehicle on the road surface, a circular shape of a high-intensity region extracted from the captured image by known binarization processing and labeling processing There has been proposed an image processing device that calculates a degree and discriminates a high luminance area into a vehicle lamp area and a road surface reflection area based on the circularity (see, for example, Patent Document 1). Further, for example, in an in-vehicle display system, it is conceivable to improve the visibility of the display by reducing the luminance of the road surface reflection area identified by the image processing apparatus.
JP 2007-193702 A

  However, in the conventional in-vehicle display system, since it is necessary to recognize the road surface reflection area by the image processing device, the recognition is not always correct, and the road surface reflected light may not be appropriately removed from the captured image. There was a problem that there was.

  In order to solve the above-described problems, an object of the present invention is to provide an image processing apparatus that can suitably suppress the influence of road surface reflected light in a captured image.

  The image processing apparatus according to claim 1, which is an invention made to achieve the above object, is an apparatus mounted on a vehicle, and is connected to a plurality of imaging apparatuses that image the same road surface region from different positions. It is configured. In the following, one of the plurality of imaging devices is referred to as a reference imaging device, and one or more devices other than the imaging device are referred to as target imaging devices.

  Specifically, the image processing apparatus of the present invention focuses on the basic principle described in detail below. That is, as shown in FIG. 7, for example, three surfaces (plane I, plane II, plane III) parallel to each other in the three-dimensional space, objects X, Y, and object X arranged on the plane I and plane II, respectively. , Y is assumed to be an image pickup device E and an image pickup device F that pick up images from different positions on the plane III. Then, when the image B is translated (translated) and synthesized with the image A so that the object X in the image B matches the object X in the image A, the plane I becomes the reference plane in the synthesized image, and the object on the reference plane becomes Although they match, the object Y on the plane II different from the reference plane does not match (shifts). On the contrary, when the object Y on the image B is matched with the object Y on the image A, the plane II becomes the reference plane in the composite image, and the object X on the plane I different from the reference plane is shifted.

  When such a basic principle is applied to the present invention, it is as follows. Of the captured images obtained by the plurality of imaging devices described above, the captured image by the reference imaging device is the reference image, the captured image by the target imaging device is the target image, and each captured image (reference image and target image) is the road surface area. A corresponding image area is defined as an image road surface area. Of these, an image road surface area in the reference image is defined as a reference road surface area, and an image road surface area in the target image is defined as a target road surface area.

  That is, in the image processing device of the present invention, the image acquisition unit acquires each captured image from a plurality of imaging devices (reference imaging device and target imaging device), and the image synthesis unit matches the target road surface area and the reference road surface area. As described above, the target image is subjected to viewpoint conversion, and the converted target image is used as a converted image to perform a process of combining the converted image (hereinafter referred to as a combining process). The combining process is not limited to the process of combining the converted image and the reference image, and may be a process of combining only a plurality of converted images.

  Thus, by setting the road surface area in the three-dimensional space as the reference plane, an object on a plane different from the reference plane is shifted in the synthesized image. What is important here is not only an object having a positive height (including street light) but also a object having a negative height (including road surface reflected light) with respect to the reference plane. (See FIG. 8).

  To explain this reason in an easy-to-understand manner, as shown in FIG. 9, for example, road surface reflected light in a captured image is formed on a road surface region (reference plane), but has a positive height with respect to the reference plane. This is because the light is reflected (and irregularly reflected) from the streetlight and projected onto the imaging device, and can be rephrased as that projected directly from the streetlight having the same negative height onto the imaging device. For this reason, in the synthesized image after the viewpoint conversion, the division lines (for example, white lines) on the road surface area coincide with each other, but the road surface reflected light is shifted (see FIG. 8).

  Note that the viewpoint transformation is the well-known “translation” (translation), “rigid transformation” (rotation and translation), “similar transformation” (rotation, translation and scale change), “affine transformation” (similar transformation) This is an image conversion in which a subgroup is a slanted distortion). In addition, a synthesized image generally means that each captured image (reference image and converted image) is synthesized at a 1 / N composition ratio, where N is the number of captured images (reference image and converted image) before synthesis. Point to.

  Further, in the image processing apparatus of the present invention, the image synthesized by the image synthesizing means is the synthesized image, the image road surface area in the synthesized image is the synthesized road surface area, and the reference road surface area in the reference image is replaced with the synthesized road surface area. An image is generated by performing (hereinafter referred to as a replacement process).

  In the image processing apparatus configured as described above, the road surface area becomes a reference plane, and an object on a plane different from the reference plane (such as street lamp light or road surface reflected light) is shifted in the composite image, whereby the brightness of the object is predetermined. It is diluted with the synthesis ratio (1 / N). Then, since the road surface area in the composite image is used as a composite road surface area, and only the road surface area is replaced with the composite road surface area in the reference image, an area other than the road surface area in the generated image has an area in the reference image. An object (such as a streetlight) remains in the state before synthesis.

  Therefore, according to the image processing apparatus of the present invention, only the brightness of an object (road surface reflected light or the like) in the road surface area is reduced (dispersed) in the generated image, and consequently the influence of the road surface reflected light on the captured image is reduced. It can suppress suitably.

  The imaging device may be configured, for example, to be installed on the roadside (or mounted on another vehicle) and transmit the captured image to the vehicle. It is desirable to be mounted on the own vehicle.

  In this case, since it is not necessary to receive a captured image from the outside of the vehicle, there is no need to consider communication errors, image degradation, a reduction in communication speed, and the like, and consequently, a reduction in accuracy and processing speed related to image processing is suppressed. Can do.

  By the way, for example, when the image processing apparatus of the present invention is used in an in-vehicle display system, objects other than road surface reflected light (for example, other vehicles, etc .; hereinafter collectively referred to as obstacles) are included in the generated image. If this happens, information important to the driver when viewing the display will be removed.

  Therefore, as described in claim 3, the image processing apparatus is connected to an obstacle detection apparatus that detects an obstacle on the road surface area, and the distance acquisition unit is information indicating a relative distance between the obstacle and the vehicle ( The replacement prohibiting means prohibits the replacement process by the image replacement means for the area whose position on the reference road surface area exceeds the above relative distance based on the distance information acquired by the distance acquiring means. It is desirable to be configured to do so. It is assumed that the position on the reference road surface area is associated in advance with a distance on the road area starting from the vehicle based on calibration of a plurality of imaging devices (known camera calibration). .

  In this case, for example, it is possible to suppress only the influence of road surface reflected light in the display image of the display and leave an obstacle such as a vehicle in a state before synthesis. As the obstacle detection device, a known radar device using millimeter waves or ultrasonic waves may be used. In addition, the position on the reference road surface area is calculated in advance from the calibration of the obstacle detection device and the camera image, and camera mounting position information.

  Here, as described in claim 4, the image road surface area may be fixed for each imaging device based on an internal parameter and an external parameter (that is, a camera parameter) of the imaging device. The camera parameter is specified in advance by camera calibration. Among these, the internal parameter refers to the one based on the focal length of the imaging device, and the external parameter refers to the one based on the position or orientation of the imaging device. Point to.

  In this case, since it is only necessary to perform viewpoint conversion of the target image based on a road surface area (fixed road surface area) fixed in advance in each captured image, for example, a lane line (white line or the like) on the fixed road surface area or its lane line It is conceivable that image conversion is performed so that the vanishing points coincide with each other, and image processing can be facilitated. On the other hand, for example, when the vehicle is traveling on a curve, there is a concern that the road surface area in the actual captured image may be shifted from the fixed road surface area.

  Therefore, as described in claim 5, the image processing apparatus includes a lane marking detection unit that detects a lane marking on the road surface area, and the image road surface area is imaged based on the lane marking detected by the lane marking detection unit. It is desirable to set for each image. In addition, as the lane marking detection means, a feature point is generally extracted by using well-known edge detection, Hough transform, or the like, and a lane marking is detected based on the connection of the feature points in the entire captured image (detection processing). Do. Further, such a lane marking detection unit may detect lane markings for all captured images. Alternatively, if a lane marking is detected for one of the captured images (for example, the reference image), the image road surface area (for example, the target road surface area) in the other captured image (for example, the target image) And can be set based on the camera parameters of each imaging apparatus.

  In these cases, the influence of the road surface reflected light in the captured image can be suitably suppressed regardless of the behavior of the vehicle. On the other hand, for example, when the image processing apparatus of the present invention is used in a vehicle control system that performs control related to driving support of a vehicle such as parking support or lane keeping, the above detection processing is performed due to the influence of road surface reflected light in a captured image. Since there is a possibility that the accuracy may decrease, it may be possible that the lane markings cannot be detected properly.

  Accordingly, in the image processing apparatus, the road marking detection unit detects the road marking on the road surface area based on the image replaced by the image replacement unit (replacement image) or the composite image. It may be configured as follows. The road marking is a generic term for what has been painted on the road surface, and includes road markings representing speed limits, primary stops, etc., and lane markings representing the boundaries of traveling lanes such as white lines.

  In the image processing apparatus configured as described above, when detecting the road marking, an image in which the influence of the road surface reflected light is suppressed by the image replacement unit is used, so that the detection accuracy can be improved. Therefore, in the vehicle control system, based on the processing result by the image processing device, for example, the road surface information such as the position of the white line, the speed limit, and the temporary stop is appropriately notified to the driver, or various travel controls are performed based on the road surface information. Can be performed appropriately.

  Embodiments of the present invention will be described below with reference to the drawings.

<Overall configuration>
FIG. 1 is a block diagram illustrating an overall configuration of a vehicle control system 1 including an image processing apparatus to which the present invention is applied, and FIG. 2 is a schematic diagram illustrating an arrangement of a camera array 10 as a component of the vehicle control system 1. is there.

  As illustrated in FIG. 1, the vehicle control system 1 includes a plurality (9 in this embodiment) of imaging devices (for example, CCD cameras) 11 that capture an image including a road surface on the front side (front of the vehicle) of the vehicle. 19, a radar apparatus 20 that detects an object located within a predetermined detection range in front of the vehicle, an image processing apparatus 30 that processes an image (captured image) captured by the camera array 10, and various types A display device 40 that displays an image and a control device 50 that performs various controls related to driving support of the vehicle, and these devices 10 to 50 are connected via a communication line such as a video line or an in-vehicle LAN (Local Area Network). Has been configured.

  Among these, as shown in FIG. 2, for example, the camera array 10 has a plurality of imaging devices 11 to 19 arranged in a straight line so that the arrangement direction thereof is the vehicle width direction (the lateral direction of the vehicle), for example. Each imaging device 11-19 is attached to the ceiling part in a vehicle so that the same road surface area may be imaged from a different position through a windshield. And it is comprised so that each captured image by these each imaging devices 11-19 may be output to the image processing apparatus 30. FIG. Note that internal parameters (parameters based on the focal length of the camera) and external parameters (parameters based on the camera position, orientation, etc.) of each of the imaging devices 11 to 19 are set in advance by well-known camera calibration.

  Returning to FIG. 1, the radar apparatus 20 is configured as a so-called “millimeter wave radar” of the FMCW system, for example. Specifically, by transmitting and receiving a frequency-modulated millimeter-wave band radar wave, objects such as other vehicles, obstacles, and pedestrians (hereinafter collectively referred to as forward obstacles) in front of the vehicle are detected ( And target information related to the front obstacle is generated based on the recognition results, and the target information is output to the image processing apparatus 30.

  The display device 40 is configured as any of a known liquid crystal display, organic EL display, CRT, head-up display, and the like, and displays an input image from the image processing device 30 (and the control device 50).

  The control device 50 outputs a generated image to the display device 40 based on road surface information (described later) input from the image processing device 30, or road surface information and various sensors (a vehicle speed sensor, an acceleration sensor, a steering wheel) (not shown). Lane maintenance control is performed based on the detection result by a sensor or the like. Note that, as the lane keeping control, the control device 50 predicts, for example, the possibility that the vehicle departs from the traveling lane (probability of departure), and displays a caution image representing a warning on the display device 40 according to the possibility of departure. Or a weak steering torque for warning is applied to the steering in the direction opposite to the departure direction.

<Configuration of image processing apparatus>
Next, FIG. 3 is a block diagram showing a configuration of an image processing apparatus to which the present invention is applied.

  As shown in FIG. 3, the image processing device 30 is input via the image input unit 31 and an image input unit 31 for inputting each captured image (video signal) from the imaging devices 11 to 19 in the camera array 10. A / D conversion unit 32 that converts the video signal (analog signal) to be converted into a digital signal and outputs the digital signal, and an image memory unit 33 that temporarily holds the digital signal output from the A / D conversion unit 32 as image data An image processing unit 30a that reads and processes image data from the image memory unit 33, and an interface (I / F) unit 34 for outputting a processing result by the image processing unit 30a to the display device 40 (and the control device 50). And. Various behavior detection sensors 35 for detecting the behavior of the vehicle, such as a yaw rate sensor and a steering sensor, are connected to the image processing device 30 (specifically, the image processing unit 30a).

  The image processing unit 30a is configured around a known microcomputer including a CPU, ROM, RAM, I / O, bus line, and the like. Among these, the CPU stores a program stored in the ROM using the RAM as a work area. Based on the above, the image control process described in detail below is executed. In the ROM, camera parameters (internal parameters and external parameters) for each of the imaging devices 11 to 19 in the camera array 10 are stored. The RAM also has a captured image read from the image memory unit 33, a converted image described later, a read image region for temporarily holding a composite image, a converted image region, and a composite image region.

<Image control processing>
Next, FIG. 4 is a flowchart showing details of image control processing executed by the CPU of the image processing unit 30a. This process is started when the image processing apparatus 30 (or the control apparatus 50) is turned on, and is repeatedly executed until the power supply (or the IG switch of the vehicle) is turned off.

  First, when this process is started, in S110, each captured image (see FIG. 5) by the imaging devices 11 to 19 in the camera array 10 is read from the image memory unit 33 and stored in the read image area of the RAM. The process proceeds to S120. In the image storage area, an image picked up by one predetermined device (hereinafter referred to as a reference image pickup device) 10a among the image pickup devices 11 to 19 in the camera array 10 (hereinafter referred to as a reference image; see FIG. 5) and Each of the captured images (hereinafter referred to as target images) by a plurality (eight in the present embodiment) of devices (hereinafter referred to as target imaging devices) 10b other than the reference imaging device 10a is stored so as to be identifiable.

  In S120, target information is acquired from the radar apparatus 20, and the process proceeds to S130. The target information includes presence / absence information indicating detection or non-detection of a front obstacle and distance information indicating a relative distance between the front obstacle and the vehicle at the time of detection.

  In S130, in each captured image (reference image and target image) stored in S110, an image area corresponding to a road surface area (hereinafter referred to as an image road surface area) is preset based on camera parameters stored in the ROM. And converting the viewpoint of each target image stored in the read image area of the RAM so that the target road area matches the reference road area based on the set image road area. . The target road surface area refers to the image road surface area in the target image, and the reference road surface area refers to the image road surface area in the reference image. In the conversion process, for example, image conversion is performed so that the white line on the image road surface region and the vanishing point of the white line coincide. Since such image conversion is well known (see, for example, Japanese Patent Application Laid-Open No. 2002-352225), description thereof is omitted. Further, the target image converted here (hereinafter referred to as a converted image) is stored in a converted image area of the RAM.

  In subsequent S140, a process of combining the converted image stored in the converted image area of the RAM (generated in S130) and the reference image stored in the read image area of the RAM (read out in S110). (Hereinafter referred to as synthesis processing). In the combining process, each captured image (reference image and converted image) is combined at a combining ratio equal to the number of captured images (reference image and converted image) before combining (9 images in the present embodiment). The captured image synthesized here (hereinafter referred to as a synthesized image; see FIG. 5) is stored in a synthesized image area of the RAM.

  In subsequent S150, based on various signals input from the behavior detection sensor 35, it is determined whether or not a detection value (for example, yaw rate) indicating the behavior of the vehicle is below a predetermined threshold value, and when an affirmative determination is made here. Since the vehicle is traveling straight or stopped (the image road surface area is fixed), the process proceeds to S160. On the other hand, if a negative determination is made, it is assumed that the vehicle is traveling on a curve (the image road surface area is changing), and the process proceeds to S170.

  In S160, the image road surface area set in S130 is fixed as it is, and the process proceeds to S190. That is, in the synthesized image synthesized in S140, an image area corresponding to the road surface area (hereinafter referred to as a synthesized road area; including the reference road area and the target road area) is fixed in advance based on the camera parameters stored in the ROM. Area (hereinafter referred to as a fixed road surface area). The fixed road surface area is specified in advance based on the position and orientation of the camera array 10 (imaging devices 11 to 19), the focal length, and the like.

  On the other hand, in S170, a white line detection process for detecting a white line position on the road surface area is performed based on the composite image synthesized in S140. In the white line detection process, for example, a pixel point (edge point) whose luminance level changes with a level difference equal to or greater than a predetermined threshold for each horizontal line in the composite image is extracted, and is sandwiched between two adjacent edge points Among these, pixel points belonging to the region with the higher luminance level are set as white line candidate points. Then, the white line position is estimated based on the connection of white line candidate points in the entire reference image. That is, here, it is applied that edge points can be easily extracted by using a composite image in which the luminance other than the white line is relatively thin.

  In subsequent S180, the image road surface area set in S130 is changed based on the processing result of the white line detection processing in S170. Specifically, for example, an image road surface area (reference road surface area) in the reference image is set by specifying an area surrounded by a plurality of white line positions (and white line vanishing points) in the reference image. Further, by specifying the pixel point corresponding to the reference road surface area in the composite image synthesized in S140 based on the set reference road surface area and the relative relationship between each camera parameter stored in the ROM, An image road surface area (target road surface area) in the target image is set, and the process proceeds to S190.

  In S190, the composite road surface area set in S160 or S180 from the composite image stored in the composite image area of the RAM (generated in S140) is an image replacement target area (hereinafter referred to as “image replacement target”). , Referred to as a replacement road surface area), and the process proceeds to S200.

  In S200, based on the target information (specifically, presence / absence information) acquired in S120, it is determined whether or not a forward obstacle exists on the road surface area. If an affirmative determination is made here, the process proceeds to S210. If a negative determination is made, the process proceeds to S220.

  In S210, based on the target information (specifically, distance information) acquired in S120, a pixel point (position) on the replacement road surface area extracted from S190 is a front obstacle on the road surface area. By removing (cutting) the area exceeding the relative distance between the vehicle and the vehicle (that is, the upper area), the replacement road surface area is adjusted so that no front obstacle is included, and the process proceeds to S220.

  In S220, a process of replacing the reference road area in the reference image stored in the read image area of the RAM with the replacement road area extracted in S190 (hereinafter referred to as a replacement process) is performed, and the process proceeds to S230. To do. Here, for example, when the replacement road surface area is adjusted in S210, an area in which the upper area is cut is applied to the reference road surface area to be replaced here, as in the case of the replacement road surface area.

  In S230, the image replaced in S220 (hereinafter referred to as a replacement image; see FIG. 5) is output to the display device 40 via the I / F unit 34, and the process proceeds to S110. Here, in addition to the output to the display device 40, white line detection processing similar to that in S150 is performed based on the replacement image, road surface information representing the position of the white line on the road surface region, etc. is generated, and the road surface information is used as the control device. 50 is also output.

  In the above embodiment, S110 is an image acquisition unit, S130 and S140 are image synthesis units, S190 and S220 are image replacement units, S120 is a distance acquisition unit, S210 is a replacement prohibition unit, and white line detection processing (S170, S230) is performed. It corresponds to lane marking detection means and road marking detection means.

<Effect>
As described above, in the image processing apparatus 30 according to the present embodiment, each target image is subjected to viewpoint conversion using the road surface area in the three-dimensional space as a reference plane (S130), and each converted image and the reference image are synthesized. (S140), the objects having a positive height with respect to the reference plane (such as streetlights, streetlights, and other vehicles) and the objects having a negative height (such as road surface reflected light) are shifted, and the brightness of these objects Is reduced to a predetermined synthesis ratio (in this embodiment, 1/9).

  Then, by extracting a road surface area (replacement road surface area) in the composite image (S190, and if necessary, S210), an object having a positive height with respect to the reference plane (a streetlight, a streetlight, other vehicles, etc.) ) Is removed from the composite image, and the reference road surface area in the reference image is replaced with the composite road surface area (S220), thereby generating a replacement image.

  Therefore, according to the image processing apparatus 30 of the present embodiment, it is possible to generate a replacement image in which only the brightness of an object (such as road surface reflected light) having a negative height with respect to the reference plane is reduced, and consequently. The influence of the road surface reflected light in the captured image can be suitably suppressed.

  Further, in the image processing apparatus 30, if the vehicle is traveling straight (or stopped) (S150; YES), the vehicle is traveling in a curve on the basis of an image road surface area fixed in advance for each captured image (S160). If so (S150; NO), the image road surface area set for each captured image by the white line recognition process is used as a reference (S170, S180), and each target image is selected so that the target road surface area matches the reference road surface area. The viewpoint will be changed.

  Therefore, according to the image processing device 30, the setting method of the image road surface area (the reference road surface area and the target road surface area) can be suitably changed in accordance with the behavior of the vehicle. The effect of road surface reflected light can be accurately suppressed.

  Furthermore, since the image processing device 30 performs the white line detection process based on the replacement image in which the brightness of the road surface reflected light is reduced, the detection accuracy can be improved. As a result, the control device 50 determines the detection result based on the detection result. It is possible to improve the accuracy of various controls (lane maintenance control, etc.) to be performed.

[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  For example, the camera array 10 of the above-described embodiment is configured to be arranged in a straight line so that the nine imaging devices 11 to 19 image the same road surface area on the front side of the vehicle through the windshield. However, the present invention is not limited to this, and it is only necessary that two or more imaging devices are arranged so as to image the same road surface area (at least a part thereof) outside the vehicle.

  In addition, as the number of such imaging devices is installed (that is, the number of captured images is large), the brightness of the road surface reflected light can be further reduced in the generated image (replacement image) by the image control process. In this case, it is conceivable that the installation cost including the manufacturing cost will increase.

  Therefore, for example, as shown in FIG. 5, one image pickup device is installed in each of the central portion of the vehicle and the left and right side mirrors. The captured image (target image) by the device (target image capturing device) may be subjected to viewpoint conversion, synthesis, and replacement processing (image control processing) in the same manner as in the above embodiment. In this case, if the road surface reflected light is imaged by any imaging device of each side mirror, the brightness of the road surface reflected light is reduced in the image (generated image) generated by the image control process. . The left and right side mirror imaging devices may be installed so that at least the left half and the right half of the road surface area can be imaged. In this case, if the road surface reflected light is captured in the captured image (reference image) by the imaging device in the center, the imaging device of either the left or right side mirror captures the road surface reflected light. It is possible to reliably suppress the influence of road surface reflected light on the image.

  Moreover, these imaging devices (including the camera array 10 of the above embodiment) may be installed so as to image the same road surface area on the rear side of the vehicle, for example. In this case, the vehicle control system 1 can be applied as a well-known parking assistance control system, for example, when the control device 50 performs travel control for the reverse of the vehicle based on the road surface information input from the image processing device 30. .

  Regarding the installation of a plurality of imaging devices, for example, the larger the installation interval of the imaging devices 11 to 19 in the camera array 10, the greater the deviation of the road surface reflected light in the generated image, and the smaller the installation interval, Since road surface reflected light accumulates, it is preferable to adjust appropriately considering these balances. For example, priority may be given to the visibility of the image in the display apparatus 40, and it may be adjusted so that the road surface reflected light with respect to the light source (streetlight light) of the predetermined distance from a vehicle may be collected | stacked moderately at the time of an image synthesis, or a white line recognition process ( The road surface reflected light may be adjusted so as to be surely dispersed by giving priority to generation of road surface information).

1 is a block diagram illustrating an overall configuration of a vehicle control system 1 including an image processing device 30 to which the present invention is applied. FIG. 2 is a schematic diagram showing an arrangement of a camera array 10 as a component of the vehicle control system 1. 1 is a block diagram showing a configuration of an image processing apparatus 30 to which the present invention is applied. The flowchart which shows the detail of the image control process which CPU of the image process part 30a performs. The image figure for demonstrating the image (composite image, replacement image) produced | generated by image control processing. The schematic diagram and image figure for demonstrating the installation position of each imaging device, each captured image, and replacement image in other embodiment. The first explanatory view for explaining the basic principle of the present invention. The 2nd explanatory view for explaining the basic principle of the present invention. The 3rd explanatory view for demonstrating the basic principle of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Vehicle control system, 10 ... Camera array, 20 ... Radar apparatus, 30 ... Image processing apparatus, 30a ... Image processing part, 31 ... Image input part, 32 ... A / D conversion part, 33 ... Image memory part, 34 ... I / F section, 35 ... behavior detection sensor, 40 ... display device, 50 ... control device.

Claims (6)

In an image processing apparatus mounted on a vehicle,
Image acquisition means for acquiring each captured image from a plurality of imaging devices that capture the same road surface region from different positions;
In each captured image acquired by the image acquisition unit, an image region corresponding to the road surface region is an image road surface region, one of the plurality of imaging devices is a reference imaging device, and an image captured by the reference imaging device is a reference image. The image road surface area in the reference image is a reference road surface area, and an image captured by an imaging device other than the reference imaging device among the plurality of imaging devices is used as a target image.
Image synthesizing means for performing viewpoint conversion of the target image such that the target road surface area that is the image road surface area in the target image matches the reference road surface area, and synthesizing the converted image that is the target image after the conversion;
Image replacement is performed for replacing the reference road surface area of the reference image with the combined road surface area, using the image combined by the image combining means as a combined image, the image road surface area in the combined image as a combined road surface area. Means,
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the plurality of imaging devices are mounted on the vehicle. The position on the reference road surface area is associated in advance with a distance on the road surface area starting from the vehicle based on calibration of the plurality of imaging devices,
Distance acquisition means for acquiring a relative distance between the obstacle and the vehicle from an obstacle detection device that detects an obstacle on the road surface area;
Replacement prohibiting means for prohibiting replacement processing by the image replacement means for an area where the position on the reference road surface area exceeds the relative distance acquired by the distance acquisition means;
The image processing apparatus according to claim 2, further comprising:   The image processing apparatus according to claim 1, wherein the image road surface area is fixed for each captured image based on an internal parameter and an external parameter of the imaging apparatus. Comprising lane marking detection means for detecting a lane marking on the road surface area;
The image processing apparatus according to claim 2, wherein the image road surface area is set for each of the captured images based on a lane line detected by the lane line detection unit.   The road marking detection means for detecting a road marking on the road area based on the replacement image or the composite image using the image replaced by the image replacement means as a replacement image. Item 5. The image processing device according to any one of Items 4 to 5.