JP5546321B2 - In-vehicle display device using multiple camera images - Google Patents

Description

  The present invention uses images taken by a plurality of cameras that photograph the outside of the vehicle to display an image in which a viewpoint exists above the vehicle or a photographed image of a single camera using the same image data, Multi-camera image display device that detects white lines and obstacles and highlights them, especially when water droplets adhere to some of the camera lens or when it becomes dirty. The present invention relates to a multi-camera image using vehicle outside display device that can interpolate the display of white lines and obstacles in this part from images taken by other cameras.

  When putting a vehicle in a narrow garage or parking lot, the driver needs to be very careful, and even with extreme care, accidents such as rubbing surrounding objects with the vehicle may occur. Also, when taking a vehicle out of a garage or parking lot, it is necessary to pay close attention to surrounding objects and people. In addition to entering and exiting garages and parking lots, when driving on extremely narrow roads or passing oncoming vehicles on narrow roads, contact with surrounding objects, or prevent wheels from dropping into grooves, etc. There are many cases where sufficient attention is required around the vehicle.

  As a countermeasure, the rear camera that has been widely installed in vehicles in recent years, images the rear of the vehicle and displays it on the monitor, and makes it easy to put in the garage or park, When driving out of a narrow road with poor visibility, use a vehicle-side camera, such as a vehicle side-view camera, to check the vehicle traveling on the other road, monitor the surroundings of the vehicle, and drive with sufficient confirmation It has also been done.

  However, even if an ultra-wide-angle camera is used as such an outside-camera camera, for example, when a garage is placed while the vehicle is retracted, and there are obstacles around the rear side of the vehicle, the rear-facing camera is used when the vehicle is retracted. You will not be able to know the state of the obstacle. As a countermeasure, it is also conceivable to provide super wide-angle cameras on both sides of the vehicle so that the portion outside the field of view of the rear-view camera can be seen by the side-view camera.

  However, in the image of the ultra-wide-angle camera as described above, the appearance of the object is greatly different between the central part and the peripheral part of the photographed image, and the positional relationship between the object and the vehicle is clear just by looking at the photographed image. Instead, it may cause unexpected collisions and contact accidents. In particular, even when switching from an image taken from the rear camera to an image taken from the side camera, the image shape and position of objects such as obstacles displayed on the screen at that time are very different, and the actual state of the object is grasped. It is not easy.

  Therefore, a plurality of outside-camera cameras are installed in the vehicle, and the images taken by these cameras are combined, and an image is created as if the surroundings of the vehicle were photographed from the camera installed above the vehicle. Display has already been implemented. When using such a plurality of cameras, for example, as shown in FIG. 5, a left side camera 32 for photographing the left side of the vehicle 31, a right side camera 33 for photographing the right side of the vehicle 31, and a rear side of the vehicle 31. A total of four cameras, a vehicle rear-view camera 34 that captures images and a vehicle front-view camera 35 that captures the front of the vehicle 31, are mounted, and all the images of these cameras are combined. For example, FIG. As shown, an image as if the entire periphery of the vehicle 31 was photographed by a single camera provided directly above the vehicle is displayed on the monitor. This method is also referred to as a top view system, and two or three cameras for photographing the outside of the vehicle are appropriately selected as necessary, and the installation positions are also selected and installed at various positions.

  Further, according to this system, for example, a camera that captures the rear of the vehicle can be displayed on a monitor as an image of a normal rear camera, and further, a camera that captures the front of the vehicle using image data of a plurality of cameras. It is possible to display the image of the front of the vehicle including the side part of the vehicle by the photographed image and the image of the camera that captures both sides of the vehicle. Similarly, the image is displayed including the side part of the vehicle behind the vehicle. For example, since an image can be displayed in various modes, it is sometimes called a multi-view camera system. In such a multi-view camera system, among the images taken by each camera, the white line indicating the traveling range of the vehicle and the danger that the vehicle will collide with or come into contact with the vehicle, and further climb A white line obstacle detecting means for detecting a characteristic obstacle is provided, and a white line or an obstacle is highlighted in a captured image of each camera.

  The top view system shown in FIG. 5 shows a state when the vehicle is moving backward to enter the parking space, and there are white lines 36 and 37 serving as an indication of the parking space on both sides of the vehicle, and An example is shown in which a pole 38 as an obstacle exists on the outer side of the rear side, and car stops 39 and 40 are provided. In other words, the front of the vehicle is always easy to see in a wide range, and even if the surroundings are dark, it is designed so that it can be seen in a very bright state by turning on the headlamp, whereas the rear of the vehicle is extremely Even if you use a rear-view camera with a narrow viewing range, lighting such as headlights is prohibited when the surroundings are dark, and white lines and obstacles can be identified using an infrared camera. Is extremely difficult. Therefore, FIG. 5 illustrates a state when the vehicle moves backward, in which safe driving of the vehicle is particularly difficult. However, for example, the same problem may occur also in a vehicle front photographing camera or the like.

  In this state, each camera shoots with a super-wide-angle fisheye lens. Therefore, as shown in FIG. 5C showing the shooting range of each camera, the vehicle rear-side shooting camera 34 and the vehicle left-side shooting camera 32 are the same. As shown in the lower left corner, there is an overlapping shooting range 42. Similarly, as shown in the lower right corner of the figure, an overlapping shooting range 43 is generated between the vehicle rearward shooting camera 34 and the vehicle rightward shooting camera 33. Further, as shown in the upper right corner of the figure, there is an overlapping shooting range 44 in the images taken by the vehicle right side camera 33 and the vehicle front side camera 35. The vehicle front side camera 35 and the vehicle left side camera 32 are shown in FIG. As shown in the upper left corner, there is an overlapping shooting range 45, and there are a total of four overlapping shooting ranges.

  With respect to the photographing range of each camera as shown in FIG. 5C, the boundary line of this image can be set to an appropriate range. At that time, for example, as shown in FIG. 4D, the images taken by the left side camera 32 and the rear camera 34 are set and divided as indicated by the lower left corner display boundary line 46. Similarly, the captured images of the rear camera 34 and the right camera 33 are set and divided at the lower right corner display boundary line 47, and the right camera 33 and the front camera 35 have their image ranges at the upper right. A corner display boundary line 48 is set and divided, and similarly, a captured image range of the vehicle front shooting camera 35 and the vehicle left side shooting camera 32 is set and divided at an image upper left corner display boundary line 49, It is displayed on the camera.

  Therefore, during normal operation, for example, as shown in FIG. 5D, the surroundings are displayed by images taken by the respective cameras around the vehicle, and the white line 36 detected by the white line obstacle detecting means as described above. , 37, indicating that it is only necessary to park between these lines, highlighting that there is an obstacle 38 nearby, so that the driver can park securely by looking at this image Can do. At this time, if necessary, the car stops 39 and 40 can also be detected as obstacles and can be highlighted. Such processing is not limited to when the vehicle is parked as described above, but is detected as an obstacle even when traveling on a narrow road where a gutter or the like exists outside the range of the normal field of view for the driver. Then, the highlighting operation is performed, and the vehicle can be safely guided.

  Note that, in the top view display, when it is confirmed that any image signal of the imaging camera is abnormal, a technique for changing the area where the imaging camera image signal is adopted and changing the boundary line is disclosed in Patent Document 1. ing.

JP 2007-090882 A

  In the above-described vehicle outside display device using a plurality of camera images, a desired composite image can be obtained during normal operation, and all white lines and obstacles can be highlighted by the white line obstacle detection means, which is extremely effective. . However, the external camera has at least the lens part or its cover part exposed to the outside of the car, and is normally exposed to wind and rain, and is easily contaminated with dust and mud. Sometimes. In this case, as shown in FIG. 6A as an example of the vehicle rear-view camera 34, as a result of the water droplet 50 adhering, as shown in FIG. The improper image portion 51 exists in the captured image range. The same applies to the inappropriate photographed image example 2 shown in FIG. 6C, which is an example when the water droplet 50 is relatively small.

  For this reason, the image behind the white line 36 detected and highlighted by the white line obstacle detection means, which is the most important display element, is displayed indefinitely or not at all. As a result, as described in the description of FIG. 5D, in FIG. 6B and FIG. 6C, the rear end portion of the white line 36 to be highlighted is lower in the drawing than the lower left corner display boundary line 46. The part of is not displayed. In addition to the case of the top view system as described above, this problem is particularly noticeable when, for example, a captured image of only the vehicle rear-view camera 34 as shown in FIG. 11 is displayed.

  Therefore, the present invention captures the outside of the vehicle with a plurality of cameras, displays an image as if it was captured with a single camera, displays other various images as necessary, and removes white lines and obstacles from the captured image. When detecting that there is one improper shooting camera in a multi-camera image using vehicle outside display device that detects and highlights, whether or not the improper shooting camera image range is duplicated by another camera is determined. When there is a proper shooting camera that has been detected and overlapped, multiple camera images are displayed outside the vehicle so that white lines and obstacles to be highlighted can be easily interpolated using the captured images of that camera. The main purpose is to provide a device.

  In order to solve the above-described problem, the multi-camera image using vehicle outside display device according to the present invention has a plurality of cameras set so that there is a range in which images are taken overlapping each other, and white lines or obstacles depending on the captured images of the cameras. A plurality of cameras in an on-vehicle display device using a multi-camera image, comprising: white line obstacle detecting means for detecting an object; and white line obstacle highlighting means for highlighting a white line or an obstacle detected by the white line obstacle detecting means. A multi-camera overlapping photographing detecting means for detecting overlapping photographing portions that are photographed overlapping each other, and a water droplet adhesion for detecting an image in which water droplets are attached to the lens in the overlapping photographing portion detected by the multiple camera overlapping photographing detecting means When the image detection means and the water droplet adhesion image detection means detect an image with water droplets attached to the lens, the image with water droplets attached is output. An improper shooting camera specifying means for specifying the camera as an improper shooting camera, and an outer edge of a predetermined range including a water droplet adhesion image portion from the shot image of the camera specified as the improper shooting camera by the improper shooting camera specifying means. Inappropriate image range outer edge specifying means for specifying, and conversion processing means for converting the coordinates of the improper photographing camera of the outer edge specified by the improper image range outer edge specifying means into the coordinates of the appropriate photographing camera for photographing the same overlapping photographing range And the white line obstacle image data obtained by the white line obstacle highlighting means within the outer edge of the image of the appropriate photographing camera converted by the conversion processing means, and this image data is taken as the inappropriate camera photographing image data. A data interpolation processing means for improper photographing camera that converts and interpolates into white line obstacle image data of the improper camera photographing image range part It is characterized in.

  According to another aspect of the present invention, there is provided a multi-camera image using vehicle outside display device, wherein the water droplet adhesion image detecting means in the multiple camera image using vehicle outside display device, It is characterized in that it is determined that the image is attached.

  In addition, the multi-camera image using vehicle outside display device according to the present invention detects a white line or an obstacle by each of a plurality of cameras set so that there is a range to be photographed overlapping each other, and a photographed image of each camera. In a multi-camera image using vehicle outside display device comprising a white line obstacle detecting means for performing and a white line obstacle highlighting means for highlighting a white line or an obstacle detected by the white line obstacle detecting means, the plurality of cameras overlap each other Multi-camera overlapping photographing detecting means for detecting the overlapping photographing part being photographed, and the white line or obstacle of each camera detected by the white line obstacle detecting means in the overlapping photographing part detected by the multiple camera overlapping photographing detecting means The white line obstacle image difference detection means for comparing the images of the objects and detecting the difference between the images, and the white line obstacle image difference detection means It is detected that the images are different from each other by a predetermined amount, and the white line obstacle image of one camera is a series of appropriate images, and the other camera lacks at least a part of the series of appropriate images, or exceeds a predetermined value. When it is detected that the camera is deformed, an improper shooting camera specifying unit that specifies the other camera as an improper shooting camera, and a captured image of the camera specified as the improper shooting camera by the improper shooting camera specifying unit Incorrect image range outer edge specifying means for specifying an outer edge of a predetermined range including an image part in which a white line obstacle image is missing or deformed, and an inappropriate outer edge specified by the inappropriate image range outer edge specifying means Conversion processing means for converting the coordinates of the photographing camera into coordinates of an appropriate photographing camera for photographing the same overlapping photographing range, and within the outer edge of the image of the proper photographing camera converted by the conversion processing means. The white line obstacle image data obtained by the white line obstacle highlighting means is captured, the image data is converted into the inappropriate camera photographed image data, and interpolated into the white line obstacle image data in the inappropriate camera photographed image range portion. And a data interpolation processing means for the inappropriate photographing camera.

  In addition, the multi-camera image using vehicle outside display device according to the present invention detects a white line or an obstacle by each of a plurality of cameras set so that there is a range to be photographed overlapping each other, and a photographed image of each camera. In a multi-camera image using vehicle outside display device comprising a white line obstacle detecting means for performing and a white line obstacle highlighting means for highlighting a white line or an obstacle detected by the white line obstacle detecting means, the plurality of cameras overlap each other In the multiple camera overlapping shooting detection means for detecting the overlapping shooting portion being shot and the overlapping shooting portion detected by the multiple camera overlapping shooting detection means, the captured images of each camera are compared, and the difference between the images is determined. When it is detected by the camera-captured image difference detection means to detect and the camera-captured image difference detection means that the images are different from each other by a predetermined amount, From the captured image of the camera specified as the inappropriate shooting camera by the inappropriate shooting camera specifying means for specifying whether the misaligned camera is outputting the inappropriate image, the inappropriate image An inappropriate image range outer edge specifying means for specifying an outer edge of a predetermined range including a portion, and an appropriate shooting camera for shooting the same shooting range with the coordinates of the inappropriate imaging camera of the outer edge specified by the inappropriate image range outer edge specifying means A white line obstacle image data obtained by the white line obstacle highlighting means within the outer edge of the image of the appropriate photographing camera converted by the conversion processing means, and converting the image data into the coordinates; Data interpolating process for improper camera that converts to improper camera image data and interpolates into white line obstacle image data of improper camera image area Characterized by comprising a means.

  Further, in the other multi-camera image using vehicle outside display device according to the present invention, in the plural camera image using vehicle outside display device, the inappropriate camera specifying means detects that images taken by the cameras are different from each other by a predetermined amount or more. Then, after confirming which camera image is inappropriate, the inappropriate camera is specified.

  In addition, another multi-camera image using vehicle outside display device according to the present invention is the multi-camera image using vehicle outside display device, in which the coordinate conversion processing unit is configured for one of the cameras created in advance for the overlapping photographing ranges of the plurality of cameras. A coordinate conversion table that directly converts fisheye coordinates to fisheye coordinates of the camera on the other side is used.

  Since the present invention is configured as described above, in a multi-camera image using vehicle exterior display device that captures the outside of a vehicle by a plurality of cameras and displays a photographed image, and detects and highlights white lines and obstacles. When it is detected that there is an inappropriate camera with two cameras, it is detected whether the image range of the inappropriate camera is overlapping with other cameras, and there is an appropriate camera that is shooting twice. In some cases, it is possible to interpolate the display of white lines and obstacles in the inappropriate shooting range with the white lines and obstacle display images in the appropriate shooting area.

It is a functional block diagram of the Example of this invention. It is an operation | movement flowchart of the Example. In the same Example, it is an operation | movement flowchart which shows the example of the process which produces a data conversion table about an overlapping imaging | photography range. In the same Example, it is a figure which shows the basic example of the corresponding pixel calculation method between cameras. It is explanatory drawing of the example of the multi-camera image use vehicle exterior display apparatus with which this invention is applied. It is explanatory drawing when an improper image is displayed by a water droplet adhering to the lens of a vehicle rear side camera. It is explanatory drawing when performing a process in order by the Example of this invention at the time of an improper image display. It is explanatory drawing when performing a process in order by the Example. It is explanatory drawing when processing is further performed in order according to the embodiment. It is a figure which shows the example of a data conversion table. It is an example of the interpolation display of the white line obstruction image by the appropriate photography camera of the improper image display part at the time of vehicle rear photography camera single image display.

  Embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a functional block diagram of an apparatus for carrying out the present invention. In the functional block diagram shown in FIG. 1, functional units that perform the respective functions can be said to be means for performing the respective functions. Inappropriate camera white line obstacle image interpolation processing unit 1 in FIG. 1 includes an inadequate camera specifying processing unit 2 for specifying an inadequate camera. Detect and identify whether it is. The improper camera identification processing unit 2 includes a multiple camera overlapping shooting part detection unit 3 that detects overlapping shooting parts of a plurality of cameras, so that the duplication exists at the four corners in FIG. 5C. The photographing parts 42, 43, 44, 45 are detected. In addition, when the apparatus of the present invention is mounted on a specific vehicle, the overlapped image range is implemented by capturing the value of the initial setting value when there is data regarding the overlapping shooting range in relation to the initial setting of the camera. You can also

  In the multiple camera overlapping captured partial image difference detection unit 4, the camera captured image comparison unit 5 therein directly captures the captured image of the camera captured image capturing unit 20 that captures the captured image of each camera, or the camera captured image thereof. The processing image of the camera-captured image display processing unit 19 that performs processing for display is captured, and the captured images of the two cameras are compared for the overlapped captured portion detected by the multiple-camera overlapping captured-portion detection unit 3, Detect image differences.

  The white line obstacle image difference detection unit 6 takes in the white line obstacle image data from the white line obstacle detection unit 18 or the white line obstacle highlight display unit 17 and compares the images. As a result of the comparison, white lines and obstacles are displayed properly and continuously in the image of one camera, while the image of the camera on the other side is not discontinuous, deformed, lack of image, etc. Whether or not the appropriate image is present is detected.

  The improper shooting camera confirmation specifying unit 7 finally specifies a camera that is shooting improperly. In this case, the white line obstacle image difference detection unit 5 uses one of the cameras as described above. In the image of, white lines and obstacles are displayed properly and continuously, but when the image of the other camera detects that there is an inappropriate image in that part, the camera is inappropriate It can be immediately identified as a shooting camera. In addition, in the water droplet adhesion image detection unit 8 in the inappropriate photographing camera confirmation specifying unit 7, a characteristic that appears when water droplets are attached to the overlapping photographing part by the image of the camera photographing image capturing unit 20 or the camera photographing image display processing unit 19. When it is detected that there is an image with a brightness change or a distorted image peculiar to water droplets, the camera displaying such an image is immediately identified as an inappropriate photographing camera.

  In the inappropriate photographing camera confirmation specifying unit 7, even when the camera photographed image difference detection unit 5 performs comparison processing on photographed images of a plurality of cameras, even when it is detected that the photographed images are different from each other by a predetermined amount or more. Since it is not always clear which camera is improper or both are improper, the confirmation process is performed. That is, in the case where there are relatively many features of the image of the improper image part, such as when water droplets adhere to the surface of the camera, this image is taken when it matches the pattern prepared in advance. It can be seen with a considerable probability that a water droplet has adhered to the camera that is being displayed, and at least a different image from that of the other camera that is shooting this portion is being displayed. On the other hand, improper images due to dust and dirt adhering to the lens are difficult to understand at a glance, and it is necessary to check various images taken by the two cameras that capture the overlapping area.

  In that case, for example, as described above, even when it is determined that the image of the vehicle rear-view camera 34 is inappropriate in the overlapping shooting portion, the vehicle left-side shooting camera shooting the image in the overlapping shooting range is The overlapping photographing range 45 with the camera 35 is checked again, and it is checked whether image differences with the same tendency have occurred. When a similar image difference has occurred here, the difference in the image in the overlapping shooting range with the vehicle rear camera is not simply a dirt, etc., it is highly likely that it is a camera installation change, adjustment change, etc. The countermeasures will be taken after that.

  In addition, in the inappropriate photographing camera confirmation specifying unit 7, for example, in FIG. 5C, when an image difference of a predetermined value or more is found in the overlapping photographing range 42 between the vehicle rear side photographing camera 34 and the vehicle left side photographing camera, When the photographing camera 34 is checked again for the overlapping photographing range of the other right photographing camera 43 which is another overlapping photographing range 43 and the difference is the same tendency difference, the vehicle rear photographing camera 34 causes a misalignment or the like. Judge that it is. On the other hand, when it is detected that there is almost no difference in the overlapping shooting range 43 with the vehicle right-side shooting camera 33, particularly when the vehicle right-side shooting camera 33 is confirmed to be almost correct by the above-described checking method, It can be estimated almost accurately that a part of the camera has water drops, dirt, or the like.

  Furthermore, for example, it is possible that an improper image happens to appear as a white line. Considering such a case, refer to the overlapping area of the left camera and the rear camera and refer to the same area. Whether or not a pattern is found is detected, and a process for increasing the accuracy that the left camera on the vehicle is outputting the correct video is performed. As a result of such processing, it is possible to increase the accuracy of the vehicle left-side camera outputting a correct captured image.

  In the inappropriate camera specification processing unit 2 in FIG. 1, when a camera that performs inappropriate shooting is specified by the series of processes as described above, the inappropriate camera overlapping shooting range inappropriate image range outer edge specifying unit 9 is identified. Thus, the process of specifying the outer edge of the inappropriate image range is performed. That is, for example, a water droplet 50 as shown in FIG. 6A adheres to the vehicle rear-facing camera 34, and the image of this portion is displayed with the luminance of a specific image portion greatly changed or greatly deformed. Become. When this is divided and displayed in the same range as in the prior art, the rear portion of the white line 36 exists in the vehicle rearward shooting range, and therefore cannot be displayed substantially.

  Therefore, in the example shown in FIG. 1, first, the improper camera overlapping photographing range improper image range outer edge specifying unit 9 first identifies the outer edge of the improper image range among the images photographed by the vehicle rear photographing camera 34. Processing is in progress. In this case, for example, when an improper shooting range 51 is detected in the overlapping shooting range 42 with the left side camera in the shooting range of the vehicle rear shooting camera as shown in FIG. 7C, the outer edge of this range is specified. In this case, the outer edge (D) of the overlapping photographing display dividing line as shown in FIG. 10D is specified by the function for setting the overlapping photographing display dividing line 10 in FIG.

  Further, the outer edge (H) in FIG. 4D is specified by the function of setting the overlapping camera photographing range 11. Moreover, the outer edge (F) in the same figure (d) is specified by the function which sets the display range 12 within. Further, the outer edge (E) of the improper photographing range 51 shown in FIG. 5C is set by the function of setting the improper image area outer edge 13. As a result, the improper image outer edge within the vehicle rear-view camera overlapping shooting range as shown in FIG.

  The range surrounded by the outer edge in the rear camera image of the vehicle identified by the improper camera overlapping photographing range outer edge specifying unit 9 in FIG. 1 is converted to the outer edge of the improper image range. In the processing unit 14, a process for converting to the outer edge of the appropriate captured camera image is performed. In this case, the example shown in FIG. 1 shows an example in which conversion is performed using the coordinate conversion table 15.

  This coordinate conversion table 15 is actually obtained by performing a number of coordinate conversions as shown in FIG. 4B in accordance with the basic concept of the corresponding pixel calculation method between cameras shown in FIG. This is implemented by processing to obtain replacement data. That is, in the first place, in the technical field of the present invention, world coordinates (coordinates in real life), camera coordinates (coordinates with the camera as the origin), fisheye coordinates (coordinates on the fisheye image) displayed on the vehicle monitor. Can convert data between each. FIG. 4B shows this in more detail. In the example shown in the drawing, the coordinates of point A on the fisheye image of the front camera are first detected in step S31.

  Next, in step S32, the fisheye coordinates of the selected point A are converted into front camera coordinates. In step S33, the front camera coordinates of point A are converted to world coordinates. Thereafter, in step S34, the world coordinates of point A are converted into vehicle right-side shooting camera coordinates. Thereafter, the right-side shooting camera coordinates of point A are finally converted into fish-eye coordinates of the right-side shooting camera.

  With this series of conversion processing, the coordinates of the point A on the front-shot camera fisheye image can be converted to the fish-eye coordinates of the right-side camera, but the right-side shot is taken on the front-forward camera fisheye image. Regarding the overlapping shooting range with the camera, a data table of points corresponding to each other for converting the coordinates of the points of the right shooting camera into the coordinates of the front shooting camera is prepared in advance for all the points. As a result, conversion processing is performed using the A-point-to-B conversion table as described above, as shown in FIG. 4B, without performing any special setting or performing many operations. The replacement data can be obtained extremely easily and at high speed.

  A more specific example of the data conversion tail at this time is shown in FIG. In the example of the data conversion table shown in the figure, in each part of the overlapping shooting ranges 42 to 45, data for converting the fisheye coordinates of each camera that is being shot in duplicate into the fisheye coordinates of other cameras. The fisheye data of one camera can be converted into the points and ranges of fisheye coordinates of other cameras at each point and each range without any particular conversion process.

  In the example shown in FIG. 1, a data interpolation processing unit 16 for appropriately duplicating shooting camera range white line obstacle image to an improper shooting camera is provided, and the conversion processing unit 14 for converting the outer edge of the improper image range to an outer edge of the appropriate shooting camera. For the image range of the appropriate photographing camera that has been subjected to the conversion process, a process for interpolating the white line obstacle image in this range portion by the proper photographing camera into the white line obstacle image in the photographed image of the inappropriate camera is performed. At this time, the white line obstacle highlight display image obtained by the white line obstacle highlight display unit 17 to be described later is subjected to interpolation processing without performing any special calculation by using the table of the coordinate conversion table 15. It can be carried out.

  On the other hand, as described above, the images of the cameras that capture the front, rear, left and right of the vehicle are captured by the camera captured image capturing unit 20, and the camera captured image display processing unit 19 performs processing for displaying the captured images on the monitor. . At this time, when each camera is an image of a fisheye lens, coordinate conversion processing is performed to display the image on the monitor in an easily understandable manner. Also at this time, conversion processing can be performed using a similar coordinate conversion table different from the coordinate conversion table 15. Based on the camera-captured image data input to the camera-captured image display processing unit 19, the white-line obstacle detection unit 18 applies white lines and obstacles to the captured images of the cameras using various conventionally proposed methods. To detect. Based on the detection result, the white line obstacle highlighting display unit 17 emphasizes the white line and the obstacle in the camera image displayed by the camera-captured image display processing unit 19 on the monitor and displays it clearly.

  In the embodiment of the present invention comprising the functional blocks of FIG. 1, the present invention can be implemented according to the operation flow shown in FIG. 2, for example. In the example of the appropriate white line obstacle image interpolation processing of the improper photographing camera shown in FIG. 2, an example in which the present invention can be implemented in various aspects is shown, and particularly as shown in step S <b> 7. In order to perform the process of specifying an inappropriate image of an inappropriate camera, various modes are shown in FIG.

  In the example shown in FIG. 2, it is determined whether or not the detection process for the first water droplet adhesion specific image is performed (step S <b> 1). In this determination, the determination is made based on whether or not a setting for performing such processing is performed in advance. This process is shown as one process in the improper photographing camera confirmation specifying unit 7 in FIG. 1, and is the water droplet adhesion image detecting unit 8 and the image data of the camera photographing image capturing unit 20 or the camera photographing image display processing unit 19. Thus, for example, when water droplets adhere to the lens portion of the camera due to rain or a car wash as shown in FIG. 6A, it is detected that an improper image as shown in FIGS. To do.

  As a method for determining whether or not the image is an improper image at that time, it is a typical improper image due to water droplet adhesion when it is detected that a large luminance change has occurred in a captured image of one camera. That is, this camera can detect that an inappropriate image due to water droplets is being output at this portion. In addition, when detecting improper images due to water droplet adhesion, various detection methods such as detecting that a water droplet adheres to a part of the lens and outputting a specially distorted image accompanying water droplet adhesion are used. It can also be adopted.

  When it is determined in step S1 of FIG. 2 that the detection process of the water droplet adhesion specific image is performed, it is determined in step S13 whether or not the water droplet adhesion specific image is detected. Here, when it is determined that a unique image with water droplets is detected by a technique such as that described above, a brightness change peculiar to water droplet adhesion is detected, the inappropriate image of the inappropriate camera is identified in step S7. Done. That is, in the above case, a camera that outputs an image peculiar to water droplet adhesion is specified as an inappropriate camera, and in the above example, a range in which a large luminance change occurs is specified as an inappropriate image portion.

  When it is determined in step S13 that a water droplet adhesion specific image has not been detected, in the example of FIG. 2, whether or not the water droplet adhesion specific image detection processing is performed in step S1, and the white line obstacle image comparison processing is performed. It is discriminate | determined (step S3). This determination is also made based on whether or not the setting for performing this process has been made in advance. In the example of FIG. 2, in the various improper camera identification methods according to the present invention, a water drop adhesion specific image is detected as a technique that can identify an improper camera by a very simple process. In this example, even if no image is detected, it is detected whether there is a camera that outputs an inappropriate image by another method.

  When it is determined in step S2 that the white line obstacle image comparison process is to be performed, in step S14, it is determined that the white line obstacle image is obtained in the overlapping photographing range while one of the cameras is appropriate. It is determined whether or not the camera outputs a white line obstacle image different from this. That is, the white line obstacle image obtained from the images taken by both cameras in the overlapping photographing range is compared, and the white line obstacle image of one camera is photographed on the left side of the vehicle as shown in FIG. The camera outputs a predetermined series of images with respect to the white line, whereas the white line obstacle image obtained by the rear camera, which is the other camera, does not exist or is partially missing, or When the image is deformed more than a predetermined amount, it is determined in step S14 that an inappropriate image is being output. This processing is performed using the image data of the white line obstacle detection unit 18 or the white line obstacle highlighting display unit 17 in the white line obstacle image difference detection unit 6 in the multiple camera overlapping photographing partial image difference detection unit 4 of FIG. ing.

  In step S14, when a camera outputting an inappropriate white line obstacle image as described above is detected, an inappropriate image of the inappropriate camera is specified in step S7. When it is determined that such an improper white line obstacle image does not exist, in the example of FIG. 2, the process proceeds to step S3 when it is determined that the comparison process of the white line obstacle particle image is not performed in step S2. Then, it is determined whether or not to perform comparison processing of the camera captured image itself.

  That is, in the example of FIG. 2, in step S13 and step S14, or in any one of them, a technique for specifying a camera that outputs an inappropriate image as easily as possible is adopted. In order to specify a camera that outputs an inappropriate image, in step S3, it is determined whether or not to perform processing for comparing the captured images of both cameras in the overlapping imaging range. However, when it is determined in step S1 that the water droplet adhesion specific image is not detected, and in step S2, it is determined that the white line obstacle image comparison process is not performed, the camera captured image itself is compared in step S3. Shall be determined.

  If it is determined in step S3 that such a comparison process is not performed, the process returns to step S1 and the above process is repeated. That is, for example, when it is determined in step S1 that the water droplet adhesion specific image detection process is not performed and the white line obstacle image comparison process is performed, or in step S1, the water droplet adhesion specific image detection process is performed, and step S13 is performed. In step S2, it is determined that the comparison process of the white line obstacle image is not performed, and when it is determined that the respective processes are performed in step S1 and step S2, It is possible to perform the process of returning to step S1 and repeating the operation when setting is made so as not to perform the subsequent process of comparing the camera-captured images themselves.

  In step S3, when it is determined that the camera-captured image itself is to be compared, it is determined whether or not there is an image difference of a predetermined value or more in the overlapped imaging range among the captured images of the plurality of cameras (step S4). This processing is performed by the camera-captured image difference detection unit 5 in the multiple-camera overlapping-photographed partial image difference detection unit 4 of FIG. 1 using the image data of the camera-captured image capture unit 20 or the camera-captured image display processing unit 19. ing.

  In this process, when it is determined in step S4 that there are no more than a predetermined image difference in the overlapping shooting range among the images shot by the plurality of cameras, the process returns to step S1 and the above operation is repeated. Here, when it is determined that there is a difference between the images of a predetermined value or more, an inappropriate camera confirmation process is performed (step S5). That is, in step S1, even if one of the duplicate images is incorrect due to an inappropriate image, it is not clear which camera's captured image is inappropriate. There is a need to.

  For this reason, for example, water droplets 50 are attached to the vehicle rear-view camera 34 as shown in FIG. 6A, and as a result, as shown in FIG. When there is a large range of inappropriate images 51 in the captured image range of the imaging camera 34, the imaging range of the left-side camera 32 is present in the range including the inappropriate images 51, and the overlapping captured portion 42 is present. Will exist. In addition, the shooting range of the left side shooting camera 33 exists in the range including the inappropriate image 51, and the overlapping shooting portion 43 exists.

  In such an example, the images captured by the two cameras are compared for all the overlapping shooting ranges to check whether there is a difference of a predetermined value or more, and the vehicle rear shooting camera 34 and the vehicle left-side shooting camera 32 in the overlapping shooting range 42. It is detected that the captured images are different from each other by a predetermined amount. At this time, when overlapping images are compared for the other overlapping shooting ranges 45 in the vehicle left-side shooting camera 32, there is no significant difference, while in the shooting range of the vehicle rear-side shooting camera 34 overlaps with the vehicle right-side shooting camera 33. With regard to the shooting range 43, it is possible to confirm that the image difference in the overlapped shooting range 42 is due to the vehicle rear shooting camera 34 by detecting a difference in image more than a predetermined value. Similarly, when there is no large image difference in the other overlapping shooting range 44 of the vehicle right side shooting camera 33, the difference in the image in the overlapping shooting range 43 is caused by the vehicle rear shooting camera 34 shooting an inappropriate image. You can confirm that this is because

  On the other hand, for example, as shown in FIG. 6 (c), when the improperly captured image 51 by the vehicle rear-facing camera 34 is relatively small, the images captured by the two cameras are compared in this overlapping shooting range 42, When it is detected that there is a large difference in the images and which camera is taking an inappropriate image, it may be confirmed by a method such as the example shown in FIG. 6B. Can not.

  In such a case, for example, as shown in FIG. 6D, in the image of the vehicle left side camera 36, the white line 36 is continuously captured including the image area of the vehicle rear camera, or A portion presumed that the same obstacle 38 is shot in the overlapping shooting range of both cameras is present in the shot images of the respective cameras, and the image of the rear camera 34 of the vehicle is as shown in FIG. However, when the whole is continuously photographed in the image of the vehicle left side photographing camera 32 as shown in FIG. 4D, the whole image is not continuously photographed. It is confirmed that the image is appropriate and that the image from the rear camera is not appropriate. In particular, by confirming that the same image of the obstacle 38 is captured by the two cameras, for example, it is possible to further confirm that all of the captured images of the left camera 32 have been captured. it can.

  In step S5 of FIG. 2, the above-described inappropriate camera confirmation process is performed by various methods. In step S6, it is determined whether or not an inappropriate image of the inappropriate camera has been confirmed (step S6). If it is determined that an inappropriate image of the inappropriate camera has been confirmed, the inappropriate image of the indeterminate photographing camera is specified in step S7. If it is determined in step S6 that an inappropriate camera has not been confirmed by the above-described various methods, the process returns to step S1 and the operation is repeated.

  Thereafter, the outer edge of the inappropriate shooting range of the inappropriate camera is specified (step S8). This operation is performed in the improper image overlap improper image range outer edge specifying unit 9 in FIG. In that case, for example, an improper photographed image portion 51 exists in the image of the vehicle rearward photographing camera 34 as shown in FIG. 7A, and this is shown in the overlapping photographed image portion as shown in FIG. The improper image portion is specified, and further, as shown in FIG. 4D, the outer edge of the improper image is specified in consideration of the image drawing division range of the left camera in the overlapping shooting range.

  In the example of FIG. 2, the indeterminate photographed image outer edge of the inappropriate photographing camera is then converted into the outer edge of the proper photographing camera image (step S <b> 9). That is, after the outer edge of the improper captured image range of the improper camera is specified as shown in FIG. 7D, this is taken as an example of rearward shooting of the overlapping photographing range 42 as shown in FIG. The camera fisheye coordinates are applied as shown in (e) in the figure, thereby obtaining the coordinates of the necessary interpolation range of the white line obstacle as shown in (f) in the figure, and the fisheye coordinates in the vehicle rear-view camera. Determine the outer edge of. Thereafter, the outer edge of the improper image range as shown in FIG. 7 (f) and FIG. 8 (a) is changed to the fish-eye coordinates of the vehicle left side photographing camera that outputs the appropriate image as shown in FIG. 8 (b). Convert.

  Next, in step S10, white line obstacle image data included in the outer edge within the range of the image captured by the appropriate camera converted as described above is captured. This process determines the outer edge of the rear camera fisheye coordinates as shown in FIG. 8A, and converts the outer edge converted into the outer edge in the left camera fisheye coordinates as shown in FIG. 8B. Within the range, from the white line obstacle image data of the overlapping shooting range of the left side camera as shown in FIG. 8C, the left camera fisheye coordinates are applied as shown in FIG. This is done by fetching data as shown in e).

  In step S11, the coordinate conversion table is used to interpolate the white line obstacle image in the inappropriate photographed image range of the inappropriate camera. This processing is performed using the data of the white line obstacle emphasis display unit 17 and the coordinate conversion table 15 in the interpolation processing unit 16 of the appropriate photographing camera range white line obstacle image of FIG. In that case, for example, based on the white line obstacle image of the left camera of the vehicle as shown in FIG. 8 (e), the interpolated image to be captured is determined and the fish-eye coordinate data is obtained, and the image is obtained. As shown in FIG. 9 (f), the rear camera is converted into fish-eye coordinates, and as shown in FIG. 9 (a), it is not properly captured by an inappropriate image of the vehicle rear camera. The white line obstacle image shown in FIG. As a result, a white line obstacle image of the overlapping photographing portion as shown in FIG. 6C is obtained, and an inappropriate photographing image of the rear photographing camera is obtained so that the entire white line obstacle image is shown in FIG. It is possible to obtain a white line obstruction image in which there is substantially no.

  In step S12 of FIG. 2, an appropriate white line obstruction is obtained by interpolating an inappropriate photographing portion of one camera in the overlapping photographing range with a white line obstacle image based on a proper photographing image of another camera by the series of processes as described above. An object image can be obtained, and this process is performed by interpolating the white line obstacle image in the improper captured image of all the cameras with the white line obstacle image obtained by the image captured by the other appropriate photographing camera. Terminate the process.

  The coordinate conversion table 15 of FIG. 1 used in the embodiment of the present invention can be implemented without any special calculation using the coordinate conversion table of each overlapping range of FIGS. 10 (a) to 10 (e). However, the coordinate conversion table can be generated based on the concept of the coordinate conversion system of the present invention described with reference to FIG. 4, for example, according to the operation flow shown in FIG. That is, in the process of creating the data conversion table for the overlapping shooting range shown in FIG. 3, first, the pixels of one camera are sequentially selected for one overlapping shooting range (step S21). Next, the fish-eye image coordinates of the selected pixel are converted into plane camera coordinates (step S22).

  Thereafter, the converted planar camera coordinates are converted into world coordinates of the entire camera (step S23). Next, the converted world coordinates are converted into plane camera coordinates of another camera that captures the overlapping range (step S24). Thereafter, the converted planar camera coordinates are converted into fish-eye coordinates of another camera (step S25). Next, it is determined whether or not the coordinate conversion of the pixels of one camera in one overlapping photographing range has been completed (step S26). In this determination, when it is determined that the coordinate conversion has not been completed yet, the process returns to step S21 to perform processing for sequentially selecting the pixels of one camera for this one overlapping photographing range, and thereafter the same operation is repeated.

  On the other hand, when it is determined in step S26 that the coordinate conversion of the pixels of one camera in one overlapping shooting range has been completed, it is determined in step S27 whether the coordinate conversion table for the overlapping shooting ranges at the four corners has been completed. Determine. Here, when it is determined that the coordinate conversion table for all of the four corners is not yet completed, in step S28, another overlapping photographing range that has not been completed is selected, and the process returns to step S21 again. Repeat the above operation. When such processing is continued and it is determined in step S27 that the coordinate conversion table for the overlapping photographing ranges at the four corners has been completed, this conversion processing ends (step S29).

  By using the coordinate conversion table as described above, an image with one fisheye lens can be easily converted into an image with the other fisheye lens for the overlapping shooting range, and a white line obstacle due to an improper image of one camera The image can be easily interpolated by the white line obstacle image of the appropriate photographing camera on the other side, and the easy-to-see white line obstacle image can be displayed easily and clearly.

  Moreover, in the said Example, although the example at the time of synthesize | combining and displaying the image of a some camera was shown, the example at the time of displaying the image of a vehicle rear-view camera on a monitor other than that is shown, for example in FIG. As described above, although it is determined that it is better not to display the synthesized image as described above even though it has a plurality of cameras, when it is decided to display a photographed image of a single camera by a multi-view system, As shown in FIG. 5A, when the vehicle rear-view camera is attached with water droplets or the like as described above, even when an inappropriate image is generated, the image is not displayed but the image is taken. Using the white line obstacle image obtained by the proper operation camera of the camera, for example, as shown in FIG. 11B, the white line obstacle image in the overlapping photographing range portion of the inappropriate photographing camera is used as the white line obstacle of the proper photographing camera. It is also possible to display the interpolated image. Since the operation at this time can be performed in the same manner as described above, a detailed description thereof will be omitted.

  In the above-described embodiment, for example, for an improper image captured by the vehicle rear-facing camera, the improper image range outer edge specifying means identifies the outer edge of the improper image capturing range, and only this range is duplicated. Although an example of capturing and interpolating a white line obstacle image by a photographing camera has been shown, other than that, for example, when it is detected that an improper photographed image is present in a vehicle rear photograph camera, the undefined photographed image exists. It is also possible to set a rectangular range of an appropriate size including the portion as the outer edge of the inappropriate image range. Furthermore, it is possible to specify the entire overlapping photographing range where the improper image exists as an outer edge, and to perform interpolation processing on the white line obstacle image by the proper photographing camera. In such a case, it is necessary to perform unnecessary interpolation processing when the improper shooting range is narrow, but it is necessary to specify the outer edge as described above, or to select white line obstacle image data only for the specified outer edge. There is an advantage that it is not necessary to perform complicated processing.

DESCRIPTION OF SYMBOLS 1 Inappropriate camera white line obstacle image interpolation process part 2 Inappropriate camera specific process part 3 Multiple camera duplication imaging | photography part detection part 4 Multiple camera duplication photography partial image difference detection part 5 Camera photography image difference detection part 6 White line obstacle image difference detection Unit 7 Inappropriate shooting camera confirmation specifying unit 8 Water droplet adhesion image detection unit 9 Inappropriate camera overlapping imaging range inappropriate image range outer edge specifying unit 10 Duplicate shooting display dividing line 11 Overlapping camera shooting range 12 Within display range 13 Inappropriate Inner image range outer edge 14 Conversion processing unit 15 for inappropriate image range outer edge to appropriate shooting camera image outer edge Coordinate conversion table 16 Interpolation processing unit 17 for appropriate shooting camera range white line obstacle image to inappropriate shooting image range White line obstacle enhancement Display unit 18 White line obstacle detection unit 19 Camera shot image display processing unit 20 Camera shot image capture unit 32 Vehicle left side shooting camera 33 Vehicle right side shooting Camera 34 Vehicle rear camera 35 Vehicle front camera

Claims (6)

A plurality of cameras set so that there is a range to shoot overlapping each other,
White line obstacle detection means for detecting a white line or an obstacle by the captured image of each camera;
In a multi-camera image using vehicle outside display device comprising a white line detected by the white line obstacle detection means or a white line obstacle highlighting display means for highlighting an obstacle,
A multi-camera overlapping shooting detection means for detecting an overlapping shooting portion where a plurality of cameras are shooting overlapping each other;
In the overlapping photographing part detected by the multiple camera overlapping photographing detecting means, a water droplet adhesion image detecting means for detecting an image in which water droplets adhere to the lens,
An improper shooting camera specifying means for specifying, as an improper shooting camera, a camera that outputs an image with water droplets attached thereto when an image in which water droplets are attached to the lens is detected by the water drop adhesion image detecting means;
An improper image range outer edge specifying means for specifying an outer edge of a predetermined range including a water droplet adhesion image portion from a photographed image of the camera specified as an improper shooting camera by the improper shooting camera specifying means;
Conversion processing means for converting the coordinates of the improper photographing camera of the outer edge specified by the improper image range outer edge specifying means into the coordinates of the image of the appropriate photographing camera for photographing the same overlapping photographing range;
The white line obstacle image data obtained by the white line obstacle highlighting means within the outer edge of the image of the appropriate photographing camera converted by the conversion processing means is captured, and the image data is converted into inappropriate camera photographing image data. A multi-camera image in-vehicle display device comprising: a data interpolation processing means for an improper photographing camera that interpolates white line obstacle image data in an improper camera photographed image range portion. The multi-camera image using vehicle exterior display device according to claim 1, wherein the water droplet adhesion image detecting means determines that the image is a water droplet adhesion image when a luminance change of a predetermined level or more is present in the photographed image. A plurality of cameras set so that there is a range to shoot overlapping each other,
White line obstacle detection means for detecting a white line or an obstacle by the captured image of each camera;
In a multi-camera image using vehicle outside display device comprising a white line detected by the white line obstacle detection means or a white line obstacle highlighting display means for highlighting an obstacle,
A multi-camera overlapping shooting detection means for detecting an overlapping shooting portion where a plurality of cameras are shooting overlapping each other;
The white line obstacle image difference for comparing the white line or obstacle image of each camera detected by the white line obstacle detection means and detecting the difference between the images in the overlapping photographing part detected by the multiple camera overlapping photographing detection means Detection means;
The white line obstacle image difference detection means detects that the images are different from each other by a predetermined amount,
When the white line obstacle image of one camera is a series of proper images and the other camera detects that at least a part of the series of proper images is missing or deformed more than a predetermined value, Improper shooting camera specifying means for specifying the camera on the side as an inappropriate shooting camera;
An improper image range outer edge that identifies an outer edge of a predetermined range including an image portion in which a white line obstacle image is missing or deformed from a photographed image of the camera identified as an improper photographing camera by the inappropriate photographing camera specifying means Specific means,
Conversion processing means for converting the coordinates of the improper photographing camera of the outer edge specified by the improper image range outer edge specifying means into the coordinates of the image of the appropriate photographing camera for photographing the same overlapping photographing range;
The white line obstacle image data obtained by the white line obstacle emphasis display means in the outer edge of the image of the appropriate photographing camera converted by the conversion processing means is captured, and this image data is converted into inappropriate camera photographing image data, A multi-camera image in-vehicle display device comprising data interpolation processing means for an improper photographing camera that interpolates white line obstacle image data in an improper camera photographing image range portion. A plurality of cameras set so that there is a range to shoot overlapping each other,
White line obstacle detection means for detecting a white line or an obstacle by the captured image of each camera;
In a multi-camera image using vehicle outside display device comprising a white line detected by the white line obstacle detection means or a white line obstacle highlighting display means for highlighting an obstacle,
A multi-camera overlapping shooting detection means for detecting an overlapping shooting portion where a plurality of cameras are shooting overlapping each other;
In the overlapping shooting part detected by the multiple camera overlapping shooting detection means, the camera shot image difference detection means for comparing the shot images of each camera and detecting the difference between the images,
An improper photographing camera specifying means for identifying which camera is outputting an improper image when it is detected by the camera photographed image difference detecting means that the images are different from each other by a predetermined amount;
An improper image range outer edge specifying means for specifying an outer edge of a predetermined range including an improper image portion from a captured image of the camera specified as an improper shooting camera by the improper shooting camera specifying means;
Conversion processing means for converting the coordinates of the improper photographing camera of the outer edge specified by the improper image range outer edge specifying means into the coordinates of the image of the appropriate photographing camera for photographing the same overlapping photographing range;
The white line obstacle image data obtained by the white line obstacle emphasis display means in the outer edge of the image of the appropriate photographing camera converted by the conversion processing means is captured, and this image data is converted into inappropriate camera photographing image data, A multi-camera image in-vehicle display device comprising data interpolation processing means for an improper photographing camera that interpolates white line obstacle image data in an improper camera photographing image range portion. The improper photographing camera specifying means identifies an improper camera after confirming which of the images of the cameras is improper after detecting that the captured images of the cameras are different from each other by a predetermined amount or more. The multi-camera image using vehicle exterior display device according to claim 4, wherein the display is performed. The conversion processing unit uses a coordinate conversion table that directly converts the fisheye coordinates of one camera created in advance into the fisheye coordinates of the other camera for the overlapping shooting ranges of a plurality of cameras. The multi-camera image use outside vehicle display apparatus as described in any one of 1-5.

Images