JP2013020308A - Vehicle surrounding image pickup system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly correct a surrounding image without requiring a high performance arithmetic processor in a vehicle surrounding image pickup system.SOLUTION: When difference γi (an inclination angle differential value) between an inclination angle αi of a camera 10 detected by a camera inclination angle sensor 20 (first detector) and an inclination angle β of a vehicle 200 detected by a vehicle inclination angle sensor 40 (second detector) changes from a preset reference value Γi (initial value, a design value, etc.), an overhead view image (surrounding image) is corrected according to variation Δγi (a shift amount), a high performance arithmetic processor is not required since arithmetic processing for calculating the difference γi is light load, and the overhead view image is properly corrected since the overhead view image is not corrected according to only change of the inclination angle αi of the camera 10 but corrected according to change of correlation γi with the inclination angle β of the vehicle 200.

Description

  The present invention relates to a vehicle surrounding image capturing system that obtains a vehicle surrounding image by synthesizing images captured by a plurality of imaging units provided in a vehicle.

  By displaying an image of the rear of the vehicle captured by the camera mounted on the vehicle on the in-vehicle monitor, the situation behind the vehicle, which is likely to be a blind spot of the driver, can be viewed as an image displayed on the in-vehicle monitor. It has been done to improve the visibility.

  In addition, images taken by a plurality of cameras directed in different directions are converted (viewpoint conversion) into, for example, an image looked down from a viewpoint virtually provided above the vehicle, and these images are converted into one image. To obtain a surrounding image of the vehicle (also referred to as an overhead image or the like).

  In such viewpoint conversion and composition processing for obtaining surrounding images, each camera is installed in the vehicle, and its posture (inclination angle) is set according to a preset design value. Necessary, and if the posture deviates from the design value, the surrounding image obtained by the viewpoint conversion and synthesis processing becomes inaccurate.

  The posture of the camera is almost the same as the designed value at the initial stage when the camera is installed in the vehicle, but it gradually changes over time due to the vibration of the vehicle, etc. And may change due to vehicle accidents.

  By the way, even if the road surface on which the vehicle travels is inclined instead of the camera, the surrounding image becomes inaccurate due to the inclination of the road surface, but a technique for adjusting the surrounding image in that case has been proposed. (Patent Document 1).

  In this technique, the inclination of the vehicle is detected by a gyro sensor attached to the vehicle, and the surrounding image is corrected according to the image recognition result of the obtained surrounding image and the inclination of the vehicle by the gyro sensor.

JP 2010-245821 A

  However, this technique corresponds to the inclination of the road surface and does not correspond to the inclination of the camera.

  In addition, since this technique requires immediate image correction in response to the inclination of the road surface on which the vehicle is traveling, a high-performance arithmetic processing device that requires a high calculation speed for adjusting the image is required.

  In addition, although the slope of the road surface on which the vehicle travels changes every moment, in order to display an image in response to this change, it is necessary to constantly perform image correction. It is required to be performance.

  The present invention has been made in view of the above circumstances, and does not require a high-performance arithmetic processing device, and can correct a surrounding image appropriately when it is necessary to correct the surrounding image due to a change in the posture of the camera. An object of the present invention is to provide a vehicle surrounding image capturing system that can be used.

  In the vehicle surrounding imaging system according to the present invention, the difference between the tilt angle of the camera detected by the first detector and the tilt angle of the vehicle detected by the second detector (inclination angle difference value) is determined in advance. When it changes from the set reference value (initial value or design value that is the initial value when the camera is installed in the vehicle), the surrounding image is corrected according to the change amount (deviation amount), Since the arithmetic processing for calculating the inclination angle difference value is lightly loaded, a high-performance arithmetic processing device is not required.

  In addition, the peripheral image is not corrected only according to the change in the camera tilt angle, but is corrected according to the change in the relative relationship (difference) with the vehicle tilt angle. It is to correct.

  That is, the vehicle surrounding image capturing system according to the present invention generates a single surrounding image by combining a plurality of cameras attached to a vehicle and a plurality of images obtained by the plurality of cameras by performing viewpoint conversion. Detected by the processing unit, a first detector for detecting an angle of inclination of each of the plurality of cameras, a second detector for detecting an angle of inclination of the vehicle, and the first detector An inclination angle difference value, which is a difference between a camera inclination angle and a vehicle inclination angle detected by the second detector, is obtained, and the obtained inclination angle difference value is preset with respect to the inclination angle difference value. And a control unit that changes viewpoint conversion and synthesis processing by the image processing unit in accordance with a shift amount that is a difference with respect to a reference value.

  According to the vehicle surrounding image capturing system according to the present invention, a high-performance arithmetic processing device is not required, and the surrounding image is corrected appropriately when it is necessary to correct the surrounding image due to a change in posture of the in-vehicle camera. be able to.

1 is a block diagram illustrating a vehicle surrounding image capturing system according to an embodiment of the present invention. It is a table (table | surface) which shows the correspondence of deviation | shift amount and mapping data memorize | stored in flash memory. It is a flowchart which shows the effect | action of the vehicle surrounding image imaging system of embodiment shown in FIG.

  Embodiments of a vehicle surrounding image capturing system according to the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a vehicle surrounding image capturing system 100 according to an embodiment of the present invention.

  The illustrated vehicle surrounding image capturing system 100 includes four cameras 10 that capture a region around the vehicle 200 including a part of the vehicle 200 (the front camera 11 that captures the front of the vehicle 200 and the rear of the vehicle 200. The rear camera 12, the right camera 13 for photographing the right side of the vehicle 200, the left camera 14 for photographing the left side of the vehicle 200, and the inclinations of the cameras 11 to 14 mounted on the cameras 11 to 14, respectively. Camera angle sensor 20 (an example of a first detector: camera angle sensor 21 for front camera 11, camera angle sensor 22 for rear camera 12, camera for right camera 13) The tilt angle sensor 23, the camera tilt angle sensor 24 for the left camera 14, and the images captured by the respective cameras 11 to 14 are converted into viewpoints (for example, right above the vehicle 200). A viewpoint conversion module 30 (an example of an image processing unit) that generates a single bird's-eye view image (an example of a surrounding image) that is synthesized by conversion to an image at a viewpoint looking down from the viewpoint, and an inclination angle β of the vehicle 200 is detected. Detected by the vehicle inclination angle sensor 40 (an example of a second detector) and the corresponding camera inclination angles α1 to α4 detected by the camera inclination angle sensors 21 to 24 and the vehicle inclination angle sensor 40. The inclination angle difference values γ1 to γ4 (γ1 = α1-β, γ2 = α2-β, γ3 = α3-β, γ4 = α4-β), which are the differences from the inclination angle β of the vehicle 200, are obtained and obtained. Deviation amounts Δγ1 to Δγ4 (Δγ1 = γ1-Γ1, Δγ2 = γ2-Γ2), which are the differences between the obtained inclination angle difference values γ1 to γ4 and the reference values Γ1 to Γ4 set in advance for the inclination angle difference values γ1 to γ4, respectively. , Δγ3 = γ3-Γ3, Δγ4 = 4-Γ4), a display control module 50 (an example of a control unit) that changes the process of viewpoint conversion and synthesis by the viewpoint conversion module 30, a display device 90 (an example of a display unit) that displays an image, and a display This is a configuration including a video display module 60 (an example of a configuration for realizing a partial function (a function for displaying a warning) of a control unit) that controls display on the device 90, and a flash memory 70 that stores various data. .

  Here, the front camera 11 is installed at the front of the vehicle 200, the rear camera 12 is installed at the rear of the vehicle 200, the right camera 13 is installed at the right side of the vehicle 200, and the left camera 14 is installed at the left side of the vehicle 200. Is installed.

  The viewpoint conversion module 30, the vehicle tilt angle sensor 40, the display control module 50, the video display module 60, and the flash memory 70, as a single camera electronic control unit 80, are places that are not easily noticeable by passengers in the vehicle interior of the vehicle 200. Is installed.

  On the other hand, the display device 90 is set separately from the camera electronic control unit 80 and in a place (for example, an instrument panel, a dashboard, or the like) that is easily visible to the occupant in the passenger compartment.

  As each of the camera tilt angle sensors 21 to 24 and the vehicle tilt angle sensor 40, for example, a gyro sensor can be applied. In the vehicle surrounding image capturing system of the present invention, the first detector and the second detector are used. As long as the sensor has a function of detecting the inclination angle, any sensor other than the gyro sensor may be applied.

  The inclination angle β of the vehicle 200 detected by the vehicle inclination angle sensor 40 in the camera electronic control unit 80 is the installation attitude of the vehicle inclination angle sensor 40 in the camera electronic control unit 80 and the installation attitude of the camera electronic control unit 80 with respect to the vehicle 200. Are combined.

  On the other hand, the inclination angles α1 to α4 of the cameras 11 to 14 detected by the camera inclination angle sensors 21 to 24 installed in the cameras 11 to 14 are the installation positions of the camera inclination angle sensors 21 to 24 corresponding to the corresponding cameras 11 to 14, respectively. The posture and the installation posture of the cameras 11 to 14 with respect to the vehicle 200 are combined.

  Here, the display control module 50 performs the above-described viewpoint conversion and single image composition processing by conversion using a preset mapping table. This mapping table is stored in the flash memory 70. It is stored as mapping data 71.

  Normally, the mapping table stored in the flash memory 70 as the mapping data 71 is a tilt angle (design value or initial value when each camera 11 to 14 is installed in the vehicle 200). Only one of the standards corresponding to the initial value which is a large tilt angle.

  However, the vehicle surrounding image capturing system 100 according to the present embodiment stores a plurality of different mapping data 71 in the flash memory 70 in addition to the standard mapping data 71 (mapping data 71g described later).

  Since the plurality of mapping data 71 are set to be different from each other, the viewpoint conversion and synthesis by the viewpoint conversion module 30 can be performed by replacing the mapping data 71 applied to the viewpoint conversion and synthesis processing of the viewpoint conversion module 30. The contents of processing can be changed.

  The mapping data 71 other than the standard mapping data 71g is used when one or more of the cameras 11 to 14 are deviated from the reference inclination angle described above (strictly speaking, the cameras 11 to 14 are relative to the vehicle 200). Correct inclination angle (when the inclination angle difference value γi (i = 1 to 4) is deviated from the reference value Γi (i = 1 to 4)), the deviation is corrected and there is no deviation (inclination angle difference value). This is a mapping table for conversion to an overhead image that would be obtained if γi is a reference value Γi).

  Then, one of the plurality of mapping data 71 is selected by the display control module 50, read from the flash memory 70, sent to the viewpoint conversion module 30, and the viewpoint conversion module 30 receives the sent 1 Based on one mapping data 71, viewpoint conversion and composition processing are performed.

  The selection of the specific mapping data 71 by the display control module 50 includes the inclination angles α1 to α4 of the cameras 11 to 14 detected by the camera inclination angle sensors 21 to 24 and the inclination of the vehicle 200 detected by the vehicle inclination angle sensor 40. The inclination angle difference values γ1 to γ4, which are differences from the angle β, and the deviation amounts Δγ1 to Δγ4, which are differences between the inclination angle difference values γ1 to γ4 and the reference values Γ1 to Γ4 set in advance. .

  Here, since the tilt angles α1 to α4 are tilt angles of the cameras 11 to 14, even if the cameras 11 to 14 themselves are simply tilted, the tilt angles α1 to α4 naturally change. Even when the cameras 11 to 14 themselves are not tilted with respect to the vehicle 200, for example, when the vehicle 200 is on a slope, the vehicle 200 itself tilts and the tilt angles α1 to α4 also change.

  Therefore, even if the inclination angles α1 to α4 change from the reference values such as initial values and design values at the time of manufacture, the installation state of the cameras 11 to 14 with respect to the vehicle 200 does not necessarily change.

  On the other hand, since the inclination angle difference values γ1 to γ4 are relative inclination angles of the cameras 11 to 14 with respect to the vehicle 200, the same value is obtained whether the vehicle 200 is on a flat ground or on a slope. Is shown.

  On the other hand, the inclination angle difference values γ1 to γ4 are the reference values Γ1 to Γ4 set for the inclination angle difference values γ1 to γ4 (the inclination angle when the cameras 11 to 14 and the camera electronic control unit 80 are installed in the vehicle 200). When the difference value is not equal to the initial value of the difference value or the design value of the inclination angle difference value (when the deviation amounts Δγ1 to Δγ4 are no longer zero), the camera 11 whose deviation amount Δγ is no longer zero. About -14, the display control module 50 can determine that the inclination angle with respect to the vehicle 200 has changed.

  The display control module 50 is configured to select one mapping data 71 corresponding to the magnitudes of the shift amounts Δγ1 to Δγ4 from the plurality of mapping data 71 stored in the flash memory 70. Yes.

  Here, in order to specify one mapping data 71 corresponding to the magnitude of the deviation amount Δγi, one mapping data 71 may be stored in association with each value of the deviation amount Δγi. The amount of mapping data to be stored becomes enormous.

  Therefore, in the vehicle surrounding image capturing system 100 of the present embodiment, the mapping data 71 is grouped for each range obtained by dividing the deviation amount Δγi for each predetermined numerical range, and one mapping table is associated with each range. The number of mapping data 71 stored in the flash memory 70 is reduced, and the storage capacity of the flash memory 70 is saved.

Specifically, as shown in the list of FIG.
(1) When the deviation amount Δγi is in the range of −0.5 to +0.5 [deg], standard mapping data 71g with no change in the posture of the camera 10 (mapping data when the deviation amount Δγi is 0) Are associated.
(2) When the shift amount Δγi is in the range of −1.0 to −0.5 [deg], the image in the state where the posture of the camera 10 is shifted from the standard by the tilt angle −0.5 [deg] is obtained. On the other hand, mapping data 71f for obtaining a bird's-eye view image is associated.
(3) When the shift amount Δγi is in the range of +0.5 to +1.0 [deg], the camera 10 has a bird's-eye view with respect to an image in a state where the posture is shifted by an inclination angle +0.5 [deg] from the standard. Mapping data 71h for obtaining an image is associated.
(4) When the shift amount Δγi is in the range of −1.5 to −1.0 [deg], the image in a state where the posture of the camera 10 is shifted by the tilt angle −1.0 [deg] from the standard is obtained. On the other hand, mapping data 71e for obtaining an overhead image is associated.
(5) When the shift amount Δγi is in the range of +1.0 to +1.5 [deg], the camera 10 has a bird's-eye view with respect to an image in a state where the posture is shifted by an inclination angle +1.0 [deg] from the standard. Mapping data 71i for obtaining an image is associated.
(6) When the shift amount Δγi is in the range of −2.0 to −1.5 [deg], the image in the state where the posture of the camera 10 is shifted from the standard by the tilt angle −1.5 [deg] is obtained. On the other hand, mapping data 71d for obtaining a bird's-eye view image is associated.
(7) When the shift amount Δγi is in the range of +1.5 to +2.0 [deg], an overhead view is obtained with respect to an image in a state where the posture of the camera 10 is shifted by an inclination angle +1.5 [deg] from the standard. Mapping data 71j for obtaining an image is associated.
(8) When the shift amount Δγi is in the range of −2.5 to −2.0 [deg], the image of the camera 10 is shifted by an inclination angle −2.0 [deg] from the standard. On the other hand, mapping data 71c for obtaining a bird's-eye view image is associated.
(9) When the shift amount Δγi is in the range of +2.0 to +2.5 [deg], the camera 10 has a bird's-eye view with respect to an image in a state where the posture is shifted by an inclination angle +2.0 [deg] from the standard. Mapping data 71k for obtaining an image is associated.
(10) When the shift amount Δγi is in the range of −3.0 to −2.5 [deg], the image in the state where the posture of the camera 10 is shifted from the standard by the tilt angle −2.5 [deg] is obtained. On the other hand, mapping data 71b for obtaining a bird's-eye view image is associated.
(11) When the shift amount Δγi is in the range of +2.5 to +3.0 [deg], the camera 10 has a bird's-eye view with respect to an image in a state where the posture is shifted by an inclination angle +2.5 [deg] from the standard. Mapping data 71l for obtaining an image is associated.
(12) When the shift amount Δγi is in the range of −3.0 [deg] or less, the bird's-eye view image with respect to the image in a state where the posture of the camera 10 is shifted from the standard by the tilt angle −3.0 [deg]. Mapping data 71a for obtaining the data is associated.
(13) When the shift amount Δγi is in a range of +3.0 [deg] or more, an overhead image is obtained with respect to an image in a state where the posture of the camera 10 is shifted by an inclination angle +3.0 [deg] from the standard. Mapping data 71m for this purpose is associated.

  The mapping data 71 is set to convert four images obtained from the four cameras 11 to 14 into a single overhead view image. If Δγ is not zero (in the correspondence table shown in FIG. 2, the shift amount Δγ1 is outside the range of −0.5 to 0.5), and there is one camera 10, other than the standard mapping data 71g Mapping data 71 is applied, and in the correspondence table shown in FIG. 2, it is described that one mapping data 71 is simply associated with the shift amount Δγ for one camera 10. Actually, one mapping data 71 is associated with each deviation amount Δγ1 to Δγ4 of each camera 11-14.

  In addition, the inclination angles α1 to α4 of the cameras 11 to 14 are respectively defined by three orthogonal axes (vertical axes extending in the vertical direction (z-axis) in order to uniquely specify the inclination angles in the three-dimensional space. , Yaw axis), angle of inclination (roll angle) with respect to the longitudinal axis (x-axis, roll axis) extending horizontally along the vehicle longitudinal direction, horizontal along the vehicle lateral direction Is defined as an angle of inclination (pitch angle) with respect to the left and right axes (y-axis, pitch axis) extending in the direction of, the inclination angle α1 of the camera 11 is an inclination angle αx1 with respect to the x-axis and an inclination angle with respect to the y-axis. The tilt angle α2 of the camera 12 is defined by αy1 and the tilt angle αz2 with respect to the z axis. The tilt angle α2 of the camera 12 is set to the tilt angle αx2 with respect to the x axis, the tilt angle αy2 with respect to the y axis, and the tilt angle αz2 with respect to the z axis. The tilt angle α3 of the camera 13 is defined by the tilt angle αx3 with respect to the x-axis, the tilt angle αy3 with respect to the y-axis, and the tilt angle αz3 with respect to the z-axis, and the tilt angle α4 of the camera 14 is An inclination angle αx4 with respect to the x axis, an inclination angle αy4 with respect to the y axis, and an inclination angle αz4 with respect to the z axis are defined.

  Accordingly, inclination angle difference values γ1 to γ4, reference values Γ1 to Γ4, and deviation amounts Δγ1 to Δγ4 are also defined for each of the three axes.

  That is, of the tilt angle difference value γ1 of the camera 11 with respect to the vehicle 200, the tilt angle difference value γx1 with respect to the x axis, the tilt angle difference value γy1 with respect to the y axis, the tilt angle difference value γz1 with respect to the z axis, and the reference value Γ1 with respect to the x axis. Of the reference value Γx1, the reference value Γy1 for the y axis, the reference value Γz1 for the z axis, and the deviation amount Δγ1, the deviation amount Δγx1 for the x axis, the deviation amount Δγy1 for the y axis, and the deviation amount Δγz1 for the z axis are defined. Of the tilt angle difference values γ2 of the camera 12, the tilt angle difference value γx2 with respect to the x axis, the tilt angle difference value γy2 with respect to the y axis, the tilt angle difference value γz2 with respect to the z axis, and the reference value Γx2, y with respect to the x axis of the reference value Γ2. A reference value Γy2 for the axis, a reference value Γz2 for the z-axis, a deviation amount Δγx2 for the x-axis out of the deviation amount Δγ2, and a deviation for the y-axis Δγy2, a deviation amount Δγz2 with respect to the z axis is defined, and among the tilt angle difference values γ3 of the camera 13 with respect to the vehicle 200, an inclination angle difference value γx3 with respect to the x axis, an inclination angle difference value γy3 with respect to the y axis, and an inclination angle difference value with respect to the z axis Of γz3, reference value Γ3, reference value Γx3 for the x axis, reference value Γy3 for the y axis, reference value Γz3 for the z axis, of the deviation amount Δγ3, deviation amount Δγx3 for the x axis, deviation amount Δγy3 for the y axis, for the z axis A deviation amount Δγz3 is defined, among the tilt angle difference values γ4 of the camera 14 with respect to the vehicle 200, the tilt angle difference value γx4 with respect to the x axis, the tilt angle difference value γy4 with respect to the y axis, the tilt angle difference value γz4 with respect to the z axis, and the reference value Γ4. Of the reference value Γx4 for the x axis, the reference value Γy4 for the y axis, the reference value Γz4 for the z axis, and the deviation amount Δγ4 to the x axis. The deviation amount Δγx4 to be performed, the deviation amount Δγy4 with respect to the y axis, and the deviation amount Δγz4 with respect to the z axis are defined.

  As a result, in the correspondence table shown in FIG. 2, it is assumed that one mapping data 71 is simply associated with the shift amount Δγ about one axis (for example, pitch axis) for one camera 10. Although described, actually, the amount of deviation Δγx1, Δγy1, Δγz1, Δγx2, Δγy2, Δγz2, Δγx3, Δγy3, Δγz3, Δγx4, Δγy4, and Δγz4 with respect to each axis of each camera 11 to 14 is mapped. Data 71 is associated one by one.

  In addition to the reference values Γ1 (Γx1, Γy1, Γz1), Γ2 (Γx2, Γy2, Γz2), Γ3 (Γx3, Γy3, Γz3), Γ4 (Γx4, Γy4, Γz4), the inclinations of the respective cameras 11 to 14 Reference values for the angles α1 to α4 (initial values of the inclination angles α1 to α4 when the cameras 11 to 14 are installed in the vehicle 200, design values of the inclination angles α1 to α4, etc. Angle reference values Α1, Α2, Α3, and Α4) and a reference value for the inclination angle β of the vehicle 200 (the initial value of the inclination angle β when the camera electronic control unit 80 is installed in the vehicle 200, or The design value of the inclination angle β, etc., hereinafter referred to as a vehicle inclination angle reference value）) is also stored in the flash memory 70.

  In the present embodiment, in order to avoid complicated explanation, it is assumed that one mapping data 71 is associated with one camera 10 with respect to a deviation amount Δγ about one axis. As described above, the mapping data for each combination of the deviation amounts Δγx1, Δγy1, Δγz1, Δγx2, Δγy2, Δγz2, Δγx3, Δγy3, Δγz3, Δγx4, Δγy4, and Δγz4 with respect to the respective axes of the cameras 11 to 14 is implemented. 71 is assumed to be associated one by one, and one of them may be selected.

  The OSD data 72 stored in the flash memory 70 is visible information representing a warning superimposed on the overhead image on the display device 90 by the on-screen display function of the video display module 60. For example, character information such as “there is a deviation in the camera mounting angle” can be applied.

  This warning is displayed, for example, when the deviation amount Δγi shown in FIG. 2 is −3.0 or less or 3.0 or more.

  That is, when the detected deviation amount Δγi exceeds 3.0, which is the upper limit value of the preset deviation amount Δγi (the threshold, the maximum value assumed as the deviation amount), the display control module 50 The mapping data 71m corresponding to the deviation amount Δγi = 3.0 of the upper limit value is selected, and the viewpoint conversion module 30 performs viewpoint conversion on the four images obtained by the cameras 11 to 14 according to the selected mapping data 71m. The image display module 60 reads out the OSD data 72 stored in the flash memory 70 and superimposes the OSD data 72 on the overhead image obtained by the viewpoint conversion module 30. The display device 90 outputs the display.

  Similarly, when the detected deviation amount Δγi falls below −3.0 which is a preset lower limit value (maximum value (as an absolute value) assumed as the threshold, deviation amount) of the deviation amount Δγi, The display control module 50 selects the mapping data 71a corresponding to the shift amount Δγi = −3.0 of the lower limit value, and the viewpoint conversion module 30 obtains each camera 11-14 according to the selected mapping data 71a. The four images thus obtained are subjected to viewpoint conversion and composition processing to obtain a bird's-eye view image, and the video display module 60 reads out the OSD data 72 stored in the flash memory 70 to obtain the bird's-eye view image obtained by the viewpoint conversion module 30. The OSD data 72 is superimposed and displayed on the display device 90.

  When the detected deviation amount Δγi is a value in a range between −3.0 which is a lower limit value of preset deviation amount Δγi and 3.0 which is an upper limit value, the display control module 50 detects the deviation amount Δγi. The mapping data 71 (any one of the mapping data 71b to 71l) corresponding to the range (the range described in the top row of the table in FIG. 2) corresponding to the value of the shift amount Δγi is selected, and the viewpoint is converted. The module 30 obtains a bird's-eye view image by performing viewpoint transformation and synthesis processing on the four images obtained by the cameras 11 to 14 according to the selected mapping data 71, and the video display module 60 is obtained by the viewpoint transformation module 30. The overhead image is displayed on the display device 90.

  Next, the operation of the vehicle surrounding image capturing system 100 of this embodiment will be described according to the flowchart shown in FIG.

  First, from each camera 11-14, the image image | photographed with each camera 11-14 is input into the viewpoint conversion module 30 (S1).

  That is, an image of the front of the vehicle photographed by the front camera 11, an image of the rear of the vehicle photographed by the rear camera 12, an image of the right side of the vehicle photographed by the right camera 13, and a vehicle photographed by the left camera 14. The image on the left side is input to the viewpoint conversion module 30 of the camera electronic control unit 80.

  On the other hand, the camera inclination angle sensors 21 to 24 attached to the cameras 11 to 14 detect the inclination angles α1 to α4 of the corresponding cameras 11 to 14, respectively, and these inclination angles α1 to α4 are electronically controlled by the camera. The data is input to the display control module 50 of the unit 80 (S2).

  Here, each camera tilt angle sensor 21 to 24 detects three tilt angles (yaw angle, roll angle, and pitch angle) for each camera 11 to 14, but in the following description, as described above. In order to simplify the description, the inclination angles of the cameras 11 to 14 are represented by single inclination angles α1 to α4, respectively.

  When the camera tilt angle sensors 21 to 24 detect the tilt angles α1 to α4 of the cameras 11 to 14 and input the results to the display control module 50, images taken by the cameras 11 to 14 are input to the viewpoint conversion module 30. Although it is preferable to synchronize with the input timing, it does not necessarily have to synchronize.

  Further, the vehicle inclination angle sensor 40 of the camera electronic control unit 80 detects the inclination angle β of the vehicle 200, and this inclination angle β is also input to the display control module 50 (S3). The input timing of the tilt angle β to the display control module 50 is preferably synchronized with the input timing of the tilt angles α1 to α4 to the display control module 50, but it is not always necessary.

  The display control module 50 reads the camera tilt angle reference values Α1 to Α4 from the flash memory 70, and calculates the difference Δα1 between the input camera tilt angles α1 to α4 and the camera tilt angle reference values Α1 to Α4 ( = Α1-Α1), Δα2 (= α2-Α2), Δα3 (= α3-Α3), Δα4 (= α4-Α4) are calculated (S4).

  The display control module 50 reads the vehicle inclination angle reference value Β from the flash memory 70, and calculates the difference Δβ between the input vehicle inclination angle β and the vehicle inclination angle reference value （(S5).

  Next, the display control module 50 compares the differences Δα1, Δα2, Δα3, Δα4 with the difference Δβ (S6).

  Here, when the differences Δα1 to Δα4 and the difference Δβ for all the cameras 11 to 14 are all the same value (in the case of No in S6), all the cameras 11 to 14 have an abnormal inclination angle in the special state. In this case, the display control module 50 selects the standard mapping data 71g from the mapping data 71 stored in the flash memory 70 (S7), and uses the selected mapping data 71g. Input to the viewpoint conversion module 30.

  The viewpoint conversion module 30 performs a viewpoint conversion process and a synthesis process on the images input from the cameras 11 to 14 according to the input mapping data 71g to generate a single overhead image (S8). The generated overhead image is input to the video display module 60.

  The video display module 60 performs encoding corresponding to the display device 90 on the overhead image so that the input overhead image is displayed as a visible image on the display device 90, and causes the display device 90 to display the overhead image. (S9).

  On the other hand, in the magnitude comparison between the differences Δα1, Δα2, Δα3, Δα4 and the difference Δβ in step 6 (S6), the differences Δα1, Δα2, Δα3, Δα4 for any one of the cameras 11 to 14 are different from the difference Δβ for the vehicle 200. When they do not match (Yes in S6), the display control module 50 calculates inclination angle difference values γ1 to γ4 for the cameras 11 to 14 (S10).

  Further, the display control module 50 calculates the shift amounts Δγ1 to Δγ4 for the cameras 11 to 14 (S11).

  Then, the display control module 50 selects one mapping data 71 corresponding to the combination of the shift amounts Δγ1 to Δγ4 for each camera 11 to 14 from the correspondence table shown in FIG. 2 (S12), and the selected mapping Data 71 is input to the viewpoint conversion module 30.

  Here, in the table shown in FIG. 2, one mapping data 71 is described corresponding to one deviation amount Δγ, but as described above, actually, the four deviation amounts Δγ1 to Δγ1. Since the deviation amounts Δγxi, Δγyi, Δγzi around three axes (yaw axis, roll axis, pitch axis) are set for the combination of Δγ4 and each deviation amount Δγi, strictly speaking, the 12 deviation amounts Δγx1, Δγy1 , Δγz1, Δγx2, Δγy2, Δγz2, Δγx3, Δγy3, Δγz3, Δγx4, Δγy4, and Δγz4, one mapping data 71 is set.

  The display control module 50 also determines that the calculated deviation amount Δγi exceeds the assumed upper limit of 3.0 or falls below the assumed lower limit of −3.0 (Yes in S13). In the case), a warning flag indicating that fact is set (S14), the warning flag is set to the selected mapping data (the mapping data selected when this warning flag is set is the mapping data 71a or the mapping data 71m). And input to the viewpoint conversion module 30.

  When the calculated shift amount Δγi is between the assumed upper limit value 3.0 and lower limit value −3.0 (No in S13), the selected mapping data 71 is input to the viewpoint conversion module 30. To do.

  In accordance with the input mapping data 71, the viewpoint conversion module 30 performs a viewpoint conversion process and a synthesis process on the images input from the cameras 11 to 14 to generate a single overhead image (S8). The generated overhead image is input to the video display module 60.

  When the warning flag is set (S14), the warning flag is also input to the video display module 60.

  The video display module 60 performs encoding corresponding to the display device 90 on the overhead image so that the input overhead image is displayed as a visible image on the display device 90, and causes the display device 90 to display the overhead image. (S9).

  Here, when the warning flag is set (in the case of Yes in S15), the video display module 60 reads the OSD data 72 stored in the flash memory 70 and is obtained from the viewpoint conversion module 30. The OSD data 72 is superimposed on the image and displayed on the display device 90 (S16).

  When the warning flag is not set, the process is terminated without causing the display device 90 to superimpose and display the OSD data 72.

  As described above in detail, according to the vehicle surrounding image capturing system 100 according to the present embodiment, the inclinations of the cameras 11 to 14 detected by the camera tilt angle sensors 21 to 24 provided in the cameras 11 to 14 are determined. Inclination angle difference values γ1 to γ4, which are differences between the angles α1 to α4 and the inclination angle β of the vehicle 200 detected by the vehicle inclination angle sensor 40 provided on the vehicle 200, are calculated, and the reference values Γ1 to Γ4 are calculated. This is a simple calculation by simply selecting one of the preset mapping data 71 in accordance with the deviations Δγ1 to Δγ4, which is the difference between the two, and is caused by the posture change (change in the tilt angle) of the cameras 11-14. A bird's-eye view image with the error corrected can be obtained.

  Therefore, since it is not necessary to perform advanced arithmetic processing as in the prior art, a high-performance arithmetic processing device is not required.

  In addition, generation of the overhead view image is performed by conversion using preset mapping data 71, and it is possible to reduce the load of calculation processing for obtaining the overhead view image.

  Further, according to the vehicle surrounding image capturing system 100 of the present embodiment, it is determined whether or not it is necessary to correct the overhead image based only on the changes in the postures of the cameras 11 to 14 (tilt angles α1 to α4). Since the necessity of correction is determined based on the magnitude of the relative relationship with the inclination angle β of the vehicle 200, there is a problem with the installation posture of the cameras 11 to 14 with respect to the vehicle 200. In spite of the fact that the vehicle 200 is running on a slope, for example, there is no longer erroneous determination such as a change in the installation posture of the cameras 11 to 14, and it is truly an overhead image correction When it is necessary, the overhead view image can be corrected appropriately.

Further, when the deviation amount Δγi exceeds an upper limit (3.0) that is a preset threshold value, the mapping corresponding to the maximum value 3.0 assumed as the deviation amount Δγi among the plurality of mapping data 71. When the data 71m is selected, or when the deviation amount Δγi falls below a lower limit (−3.0) that is a preset threshold, the minimum amount assumed as the deviation amount Δγi among the plurality of mapping data 71 When the mapping data 71a corresponding to the value −3.0 is selected, the OSD data that is visible information indicating a warning is displayed on the overhead image generated based on the selected mapping data 71m or 71a and displayed on the display device 90. Since 72 is also displayed, it is visually displayed on the display device 90 that the installation posture of the cameras 11 to 14 is not large from the reference value. The SD data 72 allows the user of the vehicle 200 to visually recognize, thereby prompting early inspection and repair of the installation state of the cameras 11 to 14.

  The display device 90 has an on-screen display function, and the visible information representing the warning described above is displayed on the image data by being superimposed on the overhead image by the on-screen display function. Thus, it is not necessary to perform the superimposing process of FIG.

  In the vehicle surrounding image capturing system 100 according to the present embodiment, the timing of performing the correction in Step 2 (S2) to Step 16 (S16) in the above-described processing is when there is a physical shift in the camera tilt angle αi. (For example, when the camera 10 physically changes its inclination angle αi due to contact or the like) or when the inclination angle of the vehicle 200 seems to change relatively in relation to the inclination angle β of the vehicle 200. The timing is set as follows assuming that the road on which the vehicle 200 travels is inclined and the image displayed on the display device 90 appears to be shifted.

That is, it is set to be performed at at least one of the following (1) to (3).
(1) When the power to the camera electronic control unit 80 is turned on (2) During low-speed driving, the tilt angle αi of the camera 10 is detected at a predetermined cycle, and the tilt angle αi does not change within a predetermined time. When (3) When the vehicle stops, the inclination angle αi of the camera 10 is detected at a predetermined cycle, and the inclination angle αi does not change within a predetermined time. And, the vehicle surrounding image capturing system 100 of the present embodiment Since the correction is performed only at the timing described above, the processing load can be reduced as compared with the prior art that performs the process for the constant correction.

Example 1
In the embodiment described above, the camera tilt angle reference value Αi is, for example, 20 [degrees], the vehicle tilt angle reference value 例 え ば is, for example, 5 [degrees], and the detected tilt angle αi of the camera 10 is, for example, 25 [degrees]. When the inclination angle β of the vehicle 200 is, for example, 10 [degrees], the difference Δα for the camera 10 and the difference Δβ for the vehicle 200 are both 5 [degrees]. In step 6 (S6), it is determined that there is no difference, standard mapping data 71g is selected (S7), and an overhead image defined by the mapping data 71g is displayed (S9).

  At this time, there is no superimposed display (S16) of the OSD data 72.

(Example 2)
In the embodiment described above, the camera tilt angle reference value Αi is, for example, 20 [degrees], the vehicle tilt angle reference value Β is, for example, 5 [degrees], and the detected tilt angle αi of the camera 10 is, for example, 26 [degrees]. When the inclination angle β of the vehicle 200 is, for example, 10 [degrees], the difference Δα for the camera 10 is 6 [degrees], and the difference Δβ for the vehicle 200 is 5 [degrees]. In step 6 (S6), it is determined that there is a difference. In step 7 (S7), an inclination angle difference value γi is calculated, and 16 [degrees] is obtained as a calculation result (S10).

  Next, the deviation amount Δγi is calculated. Since the reference value Γi is 15 [degrees] which is the difference between the camera inclination angle reference value Αi and the vehicle inclination angle reference value Β, the deviation amount Δγi is calculated as 1 [degree]. (S11).

  As a result, the mapping data 71i corresponding to the shift amount Δγi of 1 [degree] is selected as the mapping data 71 according to the table shown in FIG. 2 (S12), and the overhead image defined by the mapping data 71i is determined. It is displayed (S9).

  At this time, there is no superimposed display (S16) of the OSD data 72.

10, 11, 12, 13, 14 Camera 20, 21, 22, 23, 24 Camera tilt angle sensor 30 View point conversion module 40 Vehicle tilt angle sensor 50 Display control module 60 Video display module 70 Flash memory 71, 71a to 71m Mapping data 72 OSD data 80 Camera electronic control unit 90 Display device 100 Vehicle surrounding imaging system 200 Vehicle

Claims (5)

A plurality of cameras attached to the vehicle;
An image processing unit that generates a single surrounding image obtained by performing viewpoint conversion on a plurality of images obtained by the plurality of cameras; and
A first detector for detecting an angle of inclination of each of the plurality of cameras;
A second detector for detecting an angle of inclination of the vehicle;
An inclination angle difference value, which is a difference between the camera inclination angle detected by the first detector and the vehicle inclination angle detected by the second detector, is obtained, and the obtained inclination angle difference value is obtained. A vehicle periphery comprising: a control unit that changes viewpoint conversion and synthesis processing by the image processing unit in accordance with a deviation amount that is a difference with respect to a reference value set in advance with respect to the inclination angle difference value Imaging system. The image processing unit performs the viewpoint conversion and composition processing by conversion using a mapping table,
As the mapping table, a plurality of mapping tables different from each other corresponding to the amount of deviation,
The control unit selects the mapping table used for the viewpoint conversion and composition processing by the image processing unit by selecting a mapping table corresponding to the shift amount from the plurality of mapping tables, thereby performing the viewpoint conversion and The vehicle surrounding image capturing system according to claim 1, wherein the composition processing is changed.   The vehicle according to claim 2, wherein the mapping table is set corresponding to each range of the deviation amount, and the control unit selects a mapping table corresponding to the range of the deviation amount. Ambient imaging system. A display unit for displaying the surrounding image generated by the image processing unit;
When the deviation amount exceeds a preset threshold value, the control unit selects a mapping table corresponding to the maximum value assumed as the deviation amount from the plurality of mapping tables and the selected one. The vehicle surrounding image imaging system according to claim 2, wherein visible information representing a warning is displayed together with the surrounding image generated based on the mapping table and displayed on the display unit.   The said display part has an on-screen display (On-Screen Display) function, The visual information showing the said warning is superimposed and displayed on the said surrounding image by the said on-screen display function, The display is characterized by the above-mentioned. The vehicle surrounding image capturing system described in 1.