JP2011013929A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of improving the quality of a composited image.SOLUTION: Cameras CM_0 to CM_3 attached to a vehicle 100 obliquely capture a road surface by attitudes that partially have common vidual fields CVW_0 to CVM_3. A CPU converts field images P_0 to P_3, outputted from these cameras CM_0 to CM_3, into bird's-eye images BEV_0 to BEV_3 expressing the states of viewing the load surface from upper positions. Furthermore, the CPU detects positions of images expressing solid objects existing in the common visual field CVW_0 to CVM_3, as excluding the positions from respective bird's-eye images BEV_0 to BEV_3 and excludes the images on the excluding positions from the bird's-eye images BEV_0 to BEV_3, in common. Furthermore, the CPU combines the bird's-eye images BEV_0 to BEV_3, in a mode for suppressing the differences in luminances and/or chromaticities among a plurality of images left after the excluding processing. According to the present invention, quality of the combined bird's-eye image is improved.

Description

  The present invention relates to an image processing apparatus, and more particularly, to an image processing apparatus that converts a scene image output from each of a plurality of cameras that capture a reference plane obliquely with a posture partially having a common visual field into a bird's-eye view image. .

  An example of this type of device is disclosed in Patent Document 1. According to this background art, the luminance values of two camera images partially having a common visual field are compressed by the compression unit. Further, the luminance difference of the partial images representing the common visual field is obtained by the luminance difference analysis unit. The shift unit shifts the luminance value of at least one of the two camera images of interest so that the luminance difference obtained by the luminance difference analysis unit is eliminated. The shifted luminance value is expanded by the expansion unit so that the luminance value of all pixels falls within the absolute white level and the absolute black level. As a result, the difference in luminance level in the common visual field is suppressed, and the corrected luminance level is within the range from the absolute white level to the absolute black level.

JP 2007-72750 A

  However, in correcting the luminance level, it is not considered whether the three-dimensional object exists in the common visual field. For this reason, in the background art, the quality of the composite image is limited.

  Therefore, a main object of the present invention is to provide an image processing apparatus capable of improving the quality of a composite image.

  An image processing apparatus according to the present invention (10: reference numeral corresponding to the embodiment; the same applies hereinafter) includes a plurality of images output from a plurality of cameras (CM_0 to CM_3) that capture the reference plane obliquely with a posture partially having a common visual field. Converting means (S3) for converting the object scene image into a plurality of bird's-eye images relative to the reference plane, the position of the image representing the three-dimensional object existing in the common visual field excluded from each of the plurality of bird's-eye images converted by the converting means First detection means (S23 to S25, S39 to S41) for detecting as, a removal means (S27, S33) for commonly removing images at exclusion positions from a plurality of bird's-eye images converted by the conversion means, and a conversion means Combining means (S7 to S11) for synthesizing the converted plurality of bird's-eye view images in such a manner that the difference in luminance and / or chromaticity between the plurality of images remaining after the removing process of the removing means is suppressed.

  Preferably, the exclusion position corresponds to a pixel position where a luminance difference and / or a chromaticity difference between the plurality of bird's-eye images exceeds the first reference.

  Preferably, a second detection means (S23, S31, S39 to S41) for detecting the position of the sharp image corresponding to the common visual field as the exclusion position is further provided.

  More preferably, it further comprises adjusting means (S29) for adjusting the second standard related to the sharpness of the image with reference to the luminance of the object scene image output from each of the plurality of cameras, and the second detecting means. Performs the detection process with reference to the second standard.

  Preferably, the synthesis means detects detection means (S53 to S59, S65, S81 ... S87, S93, S111 to S121, S141 to S151), and adjusting means (S69 to S79, S97 to S107, S123) for adjusting the luminance and / or chromaticity of a plurality of bird's-eye images based on the differences detected by the detecting means ~ S139, S153 ~ S169).

  More preferably, the adjustment means includes definition means (S67, S95) for defining the brightness adjustment range with reference to the number of pixels of the plurality of images remaining after the exclusion process of the exclusion means.

  According to the present invention, since the plurality of cameras capture the reference plane obliquely with an attitude partially having a common visual field, the posture of the three-dimensional object that appears in the common visual field is between the bird's-eye images respectively corresponding to the multiple cameras. Is different. The position of the three-dimensional object appearing in the common visual field is detected from each of the plurality of bird's-eye images, and the image at the detected position is commonly excluded from the plurality of bird's-eye images. The synthesis processing of the plurality of bird's-eye images is executed in such a manner that the difference in luminance and / or color between the plurality of images remaining after such exclusion processing is suppressed. Thereby, the quality of the synthesized bird's-eye view image can be improved.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

It is a block diagram which shows the basic composition of this invention. It is a block diagram which shows the structure of one Example of this invention. FIG. 3 is a perspective view showing an example of a vehicle on which the embodiment in FIG. 2 is mounted. It is an illustration figure which shows the visual field captured by the some camera attached to the vehicle. It is an illustration figure which shows a part of creation operation | movement of the bird's-eye view image based on the output of a camera. It is an illustration figure which shows an example of the driving assistance image displayed by a display apparatus. It is an illustration figure which shows an example of the area referred for pixel selection operation | movement. It is an illustration figure which shows an example of the image of the partial overlap area referred for pixel selection operation | movement. (A) is an illustration figure which shows an example of the pixel selection operation | movement with respect to the bird's-eye view image BEV_0, (B) is an illustration figure which shows an example of the pixel selection operation | movement with respect to the bird's-eye view image BEV_1. It is an illustration figure which shows a part of pixel selection operation | movement paying attention to chromaticity. It is an illustration figure which shows a part of pixel selection operation | movement paying attention to a brightness | luminance. It is an illustration figure which shows another part of pixel selection operation | movement which paid its attention to the brightness | luminance. It is an illustration figure which shows a part of gain calculation operation | movement for correct | amending a brightness | luminance. It is an illustration figure which shows an example of a brightness correction characteristic. It is an illustration figure which shows a part of calculation operation | movement of the offset for correct | amending chromaticity. It is a flowchart which shows a part of operation | movement of CPU applied to the FIG. 2 Example. It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example.

Embodiments of the present invention will be described below with reference to the drawings.
[Basic configuration]

  Referring to FIG. 1, the image processing apparatus of the present invention is basically configured as follows. The converting means 2 converts a plurality of object scene images output from a plurality of cameras 1, 1,... Capturing the reference plane obliquely with an attitude partially having a common visual field into a plurality of bird's-eye images with respect to the reference plane. . The first detection unit 3 detects the position of the image representing the three-dimensional object existing in the common visual field from each of the plurality of bird's-eye images converted by the conversion unit 2 as an exclusion position. The exclusion unit 4 commonly excludes the image at the exclusion position from the plurality of bird's-eye images converted by the conversion unit 2. The synthesizing unit 5 synthesizes the plurality of bird's-eye images converted by the converting unit 2 in a manner in which differences in luminance and / or chromaticity between the plurality of images remaining after the exclusion process of the excluding unit 4 are suppressed.

Since the plurality of cameras 1, 1,... Capture the reference plane from an angle with an attitude that partially has a common field of view, the posture of the three-dimensional object that appears in the common field of view corresponds to the plurality of cameras 1, 1,. It differs between bird's-eye images. The position of the three-dimensional object appearing in the common visual field is detected from each of the plurality of bird's-eye images, and the image at the detected position is commonly excluded from the plurality of bird's-eye images. The synthesis processing of the plurality of bird's-eye images is executed in such a manner that the difference in luminance and / or color between the plurality of images remaining after such exclusion processing is suppressed. Thereby, the quality of the synthesized bird's-eye view image can be improved.
[Example]

  The steering assistance device 10 of this embodiment shown in FIG. 2 includes four cameras CM_0 to CM_3. The cameras CM_0 to CM_3 respectively output the object scene images P_0 to P_3 every 1/30 seconds. The outputted scene images P_0 to P_3 are given to the image processing circuit 12.

  Referring to FIG. 3, camera CM_0 is installed on the front upper side portion of vehicle 100 in a posture in which the optical axis of camera CM_0 extends obliquely downward in front of vehicle 100. The camera CM_1 is installed on the upper right side of the vehicle 100 in a posture in which the optical axis of the camera CM_1 extends obliquely downward to the right of the vehicle 100. The camera CM_2 is installed on the rear upper side of the vehicle 100 in a posture in which the optical axis of the camera CM_2 extends obliquely downward rearward of the vehicle 100. The camera CM_3 is installed on the upper left side of the vehicle 100 so that the optical axis of the camera CM_3 extends obliquely downward to the left of the vehicle 100. The field around the vehicle 100 is captured from such a direction that obliquely intersects the road surface by the cameras CM_0 to CM_3.

  As shown in FIG. 4, the camera CM_0 has a visual field VW_0 that captures the front direction of the vehicle 100, the camera CM_1 has a visual field VW_1 that captures the right direction of the vehicle 100, and the camera CM_2 has a visual field that captures the backward direction of the vehicle 100. VW_2, and the camera CM_3 has a field of view VW_3 that captures the left direction of the vehicle 100. The visual fields VW_0 and VW_1 have a common visual field CVW_0, the visual fields VW_1 and VW_2 have a common visual field CVW_1, the visual fields VW_2 and VW_3 have a common visual field CVW_2, and the visual fields VW_3 and VW_0 have a common visual field CVW_3.

  Returning to FIG. 2, the CPU 12p provided in the image processing circuit 12 generates a bird's-eye view image BEV_0 based on the object scene image P_0 output from the camera CM_0, and generates the object scene image P_1 output from the camera CM_1. Based on this, a bird's-eye view image BEV_1 is generated. The CPU 12p also generates a bird's-eye image BEV_2 based on the object scene image P_2 output from the camera C_2, and generates a bird's-eye image BEV_3 based on the object scene image P_3 output from the camera C_3.

  As can be seen from FIG. 5, the bird's-eye view image BEV_0 corresponds to an image captured by a virtual camera looking down the visual field VW_0 in the vertical direction, and the bird's-eye view image BEV_1 corresponds to an image captured by a virtual camera looking down the visual field VW_1. . The bird's-eye view image BEV_2 corresponds to an image captured by a virtual camera looking down the visual field VW_2 in the vertical direction, and the bird's-eye view image BEV_3 corresponds to an image captured by a virtual camera looking down the visual field VW_3 in the vertical direction. The generated bird's-eye view images BEV_0 to BEV_3 are held in the memory 12m.

  Subsequently, the CPU 12p defines cutout lines CT_0 to CT_3 corresponding to the reproduction block BLK shown in FIG. 4 on the bird's-eye view images BEV_0 to BEV_3, and a part of the images existing inside the defined cutout lines CT_0 to CT_3. Create an all-around bird's-eye view image. A vehicle image GL1 simulating the upper part of the vehicle 100 is attached to the center of the created all-around bird's-eye view image. Thereby, the driving assistance image shown in FIG. 6 is completed.

  According to FIG. 6, the overlapping area DP_0 is formed corresponding to the common visual field CVW_0, and the overlapping area DP_1 is formed corresponding to the common visual field CVW_1. Further, the overlapping area DP_2 is formed corresponding to the common visual field CVW_2, and the overlapping area DP_3 is formed corresponding to the common visual field CVW_3. The created driving assistance image is displayed on the monitor screen of the display device 14. The display device 14 is installed in the driver's seat, and the driver controls the vehicle 100 while referring to the monitor screen.

The bird's-eye view images BEV_0 to BEV_3 have a luminance difference and / or a chromaticity difference due to an error in setting of the cameras CM_0 to CM_3. Therefore, in this embodiment, before the all-round bird's-eye view image is created, the brightness and / or chromaticity of the bird's-eye view images BEV_0 to BEV_3 are corrected as described below.
Restriction of reference area location

An evaluation area EVA having an extent smaller than the extent of the reproduction block BLK is defined around the vehicle 100 as shown in FIG. Bird's-eye view images BEV_0 to BEV_3 are reproduced as shown in FIG. 8 corresponding to the evaluation area EVA. Moreover, some areas belonging to the evaluation area EVA among the overlapping areas DP_0 to DP_3 are defined as partial overlapping areas PDP_0 to PDP_3.
Selection of pixels to be referenced for luminance and / or chromaticity correction

  Since the bird's-eye view images BEV_0 to BEV_3 are created by bird's-eye conversion based on the road surface, for example, if a three-dimensional object OBJ exists in the partial overlap area PDP_0, the posture of the three-dimensional object OBJ is as shown in FIGS. ), There is a difference between the bird's-eye images BEV_0 and BEV_1. In this method, the main object is to correct the luminance, saturation, and color by comparing the luminance difference of the same object in the adjacent camera. In addition, the accuracy of chromaticity correction may be reduced. In addition, there is a possibility that the accuracy of luminance and / or chromaticity correction may be reduced in a portion where the luminance change is sharp in the image, a portion having extremely high luminance due to halation, a portion having extremely low luminance due to low illuminance, etc. is there.

Pixels that may cause such a reduction in correction accuracy are detected in the following manner and excluded from the reference pixels.
(1) Detection and exclusion of pixels representing solid objects

  Whether or not the chromaticity difference between the two bird's-eye images BEV_j and BEV_j + 1 (j: 0 to 3) exceeds the threshold Cth exists in the coordinates (x, y) of the partial overlap area PDP_i (i: 0 to 3) The determination is made by paying attention to the two pixels.

  As shown in FIG. 10, the chromaticity difference between the bird's-eye images BEV_0 and BEV_1 is compared with the threshold Cth for the partially overlapping area PDP_0, and the chromaticity difference between the bird's-eye images BEV_1 and BEV_2 is the threshold for the partially overlapping area PDP_1. Compared with Cth. Further, for the partial overlap area PDP_2, the chromaticity difference between the bird's-eye images BEV_2 and BEV_3 is compared with the threshold Cth, and for the partial overlap area PDP_3, the chromaticity difference between the bird's-eye images BEV_3 and BEV_0 is compared with the threshold Cth. .

Specifically, it is determined whether or not Equation 1 is satisfied. In Equation 1, “Uijxy” and “Vijxy” represent the U-system chromaticity and the V-system chromaticity of the bird's-eye view image BEV_j existing at the coordinates (x, y) of the partial overlap area PDP_i, respectively. “Ui (j + 1) xy” and “Vi (j + 1) xy” represent the U-system chromaticity and the V-system chromaticity of the bird's-eye view image BEV_j + 1 existing at the coordinates (x, y) of the partial overlap area PDP_i, respectively.

If the chromaticity difference between the two pixels present at the coordinates (x, y) is large, it is highly possible that at least one of the two pixels is a pixel representing a three-dimensional object. For this reason, two pixels whose chromaticity difference exceeds the threshold value Cth are excluded from the bird's-eye view images BEV_j and BEV_j + 1, respectively. The shapes of the bird's-eye view images BEV_j and BEV_j + 1 after being excluded match each other as shown in FIGS. 9 (A) and 9 (B).
(2) Eliminating pixels with high sharpness (parts where the brightness changes drastically in the image)

  Whether or not the sum of the flatness of the bird's-eye image BEV_j and the sharpness of the bird's-eye image BEV_j + 1 exceeds the threshold Yth is determined by the two pixels existing at the coordinates (x, y) of the partial overlap area PDP_i (i: 0 to 3). Discriminated with attention.

  As shown in FIG. 11, for the partial overlap area PDP_0, the sum of the sharpness of the bird's-eye image BEV_0 and the sharpness of the bird's-eye image BEV_1 is compared with a threshold Yth, and for the partial overlap area PDP_1, the sharpness of the bird's-eye image BEV_1 The sum of the sharpness of BEV_2 is compared with the threshold value Yth. For the partial overlap area PDP_2, the sum of the sharpness of the bird's-eye image BEV_2 and the sharpness of the bird's-eye image BEV_2 is compared with the threshold Yth, and for the partial overlap area PDP_2, the sharpness of the bird's-eye image BEV_3 and the sharpness of the bird's-eye image BEV_0. The sum is compared with the threshold Yth.

Specifically, it is determined whether or not Expression 2 is satisfied. In Equation 2, “Yijxy” represents the brightness of the bird's-eye view image BEV_j existing at the coordinates (x, y) of the partial overlap area PDP_i, and “Yi (j + 1) xy” is at the coordinates (x, y) of the partial overlap area PDP_i. This represents the luminance of the existing bird's-eye view image BEV_j + 1.

  Two pixels satisfying Equation 2 are excluded from the bird's-eye view images BEV_j and BEV_j + 1, assuming that the pixels have high sharpness.

If the threshold value Yth is fixed, the discrimination ability decreases as the luminance decreases. For this reason, the magnitude of the threshold Yth is calculated according to Equation 3 (the slope S is calculated according to Equation 4).

  Thereby, pixels having the luminance of the shaded portion shown in FIG. 12 are excluded from the bird's-eye view images BEV_j and BEV_j + 1. Although illustration is omitted, the shapes of the bird's-eye view images BEV_j and BEV_j + 1 after being excluded are identical to each other as described above.

  In the present embodiment, for simplification of description, “Yijxy” represents the brightness of the bird's-eye view image BEV_j existing at the coordinates (x, y) of the partial overlap area PDP_i. Depending on the implementation of the hardware, there is a case where the all-round bird's-eye view image is directly generated from the current image without temporarily storing the image converted into the bird's-eye view image for each camera as in this embodiment.

In this case, “Yijxy” in Equation 2 represents the luminance of the bird's-eye view image BEV_j existing at the coordinates (x, y) of the partial overlap area PDP_i, but the peripheral pixel value Yij (x + m) (y + n) of Yijxy is in the current image. It may be a peripheral pixel of Yijxy. In short, it is only necessary to exclude a portion where the luminance variation near Yijxy is intense.
(3) Eliminating high and low luminance pixels

The brightness of each pixel forming the bird's-eye view image BEV_j is compared with each of the threshold values S_low and S_high. When a pixel having a luminance lower than the threshold value S_low or a pixel having a luminance higher than the threshold value S_high (a pixel belonging to a range indicated by hatching in FIG. 12) is detected, the detected pixel is excluded from the bird's-eye view image BEV_j and this detection is performed. Pixels common to the pixels are excluded from the bird's-eye view image BEV_j + 1. Although illustration is omitted, the shapes of the bird's-eye view images BEV_j and BEV_j + 1 after being excluded are identical to each other as described above.
Calculation of total brightness

基準輝度ゲインの算出 The total luminance of some of the images belonging to the overlapping area PDP_i in the bird's-eye view image BEV_j is calculated as “SUM_Yij” by the calculation of Expression 5 for pixels remaining after the pixel selection process (exclusion process) described above. .

Calculation of reference luminance gain

  Referring to FIG. 13, the right side of the short side corresponding to the front of the vehicle 100 in the reproduction block BLK is defined as an edge EDG_FR, and the right side of the short side corresponding to the rear of the vehicle 100 in the reproduction block BLK is defined as an edge EDG_BR. Define. Further, the left side of the short side corresponding to the front of the vehicle 100 in the reproduction block BLK is defined as an edge EDG_FL, and the right side of the short side corresponding to the rear of the vehicle 100 in the reproduction block BLK is defined as an edge EDG_FL.

Further, the reference luminance gain at the edge EDG_FR is defined as “Gy_FR”, and the reference luminance gain at the edge EDG_BR is defined as “Gy_BR”. Further, the reference luminance gain at the edge EDG_FL is defined as “Gy_FL”, and the reference luminance gain at the edge EDG_BL is defined as “Gy_BL”. Then, the gains Gy_FR to Gy_BL are calculated according to Equation 6.

According to Equation 6, the reference luminance gain Gy_FR indicates a magnitude corresponding to the difference between the total luminances SUM_Y00 and SUM_Y01, and the reference luminance gain Gy_BR indicates a magnitude corresponding to the difference between the total luminances SUM_Y12 and SUM_Y11. The reference luminance gain Gy_FL indicates a magnitude corresponding to the difference between the total luminances SUM_Y30 and SUM_Y33, and the reference luminance gain Gy_BL indicates a magnitude corresponding to the difference between the total luminances SUM_Y22 and SUM_Y23.
Calculation of target brightness gain

The target luminance gain to be given to each pixel of the bird's-eye image BEV_1 representing the right side of the vehicle 100 is defined as “Gy_R”, and the target luminance gain to be given to each pixel of the bird's-eye image BEV_3 representing the left side of the vehicle 100 is defined as “Gy_L”. To do. Then, the target luminance gain Gy_R is calculated by linear interpolation with reference to the reference luminance gains Gy_FR and Gy_BR and the position of the target pixel, and the target luminance gain Gy_L is calculated by linear interpolation with reference to the gains Gy_FL and Gy_BL and the position of the target pixel. To do. Specifically, the target luminance gain Gy_R is calculated according to Equation 7, and the target luminance gain Gy_L is calculated according to Equation 8.

By separately calculating the target luminance gain Gy_R corresponding to the right side of the vehicle 100 and the target luminance gain Gy_L corresponding to the left side of the vehicle 100, another vehicle passes through the right side of the vehicle 100, and the monotonous pattern on the left side of the vehicle 100 When there is a road surface, the reliability of the gain is ensured.
Limiting the target luminance gain

  If the target luminance gain is significantly different from “1”, if the magnitude is incorrect, the quality of the all-around bird's-eye view image is greatly degraded. In order to avoid such a phenomenon, the size of the target luminance gain is limited so that the upper and lower limits of the luminance level (corrected luminance level) of the pixel to which the target luminance gain is given fall within the range shown in FIG. The

  However, if the number of pixels remaining after the pixel selection process is small, there is a high possibility that a three-dimensional object exists in the overlapping area PDP_i. In such a state, it can hardly be said that the calculated target luminance gain is correct, and it can be said that it is more preferable to suppress the size of the target luminance gain. Further, the fluctuation amount of the situation around the vehicle 100 increases as the moving speed of the vehicle 100 increases, and the accuracy of luminance correction referring to the target luminance gain also decreases as the moving speed of the vehicle 100 increases.

Therefore, the upper limit and the lower limit of the corrected luminance level are adaptively adjusted based on the number of pixels remaining after the pixel selection process and the moving speed of the vehicle 100 detected by the speed sensor 18. That is, the magnitude of the target luminance gain approaches “1” as the number of pixels remaining after the pixel selection process is reduced and / or the moving speed of the vehicle 100 is increased.
Calculation of average chromaticity

The U-system total chromaticity of a part of the images belonging to the overlapping area PDP_i in the bird's-eye view image BEV_j is calculated as “SUM_Uij” by the calculation of Expression 9 for pixels remaining after the pixel selection process described above. Similarly, the V-system total chromaticity of a part of the images belonging to the overlapping area PDP_i in the bird's-eye image BEV_j is calculated as “SUM_Vij” by the calculation of Expression 10 for the pixels remaining after the pixel selection process described above. Is done.

基準オフセットの算出 Further, the U-system average chromaticity of some images belonging to the overlapping area PDP_i in the bird's-eye view image BEV_j is calculated as “AVE_Uij” by the calculation of Equation 11. Similarly, the V-system average chromaticity of some images belonging to the overlapping area PDP_i in the bird's-eye view image BEV_j is calculated as “AVE_Uij” by the calculation of Expression 12. Note that “CNTi” corresponds to the number of pixels remaining after pixel selection processing in the overlapping area PDP_i.

Calculation of reference offset

Referring to FIG. 15, the U-system reference offset at edge EDG_FR is defined as “OFSTu_FR”, and the U-system reference offset at edge EDG_BR is defined as “OFSTu_BR”. Further, the U-system reference offset at the edge EDG_FL is defined as “OFSTu_FL”, and the U-system reference offset at the edge EDG_BL is defined as “OFSTu_BL”. Then, the U-system reference offsets OFSTu_FR to OFSTu_BL are calculated according to Equation 13.

  According to Equation 13, the U-system reference offset OFSTu_FR indicates a magnitude corresponding to the difference between the U-system average chromaticity AVE_U00 and AVE_U01, and the U-system reference offset OFSTu_BR is a magnitude corresponding to the difference between the U-system average chromaticity AVE_U12 and AVE_U11. It shows. The U-system reference offset OFSTu_FL indicates a magnitude corresponding to the difference between the U-system average chromaticity AVE_U30 and AVE_U33, and the U-system reference offset OFSTu_BL indicates a magnitude corresponding to the difference between the U-system average chromaticity AVE_U22 and AVE_U23.

Similarly, the V system reference offset at the edge EDG_FR is defined as “OFSTv_FR”, and the V system reference offset at the edge EDG_BR is defined as “OFSTv_BR”. Further, the V-system reference offset at the edge EDG_FL is defined as “OFSTv_FL”, and the V-system reference offset at the edge EDG_BL is defined as “OFSTv_BL”. Then, the V system reference offsets OFSTv_FR to OFSTv_BL are calculated according to Equation 14.

According to Equation 14, the V-system reference offset OFSTv_FR indicates a magnitude corresponding to the difference between the V-system average chromaticity AVE_V00 and AVE_V01, and the V-system reference offset OFSTv_BR is a magnitude corresponding to the difference between the V-system average chromaticity AVE_V12 and AVE_V11. It shows. The V-system reference offset OFSTv_FL indicates a magnitude corresponding to the difference between the V-system average chromaticity AVE_V30 and AVE_V33, and the V-system reference offset OFSTv_BL indicates a magnitude corresponding to the difference between the V-system average chromaticity AVE_V22 and AVE_V23.
Target offset calculation

The U-system target offset and the V-system target offset to be assigned to each pixel of the bird's-eye view image BEV_1 representing the right side of the vehicle 100 are defined as “OFSTu_R” and “OFSTv_R”, respectively. Then, the U-system target offset “OFSTu_R” is calculated by linear interpolation with reference to the U-system reference offsets OFSTu_FR and OFSTu_BR and the position of the target pixel, and the V-system target offset “OFSTv_R” is calculated using the V-system reference offsets OFSTv_FR and OFSTv_BR and the target pixel. It is calculated by linear interpolation with reference to the position of. Specifically, the U-system target offset OFSTu_R is calculated according to Expression 15, and the V-system target offset OFSTv_R is calculated according to Expression 16.

色ゲインの算出 Similarly, a U-system target offset and a V-system target offset to be assigned to each pixel of the bird's-eye view image BEV_3 representing the left side of the vehicle 100 are defined as “OFSTu_L” and “OFSTv_L”, respectively. Then, the U-system target offset “OFSTu_L” is calculated by linear interpolation with reference to the U-system reference offsets OFSTu_FL and OFSTu_BL and the position of the target pixel, and the V-system target offset “OFSTv_L” is calculated using the V-system reference offsets OFSTv_FL and OFSTv_BL and the target pixel. It is calculated by linear interpolation with reference to the position of. Specifically, the U-system target offset OFSTu_L is calculated according to Equation 17, and the V-system target offset OFSTv_L is calculated according to Equation 18.

Color gain calculation

  The gain correction for the color is performed by changing the saturation of the image subjected to the color offset correction. In order to correct a difference in saturation due to a difference in exposure amount between the cameras CM_0 to CM_3, it is preferable that the saturation gain is the same as the luminance gain. However, since the luminance gain is non-linear, if the saturation gain is simply matched to the luminance gain, the U-type chromaticity magnitude relationship or the V-type chromaticity magnitude relationship may be reversed between adjacent pixels. It becomes easy to do. Further, there are cases where the U-type chromaticity or the V-type chromaticity is saturated. Therefore, the color gain (saturation gain) applied to the U-system chromaticity and the V-system chromaticity is set to K times the luminance gain (K: 0.3, for example), and saturation is eliminated by clipping.

  The driving assistance image shown in FIG. 6 is created after completing such luminance correction and / or chromaticity correction. At this time, for the overlapping area DP_0, the weighting amount of the bird's-eye image BEV_0 is set to “0”, and the weighting amount of the bird's-eye image BEV_1 is set to “1”. For the overlapping area DP_1, the weighting amount of the bird's-eye view image BEV_1 is set to “1”, and the weighting amount of the bird's-eye view image BEV_2 is set to “0”.

  Furthermore, for the overlapping area DP_2, the weighting amount of the bird's-eye image BEV_2 is set to “0”, and the weighting amount of the bird's-eye image BEV_3 is set to “1”. For the overlapping area DP_4, the weighting amount of the bird's-eye image BEV_3 is set to “1”, and the weighting amount of the bird's-eye image BEV_0 is set to “0”.

  Specifically, the CPU 12p executes processing according to the flowcharts shown in FIGS. A control program corresponding to these flowcharts is stored in the flash memory 16.

  Referring to FIG. 16, in step S1, it is determined whether or not a vertical synchronization signal Vsync for internal synchronization has occurred. When the determination result is updated from NO to YES, the process proceeds to step S3, and the object scene images P_0 to P_3 stored in the memory 12m are converted into the bird's-eye view images BEV_0 to BEV_3. Subsequently, pixel selection processing is executed in step S5, luminance correction processing is executed in step S7, and chromaticity correction processing is executed in step S9. When the process of step S9 is completed, an all-around bird's-eye image is created in step S11, and then the process returns to step S1.

  The pixel selection process in step S5 is executed according to a subroutine shown in FIGS. First, in step S21, the variable i is set to “0”, and in step S23, the top pixel of the partial overlap area PDP_i is designated. In step S25, it is determined whether or not the above equation 1 is satisfied. If a determination result is NO, it will progress to step S29 as it is, and if a determination result is YES, it will progress to step S29 through the process of step S27. In step S27, the designated pixel is excluded from both the bird's-eye view images BEV_j and BEV_j + 1.

  In step S29, the threshold value Yth is calculated with reference to equations 3 and 4, and in step S31, it is determined whether equation 2 is satisfied. If the determination result is NO, the process proceeds directly to step S35, whereas if the determination result is YES, the same exclusion process as in step S27 is executed in step S33, and then the process proceeds to step S35.

  In step S35, it is determined whether or not the luminance of the designated pixel on the bird's-eye view image BEV_j is out of a predetermined luminance range (range of S_low to S_high). If the determination result is NO, the process proceeds to step S39 as it is, while if the determination result is YES, the same exclusion process as in step S27 is executed in step S37, and then the process proceeds to step S39.

  In step S39, it is determined whether or not the designated image is the last pixel of the overlapping area PDP_i. In step S43, it is determined whether or not the variable i is “3”. If “NO” in the step S39, the next pixel of the overlapping area PDP_i is designated in a step S41, and thereafter, the process returns to the step S25. If “YES” in the step S39 and “NO” in the step S41, the variables i and j are incremented in a step S45, and thereafter, the process returns to the step S23. Variables i and j both represent a numerical value that circulates between “0” and “3”. When variable j indicates “3”, variable “j + 1” indicates “0”. If both steps S39 and S43 are YES, the process returns to the upper hierarchy routine.

  The brightness correction process in step S7 shown in FIG. 16 is executed according to a subroutine shown in FIGS.

  First, in step S51, the moving speed of the vehicle 100 is detected as "SPD" based on the output of the speed sensor 18. In step S53, a part of the image corresponding to the partial overlap area PDP_0 is specified from the bird's-eye view image BEV_0, and the total luminance of the specified image is calculated as “SUM_Y00”. In step S55, a part of the image corresponding to the partial overlap area PDP_0 is specified from the bird's-eye view image BEV_1, and the total luminance of the specified image is calculated as “SUM_Y01”.

  In step S57, a part of the image corresponding to the partial overlap area PDP_1 is specified from the bird's-eye view image BEV_1, and the total luminance of the specified image is calculated as “SUM_Y11”. In step S59, a part of the image corresponding to the partial overlap area PDP_1 is specified from the bird's-eye view image BEV_2, and the total luminance of the specified image is calculated as “SUM_Y12”.

  Note that the total luminances “SUM_Y00”, “SUM_Y01”, “SUM_Y11”, and “SUM_Y12” all correspond to the sum of the luminances of the pixels remaining after the exclusion processing in steps S27, S33, and S37 described above. 5 with reference to FIG.

  In step S61, the number of pixels remaining after the exclusion process for the partial overlap area PDP_0 is detected as “CNT1”. In step S63, the number of pixels remaining after the exclusion process for the partial overlap area PDP_1 is detected as “CNT2”.

  In step S65, a reference luminance gain Gy_FR corresponding to the edge EGD_FR and a reference luminance gain Gy_BR corresponding to the edge EGD_BR are calculated with reference to Equation 6. In step S67, the upper limit and the lower limit (luminance correction range) of the corrected luminance level shown in FIG. 14 are determined based on the moving speed SPD and the pixel numbers CNT0 and CNT1. The brightness correction range becomes narrower as the moving speed SPD increases or the number of pixels CNT0 and / or CNT1 decreases.

  When the process of step S67 is completed, the top pixel of the bird's-eye view image BEV_1 is designated in step S69. In step S71, the target luminance gain Gy_R corresponding to the designated pixel is calculated according to Equation 7. In step S73, the calculated target luminance gain Gy_R is corrected with reference to the luminance correction range determined in step S67. In step S75, the corrected gain Gy_R ′ is given to the luminance of the designated pixel. In step S77, it is determined whether or not the designated pixel is the end pixel of the bird's-eye view image BEV_1. If the determination result is NO, the next pixel of the bird's-eye view image BEV_1 is designated in step S79, and then the process returns to step S71. If a determination result is YES, it will progress to Step S81.

  In step S81, a part of the image corresponding to the partial overlap area PDP_2 is identified from the bird's-eye view image BEV_2, and the total luminance of the identified image is calculated as “SUM_Y22”. In step S83, a part of the image corresponding to the partial overlap area PDP_2 is specified from the bird's-eye view image BEV_3, and the total luminance of the specified image is calculated as “SUM_Y23”.

  In step S85, a part of the image corresponding to the partial overlap area PDP_3 is specified from the bird's-eye view image BEV_0, and the total luminance of the specified image is calculated as “SUM_Y30”. In step S87, a part of the image corresponding to the partial overlap area PDP_3 is specified from the bird's-eye view image BEV_3, and the total luminance of the specified image is calculated as “SUM_Y33”.

  As described above, the total luminances “SUM_Y22”, “SUM_Y23”, “SUM_Y30”, and “SUM_Y33” correspond to the sum of the luminances of the pixels remaining after the exclusion processing in steps S27, S33, and S37. Calculated by reference.

  In step S89, the number of pixels remaining after the exclusion process for the partial overlap area PDP_2 is detected as “CNT2”. In step S91, the number of pixels remaining after the exclusion process for the partial overlap area PDP_3 is detected as “CNT3”.

  In step S93, the reference luminance gain Gy_FL corresponding to the edge EGD_FL and the reference luminance gain Gy_BL corresponding to the edge EGD_BL are calculated with reference to Equation 6. In step S95, the luminance correction range is determined based on the moving speed SPD and the number of pixels CNT2 and CNT3.

  When the process in step S95 is completed, the top pixel of the bird's-eye view image BEV_3 is designated in step S97. In step S99, the target luminance gain Gy_L corresponding to the designated pixel is calculated according to Equation 8. In step S101, the calculated target luminance gain Gy_L is corrected with reference to the luminance correction range determined in step S95. In step S103, the corrected gain Gy_L ′ is given to the luminance of the designated pixel. In step S105, it is determined whether or not the designated pixel is the end pixel of the bird's-eye view image BEV_3. If the determination result is NO, the next pixel of the bird's-eye view image BEV_3 is designated in step S107, and then the process returns to step S99. If the determination result is YES, the process returns to the upper hierarchy routine.

  The chromaticity correction process in step S9 shown in FIG. 16 is executed according to a subroutine shown in FIGS.

  In step S111, a part of the image corresponding to the partial overlap area PDP_0 is specified from the bird's-eye view image BEV_0, and the U-system average chromaticity and the V-system average chromaticity of the specified image are calculated as “AVE_U00” and “AVE_V00”. . In step S113, a part of the image corresponding to the partial overlap area PDP_0 is identified from the bird's-eye view image BEV_1, and the U-system average chromaticity and the V-system average chromaticity of the identified image are calculated as “AVE_U01” and “AVE_V01”. .

  In step S115, a part of the image corresponding to the partial overlap area PDP_1 is specified from the bird's-eye view image BEV_1, and the U-system average chromaticity and the V-system average chromaticity of the specified image are calculated as “AVE_U11” and “AVE_V11”. . In step S117, a part of the image corresponding to the partial overlap area PDP_1 is identified from the bird's-eye view image BEV_2, and the U-system average chromaticity and the V-system average chromaticity of the identified image are calculated as “AVE_U12” and “AVE_V12”. .

  Note that the U-system average chromaticity “SUM_U00”, “SUM_U01”, “SUM_U11”, and “SUM_U12” are all the average values of U components that form the remaining pixels after the above-described steps S27, S33, and S37. And is calculated with reference to Equation 9 and Equation 11 above.

  Similarly, the V-system average chromaticities “SUM_V00”, “SUM_V01”, “SUM_V11”, and “SUM_V12” are all the averages of the V components that form the remaining pixels after the above-described steps S27, S33, and S37. It corresponds to a value and is calculated with reference to the above-mentioned formulas 10 and 12.

  In step S119, the U-system reference offset OFSTu_FR corresponding to the edge EGD_FR and the U-system reference offset OFSTu_BR corresponding to the edge EGD_BR are calculated with reference to Equation 13. In step S121, the V-system reference offset OFSTv_FR corresponding to the edge EGD_FR and the V-system reference offset OFSTv_BR corresponding to the edge EGD_BR are calculated with reference to Equation 14.

  In step S123, the top pixel of the bird's-eye view image BEV_1 is designated. In step S125, the U-system target offset OFSTu_R corresponding to the designated pixel is calculated with reference to Equation 15, and in Step S127, the V-system target offset OFSTv_R corresponding to the designated pixel is calculated with reference to Equation 16. In step S129, the calculated U system target offset OFSTu_R is added to the U system chromaticity of the designated pixel, and in step S131, the calculated V system target offset OFSTv_R is added to the V system chromaticity of the designated pixel.

  In step S133, the gain Gu_R is added to the U system chromaticity to which the U offset OFSTu_R is added. In step S135, the gain Gv_R is applied to the V system chromaticity to which the V offset OFSTy_R is added. Here, each of the gains Gu_R and Gv_R corresponds to K times the gain Gy_R applied to the same pixel.

  In step S137, it is determined whether or not the designated pixel is the last pixel of the bird's-eye view image BEV_1. If the determination result is NO, the next pixel of the bird's-eye view image BEV_1 is designated in step S139, and then the process returns to step S125. On the other hand, if a determination result is YES, it will progress to Step S141.

  In step S141, a part of the image corresponding to the partial overlap area PDP_2 is identified from the bird's-eye view image BEV_2, and the U-system average chromaticity and the V-system average chromaticity of the identified image are calculated as “AVE_U22” and “AVE_V22”. . In step S143, a part of the image corresponding to the partial overlap area PDP_2 is identified from the bird's-eye view image BEV_3, and the U-system average chromaticity and the V-system average chromaticity of the identified image are calculated as “AVE_U23” and “AVE_V23”. .

  In step S145, a part of the image corresponding to the partial overlap area PDP_3 is identified from the bird's-eye view image BEV_0, and the U-system average chromaticity and the V-system average chromaticity of the identified image are calculated as “AVE_U30” and “AVE_V30”. . In step S147, a part of the image corresponding to the partial overlap area PDP_3 is specified from the bird's-eye view image BEV_3, and the U-system average chromaticity and the V-system average chromaticity of the specified image are calculated as “AVE_U33” and “AVE_V33”. .

  The U-system average chromaticity “SUM_U22”, “SUM_U23”, “SUM_U30”, and “SUM_U33” are all average values of U components that form the remaining pixels after the above-described steps S27, S33, and S37. And is calculated with reference to Equation 9 and Equation 11 above.

  Similarly, the V-system average chromaticities “SUM_V22”, “SUM_V23”, “SUM_V30”, and “SUM_V33” are all averages of V components that form the remaining pixels after the above-described steps S27, S33, and S37. It corresponds to a value and is calculated with reference to the above-mentioned formulas 10 and 12.

  In step S149, the U-system reference offset OFSTu_FL corresponding to the edge EGD_FL and the U-system reference offset OFSTu_BL corresponding to the edge EGD_BL are calculated with reference to Equation 13. In step S151, the V-system reference offset OFSTv_FL corresponding to the edge EGD_FL and the V-system reference offset OFSTv_BL corresponding to the edge EGD_BL are calculated with reference to Equation 14.

  In step S153, the top pixel of the bird's-eye view image BEV_3 is designated. In step S155, the U-system target offset OFSTu_L corresponding to the designated pixel is calculated with reference to Equation 17, and in Step S157, the V-system target offset OFSTv_L corresponding to the designated pixel is calculated with reference to Equation 18. In step S159, the calculated U system target offset OFSTu_L is added to the U system chromaticity of the designated pixel, and in step S161, the calculated V system target offset OFSTv_L is added to the V system chromaticity of the designated pixel.

  In step S163, the gain Gu_L is added to the U-system chromaticity to which the U-system target offset OFSTu_L is added, and in step S165, the gain Gv_L is applied to the V-system chromaticity to which the V system target offset OFSTy_L is added. Here, each of the gains Gu_L and Gv_L corresponds to K times the gain Gy_L applied to the same pixel.

  In step S177, it is determined whether or not the designated pixel is the last pixel of the bird's-eye view image BEV_3. If the determination result is NO, in step S169, the next pixel of the bird's-eye view image BEV_3 is designated, and thereafter, the process returns to step S155. On the other hand, if the determination result is YES, the process returns to the upper layer routine.

  As can be seen from the above description, the cameras CM_0 to CM_3 provided in the vehicle 100 capture the road surface obliquely with a posture partially having the common visual field CVW_0 to CVW_3. The CPU 12p converts the object scene images P_0 to P_3 output from the cameras CM_0 to CM_3 into bird's-eye images BEV_0 to BEV_3 representing a state in which the road surface is viewed from above (S3). The CPU 12p also detects the position of the image representing the three-dimensional object existing in the common visual field CVW_0 to CVW_3 as the exclusion position from each of the bird's-eye images BEV_0 to BEV_3 (S23 to S25, S39 to S41), and the image at the exclusion position is a bird's-eye view. The images BEV_0 to BEV_3 are commonly excluded (S27, S33). The CPU 12p further synthesizes the bird's-eye view images BEV_0 to BEV_3 in such a manner that the difference in luminance and / or chromaticity between the plurality of images remaining after the exclusion process is suppressed (S7 to S11).

  Since the cameras CM_0 to CM_3 capture the reference plane obliquely with an attitude partially having the common visual field CVW_0 to CVW_3, the posture of the three-dimensional object that appears in the common visual fields CVW_0 to CVW_3 is the bird's-eye view image BEV_0 corresponding to each of the cameras CM_0 to CM_3. It differs between BEV_3. The positions of the three-dimensional objects appearing in the common visual fields CVW_0 to CVW_3 are detected from each of the bird's-eye images BEV_0 to BEV_3, and the images at the detected positions are commonly excluded from the bird's-eye images BEV_0 to BEV_3. The synthesis process of the bird's-eye view images BEV_0 to BEV_3 is executed in a manner in which a difference in luminance and / or color between a plurality of images remaining after such an exclusion process is suppressed. Thereby, the quality of the synthesized bird's-eye view image can be improved.

  In this embodiment, the gain correction for the color is performed after the color offset correction. However, gain correction for colors may be performed before color offset correction.

  In this embodiment, the all-round bird's-eye display for a vehicle is described as an example. However, the present technique can also be applied to a multi-camera system that performs bird's-eye conversion (viewpoint conversion) on a reference surface such as a road surface.

DESCRIPTION OF SYMBOLS 10 ... Steering assistance apparatus CM_0-CM_3 ... Camera 12 ... Image processing circuit 12m ... Memory 12p ... CPU
DESCRIPTION OF SYMBOLS 14 ... Display apparatus 16 ... Flash memory 18 ... Speed sensor 100 ... Vehicle

Claims (6)

Conversion means for converting a plurality of object scene images output from a plurality of cameras capturing the reference plane obliquely with a posture having a common visual field partially into a plurality of bird's-eye images with respect to the reference plane;
First detection means for detecting the position of an image representing a three-dimensional object existing in the common visual field as an exclusion position from each of a plurality of bird's-eye images converted by the conversion means;
Exclusion means for commonly removing the image at the exclusion position from a plurality of bird's-eye images converted by the conversion means, and a plurality of bird's-eye images converted by the conversion means left after the exclusion processing of the exclusion means An image processing apparatus comprising combining means for combining in such a manner that differences in luminance and / or chromaticity between the images are suppressed.   The image processing apparatus according to claim 1, wherein the exclusion position corresponds to a pixel position where a luminance difference and / or a chromaticity difference between the plurality of bird's-eye images exceeds a first reference.   The image processing apparatus according to claim 1, further comprising second detection means for detecting a position of a sharp image corresponding to the common visual field as the exclusion position. Adjusting means for adjusting a second standard related to the sharpness of the image with reference to the luminance of the object scene image output from each of the plurality of cameras;
The image processing apparatus according to claim 3, wherein the second detection unit performs detection processing with reference to the second reference.   The synthesizing unit detects a luminance and / or chromaticity difference between a plurality of images remaining after the exclusion process of the exclusion unit corresponding to the common visual field, and luminances of the plurality of bird's-eye images 5. The image processing apparatus according to claim 1, further comprising adjustment means for adjusting chromaticity based on a difference detected by the detection means.   The image processing apparatus according to claim 5, wherein the adjustment unit includes a definition unit that defines the luminance adjustment range with reference to the number of pixels of a plurality of images remaining after the exclusion process of the exclusion unit.

Images