JP2009141490A - Composite image generation device and composite image generation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a composite image generation device and a composite image generation method, which generates a composite image without over exposure or under exposure as much as possible in an overlapped image part without performing a complicated operation when the composite image is generated by compounding a plurality of images. <P>SOLUTION: A luminance detection part 160 detects a luminance value of the overlapped image part in which an imaging range is overlapped from each of one image, and the other image in which a part of an imaging range overlaps with one image for the plurality of image in which the different imaging ranges around a vehicle are picked up by a plurality of cameras, when only any luminance value is within a predetermined range, a luminance setting part 220 sets a luminance value within the predetermined range to the luminance value of the overlapped image part by setting the luminance value of the overlapped image part to the luminance value within the predetermined range, and the composite image without the over exposure or the under exposure as much as possible in the overlapped image part is generated. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a composite image generation apparatus and a composite image generation method, and is suitable for use in, for example, a vehicle peripheral image generation apparatus that can generate a vehicle peripheral image for driving support.

  Conventionally, a plurality of cameras are installed on the front, rear, left, and right sides of the host vehicle, and images around the host vehicle captured by the plurality of cameras are subjected to viewpoint conversion processing, thereby generating a vehicle periphery image viewed from a virtual viewpoint above the host vehicle. And the technique (henceforth a vehicle periphery image generation technique) which displays the said vehicle periphery image on display apparatuses, such as a display, is proposed. According to this technology, for example, the driver can grasp the positional relationship between the host vehicle and the obstacles around the host vehicle by confirming the vehicle periphery image displayed on the display device of the host vehicle. It is possible to drive to prevent a collision with an obstacle.

  FIG. 7 is a diagram illustrating an arrangement example of cameras installed in a vehicle. In FIG. 7, the front camera 1 a is disposed in front of the host vehicle 2 and captures an image in the front range 3. The left side camera 1 b is disposed on the left side surface of the host vehicle 2 and images the left range 5. The rear camera 1 c is disposed on the rear surface of the host vehicle 2 and captures an image in the rear range 7. The right side camera 1d is disposed on the right side surface of the host vehicle 2 and captures an image in the right range 9.

  Here, between the range 3 imaged by the front camera 1a and the range 5 imaged by the left side camera 1b (adjacent to the range 3), the images imaged by the respective cameras partially overlap. . In addition, between the range 5 captured by the left side camera 1b and the range 7 captured by the rear camera 1c (adjacent to the range 5), the images captured by the respective cameras partially overlap. In addition, between the range 7 captured by the rear camera 1c and the range 9 (adjacent to the range 7) captured by the right side camera 1d, images captured by the respective cameras partially overlap. In addition, between the range 9 captured by the right side camera 1d and the range 3 captured by the front camera 1a (adjacent to the range 9), images captured by the respective cameras partially overlap. Hereinafter, cameras having adjacent imaging ranges are referred to as “adjacent cameras”.

  In the above-described vehicle peripheral image generation technology, when the viewpoint conversion processing is performed on captured images of the vehicle periphery captured by a plurality of cameras, any one of the adjacent cameras with respect to the overlapping image portion where the imaging ranges partially overlap The image picked up by is adopted. In the conventional vehicle periphery image generation technique, it is determined in advance which of the adjacent cameras to use the image captured by the camera. Normally, an image captured by the rear camera 1c is used for an image of a portion where the imaging ranges overlap between the rear camera 1c and the left side camera 1b or the right side camera 1d. In addition, an image captured by the front camera 1a is used for an image of a portion where the imaging range overlaps between the front camera 1a and the left side camera 1b or the right side camera 1d.

  By the way, in the vehicle periphery image generation technology, in the overlapping image portion where the imaging range overlaps between a plurality of cameras, the luminance value of the image captured by one camera or another camera whose imaging range is adjacent to the one camera. Of the luminance values of the captured images, only one of them may be in a state close to whiteout (becoming an image close to white) or blackout (becoming an image close to black).

  For example, when the own vehicle enters or leaves the garage, the sunlight (light source) hits the imaging range of the adjacent camera, so that the front camera 1a or the rear camera 1c among the adjacent cameras captures images. In some cases, the luminance value of the overlapped image portion of the generated image becomes close to whiteout or blackout.

  In the conventional technology, as described above, when the vehicle periphery image is generated by combining the images around the host vehicle captured by the plurality of cameras, the front camera 1a or the adjacent camera among the adjacent cameras is generated as the overlapping image portion image. An image picked up by the rear camera 1c was fixedly adopted. As a result, an image having a luminance value close to whiteout or blackout may be selected in the overlapped image portion. Therefore, in the synthesized vehicle peripheral image, a white area is included in a partial region (overlapped image portion). There has been a problem that flying or black crushing occurs. In some cases, there has been a problem that the luminance difference between the overlapping image portion and the image portion adjacent to the overlapping image portion may increase. For this reason, the driver feels uncomfortable at the joint between the overlapping image portion and the adjacent image portion in the vehicle peripheral image.

As a technique for adjusting the brightness between a plurality of images, when a composite image is generated by connecting images captured by a plurality of cameras, a composite image that matches the brightness of each image is generated, and overexposure is performed. There is known a technique for generating an easy-to-view image without blackout (see, for example, Patent Document 1). In the technique described in Patent Document 1, among a plurality of images captured by a plurality of cameras, one image and a luminance value of an adjacent image that partially overlaps adjacent to the image are compressed, and the compressed one image And / or the brightness value of the adjacent image is shifted so that there is no difference in the brightness of the overlapping part, and the brightness value of one image and the adjacent image is the brightness of the absolute white of all pixels after expansion The image is stretched in a range that falls between the value and the luminance value of absolute black.
JP 2007-72750 A

  However, in the technique described in Patent Document 1, every time a composite image is generated by connecting images captured by a plurality of cameras, compression processing and shift processing are performed on luminance values of one image and an adjacent image. There is a problem that a lot of calculation load is applied to perform complicated calculation such as decompression processing.

  The present invention has been made to solve such problems. When a plurality of images captured by a plurality of cameras are combined to generate a composite image, a part of the plurality of images is captured. It is an object of the present invention to be able to generate a composite image with no overexposure or underexposure as much as possible in overlapping image portions whose ranges overlap each other without performing complicated calculations. Another object of the present invention is to make it possible to reduce the luminance difference between the overlapping image portion and the adjacent image portion as much as possible.

  In order to solve the above-described problem, in the present invention, for a plurality of images in which different imaging ranges around the vehicle are captured by a plurality of cameras, one image and a part of the imaging range overlap with the one image. When the luminance value of the overlapping image portion where the imaging ranges overlap is detected from each of the images, and only one of the luminance values is within the predetermined range, the luminance value of the overlapping image portion is the luminance value within the predetermined range. It is set to.

  In another aspect of the present invention, after the luminance value of the overlapping image portion is set, the luminance value different from the luminance value of the set overlapping image portion among the one image and the other image is set as the overlapping image. The non-overlapping image part obtained by removing the overlapping image part from the image included in the part is divided into a plurality of blocks, and the luminance value of the plurality of blocks is changed to the luminance value obtained by the interpolation calculation using the luminance value of the overlapping image part. Each is set.

  According to the present invention configured as described above, with respect to a plurality of images captured by a plurality of cameras, only one of the luminance values of the overlapping image portion of one image and another image has a predetermined range. Even if the brightness value is in a state close to, for example, overexposure or blackout, the brightness value of the overlapped image portion includes the brightness value of the overlapped image portion of the one image and the other image within a predetermined range. The luminance value of the one located at is set. As a result, when a composite image is generated by combining a plurality of images captured by a plurality of cameras, a composite image free from overexposure or blackout can be generated as much as possible in the overlapping image portion.

  Further, according to the present invention, the brightness value of the overlapped image portion is overlapped using a simple process in which the brightness value of the overlapped image portion of one image and the other image is set within a predetermined range. Since it is possible to generate a composite image with as little whitening and underexposure as possible in the image part, when generating a composite image by combining multiple images captured by multiple cameras, it is as complex as the conventional technology It is possible to generate a composite image with no overexposure or underexposure as much as possible in the overlapped image portion without performing.

  Further, according to another aspect of the present invention, the luminance value of the non-overlapping image portion is determined by an interpolation operation using the luminance value of the overlapping image portion for each of a plurality of blocks obtained by dividing the non-overlapping image portion. Set to That is, the luminance value of the non-overlapping image portion gradually becomes closer to the luminance value of the overlapping image portion as the image position approaches the overlapping image portion. As a result, in addition to preventing overexposure or blackout in the overlapped image portion, the luminance difference between the overlapped image portion and the adjacent image portion (non-overlapping image portion) is minimized. can do.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, a first embodiment of the invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration example of a vehicle periphery image generation device including a composite image generation device according to the first embodiment. As shown in FIG. 1, the vehicle surrounding image generation device according to the first embodiment includes an imaging unit 120 and a monitor 140 in addition to the composite image generation device 100.

  The imaging unit 120 includes four imaging devices installed on the front, rear, left, and right sides of the vehicle, and images a plurality of locations around the vehicle as a front view image, a left side view image, a right side view image, and a rear view image. Specifically, the imaging unit 120 includes a front camera for imaging the front of the host vehicle, a left side camera for imaging the left side of the host vehicle, a right side camera for imaging the right side of the host vehicle, and A rear camera for imaging the rear of the host vehicle is provided. Note that the range to be imaged by each imaging device of the imaging unit 120 is mainly the road surface around the host vehicle. Therefore, each imaging device is installed in the vehicle in such a posture that many road surfaces are included in the imaging range. The monitor 140 is for displaying the composite image generated by the composite image generation apparatus 100.

  Next, the internal configuration of the composite image generation apparatus 100 will be described. The composite image generation apparatus 100 includes an image input unit 160, a luminance detection unit 180, a luminance difference detection unit 200, a luminance setting unit 220, and a composite image generation unit 240.

  The image input unit 160 inputs four images (front view image, left side view image, right side view image, and rear view image) in which different imaging ranges around the vehicle are captured by the four imaging devices included in the imaging unit 120. To do. For the four images input by the image input unit 160, the luminance detection unit 180 overlaps the imaging range from one image and another image where the one image and a part of the imaging range overlap. Part (the combined image generation unit 240 performs viewpoint conversion processing on the above four images to generate four images (a front image, a left side image, a right side image, and a rear image), and overlaps when these are combined. (Corresponding to the image portion). Hereinafter, images with overlapping imaging ranges are referred to as “overlapping images”. For example, when one image is a left side view image, there are two images, a front view image and a rear view image, which overlap with a part of the imaging range. In this case, the luminance detection unit 180 detects the luminance value of the overlapping image portion from each of the front view image and the left side view image. In addition, the luminance detection unit 180 detects the luminance value of the overlapping image portion from each of the rear view image and the left side view image.

  Here, the luminance value of the overlapping image portion is an average luminance value obtained by averaging the luminance values of all the pixels constituting the overlapping image portion. The luminance value of each pixel is expressed by each luminance value (256 gradations) of RGB components. For example, the luminance value at the time of complete blackout is expressed as R (red) = 0, G (green) = 0, and B (blue) = 0. In addition, the brightness value at the time of complete whiteout is expressed by R (red) = 255, G (green) = 255, and B (blue) = 255. Hereinafter, for the sake of simplicity of description, the luminance value of the image is expressed by 256-level values from 0 (complete blackout) to 255 (complete whiteout).

  The luminance setting unit 220 pays attention to each of the four overlapping image portions, and only one of the luminance values in the overlapping image portion of the overlapping image detected by the luminance detecting unit 180 is within a predetermined range (for example, 60 To 220, the same applies hereinafter). When it is determined that only one of the luminance values is within the predetermined range, the luminance setting unit 220 sets the luminance value of the overlapping image portion to the luminance value within the predetermined range. Specifically, among the overlapping images (one image and another image), the overlapping image portion of the image having the luminance value of the overlapping image portion within a predetermined range is converted by the composite image generating unit 240 into the vehicle peripheral image. Is selected as the image portion to be used when generating the image, thereby setting the luminance value of the overlapping image portion to the luminance value within the predetermined range.

  If the luminance setting unit 220 determines that both of the luminance values in the overlapping image portion of the overlapping image detected by the luminance detecting unit 180 are outside the predetermined range, the luminance value of the overlapping image portion is generated as a composite image. The brightness value of the image set as default in the apparatus 100 is set. Specifically, among the overlapping images (one image and another image), the overlapped image portion of the default image (in this embodiment, the front view image or the rear view image) is converted into a composite image generation unit 240. By selecting as an image part to be used when generating a vehicle peripheral image, the luminance value of the overlapping image part is set to the luminance value in the overlapping image part of the default image.

  When the luminance setting unit 220 determines that both of the luminance values in the overlapping image portion of the overlapping image detected by the luminance detecting unit 180 are within the predetermined range, the luminance difference detecting unit 200 detects the difference in the overlapping image portion. The luminance difference between luminance values is obtained. When the luminance difference obtained by the luminance difference calculation unit 200 is greater than or equal to a predetermined value (for example, 60, the same applies hereinafter), the luminance setting unit 220 uses the luminance value of the overlapping image portion as the luminance value in the overlapping image portion of the overlapping image. Set to the higher brightness value. Specifically, when a composite image generation unit 240 generates a vehicle peripheral image for an overlapping image portion of an image having a higher luminance value of the overlapping image portion among overlapping images (one image and another image). By selecting this as the image portion to be used, the luminance value of the overlapping image portion is set to the higher luminance value.

  The luminance setting unit 220 sets the luminance value of the four overlapping image portions (selects one of the overlapping images of one image or another image), and then sets the luminance among the images overlapping in the overlapping image portion. The non-overlapping image part obtained by removing the overlapping image part from the image having the luminance value different from the luminance value of the overlapping image part set by the unit 220 in the overlapping image part is divided into a plurality of (for example, four) blocks, The luminance values of the four divided blocks are respectively set to the luminance values obtained by the interpolation calculation using the luminance values of the overlapping image portions. A specific method of interpolation calculation will be described later.

  The composite image generation unit 240 sets the luminance value of the overlapping image portion (selects one overlapping image portion of one image or another image) by the luminance setting unit 220 and sets the luminance value of the non-overlapping image portion. After setting the luminance value obtained by the interpolation calculation, based on a plurality of view images input by the image input unit 160, the overlapped image portion adopts the image portion selected by the luminance setting unit 220 and is a composite image. Is generated. In the present embodiment, the composite image generation unit 240 generates a vehicle peripheral image viewed from a virtual viewpoint above the vehicle as a composite image by combining the four view images input by the image input unit 160.

  FIG. 2 is a diagram illustrating an example of a vehicle periphery image according to the present embodiment. As shown in FIG. 2A, the vehicle periphery image is obtained by converting the front view image, the left side view image, the rear view image, and the right side view image input by the image input unit 160 from the viewpoint, respectively. A side image 320, a rear image 340, and a right side image 360 are combined to form overlapping image portions 380, 400, 420, and 440, and the vehicle image 280 is synthesized at the center thereof.

  For the overlapped image portion 380 in which the imaging range overlaps between the front camera and the left side camera, the front image 300 or the left side image 320 is selectively adopted (in the default setting, as shown in FIG. 2B). Front image 300 is employed). In addition, the left side image 320 or the rear image 340 is selectively employed for the overlapping image portion 400 in which the imaging ranges overlap between the left side camera and the rear camera (in the default setting, in FIG. 2B). Thus, the rear image 340 is adopted).

  In addition, the rear image 340 or the right side image 360 is selectively employed for the overlapping image portion 420 in which the imaging range overlaps between the rear camera and the right side camera (the default setting in FIG. 2B). Thus, the rear image 340 is adopted). Further, for the overlapping image portion 440 where the imaging range overlaps between the right side camera and the front camera, the right side image 360 or the front image 300 is selectively adopted (in the default setting, in FIG. 2B). Thus, the front image 300 is adopted).

  Next, the operation of the composite image generation apparatus 100 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of luminance adjustment processing by the composite image generation apparatus 100. In FIG. 3, four view images before the front image 300, the left side image 320, the rear image 340, and the right side image 360 are generated by the viewpoint conversion process are respectively represented as a front view image 300 ′ and a left side view image 320. ', A rear view image 340', and a right side view image 360 '. The vehicle 280 does not exist at the stage of detection and setting of the luminance value, but is illustrated for easy understanding of the positional relationship between the view images.

  FIG. 3 shows an example in which the luminance value is set by paying attention only to the overlapping image portion 400 ′ on the left rear side of the vehicle 280 and the non-overlapping image portion of the rear view image 340 ′. The luminance values are sequentially set for the four overlapping image portions 380 ′, 400 ′, 420 ′, and 440 ′ of the vehicle peripheral image, and the luminance values are set for necessary ones of the four non-overlapping image portions. I do.

  FIG. 3A shows that the luminance value of the overlapping image portion 400 ′ on the left rear side of the vehicle 280 (when the rear view image 340 is adopted) is whiteout (the luminance value is outside the predetermined range). Yes. As a result, the luminance difference between the overlapping image portion 400 ′ and the non-overlapping image portion of the left side view image 320 ′ adjacent thereto is increased.

  At this time, first, the luminance detection unit 180 detects the luminance value in the overlapping image portion 400 ′ from each of the left side view image 320 ′ and the rear view image 340 ′ in which a part of the imaging range overlaps. Next, the luminance setting unit 220 selects one of the luminance value in the overlapping image portion 400 ′ of the left side view image 320 ′ detected by the luminance detecting unit 180 and the luminance value in the overlapping image portion 400 ′ of the rear view image 340 ′. It is determined whether or not only is within a predetermined range. Here, for example, when the luminance setting unit 220 determines that only the luminance value in the overlapping image portion 400 ′ of the left side view image 320 ′ is within the predetermined range, the luminance setting unit 220 sets the luminance value of the overlapping image portion 400 ′ to the predetermined range. The luminance value in the overlapping image portion 400 ′ of the left side view image 320 ′ is set. FIG. 2B shows a state in which the luminance value of the overlapping image portion 400 ′ is set to the luminance value in the overlapping image portion 400 ′ of the left side view image 320 ′ within the predetermined range. The luminance setting process is similarly performed for the other three overlapping image portions 380 ′, 420 ′, and 440 ′.

  Next, the luminance setting unit 220 sets the luminance values of the four overlapping image portions 380 ′, 400 ′, 420 ′, and 440 ′, and then overlaps the images (the rear view image 340 ′ and the left side view image 320 ′). Among these, the overlap image from the rear view image 340 ′ having a brightness value different from the brightness value of the overlap image portion 400 ′ set by the brightness setting unit 220 (the brightness value of the left side view image 320 ′) in the overlap image portion 400 ′. The non-overlapping image part excluding the parts 400 ′ and 420 ′ is divided into four blocks. FIG. 3C shows a state in which the rear view image 340 ′ is divided into four blocks 460, 480, 500, and 520 by the luminance setting unit 220. Note that the block numbers of the four blocks 460, 480, 500, and 520 divided in the non-overlapping image portion of the rear-view image 340 ′ are set to increase as the image position approaches the overlapping image portion 420 from the overlapping image portion 400. Assigned.

The luminance setting unit 220 calculates the luminance values of the four blocks 460, 480, 500, and 520 by interpolation using the luminance values of the overlapping image portions 400 ′ and 420 ′ sandwiching the non-overlapping image portions. Set to each. Specifically, the luminance setting unit 220 sets the luminance value obtained by linear interpolation between the luminance value of the overlapping image portion 400 ′ and the luminance value of the overlapping image portion 420 ′ in each of the blocks 460, 480, 500, and 520. . That is, the luminance value of each block is set to a value obtained by the following equation (1).
Luminance value of block = (luminance value of overlapping image portion 400 ′) + {((luminance value of overlapping image portion 420 ′) − (luminance value of overlapping image portion 400 ′)) × (block number) / (number of blocks + 1) )} ... (1)

For example, it is assumed that the luminance value of the overlapping image portion 400 ′ is 200 and the luminance value of the overlapping image portion 420 ′ is 100. The brightness value of the overlapped image portion 400 ′ is 200 because the brightness value of the overlapped image portion 400 ′ is out of the predetermined range because the rear view image 340 ′ is blown out. The left side view image 320 ′ is the result of selection in the overlapping image portion 400 ′. Similarly, the luminance value of the overlapping image portion 420 ′ is 100 because the rear view image 340 ′ is not overexposed and the luminance value of the overlapping image portion 420 ′ is within a predetermined range. 'Is the result of selection in the overlapped image portion 420'. As described above, when the luminance values of the overlapping image portions 400 ′ and 420 ′ are set, the luminance value of each block is set to a value obtained by the following equation (2). In other words, the luminance value of each block obtained using Equation (2) is 180 for the block 460, 160 for the block 480, 140 for the block 500, and 120 for the block 520. Each is set.
Luminance value of block = 200− {100 × block number / 5} (2)

  It should be noted that the non-overlapping image portion (the overlapping image portions 380 ′ and 400 ′ from the left side view image 320 ′ are included in the left side view image 320 ′ having the luminance value set for the overlapping image portion 400 ′ by the luminance setting unit 220. With respect to (excluded portion), the luminance value of the overlapping image portion 400 ′ and the luminance value of the non-overlapping image portion of the left side image 320 ′ adjacent thereto are based on images originally captured by the same camera, and the luminance difference Is not considered to be too large. Therefore, the luminance setting unit 220 does not set the luminance value by interpolation calculation for the non-overlapping image portion of the left side view image 320 ′.

  Next, the operation of the composite image generation apparatus 100 in the first embodiment will be described. FIG. 4 is a flowchart illustrating an operation example of the composite image generation apparatus 100 according to the first embodiment. Each process in FIG. 4 is started, for example, by receiving an execution request for the vehicle periphery image generation function through an operation of an operation unit (not shown) by the user.

  First, the image input unit 160 displays four images (a front view image, a left side view image, a right side view image, and a rear view image) in which different imaging ranges around the vehicle are captured by the four imaging devices included in the imaging unit 120. Input (step S100). Next, the luminance detection unit 180 out of the four images input by the image input unit 160, one image (one image) and another image (another image) having a common overlapping image portion. Focusing on this combination, the brightness value of the overlapping image portion where the imaging ranges overlap is detected from each of the overlapping images (step S120).

  Next, the luminance setting unit 220 determines whether only one of the luminance values in the overlapping image portion of the overlapping image detected by the luminance detecting unit 180 is within a predetermined range (step S140). If the luminance setting unit 220 determines that only one of the luminance values in the overlapping image portion of the overlapping image is within the predetermined range (YES in step S140), the luminance setting unit 220 determines that the overlapping image portion Is set to a luminance value within a predetermined range (step S160). Thereafter, the process proceeds to step S280.

  On the other hand, when the luminance setting unit 220 does not determine that only one of the luminance values in the overlapping image portion of the overlapping image detected by the luminance detecting unit 180 is within the predetermined range (NO in step S140). The luminance setting unit 220 determines whether or not both of the luminance values in the overlapping image portion of the overlapping image detected by the luminance detecting unit 180 are within a predetermined range (step S180).

  If the luminance setting unit 220 does not determine that both of the luminance values in the overlapping image portion of the overlapping image are within the predetermined range (NO in step S180), both of the luminance values in the overlapping image portion are predetermined. It is out of range. In this case, the luminance setting unit 220 sets the luminance value of the overlapping image portion as a default value for the overlapping image portion (in the present embodiment, the luminance in the overlapping image portion of the front view image 320 ′ or the rear view image 340 ′). A value is set (step S200). Thereafter, the process proceeds to step S280. On the other hand, when the luminance setting unit 220 determines that both of the luminance values in the overlapping image portion of the overlapping image are within the predetermined range (YES in step S180), the luminance difference detecting unit 200 displays the overlapping image. A luminance difference between both luminance values in the overlapping image portion is obtained (step S220).

  Next, the luminance setting unit 220 determines whether or not the luminance difference obtained by the luminance difference calculating unit 200 is equal to or greater than a predetermined value (step S240). If the luminance setting unit 220 determines that the luminance difference obtained by the luminance difference calculating unit 200 is not equal to or greater than the predetermined value (NO in step S240), the luminance setting unit 220 determines the luminance value of the overlapping image portion as the value. The luminance value in the overlapping image portion of the image set as default for the overlapping image portion is set (step S200). Thereafter, the process proceeds to step S280.

  On the other hand, when the luminance setting unit 220 determines that the luminance difference obtained by the luminance difference calculating unit 200 is greater than or equal to a predetermined value (YES in step S240), the luminance setting unit 220 calculates the luminance value of the overlapping image portion. The higher luminance value of the luminance value in the overlapping image portion of one image and the luminance value in the overlapping image portion of the other image is set (step S260). Thereafter, the process proceeds to step S280.

  In step S280, the brightness setting unit 220 determines whether brightness settings have been completed for all four overlapping image portions present in the vehicle peripheral image. If luminance setting unit 220 determines that there is a place where luminance setting has not yet been completed for four overlapping image portions existing in the vehicle peripheral image (NO in step S280), the process transitions to step S120, The processes in steps S120 to S260 are similarly performed on the unprocessed part of the four overlapping image parts. That is, the processes in step S120 to step S260 are repeated until all the luminance settings are completed for the four overlapping image portions.

  If the luminance setting unit 220 determines that the luminance setting has been completed for the four overlapping image portions (YES in step S280), the luminance setting unit 220 determines that the luminance setting unit among the images that overlap in the overlapping image portion. The luminance value is set by interpolation calculation for a non-overlapping image portion obtained by removing the overlapping image portion from an image having a luminance value different from the luminance value of the overlapping image portion set in 220 in the overlapping image portion. That is, the non-overlapping image portion is divided into four blocks, and the luminance values of the divided four blocks are set to the luminance values obtained by the interpolation calculation using the luminance values of the overlapping image portions, respectively (step S300). . Finally, the composite image generation unit 240 generates a vehicle peripheral image viewed from the virtual viewpoint above the vehicle based on the four images input by the image input unit 160 and the luminance values set by the above procedure. (Step S320). Thereby, the composite image generating apparatus 100 ends the processing in FIG.

  As described above in detail, in the first embodiment, the luminance detection unit 180 overlaps the imaging range from each of the overlapping images with respect to four images in which different imaging ranges around the vehicle are captured by the four cameras. When the luminance value of the overlapping image portion is detected and only one of the luminance values is within the predetermined range, the luminance setting unit 200 calculates the luminance value of the overlapping image portion of the overlapping image portion of the overlapping images. The luminance value is set to a luminance value within a predetermined range.

  In this way, even if only one of the luminance values of the overlapping image portion of the overlapping image is outside the predetermined range, for example, the brightness value in a state close to whiteout or blackout, the overlapping image portion As the luminance value, the luminance value within the predetermined range is set. As a result, when the composite image generation unit 240 generates a vehicle peripheral image by combining the four images captured by the four cameras, the composite image generation unit 240 generates a vehicle peripheral image with no overexposure or blackout as much as possible in the overlapping image portion. be able to.

  Further, according to the first embodiment, when both of the luminance values in the overlapping image portion of the overlapping image detected by the luminance detecting unit 180 are within the predetermined range, the luminance difference between the overlapping images is a predetermined value. When this is the case, the luminance setting unit 220 sets the luminance value of the overlapping image portion to a higher luminance value. As a result, when the vehicle peripheral image is generated by synthesizing the four images captured by the four cameras, it is possible to generate a vehicle peripheral image that is brighter and easier to see when the luminance value is within an appropriate range (within a predetermined range).

  Further, according to the first embodiment, after the luminance value of the overlapping image portion is set, the luminance setting unit 200 determines the luminance value of the overlapping image portion set by the luminance setting unit 200 among the overlapping images. Divides the non-overlapping image portion obtained by removing the overlapping image portion from the image having different luminance values in the overlapping image portion into four blocks, and linearly uses the luminance values of the four blocks using the luminance values of the overlapping image portion. The luminance values obtained by the interpolation calculation are respectively set. Note that such brightness setting for non-overlapping image portions is not an essential process. However, if this process is performed, the luminance value of the non-overlapping image portion gradually becomes closer to the luminance value of the overlapping image portion as the image position approaches the overlapping image portion. Accordingly, in addition to preventing whiteout or blackout from occurring in the overlapping image portion, the luminance difference between the overlapping image portion and the non-overlapping image portion adjacent thereto can be reduced as much as possible. That is, there is no sense of discomfort at the boundary between the overlapping image portion and the non-overlapping image portion in the vehicle peripheral image, and the driver can more easily see the vehicle peripheral image.

  In addition, according to the first embodiment, the overlap image is set using a simple process of setting the brightness value of the overlap image portion within the predetermined range as the brightness value of the overlap image portion. As much as possible, the generation of a composite image with no whiteout or blackout is realized. As a result, when a composite image is generated by combining a plurality of images captured by a plurality of cameras, a composite image having no overexposure or blackout as much as possible in a duplicated image portion without performing a complicated calculation as in the prior art. Can be generated.

  Next, a second embodiment of the present invention will be described. FIG. 5 is a block diagram illustrating a configuration example of a vehicle surrounding image generation device including the composite image generation device according to the second embodiment. Also in the second embodiment, the composite image generation apparatus 600 is configured in the same manner as the composite image generation apparatus 100 of FIG. However, the processing content performed by the luminance setting unit 220 ′ is slightly different from that of the luminance setting unit 220 in the first embodiment. Only the differences from the first embodiment will be described below.

  In the second embodiment, when the luminance setting unit 220 ′ determines that only one of the luminance values of the overlapping image portion detected by the luminance detecting unit 180 is within the predetermined range for the overlapping image, When the blur of one image having a luminance value within a predetermined range is larger than the blur of the other image, the other image is adopted as the duplicate image portion, and the luminance of the duplicate image portion of the adopted image is adopted. The value is set to a luminance value within the predetermined range.

  In the present embodiment, information indicating the degree of blur between overlapping images for each overlapping image portion is stored in a storage unit (not shown) of the composite image generating apparatus 100. For example, for an overlapped image portion where the imaging range overlaps between the front camera and the left side camera, the blur condition of the front view image is smaller than the blur condition of the left side view image. In addition, regarding the overlapping image portion where the imaging ranges overlap between the left side camera and the rear camera, the degree of blur of the rear view image is smaller than the degree of blur of the left side view image.

  Further, regarding the overlapping image portion where the imaging ranges overlap between the rear camera and the right side camera, the blur condition of the rear view image is smaller than the blur condition of the left side view image. In addition, regarding the overlapping image portion in which the imaging ranges overlap between the right side camera and the front camera, the blur condition of the front view image is smaller than the blur condition of the left side view image. As described above, it is known from the distance relationship between the overlapping shooting ranges and the cameras which of the overlapping images has a smaller degree of blur.

  For example, the luminance setting unit 220 ′ determines that only the left side view image is within a predetermined range among the luminance values of the overlapped image portion detected by the luminance detection unit 180 for overlapping images (rear view image and left side view image). When the determination is made, the blurring degree of the left side view image having the luminance value within the predetermined range is larger than the blurring degree of the rear view image. Therefore, the rear view image is adopted as the overlapping image part, and the overlapping image part of the adopted rear view image is used. Is set to the luminance value in the overlapping image portion of the left side view image within the predetermined range.

  In addition, when the luminance setting unit 220 ′ determines that both of the luminance values of the overlapping image portion detected by the luminance detecting unit 180 for the overlapping image are within the predetermined range, the luminance setting unit 220 ′ is higher in the luminance value of the overlapping image portion. When the degree of blurring of the image having the lower luminance value is larger than the degree of blurring of the image having the lower luminance value, an image having the lower luminance value is adopted as the overlapping image portion, and the overlapping image of the adopted image is adopted. The luminance value of the part is set to a higher luminance value within the predetermined range.

  For example, for overlapping images (rear view image and left side view image), both of the luminance values of the overlapping image portion detected by the luminance detection unit 180 are within a predetermined range, and the luminance in the overlapping image portion of the left side view image is It is assumed that the value is higher than the luminance value in the overlapping image portion of the rear view image. In this case, the luminance setting unit 220 ′ employs the rear view image with the smaller blurring degree in the overlapping image portion, and sets the luminance value of the overlapping image portion of the adopted rear view image to the left side view having the higher luminance value. Set to the brightness value of the image. When the luminance value in the overlapping image portion of the left side view image is lower than the luminance value in the overlapping image portion of the rear view image, the luminance setting unit 220 ′ employs a rear view image with a smaller degree of blur in the overlapping image portion. Then, the luminance value of the overlapped image portion of the adopted rear view image is set to the luminance value of the rear view image having the higher luminance value.

  FIG. 6 is a flowchart illustrating an operation example of the composite image generation apparatus 600 according to the second embodiment. Also in the second embodiment, the operation procedure of the composite image generation apparatus 600 is the same as the operation procedure of the composite image generation apparatus 100 shown in FIG. However, the processing contents of steps S160 'and S260' are slightly different from those of steps S160 and S260 in the first embodiment (see FIG. 4). Only the differences from the first embodiment will be described below.

  If the luminance setting unit 220 ′ determines that only one of the luminance values of the overlapping image portion detected by the luminance detecting unit 180 is within the predetermined range for the overlapping image (YES in step S140), FIG. In step S160 ′ of 6, the overlapping image portion having the smaller blur condition is adopted among the overlapping image portions of the overlapping images, and the luminance value of the adopted overlapping image portion is set to a luminance value within the predetermined range.

  Further, the luminance setting unit 220 ′ determines that both of the luminance values of the overlapping image portions detected by the luminance detecting unit 180 are within a predetermined range for the overlapping images, and the luminance difference between them is equal to or larger than the predetermined value. If this is the case (YES in step S240), in step S260 ′ of FIG. 6, the one with the smaller degree of blur among the overlapping image portions of the overlapping images is adopted, and the luminance value of the adopted overlapping image portion is set to the predetermined range. The higher one of the two existing luminance values is set.

  As described above in detail, in the second embodiment, the luminance setting unit 220 ′ employs an image with a smaller degree of blur in the overlapping image portion, and sets the luminance value of the overlapping image portion of the adopted image as a predetermined value. The brightness value that is within the range is set. At this time, if both of the luminance values of the overlapping images are within a predetermined range, the higher luminance value is set as the luminance value of the overlapping image portion of the adopted image. As a result, when a vehicle peripheral image is generated by combining a plurality of images captured by four cameras, in addition to generating a vehicle peripheral image with no overexposure or blackout as much as possible, the overlap It is possible to prevent the degree of blur in the image portion from increasing.

  In the first and second embodiments described above, when it is determined that both of the luminance values of the overlapping image portion detected by the luminance detecting unit 180 for the overlapping images are outside the predetermined range, the luminance setting unit 220. Although (220 ′) has described the example in which the luminance value of the overlapped image portion is set to the luminance value of the default image, the present invention is not limited to this. For example, the luminance setting unit 220 (220 ′) may set the luminance value of the overlapping image portion to a luminance value having a smaller luminance difference from the boundary value of the predetermined range.

  In the first and second embodiments described above, when both the luminance values of the overlapping image portions detected by the luminance detecting unit 180 for the overlapping images are within a predetermined range, the luminance difference calculating unit 220 (220 ′ ), The luminance setting unit 220 (220 ′) sets the luminance value of the overlapping image portion to the luminance value of the default image. Although the example to do was demonstrated, it is not limited to this. For example, the luminance setting unit 220 (220 ′) may set the luminance value of the overlapping image portion to a higher luminance value.

  In the first and second embodiments described above, the non-overlapping image portion is divided into four blocks, and the luminance values of the four blocks are determined as the luminance values of the overlapping image portion sandwiching the non-overlapping image portion on both sides. The example of setting the luminance values obtained by the linear interpolation calculation using the above has been described. However, the polynomial interpolation using the luminance values of the four blocks as the luminance values of the overlapping image portions sandwiching the non-overlapping image portions on both sides You may make it set to the luminance value calculated | required by the calculation and the spline interpolation calculation, respectively.

  Further, in the first and second embodiments described above, a luminance value different from the luminance value of the overlapping image portion set by the luminance setting unit 220 (220 ′) among the images overlapping in the overlapping image portion is displayed as the overlapping image. Although the luminance value is always set by interpolation for the non-overlapping image part obtained by removing the overlapping image part from the image included in the part, the present invention is not limited to this. For example, the luminance setting for the non-overlapping image portion may be performed only when the luminance difference between the non-overlapping image portion and the overlapping image portion adjacent thereto is equal to or greater than a predetermined value.

  Further, in the first and second embodiments described above, the luminance setting unit 220 (220 ′) divides the non-overlapping image portion into four blocks, but divides it into less than four blocks. Alternatively, it may be divided into five or more blocks. It is preferable to increase the number of blocks in the non-overlapping image in order to reduce the luminance difference between the overlapping image portion and the non-overlapping image portion.

  In the first and second embodiments described above, the composite image is a vehicle peripheral image, but the present invention is not limited to this. For example, panoramic images obtained by combining images captured by multiple cameras so that they partially overlap in the horizontal direction, or images captured by multiple cameras are combined so that they partially overlap vertically and horizontally It may be a multi-image obtained in this way.

  In addition, each of the first and second embodiments described above is merely an example of a specific example for carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner. It will not be. In other words, the present invention can be implemented in various forms without departing from the spirit or main features thereof.

It is a block diagram which shows the structural example of the synthesized image production | generation apparatus by 1st Embodiment. It is a figure which shows the example of a vehicle periphery image. It is a figure which shows the example of the brightness | luminance adjustment process by a synthesized image generation apparatus. It is a flowchart figure which shows the operation example of the synthesized image generation apparatus by 1st Embodiment. It is a block diagram which shows the structural example of the synthesized image production | generation apparatus by 2nd Embodiment. It is a flowchart figure which shows the operation example of the synthesized image generation apparatus by 2nd Embodiment. It is a figure which shows the example of arrangement | positioning of the camera installed in a vehicle.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100,600 Composite image production | generation apparatus 120 Imaging part 140 Monitor 160 Image input part 180 Luminance detection part 200 Luminance difference detection part 220,220 'Brightness setting part 240 Composite image production | generation part

Claims (6)

An image input unit for inputting a plurality of images in which different imaging ranges around the vehicle are captured by a plurality of cameras;
For a plurality of images input by the image input unit, the luminance value of the overlapped image portion where the imaging range overlaps is detected from one image and another image where the imaging range partially overlaps the one image. A luminance detection unit to
When only one of the luminance value of the one image detected by the luminance detection unit and the luminance value of the other image is within a predetermined range, the luminance value of the overlapping image portion is within the predetermined range. A luminance setting unit for setting a certain luminance value;
After the luminance value of the overlapping image portion is set by the luminance setting unit, based on the plurality of images, the overlapping image portion adopts either the one image or the other image to be a composite image A composite image generation apparatus comprising: a composite image generation unit that generates The composite image generation apparatus according to claim 1,
When only one of the luminance value of the one image detected by the luminance detection unit and the luminance value of the other image is within the predetermined range, the one having the luminance value within the predetermined range When the degree of blur of the image is greater than the degree of blur of the other image, the luminance setting unit adopts the other image as the overlapping image portion, and sets the luminance value of the overlapping image portion of the adopted image, A composite image generating apparatus, wherein the brightness value is set within the predetermined range. The composite image generation apparatus according to claim 1 or 2,
A luminance difference calculation unit for obtaining a luminance difference between the luminance value of the one image and the luminance value of the other image;
When both the luminance value of the one image detected by the luminance detecting unit and the luminance value of the other image are within the predetermined range, the luminance difference obtained by the luminance difference calculating unit is equal to or greater than a predetermined value. The luminance setting unit sets the luminance value of the overlapping image portion to a higher luminance value of the luminance value of the one image and the luminance value of the other image. Image generation device. The composite image generation device according to claim 3,
When both the luminance value of the one image and the luminance value of the other image detected by the luminance detection unit are within the predetermined range, the luminance value of the one image and the luminance value of the other image When the degree of blur of the image having the higher luminance value is larger than the degree of blur of the image having the lower luminance value, the luminance setting unit selects the image having the lower luminance value in the overlapping image portion. A composite image generating apparatus that adopts and sets the luminance value of the overlapping image portion of the adopted image to the higher luminance value. In the synthetic image generation device according to any one of claims 1 to 4,
After the brightness value of the overlapped image portion is set by the brightness setting unit, the brightness setting unit is configured to set the brightness of the overlapped image portion set by the brightness setting unit among the one image and the other image. A non-overlapping image portion obtained by removing the overlapping image portion from an image having a luminance value different from the value in the overlapping image portion, and dividing the plurality of blocks into luminance values of the overlapping image portion. Set the brightness value obtained by interpolation calculation using
The composite image generation device generates a composite image based on the plurality of images after the brightness value of the non-overlapping image portion is set by the brightness setting unit. A first step of inputting a plurality of images in which different imaging ranges around the vehicle are captured by a plurality of cameras;
For the plurality of images input in the first step, the luminance value of the overlapping image portion where the imaging range overlaps from one image and the other image where the imaging range partially overlaps the one image. A second step of detecting;
When only one of the luminance value of the one image and the luminance value of the other image detected by the second step is within a predetermined range, the luminance value of the overlapping image portion is set within the predetermined range. A third step of setting the brightness value at
After the luminance value of the overlapped image portion is set in the third step, the overlapped image portion is synthesized using either the one image or the other image based on the plurality of images. A synthetic image generating method comprising: a fourth step of generating an image.