JP2020127077A - Image synthesizing device and control method - Google Patents

Abstract

To provide an image synthesizing device that generates a synthetic image that more appropriately shows a rear side of a vehicle even when an image is defective.SOLUTION: An image synthesizing device 10 includes an acquisition unit 21 that acquires two images, a detection unit 23 that performs a process of detecting a defect in the image acquired by the acquisition unit 21, and a synthesis unit that generates a synthetic image by arranging two images in a predetermined arrangement direction and placing the images in a part where the images overlap and combining the two images with a boundary line of the predetermined position as a boundary when no defect is detected by the detection unit 23 in either of the two images, and generates a synthetic image by arranging at least the other of the two images with a boundary line moved from the predetermined position in one direction in the arrangement direction as a boundary when a defect in only one of the two images is detected by the detecting unit 23, and the one direction is a direction in which, as the boundary line is moved to the one direction in the arrangement direction, the other image is included in the synthetic image more largely.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image synthesizing device and a control method.

There is an image display device that photographs the rear of the vehicle with a plurality of cameras provided in the vehicle and displays the photographed images. For example, there is a so-called electronic mirror for a vehicle, which is an image display device that allows a driver to visually recognize an image of the rear of a vehicle.

This image display device captures images of the rear left and right sides of the vehicle taken by a camera fixed near the left and right doors of the vehicle and a camera fixed near the rear bumper or trunk hood of the vehicle. The image of the rear of the center of the vehicle is acquired. Then, by combining these images, one image showing the left rear, the center rear, and the right rear is generated, and the generated image is displayed on the room mirror type display device.

Conventionally, when there is an abnormality in a part of the captured image of the rear side taken by the rear side camera, a region corresponding to the part of the abnormality is cut out from another image at a position corresponding to the part and the geometrical shape is cut out. There is a device that synthesizes and displays converted images (see, for example, Patent Document 1).

JP, 2008-107573, A

However, in the image synthesized by the device described in Patent Document 1, since the image of the portion corresponding to the abnormality is cut out from the fisheye camera image, the resolution is low, and the image is mainly used for parking, and the frame is used. Since the rate is often small, there is a problem that an appropriate rear image cannot be provided to the user.

Therefore, it is an object of the present invention to provide an image synthesizing apparatus that generates a synthetic image that more appropriately shows the rear of a vehicle even if the image has a defect.

An image synthesizing apparatus according to an aspect of the present invention includes an acquisition unit that performs an acquisition process for acquiring at least two images in which a common target is captured under different imaging conditions, and an image acquired by the acquisition unit. If a defect is not detected by either the detection unit that performs a detection process for detecting a defect of the two images or the two images, the two images are arranged in a predetermined arrangement direction and partially overlap each other. A synthesizing unit for generating a synthesized image by synthesizing the two images with the boundary line at a predetermined position as a boundary, the synthesizing unit further including one of the two images. When a defect is detected only in the image by the detection unit, at least the other image of the two images is arranged and the boundary line moved from the predetermined position to the one direction in the arrangement direction is removed. The composite image is generated as a boundary, and the one direction is a direction in which the other image is included in the composite image to a greater extent as the boundary line is moved to the one direction in the arrangement direction.

Note that these comprehensive or specific aspects may be realized by a recording medium such as a system, a method, an integrated circuit, a computer program or a computer-readable CD-ROM, and the system, the method, the integrated circuit, the computer program. It may be realized by any combination of the recording medium and the recording medium.

The image synthesizing apparatus according to the present invention can generate a synthetic image that more appropriately shows the rear of the vehicle even if the image has a defect.

FIG. 1 is a schematic diagram showing an example of a vehicle equipped with a camera that acquires an image to be combined by an image combining device according to an embodiment. FIG. 2 is an explanatory diagram showing an image before combining and an image after combining by the image combining device according to the embodiment. FIG. 3 is a block diagram showing a functional configuration of the image synthesizing apparatus according to the embodiment. FIG. 4 is an explanatory diagram showing an image acquired by the acquisition unit according to the embodiment and geometrically transformed by the synthesis unit. FIG. 5 is an explanatory diagram showing a first example of an image combined by the combining unit according to the embodiment. FIG. 6 is an explanatory diagram showing a second example of the image combined by the combining unit according to the embodiment. FIG. 7 is an explanatory diagram showing a first example of the interpolation image generated by the synthesizing unit according to the embodiment. FIG. 8 is an explanatory diagram showing a second example of the interpolation image generated by the synthesizing unit according to the embodiment. FIG. 9 is an explanatory diagram showing a third example of an image combined by the combining unit according to the embodiment. FIG. 10 is an explanatory diagram showing a third example of the interpolation image generated by the synthesizing unit according to the embodiment. FIG. 11 is an explanatory diagram showing a fourth example of the interpolation image generated by the synthesizing unit according to the embodiment. FIG. 12 is a flowchart showing the processing performed by the image synthesizing apparatus according to the embodiment.

An image synthesizing apparatus according to an aspect of the present invention includes an acquisition unit that performs an acquisition process for acquiring at least two images in which a common target is captured under different imaging conditions, and an image acquired by the acquisition unit. If a defect is not detected by either the detection unit that performs a detection process for detecting a defect regarding the above or the two images, the two images are arranged in a predetermined arrangement direction and partially overlap each other. A synthesizing unit for generating a synthesized image by synthesizing the two images with the boundary line at a predetermined position as a boundary, the synthesizing unit further including one of the two images. When a defect is detected only in the image by the detection unit, at least the other image of the two images is arranged and the boundary line moved from the predetermined position to the one direction in the arrangement direction is removed. The composite image is generated as a boundary, and the one direction is a direction in which the other image is included more in the composite image as the boundary line is moved to the one direction in the arrangement direction.

According to the above aspect, when only one of the two images to be acquired has a defect, the image composition device moves the boundary line such that the image without the problem is included in the composite image in a larger size. As a result, the non-defective image will have a larger area in the composite image. The user can more appropriately recognize the rear of the vehicle by visually observing the composite image in which the non-defective image occupies a larger area than when the non-defective image is displayed without moving the boundary line. You can Therefore, the image synthesizing apparatus can generate a synthetic image that more appropriately shows the rear of the vehicle even if the image has a defect.

For example, the acquisition unit performs the acquisition process for acquiring the two images by using two cameras that capture the rear of the vehicle and that are mounted at different positions of the vehicle. In the other image of the two images arranged by the combining unit, a part of the vehicle is shown at a position advanced from the predetermined position in the one direction, and the combining unit is The boundary line is moved within a range in which the boundary line does not overlap a part of the vehicle.

According to the above aspect, the image synthesizing apparatus generates a synthetic image in which a part of the vehicle is not reflected when the acquired image includes a part of the vehicle. If a part of the vehicle is reflected in the composite image, the rear image becomes an unnatural image, which may cause confusion for the user. Therefore, the image synthesizing apparatus can generate a synthetic image that more appropriately shows the rear of the vehicle without disturbing the user even if the image has a defect.

For example, the acquisition unit performs the acquisition processing for acquiring the two images by using cameras mounted at different positions of the vehicle for capturing the rear of the vehicle, and the image combining device further includes: A distance acquisition unit that acquires distance information from the vehicle to the object, which is obtained by a sensor that detects an object behind the vehicle, and the combining unit only includes one of the two images. In the case where a defect is detected by the detection unit, a first interpolation image showing a state behind the vehicle is further generated based on the distance information acquired by the distance acquisition unit to generate the two images. One of the two images is arranged at a position to be arranged, and the arranged first interpolation image and the other image of the two images are combined using the boundary line.

According to the above aspect, the image compositing apparatus arranges the interpolated image (first interpolated image) at the position where the defective image of the two acquired images should be displayed to generate the composite image. The first interpolation image is an image generated based on the distance to the object behind the vehicle that is actually acquired using the sensor. Therefore, the user can know the state behind the vehicle by visually recognizing the interpolated image instead of the defective image. In this way, the image synthesizing apparatus can generate a synthetic image that more appropriately shows the rear of the vehicle even if the image has a defect.

For example, the sensor includes a plurality of the sensors, and the distance acquisition unit acquires the distance information obtained by a sensor selected among the plurality of sensors according to a vehicle speed or a shift position of the vehicle. ..

According to the above aspect, the image synthesizing apparatus generates the interpolated image using the sensor selected from the plurality of sensors mounted on the vehicle according to the vehicle speed or the shift position. In this way, the image synthesizing apparatus can generate a synthetic image that more appropriately shows the rear of the vehicle by using the image generated by using the appropriate sensor even if the image has a defect.

For example, when the detecting unit detects a defect in only one of the two images, the combining unit generates a second interpolation image indicating the details of the defect. The one of the two images is arranged at a position where the one of the two images should be arranged, and the arranged second interpolation image and the other image of the two images are combined using the boundary line. ..

According to the above aspect, the image synthesizing apparatus arranges the interpolated image (second interpolated image) indicating the content of the defect at the position where the defective image of the two images to be acquired should be displayed. Generate a composite image. The second interpolation image is an image showing the content of the defect. Therefore, the user can know the content of the defect by visually recognizing the interpolated image instead of the image having the defect, and can take action for coping. In this way, the image synthesizing apparatus can provide information for coping with the deficiency even when the image has a defect, and as a result, can generate a combined image that more appropriately shows the rear of the vehicle.

An image synthesizing apparatus control method according to an aspect of the present invention is an image synthesizing apparatus control method, and is an acquisition process for obtaining at least two images of a common target under different imaging conditions. And a detection step for performing a detection process for detecting a defect in the image acquired in the acquisition step, and when no defect is detected in the detection step in either of the two images, Two images are arranged in a predetermined arrangement direction so as to partially overlap each other, and the two images are combined with the boundary line of the predetermined position as a boundary to generate a combined image. Then, when a defect is detected in only one of the two images in the detecting step, at least the other image of the two images is arranged, and the arrangement is performed from the predetermined position. The boundary line moved to one direction in the direction is used as a boundary to generate the composite image, and the one direction is set to the composite image as the boundary line is moved to the one direction in the arrangement direction. This is the direction in which the other image is included more largely.

Thereby, the same effect as that of the image synthesizing apparatus is obtained.

Note that these comprehensive or specific aspects may be realized by a recording medium such as a system, a method, an integrated circuit, a computer program or a computer-readable CD-ROM, and the system, the method, the integrated circuit, the computer program. Alternatively, it may be realized by any combination of recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that each of the embodiments described below shows a comprehensive or specific example. Numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of constituent elements, steps, order of steps, and the like shown in the following embodiments are examples, and are not intended to limit the present invention. Further, among the constituent elements in the following embodiments, the constituent elements that are not described in the independent claims showing the highest concept are described as arbitrary constituent elements.

(Embodiment)
In the present embodiment, an image synthesizing apparatus that generates a synthetic image that more appropriately shows the rear of the vehicle even if the image is defective will be described.

FIG. 1 is a schematic diagram showing an example of a vehicle 1 according to the present embodiment. FIG. 2 is an explanatory diagram showing an image before combining and an image after combining by the image combining device 10 according to the present embodiment.

As shown in FIG. 1, the vehicle 1 includes an image synthesizing device 10, cameras 12, 13, and 14, and an electronic mirror 40. In the following description, unless otherwise specified, the front-rear, left-right direction is the direction when the traveling direction of the vehicle 1 is the front direction, which can be said to be the front-rear direction and the left-right direction for the driver of the vehicle 1.

The camera 12 is a camera that is fixed near the left side door of the vehicle 1 and takes an image of the left rear side of the vehicle 1. The camera 12 photographs the left rear side of the vehicle 1 and then reverses the image horizontally for an electronic mirror to generate an image. The image generated by the camera 12 is also referred to as a left rear image. The camera 13 is a camera that is fixed near the right side door of the vehicle 1 and takes an image of the right rear side of the vehicle 1. The camera 13 photographs the right rear side of the vehicle 1 and then reverses the image horizontally for an electronic mirror to generate an image. The image generated by the camera 13 is also referred to as a right rear image.

The camera 14 is a camera that is fixed near a rear bumper or a trunk hood of the vehicle 1 and takes an image of the rear center of the vehicle 1. The camera 14 images the rear side of the center of the vehicle 1 and then reverses it horizontally for an electronic mirror to generate an image. The image generated by the camera 14 is also referred to as a center rear image. Note that the left-right inverted image generation processing may be performed not by the cameras 12, 13, and 14 but by the image synthesizing apparatus 10 (acquisition unit 21 described later) in the subsequent stage.

Part of the shooting ranges of the cameras 12 and 14 overlap, and part of the shooting ranges of the cameras 13 and 14 overlap. Therefore, a common target appears in a part of the left rear image and the center rear image. In addition, a common target appears in part of the right rear image and the center rear image.

Further, a part of the vehicle 1, more specifically, a part of the left door of the vehicle 1 is shown on the right side of the image captured by the camera 12 and horizontally inverted. Further, a part of the vehicle 1, more specifically, a part of the right door of the vehicle 1 is shown on the left side of the image captured by the camera 13 and generated by horizontally reversing.

Each of the cameras 12, 13 and 14 generates an image by capturing images under mutually different capturing conditions. Specifically, each of the cameras 12, 13 and 14 is arranged at a position different from each other and oriented in different directions, and acquires an image at, for example, 60 fps. The optical characteristics of the optical systems of the cameras 12, 13 and 14 may be different.

The electronic mirror 40 is an image display device that displays a rear image 50 that is an image of the rear of the vehicle 1. The electronic mirror 40 displays an image based on the video signal output by the image synthesizer 10. The electronic mirror 40 can be used as a substitute for a conventional room mirror that projects the rear of the vehicle 1 by utilizing the reflection of light.

The image synthesizing device 10 is a device that synthesizes the images generated by the cameras 12, 13, and 14 to generate an image displayed by the electronic mirror 40. An example of images generated by the cameras 12, 13 and 14 is shown in FIG. 2 as images 51, 52 and 53 (also referred to as “image 51 etc.”). Images 51, 52 and 53 are taken substantially simultaneously by cameras 12, 13 and 14, respectively. Note that “substantially simultaneous” includes a case of being strictly simultaneous, and may have a time difference that can be regarded as simultaneous by a human (for example, a time difference similar to an imaging interval). Further, the images 51, 52 and 53 are images generated by being photographed by the cameras 12, 13 and 14 and horizontally inverted for use in the electronic mirror.

As described above, each of the cameras 12, 13 and 14 images one target under different imaging conditions, so that the image synthesizing device 10 generates the images generated by the cameras 12, 13 and 14, respectively. A natural rear image 50 cannot be obtained only by arranging them at positions to be arranged and simply connecting them.

Here, the “position to be arranged” is the position on the left side in the rear image 50 for the image generated by the camera 12 flipping horizontally, and the rear image 50 for the image generated by flipping the camera 13 horizontally. This is the right side position in the inside, and is the center position in the left and right in the rear image 50 for the image generated by the camera 14 flipping left and right.

The natural rear image 50 is the same image as the rear view of the vehicle 1 as seen by the driver of the vehicle 1 through a conventional rearview mirror (so-called physical mirror). Specifically, in images that are simply stitched together, duplication or disappearance may occur near the stitching boundary. Duplication means that one object appears twice in the image, and disappearance means that what actually exists does not appear in the image.

Therefore, the image composition device 10 performs appropriate processing on these images according to the images generated by the cameras 12, 13, and 14 and then joins the images together, so that a natural rearward image in which duplication or disappearance is suppressed is suppressed. 50 is generated and displayed on the electronic mirror 40.

Note that the three images generated by the cameras 12, 13 and 14 may generate the rearward image 50 by combining the adjacent two of the three images and then combining the remaining one with the combined image. Further, the rear image 50 may be generated by simultaneously synthesizing the three images generated by the cameras 12, 13, and 14 in parallel.

The functions and processing of the image synthesizing device 10 will be described below.

FIG. 3 is a block diagram showing a functional configuration of the image synthesizing device 10 according to the present embodiment. Here, a case will be described as an example in which the image synthesizing apparatus 10 acquires and synthesizes a first image and a second image that are at least two captured images of a common target. The two images may be a left rear image and a center rear image, or a right rear image and a center rear image.

Hereinafter, a case where the first image is the left rear image and the second image is the central rear image will be described in detail. In this case, the image synthesizing apparatus 10 acquires the first image and the second image at, for example, 60 fps with the cameras 12 and 14, and at each acquisition, generates the composite image of the first image and the second image to generate the electronic mirror 40. To display.

As shown in FIG. 3, the image synthesizing device 10 includes an acquisition unit 21, a distance acquisition unit 22, a detection unit 23, a synthesizing unit 24, and a display control unit 25. Each of the above components may be realized by a processor executing a predetermined program using a memory or the like, or may be realized by dedicated hardware.

The acquisition unit 21 is a processing unit that performs an acquisition process for acquiring at least two images in which a common target is captured under different imaging conditions. Specifically, the acquisition unit 21 acquires image data generated by the two cameras 12 and 14 mounted on the vehicle 1 by imaging.

Here, in the acquisition process, the image may not be normally acquired. For example, when the photographing operation of the camera 12 or 14 is not normal, when an image cannot be obtained from the camera 12 or 14, or when the object cannot be normally photographed because some object is attached to the lens of the camera 12 or 14. , Images are not normally acquired.

When the acquisition process is not normally performed, the acquisition unit 21 obtains information indicating that the camera 12 or 14 has a defect (for example, a predetermined error message) or an image having a defect in the acquisition process. An image having a defect is an image in which all pixels have a predetermined color (for example, black or green), the same image as the immediately previous image while traveling, or a predetermined number (for example, 2) before in time. The same image as the image of. Obtaining the same image as the immediately previous image is a phenomenon that can occur when an object (water droplet, mud, snow, dew, or the like) is attached to the camera 12 or 14. Further, the same image as the previous image by a predetermined number in time is obtained when the image acquired by the camera 12 or 14 is sequentially stored in the predetermined number of memory areas and then sequentially read out. This is a phenomenon that can occur when some of the predetermined number of memory areas are defective.

The distance acquisition unit 22 is a processing unit that acquires a distance to an object behind the vehicle 1 by using a sensor 22A that senses an object behind the vehicle 1. The sensor 22A is a distance measuring sensor that senses at least a distance to an object behind the vehicle 1. The sensor 22A is, specifically, a sonar, a radar, a lidar (LIDAR), a TOF (Time of Flight) sensor, or the like, and the vehicle 22 may include a plurality of sensors 22A. The distance acquisition unit 22 provides the distance to the object acquired by the detection by the sensor 22A to the combining unit 24.

Note that the distance acquisition unit 22 may provide the synthesis unit 24 with the distance and the direction in which the object exists. The direction may be detected by the sensor 22A, or may be determined from the mounting position and direction of the sensor 22A.

When the vehicle 1 is provided with a plurality of sensors 22A, the distance acquisition unit 22 outputs the distance information obtained by the sensor 22A selected from the plurality of sensors 22A according to the vehicle speed or the shift position of the vehicle 1. It may be acquired and provided to the synthesis unit 24. The vehicle speed or shift position of the vehicle 1 can be acquired from an ECU (Electronic Control Unit) included in the vehicle 1.

For example, the distance acquisition unit 22 may select the radar when the vehicle speed of the vehicle 1 is high, while selecting the sonar when the vehicle speed is low. Further, the radar may be selected when the shift position of the vehicle 1 is high, while the sonar may be selected when the shift position is low or the reverse position.

In addition, the distance acquisition unit 22 may select the sensor 22A according to the weather. For example, when it is raining, the sensor 22A other than the sonar may be selected. This is because it is known that sonar detection of an object becomes unsuitable in rainy weather. In addition, in order to detect that the weather is rainy, it can be detected by operating the wiper, for example.

The detection unit 23 is a processing unit that performs detection processing for detecting a defect in the image acquired by the acquisition unit 21. The “defective image to be acquired” means that a normal image is not acquired, and is a concept including a case where the acquired image has a defect and a case where the image is not normally acquired.

Specifically, the detection unit 23 detects that there is a defect when all the pixels of the image acquired by the acquisition unit 21 are images of a predetermined color. In addition, if the image acquired by the acquisition unit 21 is the same as the previous image while traveling, or the same image as the previous image by a predetermined number in terms of time, there is a problem. To detect. Further, the detection unit 23 detects that there is a defect when the acquisition unit 21 obtains error information from the camera 12 or 14. The detection unit 23 acquires the vehicle speed from the ECU included in the vehicle 1 in order to determine whether the vehicle 1 is traveling.

The combining unit 24 is a processing unit that generates a combined image by combining the images acquired by the acquisition unit 21. The synthesizing unit 24 sets a boundary line for each of the two images acquired by the acquisition unit 21, and performs image processing for cutting and joining the two images by the boundary line. Specifically, the synthesizing unit 24 performs geometric conversion processing, which is image processing for suppressing a difference in image capturing conditions when two images are captured, and then performs image processing for joining the two images. To do. The geometric conversion process specifically includes a process of translation, enlargement, reduction, rotation, shearing, or a combination thereof.

The synthesizing unit 24 arranges the two images in a predetermined arranging direction and partially when the acquiring unit 21 acquires the two images and no defect is detected by the detecting unit 23 in either of the acquired two images. Are arranged at positions overlapping with each other, and two images are combined with a boundary line at a predetermined position as a boundary to generate a combined image.

Further, when the detecting unit 23 detects a defect in only one of the two images, the synthesizing unit 24 arranges at least the other image of the two images and arranges them from a predetermined position. A composite image is generated with the boundary line moved in one direction in the direction as a boundary. Here, the one direction is a direction in which the other image is included in the combined image to a greater extent as the boundary line is moved to the one direction in the arrangement direction.

Further, when a defect is detected in only one of the two images, the synthesizing unit 24 further displays an image showing a state behind the vehicle 1 based on the distance information acquired by the distance acquiring unit 22. (Also referred to as an interpolated image, particularly also referred to as a first interpolated image) is generated and arranged at a position where one of the two images should be arranged, and the arranged interpolated image and the other of the two images are arranged. The image and the image may be combined using a boundary line.

Further, instead of the above first interpolated image, an image showing the contents of the defect (also called an interpolated image, particularly also called a second interpolated image) is generated and one of the two images is arranged. It may be arranged at an appropriate position, and the arranged interpolation image and the other image of the two images may be combined using a boundary line. The second interpolation image may be displayed together with the first interpolation image.

Here, when moving the boundary line, the combining unit 24 moves the boundary line within a range in which the boundary line does not overlap a part of the vehicle 1 in the combined image. By doing so, it is possible to prevent a part of the vehicle 1 from being reflected in the combined image combined by the combining unit 24.

The display control unit 25 is a processing unit that controls the display of the combined image generated by the combining unit 24. The display control unit 25 outputs the pixel value of each pixel included in the composite image to the electronic mirror 40 so that the electronic mirror 40, which is a display device, displays the composite image.

Hereinafter, the image synthesizing process performed by the image synthesizing apparatus 10 will be specifically described with reference to examples of images.

Here, a case where the combination of the left rear image and the center rear image and the combination of the right rear image and the center rear image are performed in parallel will be described as an example. In this example, it can be said that the left rear image, the right rear image, and the center rear image are combined at once.

FIG. 4 is an explanatory diagram showing an image acquired by the acquisition unit 21 according to the present embodiment and geometrically transformed by the synthesis unit 24. More specifically, FIG. 4 shows an image 61 that is an example of a left rear image of the vehicle 1, an image 62 that is an example of a right rear image, and an image 63 that is an example of a center rear image. There is. The images 61, 62, and 63 are examples of images that have been imaged and horizontally flipped by the cameras 12, 13 and 14, respectively, and that have undergone geometric conversion processing by the synthesizing unit 24.

Further, FIG. 4 shows a boundary line when the images 61, 62 and 63 are combined. Specifically, the images 61 and 62 show a boundary line 65A and a boundary line 66A, respectively. The image 63 also shows the boundary lines 65B and 66B. The boundary line 66C shown in the image 63 indicates the position of the boundary line moved from the boundary line 66B, as described later.

Here, the imaging conditions at the time of imaging the images 61, 62, and 63 are different from each other. The imaging condition includes the position of the camera or the optical characteristics of the optical system. Specifically, the positions of the cameras (that is, the cameras 12, 13, and 14) that have captured the images are at least different.

Therefore, the images 61, 62, and 61 differ in the size, shape, or angle of the subject shown in the images. Therefore, it is difficult to obtain a natural rear image by joining the image 61 and the image 63 together without performing any conversion, and joining the image 63 and the image 62 together. Therefore, the synthesizing unit 24 performs geometric transformation processing on the images 61, 62, and 63, and then joins these to generate a synthetic image.

FIG. 5 is an explanatory diagram showing a combined image 60A that is a first example of an image combined by the combining unit 24 according to the present embodiment. The combined image 60A is an example of an image combined by the combining unit 24 when the image acquired by the acquisition unit 21 has no defect.

The composite image 60A includes images 61A, 62A and 63A. Further, there is a boundary line 65 at the boundary between the images 61A and 63A, and a boundary line 66 at the boundary between the images 62A and 63A.

The image 61A is an image showing a part on the left side of the boundary line 65A in the image 61 on which the geometric conversion process is performed.

The image 62A is an image showing a part on the right side of the boundary line 66A in the geometrically transformed image 62.

The image 63A is an image showing a part on the right side of the boundary line 65B and on the left side of the boundary line 66B in the geometrically transformed image 63.

When the composite image 60A is displayed on the electronic mirror 40, the user can recognize the composite image 60A from the rear of the vehicle 1, more specifically, the area from the left rear to the center rear, and further to the right rear.

FIG. 6 is an explanatory diagram showing a combined image 60B that is a second example of an image combined by the combining unit 24 according to the present embodiment. The combined image 60B is an example of an image that is combined by the combining unit 24 when the right rear image has a defect among the images acquired by the acquisition unit 21.

The combined image 60B includes images 61A and 63B and an interpolated image 72. Further, there is a boundary line 65 at the boundary between the images 61A and 63B, and a boundary line 66 at the boundary between the interpolated image 72 and the image 63B.

The image 61A is the same as the image 61A in FIG.

The image 63B is a part of the image 63 (see FIG. 4). More specifically, the boundary line 65C in the image 63 is left as it is, and the boundary line 66C is located between the boundary line 65B and the boundary line 66C when the boundary line 66C is set to a position moved rightward from the boundary line 66B. Is the image 63B in FIG. The right direction here corresponds to “one direction”. This is because the image 63B is included in the combined image 60B to a greater extent as the boundary line is moved to the right in the direction in which the image 63B and the interpolated image 72 are arranged (the horizontal direction on the paper surface).

The interpolated image 72 is an interpolated image generated by the synthesizing unit 24, and is an image that is generated based on the distance information acquired by the distance acquiring unit 22 and that shows a state behind the vehicle 1. Hereinafter, the interpolation image will be described with two examples.

FIG. 7 is an explanatory diagram showing an interpolated image 72 that is a first example of the interpolated image (first interpolated image) generated by the combining unit 24 according to the present embodiment. The interpolated image 72 is a first example of the interpolated image generated by the combining unit 24 when the right rear image acquired by the camera 13 has a defect, for example.

The interpolated image 72 is an image of the vehicle 1 and its surroundings (more specifically, a region on the right rear side of the vehicle 1) viewed from above, that is, a so-called top view image. In the interpolated image 72, the upward direction on the paper corresponds to the front of the vehicle 1, and the downward direction on the paper corresponds to the rear of the vehicle 1.

In the interpolated image 72, the vehicle C1 shows the vehicle 1 which is the own vehicle, and the vehicles C2 and C3 show the vehicles existing on the right rear side of the vehicle 1. The vehicles C2 and C3 are shown such that the relative position and distance with respect to the vehicle C1 are correctly represented on the interpolation image 72 based on the distance to the object behind the vehicle 1 acquired by the sensor 22A.

Further, in the interpolated image 72, one or more lines L indicating the standard of the distance in the front-rear direction of the vehicle 1 are drawn. The lines L are drawn, for example, at intervals corresponding to 1 m in reality.

By visually recognizing the interpolation image 72, the user can grasp the positional relationship and the relative distance between the vehicles C2 and C3 and the vehicle C1 which is the own vehicle from the top view.

The synthesizing unit 24 may further draw a mark C2A for alerting near the image of the vehicle C2 located within a predetermined distance from the vehicle 1. When the user visually recognizes the mark C2A, the user can recognize that the vehicle C2 located near the mark C2A is within a predetermined distance from the vehicle 1, which is the own vehicle, and can be noted when driving.

FIG. 8 is an explanatory diagram showing an interpolated image 74 that is a second example of the interpolated image generated by the synthesizing unit 24 according to the present embodiment. The interpolated image 74 is a second example of the interpolated image generated by the combining unit 24 when the right rear image acquired by the camera 13 has a defect, for example.

The interpolated image 74 is an image when the right rear side of the vehicle 1 is viewed from the position of the electronic mirror 40. The interpolated image 74 is an interpolated image corresponding to the image arranged at that position if the image acquired by the acquisition unit 21 is not defective.

In the interpolated image 74, vehicles C2 and C3 indicate vehicles that are located on the right rear side of the vehicle 1. Since the interpolated image 74 is an image when the right rear is viewed from the vehicle 1 as described above, the vehicle C1 indicating the own vehicle is not shown.

Further, in the interpolated image 74, one or more lines L indicating the standard of the distance in the front-rear direction of the vehicle 1 are drawn. The lines L are drawn, for example, at intervals corresponding to 1 m in reality.

Since the interpolated image 74 is an image looking backward from the position of the electronic mirror 40, the user can recognize the interpolated image 74 from the same viewpoint as the non-defective image displayed on the electronic mirror 40. With this, the user can grasp the positional relationship and the relative distance between the vehicles C2 and C3 and the vehicle 1, which is the own vehicle, by visually recognizing the interpolation image 74.

Note that, in the interpolated image 74 as well, as in the interpolated image 72, a mark C2A for calling attention may be further drawn.

Further, in FIGS. 7 and 8, the mark C2A for alerting is drawn on the vehicle C2 within a predetermined distance from the vehicle 1, but the present invention is not limited to this, and the content of the defect with respect to the image where the defect is detected The interpolation image (second interpolation image) shown may be arranged so as to be superimposed on the interpolation images 72 and 74. The content of the defect is, for example, a missing image signal, image sticking (unchanged), foreign matter adhesion, or the like, and the second interpolation image is formed by the wording or mark indicating these defects.

Further, in FIG. 7 and FIG. 8, the second interpolated image is not limited to the configuration in which the second interpolated image is arranged so as to be superimposed on the interpolated images 72 and 74. If the defect is caused by a failure, only the second interpolation image may be drawn.

FIG. 9 is an explanatory diagram showing a combined image 60C that is the third example of the image combined by the combining unit 24 according to the present embodiment. The combined image 60C is an example of an image combined by the combining unit 24 when there is a defect in the central rear image among the images acquired by the acquisition unit 21.

The combined image 60C includes images 61C and 62C and an interpolated image 76. Further, there is a boundary line 65 at the boundary between the image 61C and the interpolation image 76, and a boundary line 66 at the boundary between the image 62C and the interpolation image 76.

The image 61C is a part of the image 61 (see FIG. 4). More specifically, the boundary line 65 is set at a position moved rightward from the boundary line 65A in FIG. The right direction here corresponds to “one direction”. This is because the image 61C is included in the combined image 60C to a greater extent as the boundary line is moved to the right in the direction in which the image 61C and the interpolated image 76 are arranged (the horizontal direction on the paper surface).

The image 62C is a part of the image 62 (see FIG. 4). More specifically, the boundary line 66 is set at a position moved leftward from the boundary line 66A in FIG. The left direction here corresponds to “one direction”. This is because the image 62C is included in the combined image 60C to a greater extent as the boundary line is moved to the left in the direction in which the image 62C and the interpolated image 76 are arranged (the horizontal direction on the paper surface).

The interpolated image 76 is an interpolated image generated by the synthesizing unit 24, and is an image that is generated based on the distance information acquired by the distance acquisition unit 22 and that shows a state behind the vehicle 1. Hereinafter, the interpolation image will be described with two examples.

FIG. 10 is an explanatory diagram showing an interpolated image 76 that is a third example of the interpolated image generated by the synthesizing unit 24 according to the present embodiment. The interpolated image 76 is a first example of the interpolated image generated by the synthesizing unit 24 when, for example, the center rear image acquired by the camera 14 has a defect.

The interpolated image 76 is an image when the vehicle 1 and its surroundings (more specifically, a region behind the center of the vehicle 1) is viewed from above, that is, a so-called top view image. The interpolated image 76 is the same image as the interpolated image 72 (see FIG. 7), but shows a vehicle existing in the area behind the center of the vehicle 1 due to a defect in the image captured in the rear of the center. It is a thing.

In the interpolated image 76, similarly to the interpolated image 72 (see FIG. 7), the vehicle C1 indicates the vehicle 1 which is the own vehicle, and the vehicles C4 and C5 indicate the vehicles existing behind the center of the vehicle 1. ing. Further, in the interpolated image 76, one or more lines L indicating the standard of the distance in the front-rear direction of the vehicle 1 are drawn. The lines L are drawn, for example, at intervals corresponding to 1 m in reality.

Similarly to the interpolation image 72 (see FIG. 7), the user visually recognizes the interpolation image 76 to grasp the positional relationship and the relative distance between the vehicles C4 and C5 and the vehicle C1 which is the own vehicle from the top view. You can

FIG. 11 is an explanatory diagram showing an interpolated image 78 that is a fourth example of the interpolated image generated by the combining unit 24 according to the present embodiment. The interpolated image 78 is a second example of the interpolated image generated by the synthesizing unit 24 when there is a defect in the central rear image acquired by the camera 14, for example.

The interpolated image 78 is an image when the rear center of the vehicle 1 is viewed from the position of the electronic mirror 40. The interpolated image 78 is an interpolated image corresponding to the image arranged at that position if there is no problem in the image acquired by the acquisition unit 21.

In the interpolated image 78, vehicles C4 and C5 represent vehicles located behind the center of the vehicle 1. Since the interpolated image 78 is an image when the rear center is viewed from the vehicle 1 as described above, the vehicle C1 indicating the own vehicle is not shown.

Further, in the interpolated image 78, one or more lines L indicating the standard of the distance in the front-rear direction of the vehicle 1 are drawn. The lines L are drawn, for example, at intervals corresponding to 1 m in reality.

As in the case of the interpolation image 74 (see FIG. 8 ), the user can visually recognize the interpolation image 78 to understand the positional relationship and the relative distance between the vehicles C4 and C5 and the vehicle 1.

7 and 10, the vehicle C1 (vehicle 1) is drawn above the interpolated images 72 and 76, but this may be drawn below the interpolated images 72 and 76 according to the preference of the user. You can In this case, the interpolated images 72 and 76 are rotated 180 degrees and displayed to maintain the positional relationship with the vehicles C2, C3, C4, and C5.

The processing of the image synthesizing device 10 configured as above will be described.

FIG. 12 is a flow chart showing a process performed by the image composition device 10 according to the present embodiment.

In step S101, the acquisition unit 21 acquires two images. The two images are, for example, a left rear image and a center rear image.

In step S102, the detection unit 23 performs a detection process for detecting a defect in the image acquired by the acquisition unit 21 in step S101.

In step S103, the process branches based on the result of the detection process in step S102. If no defect is detected in the two images in the detection processing, the process proceeds to step S104. If a defect is detected in only one of the two images, the process proceeds to step S111. In other cases, the process proceeds to step S121. move on.

In step S104, the synthesizing unit 24 performs geometric transformation processing on the image acquired by the acquiring unit 21 as pre-processing for composition, and sets a boundary line at a predetermined position. The predetermined position is a position predetermined according to the vehicle type, the installation position of the camera, the installation angle, or the like. The same applies hereafter.

In step S105, the combining unit 24 generates a combined image by combining the two images using the boundary line set in step S104.

In step S106, the display control unit 25 displays the image generated by the combining unit 24 in step S105 or step S113 on the electronic mirror 40. When step S106 is completed, the series of processes shown in FIG. 12 is completed.

In step S111, the synthesizing unit 24 performs geometric conversion processing on the image having no defect and sets a boundary line at a position moved from a predetermined position. The boundary line is moved in one direction from the predetermined position. The one direction is a direction in which the composite image includes the other image more greatly as the boundary line is moved to the one direction in the image arrangement direction.

In step S112, the synthesizing unit 24 generates an interpolation image based on the distance to the rear object acquired by the distance acquiring unit 22 by the sensor 22A.

In step S113, the combining unit 24 combines the interpolated image generated in step S112 and the non-defective image using the boundary line set in step S111. Upon completion of step S113, the process proceeds to step S106.

In step S121, the image composition device 10 performs a predetermined error process. The predetermined error processing includes, for example, processing for generating and displaying an image indicating that a defect has been detected in both two images. When step S121 ends, the series of processes shown in FIG. 12 ends.

In the above description, the case where the image synthesizing device 10 executes the processing for two images, that is, the left rear image and the center rear image has been described as an example, but instead of the right rear image and the center rear image, The process may be performed on two images.

Further, the image synthesizing apparatus 10 executes the above-described series of processing for two images of the left rear image and the center rear image among the three images of the left rear image, the center rear image, and the right rear image, In addition, if the series of processes is executed for two images of the right rear image and the center rear image among the three images, the synthesizing process can be simultaneously executed for the three images. it can.

By the series of processes shown in FIG. 12, the image synthesizing device 10 can generate a synthetic image that more appropriately shows the rear of the vehicle 1 even when the image has a defect.

As described above, in the image synthesizing apparatus according to the present embodiment and the modification, when only one of the two images to be acquired has a defect, the image having no defect is included in the combined image to a larger extent. Move the border to. As a result, the non-defective image will have a larger area in the composite image. The user can more appropriately recognize the rear of the vehicle by visually observing the composite image in which the non-defective image occupies a larger area than when the non-defective image is displayed without moving the boundary line. You can Therefore, the image synthesizing apparatus can generate a synthetic image that more appropriately shows the rear of the vehicle even if the image has a defect.

Further, when a part of the vehicle is reflected in the acquired image, the image combining device generates a combined image in which the part is not reflected. If a part of the vehicle is reflected in the composite image, the rear image becomes an unnatural image, which may cause confusion for the user. Therefore, the image synthesizing apparatus can generate a synthetic image that more appropriately shows the rear of the vehicle without disturbing the user even if the image has a defect.

Further, the image synthesizing apparatus arranges the interpolation image (first interpolation image) at a position where the defective image of the two images to be acquired should be displayed to generate the synthetic image. The first interpolation image is an image generated based on the distance to the object behind the vehicle that is actually acquired using the sensor. Therefore, the user can know the state behind the vehicle by visually recognizing the interpolated image instead of the defective image. In this way, the image synthesizing apparatus can generate a synthetic image that more appropriately shows the rear of the vehicle even if the image has a defect.

Further, the image synthesizing apparatus generates an interpolated image using a sensor selected from a plurality of sensors mounted on the vehicle according to a vehicle speed or a shift position. In this way, the image synthesizing apparatus can generate a synthetic image that more appropriately shows the rear of the vehicle by using the image generated by using the appropriate sensor even if the image has a defect.

Further, the image compositing apparatus arranges an interpolated image (second interpolated image) indicating the content of the defect at a position where the image having the defect out of the two images to be acquired should be displayed to generate the combined image. To do. The second interpolation image is an image showing the content of the defect. Therefore, the user can know the content of the defect by visually recognizing the interpolated image instead of the image having the defect, and can take action for coping. In this way, the image synthesizing apparatus can provide information for coping with the deficiency even when the image has a defect, and as a result, can generate a combined image that more appropriately shows the rear of the vehicle.

In addition, in the above-described embodiment, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the image synthesizing device and the like of the above-described embodiment is the following program.

That is, the program is a control method of an image synthesizing apparatus, and the program includes an acquisition step of performing an acquisition process for acquiring at least two images in which a common target is captured under different imaging conditions. When a detection step for performing a detection process for detecting a defect in the image acquired in the acquisition step and a defect is not detected in the detection step in either of the two images, the two images are arranged in a predetermined sequence. A synthesizing step of arranging them in a direction in which they are partially overlapped with each other and synthesizing the two images by using a boundary line at a predetermined position as a boundary to generate a synthesized image. When a defect is detected in only one of the two images in the detection step, at least the other image of the two images is arranged, and the predetermined position is changed to one direction in the arrangement direction. The composite image is generated with the moved boundary line as a boundary, and the one direction is such that as the boundary line is moved to the one direction in the arranging direction, the other image becomes closer to the composite image. The control method, which is the direction that is largely included, is executed.

The image synthesizing device according to one or more aspects has been described above based on the embodiment, but the present invention is not limited to this embodiment. As long as it does not depart from the gist of the present invention, various modifications that those skilled in the art can think of in the present embodiment, and configurations constructed by combining components in different embodiments are also within the scope of one or more aspects. May be included within.

INDUSTRIAL APPLICABILITY The present invention can be used for an image synthesizing device that synthesizes images from a camera mounted on a vehicle and displays them on an electronic mirror.

1, C1, C2, C3, C4, C5 Vehicle 10 Image synthesizer 12, 13, 14 Camera 21 Acquisition unit 22 Distance acquisition unit 22A Sensor 23 Detection unit 24 Synthesis unit 25 Display control unit 40 Electronic mirror 50 Rear image 51, 52 , 53, 61, 61A, 61C, 62, 62A, 62C, 63, 63A, 63B image 60A, 60B, 60C composite image 65, 65A, 65B, 66, 66A, 66B, 66C boundary line 72, 74, 76, 78 Interpolated image C2A mark L line

Claims (6)

An acquisition unit that performs an acquisition process for acquiring at least two images in which a common target is captured under mutually different imaging conditions;
A detection unit that performs a detection process for detecting a defect in the image acquired by the acquisition unit,
When no defect is detected by either of the two images by the detection unit, the two images are arranged in a predetermined arrangement direction and arranged at positions where some of them overlap, and the two images are arranged with a boundary line of the predetermined position as a boundary. A combining unit that generates a combined image by combining the images,
The combining unit further includes
When the detection unit detects a defect in only one of the two images, at least the other image of the two images is arranged and one of the images in the arrangement direction is arranged from the predetermined position. With the boundary line moved in the direction as a boundary, the composite image is generated,
An image synthesizing apparatus in which the one direction is a direction in which the other image is included in the synthesized image to a greater extent as the boundary line is moved to the one direction in the arrangement direction. The acquisition unit is two cameras that capture the rear of the vehicle, and performs the acquisition process for acquiring the two images using two cameras mounted at different positions of the vehicle,
In the other image of the two images arranged by the combining unit, a part of the vehicle is shown at a position advanced from the predetermined position in the one direction,
The image synthesizing apparatus according to claim 1, wherein the synthesizing unit moves the borderline within a range in which the borderline does not overlap a part of the vehicle. The acquisition unit performs the acquisition process for acquiring the two images by using cameras mounted at different positions of the vehicle to capture the rear of the vehicle,
The image synthesizing device further includes
A distance acquisition unit that acquires distance information from the vehicle to the object, obtained by a sensor that senses an object behind the vehicle,
In the case where the detecting unit detects a defect in only one of the two images, the synthesizing unit may further rearward the vehicle based on the distance information acquired by the distance acquiring unit. A first interpolated image showing the above condition is generated, and one of the two images is arranged at a position where it should be arranged, and the arranged first interpolated image and the other of the two images are arranged. The image combining device according to claim 1, wherein the image is combined with the image using the boundary line. The sensor includes a plurality of the sensors,
The distance acquisition unit,
The image synthesizing apparatus according to claim 3, wherein the distance information obtained by a sensor selected according to a vehicle speed or a shift position of the vehicle among the plurality of sensors is acquired. The synthesizing unit,
When the detecting unit detects a defect in only one of the two images, the combining unit generates a second interpolated image indicating the content of the defect to generate one of the two images. 5. The one of the images is arranged at a position to be arranged, and the arranged second interpolation image and the other image of the two images are combined using the boundary line. The image synthesizing apparatus according to item 1. A method of controlling an image synthesizing apparatus, comprising:
An acquisition step of performing an acquisition process for acquiring at least two images in which a common target is captured under mutually different imaging conditions;
A detection step of performing a detection process for detecting a defect about the image acquired in the acquisition step;
When no defect is detected in either of the two images in the detection step, the two images are arranged in a predetermined arrangement direction and arranged at positions where some of them are overlapped with each other, and a boundary line of the predetermined position is used as a boundary between the two images. Synthesizing the images to generate a synthetic image, and
In the synthesis step,
When a defect is detected in only one of the two images in the detection step, at least the other image of the two images is arranged and one of the images in the arrangement direction is arranged from the predetermined position. With the boundary line moved in the direction as a boundary, the composite image is generated,
The control method according to claim 1, wherein the one direction is a direction in which the other image is included more in the composite image as the boundary line is moved to the one direction in the arrangement direction.

Images