JP7108855B2 - Image synthesizing device and control method - Google Patents

Description

The present invention relates to an image synthesizing device and a control method.

2. Description of the Related Art There is an image display device that photographs the rear of a vehicle with a plurality of cameras provided in the vehicle and displays the photographed images. For example, there is an image display device that allows a driver to view an image showing the rear of the vehicle, that is, a so-called vehicle electronic mirror.

This image display device captures images of the left rear and right rear of the vehicle taken by cameras fixed near the left and right doors of the vehicle, and images taken by a camera fixed near the rear bumper or trunk hood of the vehicle. An image of the center rear of the vehicle is acquired. Then, by connecting these images, one image showing the left rear, center rear, and right rear is generated, and the generated image is displayed on the rearview mirror type display device.

2. Description of the Related Art Conventionally, there is an image synthesizing device that synthesizes images by connecting them (see, for example, Japanese Unexamined Patent Application Publication No. 2002-100001).

JP-A-2000-215317

However, in synthesizing images by the image synthesizing device described in Japanese Patent Application Laid-Open No. 2002-200312, the doubling that occurs in the vicinity of the boundary of the images to be synthesized may cause a sense of discomfort. Duplication means that one object appears twice in an image.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image synthesizing apparatus that synthesizes images while suppressing a sense of incongruity that accompanies image synthesis.

An image synthesizing device according to an aspect of the present invention obtains a first image and a second image obtained by imaging a subject under mutually different imaging conditions, and partly obtains the obtained first image and the second image. one or more first feature points in the first image arranged by the acquisition unit; and one or more second features in the second image arranged by the acquisition unit. one first feature point among the one or more first feature points at which the same object is imaged; and one or more second feature points. a pair determination unit that determines a plurality of pairs with one of the second feature points; and a plurality of partial images obtained by dividing the arranged first image and the second image by a dividing line extending in the first direction. , calculating a first representative vector representing a vector from one first feature point to a midpoint between one first feature point and one second feature point relating to a pair included in the partial image, and calculating the pair and the first representative vector for each of the plurality of partial images calculated by the first calculation unit. calculating a first individual vector for each pixel row included in the first image and the second image by interpolating the vectors, and calculating the second representative vector for each of the plurality of partial images calculated by the first calculation unit; a second calculation unit that calculates a second individual vector for each pixel row included in the first image and the second image by interpolating the first individual vector for the pixel row included in the first image; to calculate a pixel value using the second image, calculate a pixel value for a pixel string included in the second image using the second individual vector, and then combine the first image and the second image. and

In addition, these general or specific aspects may be realized by a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. and any combination of recording media.

The image synthesizing device according to the present invention is capable of synthesizing images while suppressing a sense of incongruity that accompanies image synthesizing.

FIG. 1 is a schematic diagram showing an example of a vehicle equipped with a camera that acquires an image synthesized by an image synthesizing device according to Embodiment 1. FIG. 2A and 2B are explanatory diagrams showing an image before synthesis and an image after synthesis by the image synthesizing apparatus according to Embodiment 1. FIG. FIG. 3 is an explanatory diagram of doubling of subjects in images captured by different cameras. FIG. 4 is a block diagram showing the functional configuration of the image synthesizing device according to Embodiment 1. As shown in FIG. FIG. 5 is an explanatory diagram showing a method of determining feature points and pairs by the image synthesizing device according to the first embodiment. FIG. 6 is an explanatory diagram showing representative vectors calculated by the image synthesizing device according to the first embodiment. FIG. 7 is an explanatory diagram showing a method of calculating individual vectors by linear interpolation of representative vectors by the image synthesizing device according to the first embodiment. FIG. 8 is an explanatory diagram showing individual vectors calculated by the image synthesizing device according to the first embodiment. FIG. 9 is an explanatory diagram showing synthesis processing using individual vectors by the image synthesizing apparatus according to the first embodiment. FIG. 10 is a flow chart showing processing of the image synthesizing device according to the first embodiment. FIG. 11 is a block diagram showing a functional configuration of an image synthesizing device according to Embodiment 2. As shown in FIG. FIG. 12 is an explanatory diagram showing an example of a right rear image of the vehicle according to the second embodiment. FIG. 13 is an explanatory diagram showing an example of a central rear image of a vehicle according to Embodiment 2. FIG. FIG. 14 is an explanatory diagram showing an example of an image obtained by connecting a left rear image and a central rear image of a vehicle according to related art. FIG. 15 is an explanatory diagram showing a state in which a left rear image and a central rear image of a vehicle are partially overlapped and arranged according to the second embodiment. FIG. 16 is an explanatory diagram of an example of a boundary line for synthesizing images according to the second embodiment. 17 is an enlarged view of frame XVII in FIG. 16. FIG. FIG. 18 is an explanatory diagram showing an example of an image extracted from the left rear image of the vehicle according to the second embodiment. FIG. 19 is an explanatory diagram showing an example of an image extracted from the central rear image of the vehicle according to the second embodiment. FIG. 20 is an explanatory diagram showing an example of an image obtained by joining a left rear image and a central rear image of a vehicle according to the second embodiment. FIG. 21 is a flow chart showing processing of the image synthesizing device according to the second embodiment. FIG. 22 is an explanatory diagram showing another example of a boundary line for synthesizing a left rear image and a central rear image of a vehicle according to the second embodiment. FIG. 23 is an explanatory diagram showing individual vectors and boundary lines calculated by the image synthesizing device according to the modification of the second embodiment.

An image synthesizing device according to an aspect of the present invention obtains a first image and a second image obtained by imaging a subject under mutually different imaging conditions, and partly obtains the obtained first image and the second image. one or more first feature points in the first image arranged by the acquisition unit; and one or more second features in the second image arranged by the acquisition unit. one first feature point among the one or more first feature points at which the same object is imaged; and one or more second feature points. a pair determination unit that determines a plurality of pairs with one of the second feature points; and a plurality of partial images obtained by dividing the arranged first image and the second image by a dividing line extending in the first direction. , calculating a first representative vector representing a vector from one first feature point to a midpoint between one first feature point and one second feature point relating to a pair included in the partial image, and calculating the pair and the first representative vector for each of the plurality of partial images calculated by the first calculation unit. calculating a first individual vector for each pixel row included in the first image and the second image by interpolating vectors, and calculating the second representative vector for each of the plurality of partial images calculated by the first calculation unit; a second calculation unit for calculating a second individual vector for each pixel row included in the first image and the second image by interpolating the first individual vector for the pixel row included in the first image; to calculate a pixel value using the second image, calculate a pixel value for a pixel row included in the second image using the second individual vector, and then combine the first image and the second image. and

According to the above aspect, the image synthesizing device calculates the pixel values such that the feature points of the same object captured in the first image and the second image are brought closer to each other, and then the first Composite the image and the second image. In this calculation, after the first representative vector and the like are once calculated, an individual vector for each pixel row is calculated using the calculated first representative vector, and the pixel value is calculated using the individual vector. By once calculating the first representative vector and the like in this way, it is possible to eliminate the influence of vectors that are significantly different from others, which may be caused by incorrect matching of feature points, from affecting image synthesis. As a result, the image synthesizing device can synthesize images while suppressing a sense of incongruity that accompanies image synthesizing.

For example, for each of the plurality of partial images, the first calculation unit calculates one first feature point and one second feature point from one first feature point associated with a plurality of pairs included in the partial image. A median value of vectors extending to the midpoint is calculated as the first representative vector, and a median value of vectors ranging from one second feature point related to the pair to the midpoint is calculated as the second representative vector.

According to the above aspect, the image synthesizing device calculates the median value of the plurality of vectors as the first representative vector or the like. This makes it possible to eliminate the influence of vectors that are significantly different from each other, which can be caused by mismatching of feature points, from affecting individual vectors. Therefore, the image synthesizing device can more easily synthesize images while suppressing the sense of incongruity that accompanies image synthesis.

For example, the second calculation unit associates the central pixel row of each of the plurality of partial images with the first representative vector in the partial image for each pixel row included in the first image and the second image. , the first individual vector is calculated by linear interpolation, and the second representative vector in the partial image is added to the center pixel row of each of the plurality of partial images for each pixel row included in the first image and the second image. , and the second individual vector is calculated by linear interpolation.

According to the above aspect, the image synthesizing device calculates individual vectors by linear interpolation of representative vectors. Therefore, based on a more specific configuration, the image synthesizing device can more easily synthesize images while suppressing the sense of incongruity that accompanies image synthesis.

For example, the first calculation unit calculates the first representative vector and the second representative vector for each of the plurality of partial images, the first representative vector and the second representative vector for a partial image adjacent to the partial image, and Statistical processing is performed to reduce the difference from each, and the second calculation unit calculates the first individual vector and the second individual vector after the statistical processing by the first calculation unit.

According to the above aspect, the image synthesizing device reduces the difference between the representative vector of one partial image and the representative vector of the adjacent partial image, thereby reducing the difference between the individual vectors of the partial images, As a result, it is possible to avoid distortion in the synthesized image. Therefore, the image synthesizing device can synthesize an image while suppressing a sense of incongruity caused by image synthesizing by avoiding occurrence of distortion between portions of the image.

For example, the acquisition unit acquires a plurality of the first images and the second images captured at different times, and the first calculation unit calculates the first representative vector for each of the plurality of partial images and the Statistical processing for reducing the difference between each of the second representative vectors and each of the first and second representative vectors of a partial image located at the same position as the partial image and acquired prior to the partial image. and the second calculation unit calculates the first individual vector and the second individual vector after the statistical processing by the first calculation unit.

According to the above aspect, the image synthesizing device reduces the amount of temporal change in the individual vectors by reducing the difference between the representative vector of one partial image and the representative vector of the past partial image at the same position. As a result, it is possible to avoid distortion in the synthesized image. Therefore, the image synthesizing device can synthesize an image while suppressing a sense of incongruity caused by image synthesizing by avoiding occurrence of distortion between portions of the image.

For example, the acquisition unit acquires a plurality of the first image and the second image captured at different times, and the first calculation unit acquires the first image and the second image captured at one time. When the number of pairs included in the partial images among the plurality of partial images obtained by dividing the two images is a predetermined number or more, the first image captured at a time later than the one time and When the size of the partial image of the second image is reduced, and the number of pairs included in the partial image is equal to or less than a predetermined number, the first image captured at a time later than the one time and increasing the size of the partial image for the second image.

According to the above aspect, the image synthesizing device can keep the number of pairs of feature points included in the partial image within an appropriate range. If the number of feature points included in the partial image is larger than the appropriate range, the variation in the vectors connecting the feature points and the midpoints between the feature points increases, and the number of vectors apart from the calculated representative vector also increases. Therefore, an appropriate representative vector may not be calculated. On the other hand, if the number of feature points included in the partial image is smaller than the appropriate range, or if the number of feature points used as the basis for calculation is small and the calculated representative vector does not necessarily represent the vectors included in the partial image. can also occur. Therefore, by keeping the number of pairs of feature points included in the partial images within an appropriate range and by calculating an appropriate representative vector, it is possible to synthesize images while suppressing the sense of incongruity that accompanies image compositing.

For example, the first direction is horizontal.

According to the above aspect, the image synthesizing device generates the partial images using the horizontal dividing line. Therefore, the image synthesizing device can more easily synthesize images while suppressing the sense of incongruity that accompanies image synthesis.

For example, the second calculation unit calculates the first representative vector and the second representative vector by hardware processing by a GPU (Graphics Processing Unit), and the synthesizing unit calculates the first representative vector by hardware processing by the GPU. compositing the image and the second image;

According to the above aspect, the image synthesizing device performs image synthesizing based on the first representative vector using hardware processing by the GPU. Therefore, the image synthesizing device can synthesize images while reducing the amount of processing and the processing time, and suppressing the sense of incongruity that accompanies image synthesizing.

For example, the image synthesizing device further selects a plurality of determined pairs in which the separation distance between the one first feature point and the one second feature point of the pair is equal to or less than a predetermined distance. A curve determining unit that determines one curve passing through the vicinity of the short-distance pair, and the synthesizing unit combines the first image and the second image with the one curve determined by the curve determining unit as a boundary. Synthesize.

According to the above aspect, the image synthesizing device synthesizes the first image and the second image using the boundary line determined according to the content of the first image and the second image. This boundary line is set so as to pass through the vicinity of the pair of the feature points in the first image and the feature points in the second image in which the same object is imaged, and the separation distance between the feature points is relatively small. Therefore, the image doubling or loss that can occur near the border is suppressed. As a result, the image synthesizing device can synthesize images while suppressing a sense of incongruity that accompanies image synthesizing.

For example, the curve determining unit determines, as the one curve, one approximated curve that passes through the vicinity of the plurality of determined short distance pairs.

According to the above aspect, the image synthesizing device uses the approximation curve to determine the boundary line to be used for synthesizing the images. Therefore, the image synthesizing device can more easily synthesize images while suppressing the sense of incongruity that accompanies image synthesis.

For example, the curve determination unit determines the one curve by a nonlinear least-squares method.

According to the above aspect, the image synthesizing device uses the nonlinear least-squares method to determine the boundary line used for synthesizing the images. Therefore, based on a more specific configuration, the image synthesizing device can more easily synthesize images while suppressing the sense of incongruity that accompanies image synthesis.

For example, the curve determination unit determines, of the determined pairs, a plurality of long distances in which the distance between the one first feature point and the one second feature point of the pair is equal to or greater than a predetermined distance. Determine the one curve that does not pass through a pair of neighbors.

According to the above aspect, the image synthesizing device includes the neighborhood of the long-distance pair, that is, the neighborhood of the feature points of the first image and the feature points of the second image relating to the long-distance pair, and the straight line connecting these feature points. Set a border that does not pass through neighbors. Therefore, doubling or disappearance that can occur near the boundary is further suppressed. As a result, the image synthesizing device can synthesize images while further suppressing discomfort associated with image synthesizing.

For example, the first image and the second image are images captured simultaneously by two cameras fixed to one vehicle, and the predetermined positional relationship is determined according to the positions and orientations of the two cameras. This is a predetermined positional relationship.

According to the above aspect, the image synthesizing device uses the position and orientation of the camera mounted on the vehicle to determine in advance the positional relationship between the first image and the second image. Therefore, the image synthesizing device can synthesize images more easily by using a predetermined positional relationship while suppressing a sense of incongruity that accompanies image synthesis.

For example, the image synthesizing device may include (a) a left rear image captured by a left camera that captures the left rear of one vehicle, and (b) a right rear image captured by a right camera that captures the right rear of the one vehicle. A right rear image and (c) a center rear image captured by a center camera that reflects the center rear of the one vehicle are acquired, and one of the left rear image and the right rear image and the center obtaining the rear images as the first image and the second image, respectively, by the obtaining unit, determining each of the feature point determining unit and the pair determining unit, and determining each of the first calculating unit and the second calculating unit; and the combining unit generates a third image by combining the third image and the other of the left rear image and the right rear image, respectively, with the first image and the second image obtained as an image by the acquisition unit, determined by the feature point determination unit and the pair determination unit, respectively calculated by the first calculation unit and the second calculation unit, and synthesized by the synthesis unit to obtain a fourth Generate an image.

According to the above aspect, the image synthesizing device synthesizes the images captured by the left camera, the right camera, and the center camera attached to the vehicle. As a result, the image synthesizing device synthesizes a wide range from the rear left side of the vehicle to the rear right side through the rear center, or from the rear right side of the vehicle to the rear left side through the rear center, while suppressing a sense of incongruity accompanying image synthesis. It is possible to synthesize images that have been

In addition, the image synthesizing device according to one aspect of the present invention obtains a first image and a second image obtained by imaging one object under mutually different imaging conditions, and converts the obtained first image and the second image to one or more first feature points in the first image arranged by the acquisition unit; and one or more first feature points in the second image arranged by the acquisition unit. a feature point determination unit that determines two feature points; one first feature point among the one or more first feature points, which is a point at which the same object is imaged; and one or more second features. A pair determination unit that determines a plurality of pairs of one second feature point among the points, and the one first feature point and the one second feature related to the pair among the plurality of determined pairs a curve determining unit that determines one curve passing through the vicinity of a plurality of short-distance pairs whose distance from the point is a predetermined distance or less; and the first image and the second image with the determined curve as a boundary. and a synthesizing unit for synthesizing the

According to the above aspect, the same effects as those of the above image synthesizing device can be obtained.

A method for controlling an image synthesizer according to an aspect of the present invention is a method for controlling an image synthesizer, in which a first image and a second image obtained by imaging one target under mutually different imaging conditions are acquired, and an acquiring step of arranging the first image and the second image that have been obtained in a predetermined positional relationship in which parts overlap; one or more first feature points in the first image arranged in the acquiring step; and a feature point determination step of determining one or more second feature points in the arranged second image; a pair determination step of determining a plurality of pairs of a feature point and one second feature point out of the one or more second feature points; A vector from one first feature point to the midpoint between one first feature point and one second feature point for each of a plurality of partial images divided by the extending dividing line and calculating a second representative vector representing a vector from one second feature point of the pair to the midpoint; and the first calculation step calculating a first individual vector for each pixel row included in the first image and the second image by interpolating the first representative vector calculated for each of the plurality of partial images, and calculating in the first calculation step; a second calculating step of calculating a second individual vector for each pixel row included in the first image and the second image by interpolating the second representative vector for each of the plurality of partial images; A pixel value is calculated for a pixel row included in the image using the first individual vector, a pixel value is calculated for a pixel row included in the second image using the second individual vector, and then the and a compositing step of compositing the first image and the second image.

According to the above aspect, the same effects as those of the above image synthesizing device can be obtained.

For example, the control method may further include a plurality of near-field features in which the separation distance between the one first feature point and the one second feature point of the pair is equal to or less than a predetermined distance, among the determined pairs. A curve determination step of determining a curve passing through the vicinity of the distance pair, wherein the combining step combines the first image and the second image using the curve determined by the curve determination step as a boundary. do.

According to the above aspect, the same effects as those of the above image synthesizing device can be obtained.

For example, in a method for controlling an image synthesizing device, a first image and a second image obtained by imaging an object under mutually different imaging conditions are obtained, and a part of the obtained first image and the second image is obtained. one or more first feature points in the first image arranged in the acquisition step, and one or more second features in the second image arranged in the acquisition step a first feature point among the one or more first feature points, which is a point at which the same object is imaged, and one or more second feature points; a pair determination step of determining a plurality of pairs with one of the second feature points, and the one first feature point and the one second feature point associated with the pair among the plurality of determined pairs; a curve determination step of determining a curve passing through the vicinity of a plurality of short-distance pairs whose separation distance is less than or equal to a predetermined value; and synthesizing the first image and the second image using the determined curve as a boundary and a synthetic step of

According to the above aspect, the same effects as those of the above image synthesizing device can be obtained.

In addition, these general or specific aspects may be realized by a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. Or it may be realized by any combination of recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that the embodiments described below are all comprehensive or specific examples. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are examples and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in independent claims indicating the highest level concept will be described as optional constituent elements.

(Embodiment 1)
In the present embodiment, an image synthesizing apparatus that synthesizes images while suppressing a sense of incongruity that accompanies image synthesis will be described.

FIG. 1 is a schematic diagram showing an example of a vehicle 1 according to this embodiment. FIG. 2 is an explanatory diagram showing an image before synthesis and an image after synthesis by the image synthesizing apparatus 10 according to the present embodiment.

As shown in FIG. 1, the vehicle 1 includes an image synthesizing device 10, cameras 12, 13 and 14, and an electronic mirror 40. As shown in FIG. In the following description, unless otherwise specified, the front, rear, left, and right directions are the directions when the traveling direction of the vehicle 1 is forward, and can also be said to be the front, rear, left, and right directions for the driver of the vehicle 1 .

The camera 12 is fixed near the left door of the vehicle 1 and captures an image of the rear left side of the vehicle 1 . The camera 12 captures the rear left side of the vehicle 1 to generate an image. The image generated by the camera 12 is also called a left rear image.

The camera 13 is a camera that is fixed near the right door of the vehicle 1 and captures an image of the rear right side of the vehicle 1 . The camera 13 captures the rear right side of the vehicle 1 to generate an image. The image generated by the camera 13 is also called a right rear image.

The camera 14 is a camera that is fixed near the rear bumper or the trunk hood of the vehicle 1 and shoots the rear center of the vehicle 1 . The camera 14 captures the center rear of the vehicle 1 to generate an image. The image generated by the camera 14 is also referred to as a central rear image.

The photographing ranges of the cameras 12 and 14 partially overlap, and the photographing ranges of the cameras 13 and 14 partially overlap. Therefore, a common object appears in part of the left rear image and the central rear image. In addition, a common object appears in part of the right rear image and the central rear image.

Each of the cameras 12, 13, and 14 generates an image by taking an image under shooting conditions different from each other. Specifically, the cameras 12, 13, and 14 are arranged at different positions and facing different directions, and acquire images at, for example, 60 fps. Also, the optical characteristics of the optical systems of the cameras 12, 13 and 14 may be different.

The electronic mirror 40 is an image display device that displays a rear image 50 that is an image of the rear of the vehicle 1 . The electronic mirror 40 displays an image based on the video signal output by the image synthesizing device 10 . The electronic mirror 40 can be used as a substitute for a conventional rearview mirror that reflects the rear of the vehicle 1 using light reflection.

The image synthesizing device 10 is a device that synthesizes images generated by the cameras 12 , 13 and 14 to generate an image displayed by the electronic mirror 40 . An example of the images produced by cameras 12, 13 and 14, respectively, is shown as images 51, 52 and 53 in FIG.

As described above, each of the cameras 12, 13 and 14 captures one target under different imaging conditions. A natural rear image 50 cannot be obtained only by splicing. Specifically, with images that are simply stitched together, doubling or disappearance may occur near the boundary of stitching, and a natural rearward image 50 cannot be obtained. Here, the natural rearward image 50 refers to the same image as the scene behind the vehicle 1 seen by the driver of the vehicle 1 through a conventional room mirror (so-called physical mirror). Duplication means that one thing appears twice in the image, and disappearance means that something that actually exists does not appear in the image.

For example, when images captured by cameras installed at different positions are synthesized, the subject may be duplicated. FIG. 3 is an explanatory diagram of doubling of subjects in images captured by cameras A and B installed at two different positions.

In general, when synthesizing images, virtual projection planes are set in the virtual space at predetermined distances from the cameras A and B, respectively, and the images are projected onto the virtual projection planes for synthesizing. (a) and (b) of FIG. 3 show the cameras A and B arranged in the virtual space and the virtual projection plane.

As shown in FIG. 3A, the distances LRA and LRB from cameras A and B in the real space to the subject and the distances LVA and LVB from the cameras A and B in the virtual space to the virtual projection plane are In the case of equality, no image doubling occurs in the virtual projection plane (LRA=LVA, LRB=LVB).

On the other hand, as shown in FIG. 3B, the distances LVA and LVB from the cameras A and B in the virtual space to the virtual projection plane are calculated from the distances LRA and LRB from the cameras A and B in the real space to the subject When is large, image doubling occurs in the virtual projection plane. If the distance from the cameras A and B in the virtual space to the virtual projection plane is smaller than the distance from the cameras A and B to the subject in the real space, the image is doubled on the virtual projection plane (not shown).

Therefore, the image synthesizing device 10 applies appropriate processing to the images generated by the cameras 12, 13, and 14 and then splices them together, thereby suppressing the duplication of images that may occur due to synthesizing. A natural rear image 50 is generated and displayed on an electronic mirror 40. - 特許庁

The functions and processing of the image synthesizing device 10 will be described below.

FIG. 4 is a block diagram showing the functional configuration of the image synthesizing device 10 according to this embodiment. Here, an example will be described in which the image synthesizing device 10 obtains and synthesizes two images of one target, that is, a first image and a second image. These two images may be a left rear image and a central rear image, or may be a right rear image and a central rear image.

It should be noted that it is also possible to synthesize three images by using a method of synthesizing two images, the first image and the second image. For example, after the image synthesizing device 10 obtains an image by (1) synthesizing the left rear image and the central rear image as the first image and the second image, respectively, (2) the obtained image and the right rear image By synthesizing the images again as the first image and the second image, respectively, a rear image 50 in which the left rear image, the center rear image, and the right rear image are all joined together is finally obtained. Here, an image obtained by synthesizing the first image and the second image is also referred to as a third image, and an image obtained by synthesizing the third image and the right rear image is also referred to as a fourth image.

Here, with respect to synthesizing the three images described above, after the image synthesizing device 10 (1) synthesizes the left rear image and the central rear image as the first image and the second image, respectively, to obtain an image, , (2) The obtained image and the right rear image are synthesized again as the first image and the second image, respectively, but the method is not limited to this synthesis. For example, after the image synthesizing device 10 obtains an image by (1) synthesizing the right rear image and the central rear image as the first image and the second image, respectively, (2) the obtained image and the left rear image The images may be synthesized again as the first image and the second image, respectively. That is, the image synthesizing device 10 acquires one of the left rear image and the right rear image, and the center rear image as the first image and the second image, respectively, by the acquisition unit 21, and the feature point determination unit 31 and pair determination are performed. Determination by each of the units 32, calculation by each of the first calculation unit 33 and second calculation unit 34, and synthesis by the synthesis unit 23 to generate a third image, the third image, the left rear image, and the right rear image The other of the images is acquired by the acquisition unit 21 as the first image and the second image, respectively, determined by the feature point determination unit 31 and the pair determination unit 32, and determined by the first calculation unit 33 and the second calculation unit 34. Calculation is performed, and a fourth image is generated by synthesis by the synthesizing unit 23 . As a result, a rear image 50 is finally obtained by combining all of the left rear image, the central rear image, and the right rear image.

Hereinafter, the case where the first image is the left rear image and the second image is the central rear image will be described in detail. In this case, the image synthesizing device 10 acquires the first image and the second image at, for example, 60 fps by the cameras 12 and 14, generates a synthesized image of the first image and the second image each time the image is acquired, and the electronic mirror 40 to display.

As shown in FIG. 4 , the image synthesizing device 10 includes an acquisition unit 21 , a determination unit 22 , a synthesizing unit 23 and a display control unit 24 . Each component described above may be implemented by a processor executing a predetermined program using a memory or the like, or may be implemented by dedicated hardware.

The acquisition unit 21 acquires a first image and a second image obtained by imaging one target under mutually different imaging conditions, and arranges the acquired first image and second image in a predetermined positional relationship in which they partially overlap. processing unit. Acquisition unit 21 arranges the first image and the second image in a virtual space through computation by a computer. When the acquisition unit 21 arranges the first image and the second image in the virtual space, the images to be arranged are corrected according to the optical characteristics of the optical system such as the camera that captured the images. You can place the image in . Corrections specifically include corrections based on lens distortion, corrections for size and angle adjustments, color tone corrections, and the like. By applying this correction, the difference between the optical systems of the cameras is suppressed, and the difference between the image derived from the first image and the image derived from the second image included in the rear image 50 generated by stitching is corrected. can have smaller differences such as color or size. The predetermined positional relationship in which the first image and the second image are arranged is determined according to the position and orientation of the camera that acquired the first image and the second image. We call this static conversion.

The determination unit 22 is a processing unit that determines a boundary for synthesizing the first image and the second image. The determination unit 22 has a feature point determination unit 31, a pair determination unit 32, a first calculation unit 33, and a second calculation unit 34 as detailed components.

The feature point determination unit 31 is a processing unit that determines one or more first feature points in the first image arranged by the acquisition unit 21 and one or more second feature points in the second image arranged by the acquisition unit 21. be. The determination unit 22 extracts feature points in each of the first image and the second image. The process of extracting feature points is realized by, for example, a process of finding a pixel or a set of pixels included in the image and having a different color or shape from its surroundings. It can be done by image recognition technology.

Based on the first feature point and the second feature point determined by the feature point determination section 31, the pair determination section 32 selects one of the one or more first feature points at which the same object is imaged. A processing unit that determines a plurality of pairs of a first feature point and one second feature point out of one or more second feature points. Specifically, the color or shape of the pixel or the set of pixels found as the feature point in the first image by the feature point determination unit 31 and the pixel or the set of pixels found as the feature point in the second image, or SURF ( Speed-Up Robust Features), ORB (Oriented FAST and Rotated Brief), etc. are compared (so-called matching processing) to find images of the same object and determine them as the above pair. .

The first calculator 33 is a processor that calculates a representative vector for the pair determined by the pair determiner 32 . Specifically, the first calculation unit 33 calculates, for each of a plurality of partial images obtained by dividing the first image and the second image arranged by the acquisition unit 21 by a dividing line extending in the first direction, the image included in the partial image. A first representative vector representing a vector from one first feature point related to the pair to the middle point of one first feature point and one second feature point is calculated, and one first feature point related to the pair A second representative vector representing vectors from the two feature points to the midpoint is calculated.

When calculating the first representative vector, if there are a plurality of vectors from one feature point related to the pair to the midpoint, the plurality of vectors are statistically processed to calculate the first representative vector. Statistical processing includes median processing, average processing, and the like. The same applies to the second representative vector. A partial image is an image included in an area including a plurality of pixel rows.

Specifically, for each of a plurality of partial images, the first calculation unit 33 calculates one first feature point and one second feature point from one first feature point associated with a plurality of pairs included in the partial image. A median value of vectors reaching the midpoint of the points may be calculated as the first representative vector. Alternatively, the median value of the vectors from one second feature point to the midpoint of the pair may be calculated as the second representative vector. In the matching processing by the pair determination unit 32, there are cases where feature points where different objects are imaged are determined as a pair (also referred to as "mismatching"). In such a case, among the multiple vectors, only the vector in which mismatching occurred is significantly different from the others. By using the median value to calculate the first representative vector, etc., it is possible to eliminate the contribution of a vector that is significantly different from the others to the first representative vector, etc., which may occur due to false matching.

Note that the first direction is, for example, the horizontal direction. In other words, the dividing line is for example a horizontal dividing line. Although this case will be described below as an example, it is not limited to this.

Further, the first calculation unit 33 may perform a process of spatially averaging the first representative vectors and the like when calculating the first representative vectors and the like. That is, the first calculation unit 33 calculates the difference between the first representative vector and the second representative vector of each of the plurality of partial images and the first and second representative vectors of the partial images adjacent to the partial image. Statistical processing to reduce it may be performed. This statistical processing includes, for example, averaging or weighted averaging.

Further, the first calculation unit 33 may perform a process of temporally averaging the first representative vectors and the like when calculating the first representative vectors and the like. That is, the first representative vector and second representative vector of each of the plurality of partial images, and the first representative vector and second representative vector of the partial image located at the same position as the partial image and obtained before the partial image. Statistical processing may be performed to reduce the difference from each vector. This statistical processing includes, for example, averaging or weighted averaging.

Also, the first calculator 33 may adjust the size of the partial image so that the number of pairs included in the partial image falls within the appropriate range. That is, the first calculation unit 33 determines that the number of pairs included in the partial images out of the plurality of partial images obtained by dividing the first image and the second image captured at one time is equal to or greater than the first predetermined value. In some cases, the sizes of the partial images of the first image and the second image captured at a time later than the one time are reduced, and the number of pairs included in the partial images is equal to or less than the second predetermined value. In some cases, the sizes of the partial images of the first image and the second image captured at a time later than the one time may be increased. Here, the first predetermined value is the upper limit of the appropriate range, and can be set to about 50, for example. Also, the second predetermined value is the lower limit of the appropriate range, and can be set to about five, for example.

The second calculation unit 34 is a processing unit that calculates individual vectors using the representative vectors calculated by the first calculation unit 33 . The second calculation unit 34 calculates a first individual vector for each pixel row included in the first image and the second image by interpolating the first representative vector for each of the plurality of partial images calculated by the first calculation unit 33. Then, a second individual vector is calculated for each pixel row included in the first image and the second image by interpolating the second representative vector for each of the plurality of partial images calculated by the first calculation unit 33 .

Here, for each pixel row included in the first image and the second image, the second calculation unit 34 associates the central pixel row of each of the plurality of partial images with the first representative vector in the partial image, and performs linear interpolation. The first individual vector may be calculated by Further, the second calculation unit 34 associates the central pixel row of each of the plurality of partial images with the second representative vector of the partial image for each pixel row included in the first image and the second image, and performs linear interpolation to obtain A second individual vector may be calculated.

The combining unit 23 is a processing unit that generates a combined image by combining the first image and the second image using the individual vectors calculated by the second calculating unit 34 . The synthesizing unit 23 calculates pixel values for pixel columns included in the first image using the first individual vectors, and calculates pixel values for pixel columns included in the second image using the second individual vectors. Then, the first image and the second image are synthesized. Synthesis is performed by, in principle, cutting the first image and the second image by the boundary line and connecting them. At that time, the synthesizing unit 23 may perform processing such as obtaining a weighted average of the pixel values of the first image and the second image near the boundary.

The display control unit 24 is a processing unit that controls display of the synthesized image generated by the synthesizing unit 23 . The display control unit 24 outputs each pixel included in the composite image to the electronic mirror 40 so as to display the composite image on the electronic mirror 40, which is a display device.

The processing by the second calculation unit 34 and the processing by the synthesizing unit 23 can be performed by hardware processing by a general-purpose GPU (Graphics Processing Unit). This contributes to reduction of processing amount and processing time.

Hereinafter, the image synthesizing process will be specifically described while showing examples of images.

FIG. 5 is an explanatory diagram showing a method of determining feature points and pairs by the image synthesizing apparatus 10 according to the present embodiment.

The acquiring unit 21 acquires the images 51 and 53 using the camera 13 and the camera 14, respectively, and arranges the images in a predetermined positional relationship in which the images partially overlap each other. This positional relationship is predetermined according to the positions and directions of the cameras 13 and 14 with respect to the vehicle 1 . More specifically, as shown in FIG. 5, the image 51 is arranged on the left side of the paper, and the image 53 is arranged by shifting the image 51 rightward on the paper by a predetermined distance. Thus, the positional relationship in which the image 53 is displaced from the image 51 by a predetermined distance in the right direction of the drawing corresponds to the predetermined positional relationship. Note that the external shapes of the images 51 and 52 are distorted from rectangular due to correction and static transformation according to the optical characteristics of the optical system of the camera.

The feature point determination unit 31 determines feature points of the images 51 and 53 . The feature points of each image determined by the feature point determination unit 31 are shown in FIG. These feature points can be obtained by using a known image recognition technique or the like to find a pixel or a set of pixels contained in an image that has a different color or shape from its surroundings.

For example, a feature point p shown in FIG. 5 is a feature point determined on the image 51 by the feature point determination unit 31, and a feature point q is a feature point determined on the image 51 by the feature point determination unit 31. . In FIG. 5, black circles indicate feature points determined on the image 51, and black triangles indicate feature points determined on the image 53, respectively.

Then, the pair determination unit 32 determines pairs of the feature points determined by the feature point determination unit 31 in the images 51 and 53 arranged as shown in FIG. This pair is a pair of points where the same object is imaged. Feature points determined as pairs are connected by lines in FIG. For example, feature points p and q are a pair.

The first calculator 33 calculates a representative vector using the feature points p and q determined by the feature point determiner 31 . The first calculator 33 obtains a midpoint m between the feature points p and q, and calculates a vector u from the feature point p to the midpoint m and a vector v from the feature point q to the midpoint m. In addition, in FIG. 5 , the white circle indicates the midpoint calculated by the first calculator 33 .

Then, the first calculator 33 calculates a representative vector for each of the plurality of partial images 61, 62, 63 and 64 obtained by dividing the image 51 and the image 53. FIG. For example, the first calculation unit 33 sets the median value of the plurality of vectors u included in the partial image as the representative vector of the vector u, and the median value of the plurality of vectors v included in the partial image as the representative vector of the vector v. be a vector. Here, the representative vector of the vector u is also called the first representative vector, and the representative vector of the vector v is also called the second representative vector.

Note that if an erroneous matching occurs in the matching processing by the pair determination unit 32, the vector u or v will be significantly different from the surrounding vectors. As described above, the first calculation unit 33 has the advantage of being able to eliminate the contribution of vectors caused by erroneous matching to the representative vector by using the intermediate value of the vectors u or v as the representative vector.

FIG. 6 is an explanatory diagram showing representative vectors calculated by the image synthesizing device 10 according to the present embodiment.

In FIG. 6, vector 71 is a representative vector of vectors u included in partial image 61 . A vector 72 is a representative vector of vectors v included in the partial image 61 .

Similarly, vectors 73, 75 and 77 are representative vectors of vectors u included in partial images 62, 63 and 64, respectively. Vectors 74 , 76 and 78 are representative vectors of the vector v included in the partial image 61 .

Next, calculation processing of individual vectors by the second calculation unit 34 will be described.

FIG. 7 is an explanatory diagram showing a method of calculating individual vectors by linear interpolation of representative vectors by the image synthesizing apparatus 10 according to the present embodiment.

FIG. 7 is an enlarged view of a portion including partial images 62 and 63 in FIG.

The second calculator 34 associates the central pixel row 62c of the partial image 62 with the vector 73, which is the representative vector, and associates the central pixel row 63c of the partial image 63 with the vector 75, which is the representative vector. Here, the central pixel row 62c is a horizontal pixel row included in the partial image 62 and positioned in the center of the partial image 62 in the vertical direction. Similarly, the center pixel column 63c is a horizontal pixel column included in the partial image 63 and located in the center of the partial image 63 in the vertical direction. Although the case where the correspondence is based on the Y coordinate of the representative vector end point is described here, the correspondence may be based on the Y coordinate of the representative vector starting point or the representative vector middle point.

Then, the second calculator 34 calculates individual vectors of pixel rows between the central pixel row 62c and the central pixel row 63c by linear interpolation between the central pixel row 62c and the central pixel row 63c. That is, when the Y coordinate of the central pixel row 62c is Y1 and the Y coordinate of the central pixel row 63c is Y2, the second calculator 34 calculates the individual vector w of the pixel row 66 whose Y coordinate is Y by the following formula: Calculated by Here, let individual vector w=(wX, wY), vector 73=(X73, Y73), and vector 75=(X75, Y75).

wX=X73+{(Y−Y1)/(Y2−Y1)}×(X75−X73)
wY=Y73+{(Y−Y1)/(Y2−Y1)}×(Y75−Y73)

FIG. 8 is an explanatory diagram showing individual vectors calculated by the image synthesizing device 10 according to the present embodiment. FIG. 9 is an explanatory diagram showing synthesis processing using individual vectors by the image synthesizing apparatus 10 according to the present embodiment.

FIG. 8 shows individual vectors 81 to 84 calculated by the second calculator 34 in addition to the same vectors 71 to 78 as in FIG. Individual vectors 81 to 84 are calculated for each pixel column between the representative vectors.

Next, the synthesizing unit 23 joins the first image and the second image using the boundary line 90 as a boundary. The synthesizing unit 23 synthesizes the first image and the second image by calculating pixel values using individual vectors. Although the boundary line 90 is a vertical straight line on the paper, the shape of the boundary line 90 is not limited to this. good.

Specifically, the synthesizing unit 23 calculates pixel values of pixels in the vicinity of the boundary line 90 using individual vectors of pixel columns to which the pixels belong. For example, as shown in FIG. 9A, a pixel 91 adjacent to a boundary line 90 is calculated with the pixel value of a pixel 92 located at the starting point of an individual vector 81 ending at the pixel 91 . At this time, as shown in FIG. 9B, the pixel 92 located at the starting point of the individual vector 81 may be obtained by weakening the influence of the individual vector 81 as the distance from the boundary line 90 increases. In this case, when the composite region is within a predetermined pixel distance H from the boundary line 90, the influence Vt of the individual vector 81i on the individual vector 81 at the pixel 91i located at the pixel distance X from the boundary line 90 is
Vt=(1−X/H), (X=0 to H)
is represented by Therefore, by multiplying the individual vector 81 by this influence Vt, the pixel 92i located at the starting point of the individual vector 81i is obtained. Specifically, the position of the pixel 92i is obtained as follows.

In FIG. 9B, if the position coordinates of pixel 91i are (X91, Y91) and the individual vector 81 is wX81=(wX81, wY81), then the position coordinates (X92, Y92) of pixel 92i are:
X92=X91-wX81×Vt
Y92=Y91-wY81×Vt
is represented by

For the pixel values of the pixels 91 and 92 obtained in this way, the synthesizing unit 23 may perform an operation to overwrite the pixel value of the pixel 92 with the pixel value of the pixel 91, or the pixel value of the pixel 92 and the pixel value A calculation may be performed to obtain a weighted average taking into account the distance from the boundary line 90 to the pixel value of 91, or other calculations may be performed.

Further, the synthesizing unit 23 calculates pixel values of each of the first image and the second image, and performs a weighted average calculation of the pixel values of the first image and the second image for the overlap region. good too. Specifically, as shown in (c) of FIG. 9, for a pixel 91j, the pixel value of a pixel 92j (in the first image) located at the starting point of an individual vector 81j ending at the pixel 91j and the pixel 91j are The pixel values of the pixel 92k (in the second image) located at the starting point of the individual vector 81k, which is the ending point, may be weighted and mixed.

The processing of the synthesizing unit 23 in FIG. 9(c) can be summarized as follows. The synthesizing unit 23 calculates a position (eg, position coordinates (X92, Y92)) based on the individual vector 81j (first individual vector) for each pixel (eg, pixel 91j) in each region of the partial images 61 to 64. ) to refer to pixel 92j in the first image. Similarly, the synthesizing unit 23 calculates the second image based on the position calculated based on the individual vector 81k (second individual vector) for each pixel (for example, pixel 91j) in each region of the partial images 61 to 64. See pixel 92k in . Then, the synthesizing unit 23 synthesizes the first image and the second image by calculating (for example, weighted average) the pixel values of the referenced pixels 92j and 92k.

The pixels in the vicinity of the boundary line 90 are pixels within a predetermined distance from the boundary line 90 . Here, the predetermined distance is the width of each area of the partial images 61 to 64, and can be, for example, about 10 to 20% of the total width of the image 51 or 53 in the horizontal direction on the page.

The processing of the image synthesizing device 10 will be described below.

FIG. 10 is a flow chart showing processing of the image synthesizing device 10 according to the present embodiment.

As shown in FIG. 10, in step S101, the acquisition unit 21 acquires a first image and a second image. For example, the first image is the left rear image and the second image is the center rear image.

In step S102, the feature point determination unit 31 determines feature points in the first image and the second image acquired in step S101.

In step S103, the pair determining unit 32 determines pairs of the feature points determined in step S102.

In step S104, the first calculator 33 calculates a representative vector for each partial image in the first image and the second image acquired in step S101.

In step S105, the second calculator 34 calculates an individual vector for each pixel row using the representative vector calculated in step S104.

In step S106, the synthesizing unit 23 synthesizes the first image and the second image by calculating the pixel values in the vicinity of the boundary line 90 using the individual vector for each pixel row calculated in step S105.

In step S<b>107 , the display control unit 24 controls the electronic mirror 40 to display the synthesized image synthesized in step S<b>106 .

In this way, the image synthesizing device 10 can synthesize images while suppressing the feeling of strangeness that accompanies image synthesis.

As described above, the image synthesizing apparatus according to the present embodiment calculates pixel values so that the feature points of the same object captured in the first image and the second image are brought closer to each other. , synthesize the first image and the second image. In this calculation, after the first representative vector and the like are once calculated, an individual vector for each pixel row is calculated using the calculated first representative vector, and the pixel value is calculated using the individual vector. By once calculating the first representative vector and the like in this way, it is possible to eliminate the influence of vectors that are significantly different from others, which may be caused by incorrect matching of feature points, from affecting image synthesis. As a result, the image synthesizing device can synthesize images while suppressing a sense of incongruity that accompanies image synthesizing.

Also, the image synthesizing device calculates the median value of the plurality of vectors as the first representative vector or the like. This makes it possible to eliminate the influence of vectors that are significantly different from each other, which can be caused by mismatching of feature points, from affecting individual vectors. Therefore, the image synthesizing device can more easily synthesize images while suppressing the sense of incongruity that accompanies image synthesis.

Also, the image synthesizing device calculates individual vectors by linear interpolation of representative vectors. Therefore, based on a more specific configuration, the image synthesizing device can more easily synthesize images while suppressing the sense of incongruity that accompanies image synthesis.

Further, the image synthesizing device reduces the difference between the representative vector of one partial image and the representative vector of the adjacent partial image, thereby reducing the difference of the individual vectors between the partial images. Distortions in the image can be avoided. Therefore, the image synthesizing device can synthesize an image while suppressing a sense of incongruity caused by image synthesizing by avoiding occurrence of distortion between portions of the image.

In addition, the image synthesizing device reduces the amount of temporal change in the individual vectors by reducing the difference between the representative vector of one partial image and the representative vector of the past partial image at the same position. , distortions in the composite image can be avoided. Therefore, the image synthesizing device can synthesize an image while suppressing a sense of incongruity caused by image synthesizing by avoiding occurrence of distortion between portions of the image.

Also, the image synthesizing device can keep the number of pairs of feature points included in the partial images within an appropriate range. If the number of feature points included in the partial image is larger than the appropriate range, the variation in the vectors connecting the feature points and the midpoints between the feature points increases, and the number of vectors apart from the calculated representative vector also increases. Therefore, an appropriate representative vector may not be calculated. On the other hand, if the number of feature points included in the partial image is smaller than the appropriate range, or if the number of feature points used as the basis for calculation is small and the calculated representative vector does not necessarily represent the vectors included in the partial image. can also occur. Therefore, by keeping the number of pairs of feature points included in the partial images within an appropriate range and by calculating an appropriate representative vector, it is possible to synthesize images while suppressing the sense of incongruity that accompanies image compositing.

Also, the image synthesizing device generates a partial image using a horizontal dividing line. Therefore, the image synthesizing device can more easily synthesize images while suppressing the sense of incongruity that accompanies image synthesis.

Also, the image synthesizing device performs image synthesizing using hardware processing by a GPU. Therefore, the image synthesizing device can synthesize images while reducing the amount of processing and the processing time, and suppressing the sense of incongruity that accompanies image synthesizing.

Also, the image synthesizing device determines in advance the positional relationship between the first image and the second image using the position and orientation of the camera mounted on the vehicle. Therefore, the image synthesizing device can synthesize images more easily by using a predetermined positional relationship while suppressing a sense of incompatibility that accompanies image synthesizing.

Also, the image synthesizing device synthesizes images captured by the left camera, the right camera, and the center camera attached to the vehicle. As a result, the image synthesizing device synthesizes a wide range from the rear left side of the vehicle to the rear right side through the rear center, or from the rear right side of the vehicle to the rear left side through the rear center while suppressing a sense of incongruity accompanying image synthesis. It is possible to synthesize images that have been

(Embodiment 2)
In the present embodiment, an image synthesizing apparatus that synthesizes images while suppressing a sense of incongruity that accompanies image synthesis will be described. The image composition method according to the present embodiment may be executed independently of the image composition method according to the first embodiment, or may be executed in combination. Forms when implemented in combination are described in the following variations.

Hereinafter, functions and processing of the image synthesizing device 110 according to this embodiment will be described. An image synthesizing device 110 according to the present embodiment corresponds to the image synthesizing device 10 according to the first exemplary embodiment.

FIG. 11 is a block diagram showing the functional configuration of the image synthesizing device 110 according to this embodiment.

As shown in FIG. 11, the image synthesizing device 110 includes an acquisition unit 121, a determination unit 122, a synthesizing unit 123, and a display control unit . Each component described above may be implemented by a processor executing a predetermined program using a memory or the like, or may be implemented by dedicated hardware.

The acquisition unit 121 acquires a first image and a second image obtained by imaging one target under mutually different imaging conditions, and arranges the acquired first image and second image in a predetermined positional relationship in which a part overlaps. processing unit. Acquisition unit 121 arranges the first image and the second image in a virtual space through computation by a computer. When the acquisition unit 121 arranges the first image and the second image in the virtual space, the images to be arranged are corrected according to the optical characteristics of the optical system such as the camera that captured the images. You can place the image in . Corrections specifically include corrections based on lens distortion, as well as corrections for size and angle adjustments. By applying this correction, the difference between the optical systems of the cameras is suppressed, and the difference between the image derived from the first image and the image derived from the second image included in the rear image 50 generated by stitching is corrected. can have smaller differences such as color or size. The predetermined positional relationship in which the first image and the second image are arranged is determined according to the position and orientation of the camera that acquired the first image and the second image.

The determination unit 122 is a processing unit that determines a boundary for synthesizing the first image and the second image. The determination unit 122 has a feature point determination unit 131, a pair determination unit 132, and a curve determination unit 133 as its detailed components.

The feature point determination unit 131 is a processing unit that determines one or more first feature points in the first image arranged by the acquisition unit 121 and one or more second feature points in the second image arranged by the acquisition unit 121. be. The determination unit 122 extracts feature points in each of the first image and the second image. The process of extracting feature points is realized by, for example, a process of finding a pixel or a set of pixels included in the image and having a different color or shape from its surroundings. It can be done by image recognition technology.

Based on the first feature point and the second feature point determined by the feature point determination unit 131, the pair determination unit 132 determines one of the one or more first feature points at which the same object is captured. A plurality of pairs of the first feature point and one second feature point out of the one or more second feature points are determined. Specifically, the feature point determination unit 131 compares the color, shape, etc. of the pixel or the set of pixels found as the feature point in the first image with the pixel or the set of pixels found as the feature point in the second image. By doing so, images of the same object are found and determined as the pair.

The curve determining unit 133 selects, from among the plurality of pairs determined by the pair determining unit 132, pairs ( Determine a curve through the neighborhood of the near pair). The vicinity of a plurality of short-range pairs means the vicinity of one first feature point or one second feature point related to the short-range pair, and the vicinity of a line segment connecting the first feature point and one second feature point. including.

For example, the curve determination unit 133 determines the one curve by sequentially tracing the determined short-distance pairs. For example, the curve determining unit 133 may determine approximate curves of the determined short-distance pairs as the curves. Alternatively, this curve may be determined by a nonlinear least-squares method. In addition, the curve determination unit 133 determines the vicinity of a plurality of long-distance pairs in which the separation distance between one first feature point and one second feature point of the pair is equal to or greater than a predetermined distance, among the determined pairs. A curve that does not pass through may be determined.

The synthesis unit 123 is a processing unit that generates a synthetic image by synthesizing the first image and the second image using the boundary determined by the determination unit 122, that is, the curve determined by the curve determination unit 133 as the boundary. . Synthesis is performed by, in principle, cutting the first image and the second image by the boundary line and connecting them. At that time, processing such as obtaining a weighted average of the pixel values of the first image and the second image may be performed near the boundary.

The display control unit 124 is a processing unit that controls display of the synthesized image generated by the synthesizing unit 123 . The display control unit 124 outputs each pixel included in the composite image to the electronic mirror 40 so as to display the composite image on the electronic mirror 40, which is a display device.

Hereinafter, the image synthesizing process will be specifically described while showing examples of images.

FIG. 12 is an explanatory diagram showing an example of an image 151 that is a right rear image of the vehicle 1 according to this embodiment. An image 151 shown in FIG. 12 is an example of an image generated by imaging with the camera 13 in FIG. 1, and corresponds to the image 51 in FIG. The image 151 shows a vehicle, a road, a bridge pier, and the like existing behind the right side of the vehicle 1 . A region 161 of the image 151 is a portion that overlaps the central rear image (image 153) of the vehicle.

FIG. 13 is an explanatory diagram showing an example of an image 153 that is a central rear image of the vehicle 1 according to the present embodiment. An image 153 shown in FIG. 13 is an example of an image generated by imaging by the camera 14 in FIG. 1, and corresponds to the image 53 in FIG. The image 153 shows not only the vehicle, the road, and the bridge piers existing behind the vehicle 1 on the right side, but also the scenery behind the center of the vehicle 1 . A region 163 of the image 153 is a portion overlapping the right rear image (image 151) of the vehicle.

A related technique for synthesizing the image 151 and the image 153 will now be described. A related technique is a technique for synthesizing the image 151 and the image 153 using a straight boundary line.

FIG. 14 is an explanatory diagram showing an example of an image obtained by joining a left rear image and a center rear image of the vehicle 1 according to the related art.

Here, an example in which the image 151 and the image 153 are combined with the boundary line 170 as the boundary is shown. In FIG. 14, the image 151A is the image on the left side of the images obtained by cutting the image 151 along the boundary line 170. In FIG. Image 153A is the image on the right side of the images obtained by cutting image 153 along boundary line 170 . The image 155A, which is a composite image, is an image formed by connecting the image 151A and the image 153A with the boundary line 170 as a boundary.

The image 155A is a rearward image including the right rearward side and the central rearward side of the vehicle 1 . However, in the image 155A, the vehicle existing behind the vehicle 1 on the right side is distorted, in other words, part of the image corresponding to the vehicle existing behind the vehicle 1 on the right side disappears.

Image 155A is different from the view behind vehicle 1 as seen by the driver of vehicle 1 through a physical mirror. Therefore, when the image 155A is displayed on the electronic mirror 40, the driver may feel uncomfortable.

Next, a method for synthesizing images by the image synthesizing device 110 will be described.

FIG. 15 is an explanatory diagram showing a state in which an image 151 that is a left rear image of the vehicle 1 and an image 153 that is a center rear image of the vehicle 1 according to the present embodiment are partially overlapped.

The acquiring unit 121 acquires the image 151 and the image 153 using the camera 13 and the camera 14, respectively, and arranges the images in a predetermined positional relationship in which each image partially overlaps. This positional relationship is predetermined according to the positions and directions of the cameras 13 and 14 with respect to the vehicle 1 . More specifically, as shown in FIG. 15, an image 151 is arranged on the left side of the paper, and an image 153 is arranged by shifting the image 151 to the right on the paper by a distance M. Thus, the positional relationship in which the image 153 is displaced from the image 151 by the distance M in the right direction of the drawing corresponds to the predetermined positional relationship.

Then, the feature point determination unit 131 determines feature points of the images 151 and 153 . The feature points of each image determined by the feature point determination unit 131 are shown in FIGS. 12 and 13. FIG. The feature points in FIG. 12 are indicated by circles, and the feature points in FIG. 13 are indicated by triangles. These feature points can be obtained by using a known image recognition technique or the like to find a pixel or a set of pixels contained in an image that has a different color or shape from its surroundings.

Note that in FIG. 15, an area 161 in the image 151 and an area 163 in the image 153, which are overlapping portions of the images 151 and 153, are shown as an area 164 without performing any particular processing.

FIG. 16 is an explanatory diagram showing an example of boundary lines for synthesizing images according to the present embodiment. 17 is an enlarged view of frame XVII in FIG. 16. FIG.

The pair determination unit 132 determines pairs of feature points determined by the feature point determination unit 131 in the images 151 and 153 arranged as shown in FIG. This pair is a pair of points where the same object is imaged. For example, feature points p1 and q1 in FIG. 16 are the points where the right ends of the piers in images 151 and 153 are reflected. Therefore, the pair determination unit 132 determines the feature points p1 and q1 as a pair. Similarly, feature points p2 and q2 are determined as a pair because they are the points where the curved portion of the road lighting on the pier in images 151 and 153 are reflected. Feature points p3 and q3 are determined as a pair because they are points where a part of the body of the vehicle appears in images 151 and 153, respectively. Others are the same.

Regarding the determination of pairs of feature points detected in the two images 151 and 153, for example, the feature amounts of the detected feature points are calculated, the feature amounts are compared (matching), and the feature values having the same feature amount are calculated. Consider the points as pairs.

In addition, the pair determination unit 132 measures the distance between feature points determined to be paired. Then, the pair determination unit 132 determines pairs whose separation distance is equal to or less than the first predetermined value as short-distance pairs. Here, the first predetermined value is a distance at which the driver who visually recognizes the image displayed on the electronic mirror 40 determines that the two feature points related to the short-distance pair are the same point. Assuming that the total width L of the images 151 and 153 arranged as shown in 16 is about 10 inches in the number of full high-definition pixels, the distance can be about 10 to 20 pixels or less. Also, the pair determining unit 132 determines pairs in which the separation distance between the feature points is equal to or greater than the second predetermined value as long-distance pairs. Here, the second predetermined value is a distance at which the driver who visually recognizes the image displayed on the electronic mirror 40 judges that the two characteristic points related to the long-distance pair are clearly separate points, For example, if the total width L of the images 151 and 153 arranged as shown in FIG. can be done.

The curve determination section 133 determines a curve based on the pairs determined by the pair determination section 132 . Here, the curve determination unit 133 determines one curve passing through the vicinity of the short distance pair determined by the pair determination unit 132 . Note that the vicinity of the short-range pair includes the vicinity of the feature points (feature points p1, q1, etc.) related to the short-range pair and the vicinity of the line connecting the two feature points related to the short-range pair. Also, it is not necessary to pass through the neighborhood of all short range pairs.

Here, one curve is a curve having endpoints on the upper and lower sides of images 151 and 153 arranged as shown in FIG. A curve is thus a curve that has a starting point and an ending point on an edge of the image. In addition, one curved line may partially include a straight line portion.

As an example, a method of determining a curve by tracing feature points from the lower sides of images 151 and 153 will be described with reference to FIG.

First, the curve determination unit 133 finds the pair s located near the lower side, and starts the curve from the vicinity of the pair s on the lower side. The curve determining unit 133 searches for pairs near pair s and finds pair a and pair b. Here, the curve determination unit 133 determines to which of the pair a and the pair b the curve is extended as follows. Two pairs exist near pair a, and no pair exists near pair b. Therefore, the curve determination unit 133 determines to extend the curve toward the pair a.

Next, the curve determination unit 133 determines to which of pair c and pair d the curve should be extended as follows. In the vicinity of pair c, there are many pairs whose feature points are separated by a relatively large distance. On the other hand, in the vicinity of pair d, there are only pairs with relatively small distances between feature points. Therefore, the curve determination unit 133 determines to extend the curve toward the pair d.

In this way, the curve determination unit 133 sequentially traces the pairs to determine one curve.

Next, the synthesizing unit 123 synthesizes the images 151 and 153 . Synthesizing unit 123 synthesizes image 151 and image 153 with one curve determined by curve determining unit 133 as a boundary.

FIG. 18 is an explanatory diagram showing an example of an image extracted from an image 151 that is a left rear image of the vehicle 1 according to this embodiment. FIG. 19 is an explanatory diagram showing an example of an image extracted from the image 153, which is the central rear image of the vehicle 1 according to the present embodiment. FIG. 20 is an explanatory diagram showing an example of an image obtained by joining an image 151 that is a left rear image of the vehicle 1 and an image 153 that is a center rear image of the vehicle 1 according to the present embodiment.

The synthesizing unit 123 cuts and extracts an image 151B, which is a portion on the left side of the boundary line 171, which is the curve determined by the curve determination unit 133, from the image 151 shown in FIG. 12 (see FIG. 18). Further, the synthesizing unit 123 cuts and extracts an image 153B, which is a portion on the right side of the boundary line 171 on the paper surface, from the image 153 shown in FIG. 13 (see FIG. 19). Then, the synthesizer 123 obtains an image 155 that is a synthesized image by joining the image 151B and the image 153B arranged in a predetermined positional relationship.

Compared to the image 155A shown in FIG. 14, the distortion of the vehicle, that is, the disappearance of the image, is suppressed in the image 155. FIG. This can be said to be the result of setting the boundary for synthesizing the images in the vicinity of feature points with a relatively small separation distance between the feature points in the two images before synthesizing. As a result, the image synthesizing device 110 can generate a natural rear image 50 (see FIG. 2) in which the sense of discomfort is suppressed by synthesizing the images while suppressing the sense of discomfort that accompanies the image synthesis.

Hereinafter, the processing of the image composition device 110 will be described.

FIG. 21 is a flow chart showing processing of the image synthesizing device 110 according to this embodiment.

As shown in FIG. 21, in step S201, the acquisition unit 121 acquires a first image and a second image. For example, the first image is the left rear image and the second image is the center rear image.

In step S202, the feature point determination unit 131 determines feature points in the first image and the second image.

In step S203, the pair determining unit 132 determines pairs of the feature points determined in step S202, and further determines short-distance pairs in which the separation distance between the feature points is relatively small among the pairs.

In step S204, the curve determining unit 133 determines a curve that passes through the vicinity of the short-range pair of feature points determined in step S203.

In step S205, the synthesizing unit 123 generates a synthesized image by synthesizing the first image and the second image using the curve determined in step S204 as a boundary.

In step S<b>206 , the display control unit 124 controls the electronic mirror 40 to display the synthesized image synthesized in step S<b>205 .

In this way, the image synthesizing device 110 can synthesize images while suppressing the feeling of strangeness that accompanies image synthesis.

Note that the curve determination unit 133 may determine a curve through the vicinity of as many short-distance pairs as possible, as described below.

FIG. 22 is an explanatory diagram showing another example of a boundary line for synthesizing image 151, which is the left rear image, and image 153, which is the center rear image, of vehicle 1 according to the present embodiment.

The boundary line 171 shown in FIG. 22 is determined by the curve determination unit 133 so as to pass through the vicinity of as many short-distance pairs as possible. Such a curve can be determined by a known calculation method as a curve that minimizes the distance between the short distance pair and the curve. Specifically, one determined by a nonlinear least squares method can be adopted.

(Modification of Embodiment 2)
In this modified example, an image synthesizing apparatus executes a combination of the image synthesizing method of the second embodiment and the image synthesizing method of the first embodiment.

FIG. 23 is an explanatory diagram showing individual vectors and boundary lines calculated by the image synthesizing device according to this modification.

Boundary line 90A shown in FIG. 23 is boundary line 171 determined by curve determination unit 133 of the second embodiment. In the image synthesizing device according to the present modification, the boundary line 90A used by the synthesizing unit 23 of the first embodiment to join the first image and the second image is the Boundary 171 is used.

23 also shows individual vectors 81 to 84 calculated by the second calculator 34 in addition to the same vectors 71 to 78 as in FIG. These vectors are the same as those shown in FIG. 8, but are drawn with a parallel shift so as to end at boundary line 90A.

Then, the synthesizing unit 23 of the first embodiment joins the first image and the second image by calculating the pixel values using the individual vectors with the boundary line 90A as the boundary. The details of the bonding method are the same as those in the first embodiment, so the description is omitted.

As described above, the image synthesizing device according to the present embodiment and the present modified example synthesizes the first image and the second image using the boundary line determined according to the contents of the first image and the second image. do. This boundary line is set so as to pass through the vicinity of the pair of the feature points in the first image and the feature points in the second image in which the same object is imaged, and the separation distance between the feature points is relatively small. Therefore, the image doubling or loss that can occur near the border is suppressed. As a result, the image synthesizing device can synthesize images while suppressing a sense of incongruity that accompanies image synthesizing.

Also, the image synthesizing device uses an approximate curve to determine a boundary line to be used for synthesizing the images. Therefore, the image synthesizing device can more easily synthesize images while suppressing the sense of incongruity that accompanies image synthesis.

Also, the image synthesizing device uses the nonlinear least squares method to determine the boundary line used for synthesizing the images. Therefore, based on a more specific configuration, the image synthesizing device can more easily synthesize images while suppressing the sense of incongruity that accompanies image synthesis.

In addition, the image synthesizing device does not pass near the long-distance pair, that is, near the feature points of the first image and the second image relating to the long-distance pair, and near the straight line connecting these feature points. Set boundaries. Therefore, doubling or disappearance that can occur near the boundary is further suppressed. As a result, the image synthesizing device can synthesize images while further suppressing discomfort associated with image synthesizing.

Also, the image synthesizing device determines in advance the positional relationship between the first image and the second image using the position and orientation of the camera mounted on the vehicle. Therefore, the image synthesizing device can synthesize images more easily by using a predetermined positional relationship while suppressing a sense of incompatibility that accompanies image synthesizing.

Also, the image synthesizing device synthesizes images captured by the left camera, the right camera, and the center camera attached to the vehicle. As a result, the image synthesizing device synthesizes a wide range from the rear left side of the vehicle to the rear right side through the rear center, or from the rear right side of the vehicle to the rear left side through the rear center while suppressing a sense of incongruity accompanying image synthesis. It is possible to synthesize images that have been

Although the image synthesizing device according to the present embodiment has been described as synthesizing an image behind the vehicle, the present invention is not limited to this, and may be an image synthesizing image in front of or around the vehicle. , may be applied to any image composition other than a vehicle.

Although the image synthesizing device and the like according to one or more aspects have been described above based on the embodiments, the present invention is not limited to these embodiments. As long as it does not deviate from the spirit of the present invention, the scope of one or more embodiments includes various modifications that can be made by those skilled in the art, and configurations constructed by combining the components of different embodiments. may be included within

INDUSTRIAL APPLICABILITY The present invention can be used for an image synthesizing device or the like that synthesizes images while suppressing a sense of incompatibility that accompanies image synthesis. Specifically, it can be used for an image synthesizing device that synthesizes images from a camera mounted on a vehicle and displays them on an electronic mirror.

1 vehicle 10, 110 image synthesis device 12, 13, 14 camera 21, 121 acquisition unit 22, 122 determination unit 23, 123 synthesis unit 24, 124 display control unit 31, 131 feature point determination unit 32, 132 pair determination unit 33 First calculator 34 Second calculator 40 Electron mirror 50 Rear image 51, 52, 53, 151, 151A, 151B, 153, 153A, 153B, 155, 155A Image 61, 62, 63, 64 Partial image 62c, 63c Central pixel Column 66 Pixel Column 71, 72, 73, 74, 75, 76, 77, 78, u, v Vector 81, 81i, 81j, 81k, 82, 83, 84, w Individual Vector 90, 90A, 170, 171, 172 Boundary lines 91, 91i, 91j, 92, 92i, 92j, 92k, 93, 94 Pixels 133 Curve determination units 161, 163, 164 Regions a, b, c, d, s Pairs m Midpoints p, q Feature points p1, p2, p3, q1, q2, q3 Feature points

Claims (16)

an acquisition unit that acquires a first image and a second image obtained by imaging one target under mutually different imaging conditions, and arranges the acquired first image and the second image in a predetermined positional relationship in which the acquired first image and the second image partially overlap; ,
a feature point determination unit that determines one or more first feature points in the first image arranged by the acquisition unit and one or more second feature points in the second image arranged by the acquisition unit;
A pair of one first feature point out of the one or more first feature points and one second feature point out of the one or more second feature points, which are points at which the same object is imaged. A pair determination unit that determines a plurality of
For each of a plurality of partial images obtained by dividing the arranged first image and the second image by a dividing line extending in the first direction, from one first feature point associated with a pair included in the partial image, one calculating a first representative vector representing a vector reaching the midpoint of the first feature point and the one second feature point; a first calculation unit that calculates a representative vector;
calculating a first individual vector for each pixel row included in the first image and the second image by interpolating the first representative vector for each of the plurality of partial images calculated by the first calculation unit; a second calculation of calculating a second individual vector for each pixel row included in the first image and the second image by interpolating the second representative vector for each of the plurality of partial images calculated by the first calculation unit; Department and
calculating a pixel value for a pixel row included in the first image using the first individual vector, and calculating a pixel value for a pixel row included in the second image using the second individual vector; , and a synthesizing unit that synthesizes the first image and the second image from. The first calculation unit, for each of the plurality of partial images,
calculating, as the first representative vector, a median value of vectors from one first feature point to a midpoint between one first feature point and one second feature point relating to a plurality of pairs included in the partial image; ,
2. The image synthesizing device according to claim 1, wherein a median value of vectors from one second feature point related to said pair to said midpoint is calculated as said second representative vector. The second calculation unit
for each pixel row included in the first image and the second image, the first representative vector in the partial image is associated with the center pixel row of each of the plurality of partial images, and the first individual vector is obtained by linear interpolation; to calculate
for each pixel row included in the first image and the second image, the second individual vector is obtained by linear interpolation by associating the second representative vector in the partial image with the center pixel row of each of the plurality of partial images; 3. The image synthesizing device according to claim 1 or 2, wherein . The first calculation unit
Statistical processing for reducing differences between the first representative vector and the second representative vector of each of the plurality of partial images and the first and second representative vectors of partial images adjacent to the partial image. and
The second calculation unit
The image synthesizing device according to any one of claims 1 to 3, wherein the first individual vector and the second individual vector are calculated after the statistical processing by the first calculator. The acquisition unit acquires a plurality of the first images and the second images captured at different times,
The first calculation unit
The first representative vector and the second representative vector for each of the plurality of partial images, and the first representative vector and the second representative vector of a partial image located at the same position as the partial image and acquired before the partial image. Statistical processing is performed to reduce the difference from each of the second representative vectors,
The second calculation unit
The image synthesizing device according to any one of claims 1 to 4, wherein the first individual vector and the second individual vector are calculated after the statistical processing by the first calculator. The acquisition unit acquires a plurality of the first images and the second images captured at different times,
The first calculation unit
When the number of pairs included in the partial images among the plurality of partial images obtained by dividing the first image and the second image captured at one time is a predetermined number or more, the one reducing the size of the partial images of the first image and the second image captured at a time later than the time;
When the number of pairs included in the partial image is equal to or less than a predetermined number, increasing the size of the partial images of the first image and the second image captured at a time later than the one time. The image synthesizing device according to any one of claims 1 to 5. The image synthesizing device according to any one of claims 1 to 6, wherein the first direction is a horizontal direction. The second calculation unit calculates the first representative vector and the second representative vector by hardware processing by a GPU (Graphics Processing Unit),
The image synthesizing device according to any one of claims 1 to 7, wherein the synthesizing unit synthesizes the first image and the second image by hardware processing by the GPU. The image synthesizing device further comprises:
One curve passing through the vicinity of a plurality of short-distance pairs in which the separation distance between the one first feature point and the one second feature point of the pair is equal to or less than a predetermined distance among the determined pairs. A curve determination unit for determining
The image synthesizing device according to any one of claims 1 to 8, wherein the synthesizing unit synthesizes the first image and the second image using the one curve determined by the curve determining unit as a boundary. 10. The image synthesizing device according to claim 9, wherein the curve determining unit determines, as the one curve, one approximated curve that passes through the vicinity of the plurality of determined short distance pairs. 11. The image synthesizing device according to claim 10, wherein the curve determination unit determines the one curve by a nonlinear least-squares method. The curve determination unit selects a plurality of long-distance pairs in which a separation distance between the one first feature point and the one second feature point of the pair is equal to or greater than a predetermined distance among the determined pairs. The image synthesizing device according to any one of claims 9 to 11, wherein said one curve that does not pass through a neighborhood is determined. The first image and the second image are images captured simultaneously by two cameras fixed to one vehicle,
The image synthesizing device according to any one of claims 1 to 12, wherein the predetermined positional relationship is a positional relationship predetermined according to the positions and orientations of the two cameras. The image synthesizing device,
(a) a left rear image captured by a left camera that reflects the left rear of one vehicle;
(b) a right rear image captured by a right camera that reflects the right rear of the one vehicle;
(c) a central rear image captured by a central camera that reflects the central rear of the one vehicle;
and get
One of the left rear image and the right rear image and the central rear image are acquired by the acquisition unit as the first image and the second image, respectively, and the feature point determination unit and the pair determination unit determination by each, calculation by each of the first calculation unit and the second calculation unit, and generation of a third image by synthesis by the synthesis unit;
obtaining the third image and the other of the left rear image and the right rear image as the first image and the second image, respectively, and obtaining the feature point determination unit and the pair determination unit; The image composition according to any one of claims 1 to 9, wherein determination by each and calculation by each of the first calculation unit and the second calculation unit are performed, and a fourth image is generated by synthesis by the synthesis unit. Device. A control method for an image synthesizer, comprising:
an acquisition step of acquiring a first image and a second image obtained by imaging one target under mutually different imaging conditions, and arranging the acquired first image and the second image in a predetermined positional relationship in which a portion of the acquired first image and the second image overlap; ,
A feature point determination step of determining one or more first feature points in the first image arranged in the obtaining step and one or more second feature points in the second image arranged in the obtaining step;
A pair of one first feature point out of the one or more first feature points and one second feature point out of the one or more second feature points, which are points at which the same object is imaged. a pair determination step of determining a plurality of
For each of a plurality of partial images obtained by dividing the arranged first image and the second image by a dividing line extending in the first direction, from one first feature point associated with a pair included in the partial image, one calculating a first representative vector representing a vector reaching the midpoint of the first feature point and the one second feature point; a first calculation step of calculating a representative vector;
calculating a first individual vector for each pixel row included in the first image and the second image by interpolating the first representative vector for each of the plurality of partial images calculated in the first calculation step; a second calculation of calculating a second individual vector for each pixel row included in the first image and the second image by interpolating the second representative vector for each of the plurality of partial images calculated in the first calculation step; a step;
calculating a pixel value for a pixel row included in the first image using the first individual vector, and calculating a pixel value for a pixel row included in the second image using the second individual vector; a combining step of combining said first image and said second image from. The control method further comprises:
A curve passing through the vicinity of a plurality of short-distance pairs in which the distance between the one first feature point and the one second feature point of the pair is equal to or less than a predetermined distance among the determined pairs. a curve determination step of determining
16. The control method according to claim 15 , wherein in the synthesizing step, the first image and the second image are synthesized using the one curve determined by the curve determining step as a boundary.

Images