JP4832878B2 - Image composition processing apparatus and method - Google Patents

Description

  The present invention provides an image composition process for synthesizing a wide-angle image photographed with a short focal length and a telephoto image photographed with a long focal length from a part of the photographing region from the same photographing position as the wide-angle image. The present invention relates to an apparatus and a method.

  2. Description of the Related Art Conventionally, a technique for computer-analyzing a subject imaged in a digital image is used in the field of image measurement. There are various subjects for this image measurement. For example, landslides on cliffs and cracks in concrete structures such as bridges and piers are distributed over a fairly wide area and taken close to the subject. Many things are difficult.

  In order to photograph a subject far away from the photographing position in a wide area, it is common to photograph at a wide angle with a camera having a short focal length. However, in the wide-angle image captured in this way, the number of pixels of the photograph corresponding to the unit area of the subject is reduced, so the area of the subject corresponding to one pixel of the photograph image is increased, and as a result, There is a problem that the features in the image are blurred. This problem is particularly noticeable when a photographic image is enlarged and displayed.

  To solve this problem, use a telephoto image taken with a camera with a long focal length and take multiple photos from the same shooting position so that parts of the shooting range overlap. A method can be considered in which a plurality of images are connected to form one large connected image. According to this method, the number of pixels of the photograph shown per unit area of the subject increases, and the area of the subject projected per pixel is narrow, so that the feature shown in the photograph becomes clear.

  However, in the above method, when the optical axes of the individual photographs are different, there is a problem that a slight error occurs between the two photographs when the adjacent telephoto images are joined together. Especially in the case of photographic images, since the distortion of the subject image based on the aberration of the lens is large in the edge region of the image, a plurality of photos are taken so that the subject is common to both ends, and the plurality of photos are taken. When these are connected, the error (displacement) tends to increase. In other words, when the telephoto images are repeatedly stitched until a connected image in which the entire subject is photographed is created, the deviation of the last connected image from the original image position becomes cumulatively large. Therefore, it has been difficult to accurately capture the entire subject over a wide range and obtain a highly accurate linked (synthesized) image.

  Therefore, a technical problem to be solved by the present invention is to provide an image composition processing apparatus and method that can accurately capture an entire subject over a wide range and obtain a highly accurate image.

  In order to solve the above technical problem, the present invention provides the following image composition processing apparatus and method.

According to the first aspect of the present invention, at least one of a wide-angle image shot with a short focal length and a part of the shooting region from the same shooting position as the wide-angle image shot with a longer focal length. Image input means for inputting photographic image data with a first telephoto image of a sheet;
Reference point designating means for designating four or more sets of composite reference points that indicate the same position of the subject in common with the wide-angle image and the first telephoto image input by the image input means;
An arithmetic unit that calculates a corresponding projective transformation formula based on the coordinate value of the composite reference point specified by the reference point specifying means;
Coordinate point conversion means for projectively transforming the first telephoto image to create a second telephoto image by the projective transformation formula calculated by the arithmetic unit;
An image synthesizing means for synthesizing displayed in the corresponding position in the previous SL wide image Nozomu far image,
I have a,
The arithmetic unit calculates a scale for enlarging the second telephoto image so that the number of pixels does not decrease compared to the first telephoto image,
The coordinate point conversion means projects the first telephoto image based on the scale and the projective transformation formula to create a third telephoto image.
The image synthesizing unit synthesizes and displays the third telephoto image at a corresponding position in the wide-angle image;
An image composition processing device is provided.

According to a second aspect of the present invention, the arithmetic unit, before SL with calculates a rectangular bounding box circumscribing the second telescopic image, wherein the X-axis direction dimension and the Y-axis direction dimension of the first telescopic image 2. The image composition processing apparatus according to claim 1 , wherein an X-axis scale and a Y-axis scale respectively derived as a ratio of the X-axis direction dimension and the Y-axis direction dimension of the circumscribed frame are calculated as the scale. .

上記式において、ｘは第１望遠画像中の任意の点のＸ座標、ｙは第１望遠画像中の任意の点のＹ座標、ｘ_１６は第３望遠画像のＸ座標、ｙ_１６は第３望遠画像のＹ座標、ＳｘはＸ軸スケール、ＳｙはＹ軸スケール、Ａ，Ｂ，Ｃ，Ｄ，Ｅ，Ｆ，ａ，ｂ，ｃはそれぞれ射影変換式の定数を示す。 According to a third aspect of the present invention, the coordinate point conversion means creates the third telephoto image by projective conversion of each point of the first telephoto image according to the following equation. The image composition processing device described in 1. is provided.

In the above formula, x is the X coordinate of the X-coordinate of an arbitrary point in the first telescopic image, y is the Y coordinate of an arbitrary point in the first telescopic image, the x ₁₆ third telescopic image, y ₁₆ third The Y coordinate of the telephoto image, Sx is the X-axis scale, Sy is the Y-axis scale, and A, B, C, D, E, F, a, b, and c are constants of the projective transformation formula.

According to the fourth aspect of the present invention, at least one of a wide-angle image shot with a short focal length and a part of the shooting area from the same shooting position as the wide-angle image shot with a longer focal length. A method of combining and displaying using a telephoto image,
Selecting four or more sets of composite reference points indicating the same position of the subject in common with the wide-angle image and the telephoto image;
By projective transformation calculated based on the coordinate value of the composite reference point in the wide-angle image and the telephoto image, the first telephoto image is projectively transformed to create a second telephoto image,
Calculating a scale for enlarging the second telephoto image so that the number of pixels does not decrease compared to the first telephoto image;
The first telephoto image is projectively transformed based on the scale and the projective transformation formula to create a third telephoto image,
There is provided an image composition processing method characterized in that the third telephoto image is synthesized and displayed at a corresponding position in the wide-angle image.

According to the fifth aspect of the present invention, the calculation of the scale is performed by calculating a rectangular circumscribed frame circumscribing the second telephoto image, and calculating the X-axis direction dimension and the Y-axis direction dimension of the first telephoto image, and the The image composition processing method according to the fourth aspect is performed by calculating an X-axis scale and a Y-axis scale respectively derived as a ratio of the X-axis direction dimension and the Y-axis direction dimension of the circumscribed frame. To do.

  According to the first and fourth aspects of the present invention, the telephoto image and the wide-angle image are apparently used so that the subject image in the telephoto image matches the subject image in the wide-angle image. Synthesize. That is, the composite image displays a telephoto image for the corresponding subject image in the display of the composite photographic image. That is, since the composite photographic image uses fine information of the telephoto image for the entire subject image, fine image information can be obtained for the region.

  Even if there is a shift in the optical axis center between the wide-angle image and the telephoto image, the telephoto image is transformed as if it was taken from the optical axis of the wide-angle image by projective transformation. Thus, the subject images of both photographic images can be matched. Therefore, at the time of creating a composite photo image, it is possible to reduce the error of the subject image shift and create a highly accurate composite photo image having a subject image very close to the subject.

  According to the second and fifth aspects of the present invention, the X-axis scale and the Y-axis scale as the magnification of the telephoto image before and after conversion using the circumscribed frame circumscribing the conversion area of the telephoto image reduced by projective conversion. Thus, the projective transformation can be performed without reducing the telephoto image during the projective transformation. Accordingly, it is possible to eliminate the reduction in the number of pixels of the telephoto image due to the reduction at the time of projective conversion, and it is possible to display a high-definition image superimposed on the wide-angle image.

  Hereinafter, an image composition processing device according to an embodiment of the present invention will be described with reference to the drawings.

  The image composition processing apparatus according to the present embodiment is configured by a computer for inputting a photographic image and compositing the image. The camera may be for photo measurement with internal orientation or a general-purpose digital camera with no internal orientation. A general-purpose computer can be used, and a program for performing an image composition process as described later may be installed in the computer.

  The photographic image used in this apparatus is most preferably digital information obtained by photographing using a digital still camera or a digital video camera using, for example, a CCD (Charge Coupled Device) as an imaging element. It is also possible to use analog information obtained by photographing using a camera or an analog video camera and converted into digital information using a device such as a scanner. Photo images obtained as digital information can be directly handled by a computer. In addition, a hard copy of a photographic image formed on paper, a plastic film or the like can be directly used, and such a hard copy can be used as digital information by an image scanner or the like.

  FIG. 1 is a block diagram of an image composition processing apparatus 1 according to an embodiment of the present invention. The image composition processing apparatus 1 includes an interface unit 2 which is an example of an image input unit for connecting to a camera 7 and an input device 8 such as a keyboard and a mouse, a control arithmetic unit 3 which controls the entire device, a photographic image, and the like. A display unit 4 to be displayed, a storage unit 5 that functions as a storage area for photographic images and data to be combined, and a system program storage unit 6 that functions as a system program by installing the program are provided.

  As will be described later, the control calculation unit 3 is provided with blocks 3a to 3f that are driven by the system program and execute image synthesis. Each of these blocks functions as a functional block that performs the following processing alone or in cooperation with other blocks in the apparatus.

  Next, the flow of processing for performing image composition using the image composition processing apparatus 1 will be described with reference to FIG. FIG. 2 is a flowchart of image composition processing of the image composition processing device 1 of FIG. When the image composition processing is performed using the image composition processing device 1, the processing of each step shown below is mainly performed by the blocks 3a to 3f of the control calculation unit 3 of the device. That is, the reference point designating unit 3a, the projective transformation equation calculating unit 3b, the circumscribing frame calculating unit 3c, the scale calculating unit 3d, the coordinate point converting unit 3e, and the image synthesizing unit 3f, which are examples of calculating means, perform processing. Control.

  First, at the same photographing position, a wide-angle image and a telephoto image are photographed by the camera 7, and the images are input to the interface unit 2 of the image composition processing device 1 (step S1). The wide-angle image and the telephoto image input to the interface unit 2 are stored in the image storage unit 5 and displayed on the display unit 4 during operation.

  Note that the telephoto image is an image of a part of the wide-angle image, and may be two or more. Further, the telephoto image may be an image taken by changing the direction of the optical axis of the camera, as long as the telephoto image is taken from the same place as the wide-angle image. In the present embodiment, the telephoto image and the wide-angle image are described as being taken one by one for convenience of understanding. Further, it is desirable that the wide-angle image and the telephoto image used in the present embodiment are corrected in advance for distortion caused by lens aberration, but this is not an absolute condition.

  In this embodiment, the photographic image used for the image composition process is as shown in FIGS. 3 and 4, for example. 3 shows a wide-angle image 110 taken with the camera 7 in a short focal length state, and FIG. 4 is taken by the camera 7 with a long focal length from the same position as the wide-angle image 110 in FIG. 3 (first). A telephoto image 120 is shown. These photographic images have the same shooting position, but the position of the center of the optical axis of shooting may be shifted. That is, a part of the subject of the wide-angle image 110 is captured in the telephoto image 120. Any point in the image can be represented two-dimensionally using the XY coordinate system in units of the photographic image. In this embodiment, the longitudinal direction of the photographic image is the X axis, the short direction is the Y axis, and the origin is the upper left corner of each photographic image shown in FIGS.

  Next, the input device 8 is operated to designate four or more composite reference points indicating the same point of the subject in the wide-angle image and the telephoto image displayed on the display unit (step S2). In the present embodiment, as shown in FIG. 5, a total of eight points are designated, each of four reference points, that is, the combination reference points 11 a to 11 d in the wide-angle image 110 and the combination reference points 12 a to 12 d in the telephoto image 120. That is, four sets of synthesis reference points are designated.

The synthesis reference points 11a to 11d and 12a to 12d are designated by operating the input device 8 such as a mouse while referring to the wide-angle image 110 and the telephoto image 120 displayed on the display unit 4. The input points are recognized as the composite reference points 11a to 11d and 12a to 12d by the reference point designating unit 3a. At this time, if a feature portion of the subject such as an end portion of an object in the subject shown in the image is designated as the synthesis reference point, the same point can be easily designated for the wide-angle image 110 and the telephoto image 120. The coordinates of the composite reference point on the wide-angle image 110 are 11a ( _x11a , _y11a ), 11b ( _x11b , _y11b ), 11c ( _x11c , _y11c ), 11d ( _x11d , _y11d ), respectively. The coordinates on the telephoto image 120 are 12a ( _x12a , _y12a ), 12b ( _x12b , _y12b ), 12c ( _x12c , _y12c ), 12d ( _x12d , _y12d ), respectively. At this time, in order to obtain the coordinates of each set of designated synthesis reference points, it is possible to use a correlation process such as pattern matching described in Japanese Patent No. 3316682 for automation.

  Next, the projective transformation equation calculation unit 3b substitutes the coordinates of the four sets of composite reference points obtained by associating four or more points into the projection transformation equation shown by the following equation (1), thereby Nine constants (A, B, C, D, E, F, a, b, c) of the equation (1) are calculated (step S3).

式（１）において、ｘ_１２、ｙ_１２は望遠画像１２０側の合成基準点の座標１２ａ〜１２ｄを示し、ｘ_１１、ｙ_１１は広角画像１１０側の合成基準点の座標１１ａ〜１１ｄを示す。

In Expression (1), x ₁₂ and y ₁₂ indicate the coordinates 12a to 12d of the composite reference point on the telephoto image 120 side, and x ₁₁ and y ₁₁ indicate the coordinates 11a to 11d of the composite reference point on the wide-angle image 110 side.

  Nine constants (A, B, C, D, E, F, a, b, c) for superimposing the telephoto image on the wide-angle image can be obtained by inputting four or more sets of composite reference points. The telephoto image 120 is projected and converted to the (second) telephoto image 130 by this equation (1), and the telephoto image 130 is synthesized and displayed in the wide-angle image 110, so that the entire subject can be accurately captured and a high-accuracy composite image. Can be created.

  However, when this constant is applied as it is and the telephoto image 130 is combined with the wide-angle image 110, the number of pixels is reduced as the telephoto image 130 is reduced, so that the image definition is lost accordingly. End up. Further, as a method for solving this, a method of displaying the composite image as the original image with the telephoto image 120 as the original image while deforming from the equations (1) obtained above can be considered. Excessive computation is imposed on the device that displays the composite image, and the display speed of the composite image is reduced. For this reason, the following processing is performed.

Specifically, as shown in FIG. 6, the coordinate point conversion unit 3e converts the coordinates of the outer frame of the telephoto image 120 to the above constants (A, B, C, D, E, F, a, b, c). The value is converted using equation (1). Specifically, the projective transformation is performed on the coordinates of the four corner portions 13a to 13d of the telephoto image 120 by Expression (1). The points at which the four corner portions 13a to 13d of the telephoto image 120 are transformed by the equation (1) are set as outer frame transformation points 14a to 14d, and the coordinates thereof are 14a (X _14a , Y _14a ) and 14b (X _14b ), respectively. , Y _14b ), 14c (X _14c , Y _14c ), 14d (X _14d , Y _14d ) (step S4).

  Next, as shown in FIG. 7, the circumscribing frame calculation unit 3c circumscribes the area connecting the outer frame conversion points 14a to 14d from the coordinates of the outer frame conversion points 14a to 14d, and the horizontal direction is the X axis. And the circumscribed frame 15 having a rectangular shape along the Y axis is calculated (step S5). Specifically, among the coordinates of the outer frame conversion points 14a to 14d, the maximum value and the minimum value of the X coordinate and the maximum value and the minimum value of the Y coordinate are searched, and 15a (X maximum, Y minimum), respectively. The coordinates of four points are determined by a combination of 15b (X maximum, Y maximum), 15c (X minimum, Y maximum), 15d (X minimum, Y minimum), and the range surrounded by the points indicated by the coordinates is a circumscribed frame 15 is assumed.

  Next, the scale calculation unit 3d calculates the size Xo (the difference between 15a and 15d) of the circumscribing frame 15 and the size Yo (the difference between 15c and 15d) of the circumscribing frame 15 in the Y-axis direction. (Step S6), comparison with information on the X-axis direction size Xa (difference between 13a and 13d) of the entire telephoto image 120 and the Y-axis direction size Ya (difference between 13c and 13d) of the entire telephoto image 120 Then, the X-axis scale Sx and the Y-axis scale Sy are obtained based on the following formula (2) (step S7). Here, the X-axis scale Sx and the Y-axis scale Sy are scales for preventing loss of definition of the telephoto image.

式（２）において、ＳｘはＸ軸スケール、ＳｙはＹ軸スケール、Ｘａは望遠画像１２０全体のＸ軸方向の大きさ、Ｘｏは外接枠１５のＸ軸方向の大きさ、Ｙａは望遠画像１２０全体のＹ軸方向の大きさ、Ｙｏは外接枠１５のＹ軸方向の大きさを示す。

In Expression (2), Sx is the X-axis scale, Sy is the Y-axis scale, Xa is the size of the entire telephoto image 120 in the X-axis direction, Xo is the size of the circumscribed frame 15 in the X-axis direction, and Ya is the telephoto image 120. The overall size in the Y-axis direction, Yo indicates the size of the circumscribed frame 15 in the Y-axis direction.

Next, the converted telephoto image frame 16 is calculated by inputting the coordinates of the outer frame conversion points 14a to 14d to the equation (3) based on the X-axis scale Sx and the Y-axis scale Sy obtained in the above. . Specifically, as shown in FIG. 8, if the coordinates 14a to 14d of the outer frame conversion points are input to the equation (3), the coordinates 16a (x of the corresponding corner portions 16a to 16d of the converted telephoto image frame 16 are input. _16a , _y16a ), 16b ( _x16b , _y16b ), 16c ( _x16c , _y16c ), 16d ( _x16d , _y16d ) are calculated.

  As described above, each coordinate of the telephoto image 120 and the coordinates 14a to 14d of the outer frame conversion points have the relationship shown in the equation (1), so that the telephoto image 120 is represented by the X-axis scale Sx and the Y-axis scale Sy. The (third) telephoto image 140 can be converted without degrading the definition. That is, the relationship shown in Equation (4) derived from Equation (1) and Equation (3) is established between the coordinates of an arbitrary point in the telephoto image 120 and the coordinates of each point in the telephoto image 140.

式（４）において、ｘは望遠画像１２０中の任意の点のＸ座標、ｙは望遠画像１２０中の任意の点のＹ座標、ｘ_１６は望遠画像１４０のＸ座標、ｙ_１６は望遠画像１４０のＹ座標、ＳｘはＸ軸スケール、ＳｙはＹ軸スケールを示す。

In the formula (4), x is the X coordinate of an arbitrary point in the telephoto image 120, y is the Y coordinate of an arbitrary point in the telephoto image _120, the X coordinate of _{x 16} telephoto image _{140, y 16} is a telephoto image 140 Y coordinate, Sx represents the X axis scale, and Sy represents the Y axis scale.

  Accordingly, the coordinate point conversion unit 3e performs projective conversion on the telephoto image 120 based on Expression (4) as shown in FIG. 9 to create the telephoto image 140 (step S8). By this processing, projective transformation of an image used when the telephoto image 120 is pasted into the wide-angle image 110 can be performed without losing the definition of the image.

  Next, as shown in FIG. 10, the image composition unit 3f appropriately reduces and combines and displays the telephoto image 140 at the corresponding position in the wide-angle image 110 to create a composite image (step S9). In this case, the data (number of pixels) of the telephoto image 140 remains the same as before the reduction and is not reduced, and is displayed with the same fineness as the telephoto image 120. Thereby, the image composition process is completed.

  As described above, according to the image composition processing device according to the present embodiment, the image composition processing of a wide-angle image and a telephoto image is performed using projective transformation. Even when the optical axes of the two are largely deviated, such as when corresponding to the edge area of the image, the telephoto image is converted as if it was taken from the same optical axis as the wide-angle image, and the subject of both photo images The images can be matched. Therefore, it is possible to create a composite photo image without causing a subject image shift. Further, by using the X-axis scale and the Y-axis scale, it is possible to eliminate the reduction of the telephoto image that occurs at the time of projective transformation, and to reduce the number of pixels of the telephoto image displayed in the wide-angle image, thereby obtaining a fine composite photo image. Can do.

  In addition, this invention is not limited to the said embodiment, It can implement in another various aspect. For example, the telephoto image does not have to be one for each wide-angle image, and may be composed of a plurality of images obtained by photographing a plurality of regions of the wide-angle image.

1 is a block diagram of an image composition processing apparatus according to an embodiment of the present invention. It is a flowchart which shows the whole flow in the case of performing an image composition process using the image composition processing apparatus of FIG. It is an example of the wide-angle image image | photographed with the camera. It is an example of the telephoto image image | photographed with the camera. It is explanatory drawing of the process in selection of the synthetic | combination reference point of the image of 2 sheets synthesize | combined. It is explanatory drawing of the process which carries out projective transformation of the outer frame of a telephoto image, and produces an outer frame conversion point. It is explanatory drawing of the process which calculates an X-axis scale and a Y-axis scale based on the circumscribed frame circumscribing an outer frame conversion point. It is a conceptual diagram about a conversion telephoto image. It is explanatory drawing about the process which produces a conversion telephoto image from a telephoto image. It is explanatory drawing which synthesize | combines a conversion telephoto image in a wide angle image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image composition processing apparatus 2 Interface part 3 Control calculating part 4 Display part 5 Image memory | storage part 6 System program 7 Camera 8 Input device 11a-11d Wide angle side synthetic | combination reference point 12a-12d Telephoto side synthetic | combination reference point 13 Corner part 14 of a telephoto image Outer frame 14a-14d Outer frame conversion point 15 Outer frame 15a-15d Corner 16 of circumscribed frame Conversion telephoto image frame 110 Wide angle image 120 First telephoto image 130 Second telephoto image 140 Third telephoto image

Claims (5)

Photographic image data of a wide-angle image shot with a short focal length and at least one first telephoto image shot with a longer focal length in a part of the shooting area from the same shooting position as the wide-angle image Image input means for inputting
Reference point designating means for designating four or more sets of composite reference points that indicate the same position of the subject in common with the wide-angle image and the first telephoto image input by the image input means;
An arithmetic unit that calculates a corresponding projective transformation formula based on the coordinate value of the composite reference point specified by the reference point specifying means;
Coordinate point conversion means for projectively transforming the first telephoto image to create a second telephoto image by the projective transformation formula calculated by the arithmetic unit;
Image combining means for combining and displaying a telephoto image at a corresponding position in the wide-angle image;
Have
The arithmetic unit calculates a scale for enlarging the second telephoto image so that the number of pixels does not decrease compared to the first telephoto image,
The coordinate point conversion means projects the first telephoto image based on the scale and the projective transformation formula to create a third telephoto image,
The image synthesizing unit synthesizes and displays the third telephoto image at a corresponding position in the wide-angle image;
An image composition processing apparatus characterized by the above.   The computing device computes a rectangular circumscribed frame circumscribing the second telephoto image, and at the same time, measures the X- and Y-axis dimensions of the first telephoto image and the X- and Y-axis dimensions of the circumscribed frame. The image composition processing apparatus according to claim 1, wherein an X-axis scale and a Y-axis scale that are respectively derived as a ratio of the image are calculated as the scale. 3. The image composition processing apparatus according to claim 2, wherein the coordinate point conversion unit generates the third telephoto image by performing projective conversion on each point of the first telephoto image according to the following equation.

In the above formula, x is the X coordinate of the X-coordinate of an arbitrary point in the first telescopic image, y is the Y coordinate of an arbitrary point in the first telescopic image, the x ₁₆ third telescopic image, y ₁₆ third The Y coordinate of the telephoto image, Sx is the X-axis scale, Sy is the Y-axis scale, and A, B, C, D, E, F, a, b, and c are constants of the projective transformation formula. Using a wide-angle image shot with a short focal length and a portion of the shooting area from the same shooting position as the wide-angle image, and at least one telephoto image shot with a longer focal length A display method,
Selecting four or more sets of composite reference points indicating the same position of the subject in common with the wide-angle image and the telephoto image;
By projective transformation calculated based on the coordinate value of the composite reference point in the wide-angle image and the telephoto image, the first telephoto image is projectively transformed to create a second telephoto image,
Calculating a scale for enlarging the second telephoto image so that the number of pixels does not decrease compared to the first telephoto image;
The first telephoto image is projectively transformed based on the scale and the projective transformation formula to create a third telephoto image,
An image composition processing method comprising: combining and displaying the third telephoto image at a corresponding position in the wide-angle image. The scale is calculated by calculating a rectangular circumscribing frame circumscribing the second telephoto image, and measuring the X- and Y-axis dimensions of the first telephoto image and the X- and Y-axis dimensions of the circumscribing frame. 5. The image composition processing method according to claim 4, wherein the image composition processing method is performed by calculating an X-axis scale and a Y-axis scale respectively derived as a ratio of dimensions.

Images