JPH07200650A - Building image synthesizing device - Google Patents

Abstract

PURPOSE:To provide a building image synthesizing device which can easily calculate a visual point with no use of a reference object and can synthesize a computer graphic CG image of a building produced by a computer with an actually photographed image. CONSTITUTION:A building image synthesizing device is provided with an edge detecting part 102 which detects an edge 2 of a building site out of an actually photographed image 1 of the site, a site information input part 103 inputting the length of the side and the diagonal line of the building site and the position of the site as site information 3, a focus calculating part 104 which calculates a focus 5 based on the site information 3 and the detected edge 2, a visual point calculating part 105 which calculates a visual point based on the focus 5, the information 3 and the edge 2, a building CG input part 6, and an image synthesizing part 107 which synthesizes a building CG 7 with the image 1. In such a constitution, a virtual rectangle is formed on the site, and the image of the rectangle projected on the image screen of an image pickup device is calculated based on the input site information 3. Then the focus and the visual point of the pickup device are calculated based on the projected image of the rectangle. The building CG is converted into an image acquired from the visual point and synthesized with the image 1.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a computer graphic of a building using a computer (hereinafter referred to as CG).
The present invention relates to a device for synthesizing a real image with a real image, and particularly to a device for efficiently synthesizing the information when information about a site where a building is built can be obtained.

[0002]

2. Description of the Related Art Conventionally, a CG image of a building is combined with a photographed image as a landscape simulation. In this landscape simulation, it is possible to perform a simulation of a state in which a building is built in an actual landscape by synthesizing a computer-generated CG image with a photographed image of a landscape where the building is set up.

When synthesizing a CG image of a building with a real shot image, the CG image must be rotated and enlarged / reduced according to the viewpoint of the real shot image. For that purpose, it is necessary to know the position of the viewpoint and the focal length of the camera that performs the shooting. It is also possible to record parameters such as the zoom, height, and distance of the camera at the shooting point and use them to calculate the viewpoint. However, it is difficult for general photographers to record parameters such as camera zoom,
This is a very time-consuming task.

Therefore, there is a device that automatically detects the viewpoint of the camera. When taking a real image, it is a device that captures an image of a reference object such as a rectangle or a cube whose shape and size are known, and calculates the viewpoint of the camera using the image of the reference object. For example, when CG of a house is to be combined on a live-action image, the camera is placed at a position where the image is taken, the reference object is placed at a position to be captured by the camera, and an image including the reference object is displayed before capturing the real-image to be combined. After shooting, the reference object is removed, and a real image to be combined is shot without moving the camera. Next, the shooting parameters of the camera are calculated using the image of the reference object, the viewpoint and the focal length are calculated, the CG of the house is rotated / reduced, the translation is performed, and the composition is performed on the actual target image. . In this case, a reference object of known size must be prepared in advance, and the reference object also becomes large to some extent in order to improve accuracy.

[0005]

In the above-mentioned conventional technique, a certain reference object must be placed in order to calculate the viewpoint of the real image to be combined, which is very troublesome. There is a problem that a reference object of a certain size is required to improve the accuracy.

In view of the above points, in the present invention, when a CG of a building is combined with a live-action image, the viewpoint, line-of-sight direction, and focal length of the real-image of the CG of the building are not placed as a reference object. It is an object of the present invention to provide an apparatus for synthesizing a real image and a CG of a building, which enables easy calculation directly from the target image, saves the operator's trouble when synthesizing, and enables easy synthesizing.

[0007]

In order to solve the above problems and to achieve the object, the present invention has an image input means for inputting a real image of a site, a site information input means for inputting site information, and the image input. Edge detection means for detecting the edge of the site from the photographed image input by the means, focus information for calculating the focus using the site information input by the site information input means, and the edge detected by the edge detection means Means, viewpoint calculating means for calculating a viewpoint using the site information input by the site information inputting means, the edges detected by the edge detecting means, and the focus calculated by the focus calculating means, and a computer for a building CG input means for inputting graphics, and the viewpoint calculated by the viewpoint calculation means are input by the image input means. And shot images, characterized by comprising the image synthesizing means for synthesizing a computer graphic of a building which has been entered by the CG input means.

[0008]

According to the present invention, the image input means inputs the photographed image, and the edge detection means detects the edge image of the site from the photographed image. Next, in the site information input means, information about the site shape and information about the position where the building is built are input as site information. The focus calculation means detects coordinates on the image plane of the endpoints of the site from the edge image, selects four arbitrary endpoints of the site, and virtually creates a rectangle on the site in the real space based on the site information. To do. Next, the coordinates on the image plane of the virtual rectangle are obtained, and the focal length is obtained using the coordinates. Next, the viewpoint calculating means calculates the viewpoint from the obtained focus, the coordinates of the end points of the site on the image surface, and the coordinates in the real space. Finally, the image composition means creates a CG from the viewpoint calculated by the viewpoint calculation means from the CG of the site.
Next, at the building position input by the site information input section on the site in the photographed image, the C of the building input by the CG input means
Synthesize G. In this way, the operator can easily combine the CG of the building with the photographed image without placing the reference object.

[0009]

1 is a block diagram showing the construction of a building image synthesizing apparatus according to an embodiment of the present invention. In FIG. 1, reference numeral 101 denotes an image input unit, which has a means for inputting a photographed image (1) such as a scanner or a video reproducing device, and a frame buffer which is a means for storing this photographed image (1). An edge detection unit 102 detects the edge (2) of the photographed image (1) held by the image input unit 101. A site information input unit 103 inputs site information (3) necessary for focus, viewpoint calculation, and image composition. A focus calculation unit 104 calculates the focus (5) of the photographing device (4) that has photographed the real image (1) using the edge (2) and the site information (3). 105 is a viewpoint calculation unit, which is the edge (2), site information (3), and focus (5)
Viewpoint (6) of the photographic device (4) that captured the real image (1) using
To calculate. Reference numeral 106 denotes a CG input unit, which is a building CG (7) to be combined with the photographed image (1) held by the image input unit 101.
Enter. Reference numeral 107 denotes an image combining unit, which converts the building CG (7) into an image from the viewpoint (6) and combines the image with the real image (1).
Create (8). In this way, the building image synthesizing apparatus according to the present embodiment makes it possible to synthesize the building CG (7) with the photographed image (1). The synthesizing processing procedure will be described in detail below.

FIG. 2 is a flow chart of the image synthesizing procedure of the present invention in the building image synthesizing apparatus shown in FIG. First, the image input section 10 is provided with a live-action image (1) including the site (9) where the building is constructed.
Input by 1 (S21). Next, the edge detection unit 102 detects the edge (2) of the site image (10) in the photographed image (1) of the site (9) (S22). Next, in the site information input unit 103, site information
Input (3) (S23). This site information (3) means site (9)
It is a parameter that represents the shape of the site, and is the length of the side of the site (9) and the length of the diagonal line between the endpoints. The method of inputting the site information (3) is as follows. First, in an arbitrary side, for example, in the site image (10) in the live-action image (1), the actual data obtained from the sketch of the site (9) on the side closest to you. Enter the length of.

Then, the actual length of each side is input in the clockwise order from that side. Once you have entered the actual lengths of the site (9) sides for all sides, then enter the diagonal lengths. With respect to the diagonal lines, the lengths of all diagonal lines are input in order from the arbitrary diagonal line. When the length of the side of the site (9) and the length of the diagonal line have been entered, enter the position information for building the building. This position information is the distance from the side of the site (9). The site information (3) is not input one by one as described above, but is also input by inputting the drawing of the site (9) using a scanner and reading the site information (3) from the input data. It is possible.

Next, when the site information (3) is input, the focus calculation unit 104 uses the site information (3) and the edge (2) of the photographing device (4) that has photographed the photographed image (1). Calculate focus (5) (S2
Four). The viewpoint calculation unit 105 determines the edge (2), site information (3), and focus.
Viewpoint of the shooting device (4) that shot the real image (1) using (5)
(6) is calculated (S25). Detailed calculation means of the focus (5) and the viewpoint (6) will be described later. After calculating the viewpoint (6) in this way, the CG input unit 106 captures and holds the building CG (7) to be combined with the photographed image (1). The image composition unit 107 is a building CG held by the CG input unit 106.
(7) is combined with the photographed image (1) to create a combined image (8) (S26).

First, the method of calculating the focus (5) will be described. FIG. 3 shows a flow chart of the procedure for calculating the focus (5). Three coordinate systems are used to calculate the focus (5). Figure 4 shows the three coordinate system used for perspective calculation, 401 is a spatial coordinate system O _w -X _w Y _w Z _w in the real space. The origin O _w is, for example, an end point of the site (9) in the real space, the inclination of the XY plane is the same as that of the site (9), and the Z axis is set vertically above the site (9). 402 is a photographing device
It is the image side of (4). 403 is an image coordinate system O _g -X _g Y _g. The center of the image plane 402 is the origin O _g, and the X axis is in the horizontal direction of the image plane and the Y axis is in the vertical direction. 404 is a viewpoint coordinate system _{O e -X e Y e Z e} . The viewpoint (6) is set as the origin O _e , passes through the image origin O _g , and the Z axis is taken perpendicular to the image plane 402, and the image coordinate system
The X axis and the Y axis are taken in the same direction as 403. These three coordinate systems are also used when calculating the viewpoint (6) and when performing composition.

First, the end point of the site (9) in the real space is S _i , and the projection point of this end point S _{i on} the photographed image (1), that is, the site image (1
The end point of 0) is S _i ′. In order to calculate the focal point (5), the coordinates of the site (9) in the real space are calculated (S31). From the end point S _i of the site (9), make sure that 3 points are not on the same straight line.
A representative point is selected (S32). From any of the four representative points, P ₁ , P ₂ , P ₃ , and P ₄ are set clockwise. In the case of a quadrangular convex type or a concave type formed by P _i , the point that becomes convex or concave is set to P ₄ . FIG. 5A shows the case where the selected representative point forms a convex quadrangle.
(b) is a case where the selected representative point forms a concave quadrangle.

Next, the representative points P ₁ and P ₃ shown in FIGS.
P is the intersection of the straight line connecting P and the representative points P ₂ and P _4.
Set to ₀ . Next, Q ₁ , Q ₂ , Q ₃ , and Q ₄ are taken so as to form a virtually rectangular shape with P ₀ at the center (S 33). In this embodiment, the midpoint of P _0, P ₁ and Q _1, the P ₂ direction from P ₀ on the straight line P ₀ P _2, the distance between P ₀ is the length of the line segment P ₀ Q ₁ Let the point be Q ₂ . For Q ₃ and Q ₄ as well as Q ₂ ,
They are taken on the straight lines P ₀ P ₃ and P ₀ P ₄ , respectively. Figure 5 shows the site (9)
An example of creating a rectangle (11) is shown above. 501 and 504 are sites
(9) and 502 and 505 are rectangles (1
1). 503 and 506 is a spatial coordinate system _{O w -X w Y w Z w} .

Coordinate values (Xw _(Pi) , Yw _(Pi) , Zw in the spatial coordinate system 401 (FIG. 4) of each representative point P _i shown in FIGS. 5A and 5B.
_(Pi) ) calculates the rectangle (11) based on the actual side size of the site (9) in the site information (3) and the length of the diagonal line.
(S34). Coordinate values (Xw _(Qi) , Y in the space coordinate system 401 of Q _i
w _(Qi) and Zw _(Qi) ) can be calculated from the distance from each representative point P _i . In this embodiment, the length of the line segment P _i P _j is d _(PiPj) , the length of the line segment P _i Q _j is d _(PiQj) , and the length of the line segment Q _i Q _j is d.
_{Expressed as (QiQj)} . In this embodiment, the origin of the spatial coordinate system 401 is P _1, and the X axis of the spatial coordinate system 401 is the straight line P ₁ P ₄ P.
₄ is positive. The Y axis is perpendicular to the straight lines P ₁ P ₄ ,
The direction is such that the Y coordinate value of the point P ₂ is positive. Then,
The coordinates of each point P _i , Q _i are as follows.

[0017]

[Equation 1]

[0018] Next, P _i 'a projection point on the image plane of the P _i, Q
_Let Q _i ′ be the projection point on the image plane of _i, and let P _i ′ be the image coordinate system 4
The coordinate values at 03 (Xg _{(Pi ')} , Yg _(Pi') ) are used. Q _i ′
Coordinate values in the image coordinate system 403 (Xg _{(Qi ')} , Yg _(Qi') )
Ask for. The coordinates of P _i ′ in the image coordinate system 403 can be easily obtained from the edge (2) of the photographed image (1) because P _i is the end point of the site (9). A theory called cross ratio is used for Q _i ′.

FIG. 6 shows the relationship between the four points on the straight line in the space and the projected points on the image plane. The four points A, B, C and D on the straight line 601 in the space and the four points are shown. The straight line 601 projected on the image plane has points A ′, B ′, C ′, D ′ on the projected image 602, which is called a cross ratio (Equation 2).

[0020]

[Equation 2]

The relation of (Equation 1) is used for the straight lines P ₁ P ₃ and P ₂ P ₄ in space. In this embodiment, the line segment P _i ′ P _j ′ is used.
Of the segment P _i ′ Q _j ′, and the length of the line segment P _i ′ Q _j ′ is d _{(Pi′Pj ′)
(Pi'Qj '),} the line segment Q _i' 'a length of d _(Qi'Qj' Q _j represents _a). In the present embodiment, Q ₁ is set so that d _(P0P1) = 2d _(P0Q1), and therefore, when arranged using that, d _{(P0'Qi ')} is

[0022]

[Equation 3]

[0023] _', the point Q _i (number 3) of d _(P0'Qi)' obtained by the coordinates in the image coordinate system 403 of the (Xg _{(Qi '),} Yg _(Qi')) is (Equation 4) .

[0024]

[Equation 4]

Next, the focus (5) is obtained based on this Q _i ′.
If the temporary focus is f ', the focus (5) is obtained by the following procedure. First, the following vector m _i is calculated for Q _i ′.

[0026]

[Equation 5]

[0027] Using the above vector m _i, to Q _j '' each side Q _i of the rectangle which can be in 'Q _i, obtaining the vector n _ij. m _i × m _j represents an outer product.

[0028]

[Equation 6]

Next, M and _{M'of the} following equations are obtained from the obtained _nij .

[0030]

[Equation 7]

When the focal point (5) is f and these M and M'are used, the focal point (5) can be obtained by (Equation 8).

[0032]

[Equation 8]

As described above, according to the present embodiment, it is possible to calculate the focal point (5) (hereinafter referred to as f in this embodiment) by virtually creating a rectangle on the site. (S
35).

Next, based on the focus f, the edge (2) and the site information (3), the viewpoint (6) (in this embodiment, referred to as E hereinafter) is calculated.

FIG. 7 shows a flow chart of the viewpoint calculation procedure. From the formula for calculating the distance to the viewpoint E and the projection point of three or more representative points whose coordinates are known in the spatial coordinate system 401 (S7
1), the coordinates of the viewpoint E in the spatial coordinate system 401 can be determined. In this embodiment, a formula for _obtaining the distance d _(EPi) to the viewpoint E and the four points P _i that are the representative points among the end points of the site (9) used to obtain the focus f is derived (S72), Using this, the coordinates (viewpoint) of E in the spatial coordinate system 401 are calculated (S7
3).

FIG. 8 shows a projection view of a rectangle formed by four selected points on the site, and the point P _i and the projection point P _i ′ on the image plane thereof.
Indicates. 801 is a quadrangle formed by the points P _i . Reference numeral 802 is an image plane. 803 is a quadrangle obtained by projecting the quadrangle 801 onto the image surface 802. A viewpoint E is 804. 805 is the spatial coordinate system O _w
Is a -X _w Y _w Z _w. 806 is a viewpoint coordinate system _{O e -X e Y e Z e} . The coordinate value of the projection point P _i ′ in the viewpoint coordinate system is (Xe
_{(Pi ')} , Ye _(Pi') , f). Viewpoint E and projection point P _i ′
Straight EP _i connecting the 'distance d _(EPi'), obtained by (Equation 9).

[0037]

[Equation 9]

Next, the distance d between adjacent P _i ′ and P _j ′
_{By (Pi′Pj ′)} and d _{(EPi ′)} , adjacent straight lines EP _i ′,
When the angle θ _ij of EP _j ′ is set, cos θ _ij is calculated by (Equation 10).

[0039]

[Equation 10]

If the cosine theorem is used for the triangle EP _i P _j , (Equation 11) is obtained.

[0041]

[Equation 11]

By solving this simultaneous equation, d _(EPi)
Is required. Next, the coordinates of E in the spatial coordinate system 401 (Xw
_(E) , Yw _(E) , Zw _(E) ) are obtained. If the distance d _(EPi) is expressed using the coordinates of the viewpoints E and P _{i in} the spatial coordinate system 401, _(Equation 1
It becomes like 2).

[0043]

[Equation 12]

When (Equation 12) is rearranged, the coordinates (Xw _(E) , Yw _(E) , Zw in the space coordinate system 401 of the viewpoint E as shown in (Equation 13) are obtained.
_(E) ) is obtained.

[0045]

[Equation 13]

Here, for Zw _(E) , two solutions of + and-appear. This is where the camera is above and below the site. This is to select one of the points on the site that comes to the front and one that comes to the back, and compare the Y coordinate values of the two points in the image coordinate system 403. For example, in this embodiment, the point in the foreground is P ₁ , and the point in the back is P ₃ . When the Y coordinate value of the point P ₁ coming to the front is larger than the Y coordinate value of the point P ₃ located at the back, the position of the camera is below the site,
Negative value. Conversely, when the Y coordinate value of the front point P ₁ is smaller than the Y coordinate value of the back point P ₃ , the camera position is above the site and has a + value.

Next, a procedure for arranging the building CG (7) on the photographed image (1) will be described. FIG. 9 shows a flow chart of a procedure for synthesizing a building CG into a photographed image. First, the building CG (7) is fetched and held in the CG input unit 106 (S91). The size of the building CG (7) is changed according to the spatial coordinate system 401 (S92). For example, if “1” in the spatial coordinate system 401 is the actual size “1 m” and the building CG (7) is described as “1” with the actual size “50 cm”, the building CG (7) is 1 Create a building CG (7 ') whose size is corrected by doubling.

Next, the coordinates in the spatial coordinate system 401 of the position where the corrected building CG (7 ') is to be combined on the photographed image are determined. The building position is obtained from the construction drawing, and rotation and parallel movement are performed so that the bottom surface of the building CG (7 ') is located at the position shown in the construction drawing. In this way, the building C
G (7) can be aligned with the construction position (S93) on the site of the photographed image (1), that is, the composite position.

Next, the space coordinate system 401 is converted into the viewpoint coordinate system 404 (S94), the coordinates in the image coordinate system 403 are obtained, and the coordinates are combined with the photographed image (S95).

First, a conversion formula from the spatial coordinate system 401 to the viewpoint coordinate system 404 is obtained. Space coordinate system 4 of arbitrary point Ri in space
The coordinates at 01 are (Xw _(Ri) , Yw _(Ri) , Zw _(Ri) ), and the coordinates at the viewpoint coordinate system 404 are (Xe _(Ri) , Ye _(Ri) , Z ₎ .
e _(Ri) ), the coordinate conversion formula is as follows.

[0051]

[Equation 14]

The points P ₁ , P ₂ , used when obtaining the viewpoint
Applying (Equation 14) to P ₃ and P ₄ , t _{11 of} (Equation 14),
_{t 12, t 13, t 14} , t 21, t 22, t 23, t 24, t 31, t
₃₂ , t ₃₃ and t ₃₄ are as follows.

[0053]

[Equation 15]

Point Qi where point Pi in space is projected on the image plane
In the viewpoint coordinate system 404 of (Xe _{(Pi ')} , Ye _(Pi') , Z
e _{(Pi ')} ) is as follows.

[0055]

[Equation 16]

The X coordinate of the point Qi in the screen coordinate system 403,
The Y coordinate is (Xe _{(Pi ')} , Ye _(Pi') ) in (Equation 16), respectively.
Matches The coordinates on the image plane of the composite CG10 are (Equation 16)
Can be synthesized by substituting each point into.

Thus, according to this embodiment, the site (9)
By creating a virtual rectangle on top, it is possible to calculate the focus and viewpoint without placing a reference object,
It is possible to easily perform composition by simply inputting the site information (3) from the site information input unit 103.

[0058]

As described above, the building image synthesizing apparatus of the present invention can easily detect the viewpoint without placing a reference object as in the conventional case, and automatically arrange the CGs of the building. Since it is performed, it is possible to efficiently perform the work of synthesizing the CG of the building with the photographed image.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a building image synthesizing apparatus according to an embodiment of the present invention.

FIG. 2 is a flowchart of the entire image synthesizing procedure of FIG.

FIG. 3 is a flowchart of the focus calculation procedure of FIG.

FIG. 4 is a diagram showing three coordinate systems used for the viewpoint calculation of FIG.

5 is a diagram showing a site in the space of FIG. 1 and a rectangle created on the site.

6 is a diagram showing a relationship between four points on a straight line in the space of FIG. 1 and projected points on the image of the points.

FIG. 7 is a flowchart of the viewpoint calculation procedure of FIG.

FIG. 8 is a projection view of a rectangle formed by four selected points on the site.

9 is a flowchart of a procedure for synthesizing the building CG of FIG. 1 into a real image.

[Explanation of symbols]

101 ... Image input unit, 102 ... Edge detection unit, 103 ... Site information input unit, 104 ... Focus calculation unit, 105 ... Viewpoint calculation unit,
106 ... CG input unit, 107 ... image combining unit, 401 ... space coordinate system _{O w -X w Y w Z w} , 402 ... image surface of the imaging device taken the photographed image, 403 ... image coordinate system O _g -X _g Y _g , 40
4 ... viewpoint coordinate system _{O e -X e Y e Z e} , 501,504 ... site (9),
502, 505… Rectangular (11), 503, 506 created on the site
... space coordinate system _{O w -X w Y w Z w} , 601 ... straight line in space,
602 ... Straight line 601 projection image projected on the image plane, 801 ...
A square made up of 4 points selected from the end points of the site in the space,
802 ... image plane, 803 ... quadrangle 801 projected on the image plane
The projected image of the, 804 ... point of view E, 805 ... space coordinate system O _w -X _w
Y _w Z _w, 806 ... viewpoint coordinate system _{O e -X e Y e Z e} .

 ─────────────────────────────────────────────────── ─── Continuation of front page (72) Inventor Mitsutaka Teshima 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd.

Claims (1)

[Claims] 1. An image input unit for inputting a real image of a site, a site information input unit for inputting site information, and an edge detecting unit for detecting an edge of the site from the real image input by the image input unit. Site information input by the site information input means, focus calculation means for calculating a focus using the edges detected by the edge detection means, site information input by the site information input means and the edge detection means A viewpoint calculation means for calculating a viewpoint using the edges detected by the edge and the focus calculated by the focus calculation means; a CG input means for inputting a computer graphic of a building; and a viewpoint calculated by the viewpoint calculation means. And the real image input by the image input means and the computer of the building input by the CG input means. A building image synthesizing device comprising image synthesizing means for synthesizing a tagraphic.