JP2000030084A - Image compositing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily create a composited image of an actually photographed image and a CG image without feeling of incompatibility even without special experience. SOLUTION: The image compositing apparatus is provided with an operation part 20 to estimate visual point position information from the actually photographed image, an operation part 24 to estimate three-dimensional space information of the actually photographed image from the estimated visual point position information, an operation part 26 to estimate the size of an object from a coordinate value of the vertex of the object which is photographed on the actually photographed image and actual height of the vertex based on the three-dimensional space information, an operation part 28 to create a unit texture image to be a reference, a generating part 30 to generate the CG image for compositing from the estimated size of the object and the unit texture image, an operation part 22 to estimate light source position information from the actually photographed image and a compositing part 32 to perform perspective projection transformation for the generated CG image, to perform shade processing for the CG image based on the light source position information and to composite the CG image with the actually photographed image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus, and more particularly, to the creation of a product catalog based on a real image and the image of an interior of a house or the like after the interior has been changed. The present invention relates to an image synthesizing apparatus suitable for application to

[0002]

2. Description of the Related Art Conventionally, when image synthesis is performed using only a real image, in order to obtain a synthetic image without a sense of incongruity, a material of a real image which has been carefully calculated on the premise of the synthesis at the planning stage is prepared. These are combined by a dedicated machine such as a layout scanner for printing, an image processing station of a total scanner system, or a dedicated design system.

In recent years, buses used in houses and the like have
In order to create product catalogs for toiletries and kitchens, images are synthesized from actual background images and parts such as bathtubs created by CG (computer graphics) technology, and furniture, curtains, The interior such as wallpaper is created by CG, and the CG
2. Description of the Related Art By synthesizing an image with a real captured image in a room, an image obtained when the interior is changed from an obtained synthesized image is confirmed.

As described above, when a CG image is used as a material of a composite image, a photographed image to be used as a background is photographed under strictly determined conditions on the premise of composition, and when the photographing conditions are clear in advance, By creating a CG image using the shooting conditions, the actual captured image and the CG image are combined,
It is also possible to easily create a composite image without a sense of discomfort.

[0005]

However, in order to create a composite image without a sense of incongruity by combining a CG image with a real image that has not been photographed on the premise of composition, that is, using a real image whose photography conditions are unknown. However, since the operator generates a CG image by trial and error to find a condition suitable for a real image based on experience and intuition, and performs a combining process using the CG image, there is a problem that the combining operation is difficult.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned conventional problems. In synthesizing a CG image with an arbitrary real image, it is possible to easily generate a synthesized image without a sense of incongruity without special experience or intuition. It is an object to provide an image synthesizing device that can be created.

[0007]

SUMMARY OF THE INVENTION According to the present invention, a photographed image and C
In an image synthesizing apparatus for synthesizing a G image, a means for estimating viewpoint position information from a real shot image, a means for estimating three-dimensional space information of a real shot image from the estimated viewpoint position information, and an estimated three-dimensional space Means for estimating the size of an object to be synthesized in the real image based on the information; generating means for generating a CG image for synthesis based on the estimated size of the object; The above-mentioned problem has been solved by adopting a configuration including means for performing perspective projection conversion and synthesizing with the real image.

That is, in the present invention, a CG image to be synthesized with a real image can be created based on three-dimensional spatial information centered on a viewpoint estimated from the real image. Thus, it is possible to easily and reliably create a composite image having no sense of incompatibility by using each of these as a material without any special experience or intuition.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Generally, in order to create a composite image without a sense of incongruity, each material image used for composition must have the same shooting conditions, that is, the same viewpoint, angle, light hit, and the like. No. In order to create a composite image without a sense of incongruity using a real image whose setting conditions are unknown, it is necessary to estimate shooting conditions at the time of shooting the real image.

Therefore, in this embodiment, C
When synthesizing a G image, viewpoint position information (a viewpoint position, a viewing distance, a positional relationship of an object, and the like), which is a shooting condition, can be easily estimated from one real image without using dedicated hardware. Thus, there is provided an image synthesizing system (image synthesizing apparatus) having a function of generating a CG image for synthesis based on the estimated condition, performing perspective projection conversion of the generated CG image, and embedding the CG image into a real image. .

Further, this embodiment has a function of estimating light source position information from the actual photographed image and performing shading processing on the CG image based on the estimated light source position information.

In this embodiment, a function of estimating the actual size of a target position for synthesizing a CG image from the actual photographed image and a basic unit texture image are created from the actual photographed image serving as a texture of the CG image. It has a function and a function of creating a CG image for the target location from the unit texture image based on the estimated size information of the synthesis target (location).

Hereinafter, more specific embodiments will be described in detail with reference to the drawings. FIG. 1 is a block diagram illustrating a schematic configuration of an image composition system (image composition device) according to an embodiment of the present invention.

The image synthesizing system includes an image input device 10 such as a scanner for inputting a real image, an image holding memory 12 for storing image data of the input real image, and the like.
An image display device 14 for displaying an image based on the image data held in the memory 12;
And an image output device 16 for outputting the image data and the like after the synthesis stored in the image processing device.

The image holding memory 12 is connected to an operation unit 18 for executing various operation processes for image synthesis, which will be described in detail later, on the real image data input from the memory 12. The operation unit 18 includes a viewpoint position information operation unit 20, a light source information operation unit 22, an indoor space size information operation unit 24, a combined position size information operation unit 26, a unit texture image operation unit 28, a CG image generation unit 30, an image A data synthesizing unit 32 is included.

An information input unit 34 such as a pointing device such as a mouse is connected to the arithmetic unit 18 as a user interface. The information input unit 34 is displayed on the display of the image display device 14 while viewing the actual image and the like. The input unit 34 allows an operator to input data necessary for the image synthesis calculation process.

In this system, basic processing up to image synthesis is executed according to the flowchart shown in FIG. First, a real image is captured by the scanner 10 (step 1), the data is stored in the memory 12, and the real image is displayed on the image display device (display) 14. Then, by inputting information from the information input unit 34 while looking at the real shot image on the display 14, the viewpoint position information calculating unit 20 estimates viewpoint position information from the already read real shot image.

The estimation of the viewpoint position information executed by the viewpoint position information calculation unit 20 is performed by calculating the vanishing point (step 2), inputting the size information (step 3), and the viewpoint position information (viewpoint position) in the flowchart of FIG. , Visual distance) estimation (step 4).

The vanishing point calculation performed in step 2 is as follows:
Using the fact that parallel lines in a three-dimensional space intersect at one point, that is, a vanishing point, in a perspective view in a real image captured by a scanner, vanishing point coordinates are obtained from the parallel lines in the real image. .

That is, the lines representing the shapes of the table, window, tatami, ceiling and the like, which are projected on the actual photographed image of the room, are generally parallel in three dimensions. Therefore, if the real image is, for example, as shown in FIG. 3, the parallel line on the ceiling converges to the vanishing point.
One vanishing point is obtained as an intersection of two straight lines by designating two line segments parallel to each axis on the display.

The vanishing point coordinates at this time are obtained as a two-dimensional coordinate value which is a display coordinate system for a display.
However, since the image shown in FIG. 3 was taken with the camera installed horizontally with respect to the floor and the elevation angle = 0 as shown in FIG. 4, the vertical parallel lines are shown on the photograph screen. Since the positions are parallel to each other, there are only two left and right vanishing points, and there is no third vanishing point in the vertical direction.

The input of the size information in step 3 is to input the size information of a known object, such as the length of one side of a window, which is imprinted in the read real image, into the information input unit 34. Input. By inputting this size information, it is possible to obtain viewpoint position information such as a viewpoint position, which is a camera position at the time of shooting, and a viewing distance, which is a distance from the camera to the center of the projection plane (sight). . In this case, the more accurate the size information, the more accurately the viewpoint position can be obtained. However, it is sufficient that the size can be estimated to some extent.

In the viewpoint position information estimation in step 4, the viewpoint position and the viewing distance are calculated as central processing. Hereinafter, this will be described in detail. In addition, about this estimation method,
This is described in detail in Kondo, Kimura, and Tajima, “Estimation of viewpoint of hand-drawn perspective view and its application”, Transactions of Information Processing Society of Japan, July 1988.

First, a method for obtaining the viewpoint coordinates (viewpoint position) as the projection center will be described below. Here, it is assumed that the origin of the ground coordinate system is on a straight line connecting the viewpoint and the sight.

FIG. 5 shows the relationship between the viewpoint E and the vanishing point V, where F is the viewing distance. Consider a ray L containing a point P and having an angle α. At this time, the point P (x, y) is converted into P '(x', F) on the projection plane. When this point P is set to an infinite length on the half line L, it coincides with the vanishing point V. From this, the coordinates of the vanishing point of the straight line L are (F / tan α, F)
Becomes

FIG. 6 shows the relationship between the viewpoint coordinate system E-UVW and the ground coordinate system O-XYZ in a plan view of (A) and a side view of (B). Here, the viewpoint is E, the visual axis is V, a line segment orthogonal to the line segment V1-V2 is drawn from the viewpoint E, and the intersection point is HL. HL 'is the HL
E 'indicates the coordinates of the side view of the viewpoint, and F' indicates the distance from the viewpoint E to HL. FIG. 6 shows a state of being inclined by α around the W axis and by β around the U axis.
The vanishing points V1, V2, and V3 are as follows in the coordinate system C-UW of the screen whose origin is the visual center C.

V1 = (F ′ / tanα, Ftanβ) (1) V2 = (− F′tanα, Ftanβ) (2) V3 = (0, −F / tanβ) (3) F ′ = F / cosβ (4) HL = (0, Ftanβ) (5)

Using the above equations (1) to (5), V1,
When V2 and V3 are known, the azimuth angle α, the elevation angle β, the visual distance F, and the visual center C are obtained by the following procedure. This will be described with reference to FIG.

(1) The midpoint of the line segment V1-V2 is determined, and a circle having a diameter V1-V2 is drawn around the midpoint.

(2) An intersection HL 'between the perpendicular drawn from V3 to the straight line V1-V2 and the straight line V1-V2, and an intersection E with the circle are obtained.

(3) Line segment E-HL 'and line segment HL'-V2
The angle α is further obtained.

(4) Line segment E-HL 'and line segment HL'-V3
The viewing distance F is obtained from the calculation.

(5) The visual distance F is calculated by using the above equation (4).
And the angle β is determined from the line segment E-HL ′.

(6) The intersection of the perpendicular drawn from V1 to the line segment V2-V3 and the perpendicular drawn from V2 to the line segment V1-V3 is defined as the visual center C.

Next, a method of estimating viewpoint position information when the visual center information is known will be described with reference to FIG. 8 which is substantially the same as the two vanishing point images shown in FIG.

It is presumed that the photographed image G shown in FIG. 8 was taken with a camera installed horizontally with respect to the floor surface, and the elevation angle β is 0 °. In the case of such two vanishing points, the vanishing point is located on an extension of the eye height (line of sight) as shown in FIG. Since the image G has not been trimmed after being captured by the scanner, FIG.
As shown in FIG. 7, the visual center C, which is the center of the line of sight, is on the line connecting the two vanishing points and at the center in the x-axis direction, and the visual distance F is calculated by the following procedure.

(1) In order to find two vanishing points on the left and right,
Specify two sets of parallel lines that are parallel to each other in three-dimensional space,
Vanishing points V1 and V2 are calculated as intersections of two straight lines.

(2) The midpoint M of the line segment V1-V2 is determined, and a circle having the diameter V1-V2 is drawn around the midpoint.

(3) The center C on the line segment V1-V2 and in the x-axis direction of the actual photographed image is obtained.

(4) A perpendicular line is drawn from the visual center C to a semicircle, and the intersection point becomes the viewpoint E.

(5) Obtain the viewing distance F from the line segment EC.

On the other hand, when the visual acuity information is unknown, that is, when the visual acuity is deviated from the center of the actual image and is unknown because the image is a two vanishing point image but is trimmed, the viewpoint A method for estimating position information will be described with reference to FIG. This estimation method is described in F.S. This is described in detail in Hohenberg's Masuda translation, "Constitutive Geometry in Technology" (Vol. 1, Nihon Hyoronsha).

In the actual image G shown in FIG. 9A, a rectangular parallelepiped indicated by a thick solid line is imprinted. However, since the left end is cut off by trimming, the center of the image is unknown. However, in this case, A′B ′ in the image G
It is assumed that a rectangle corresponding to a part of the upper surface of a rectangular parallelepiped indicated by C′D ′ (D ′ is not visible) is a square ABCD whose dimensions a and b are clear as shown in FIG.

In the actual three-dimensional space, two line segments which are parallel to the horizontal line and are orthogonal to each other in the image, the lengths of the line segments AB and AC in FIG. As described above, the viewpoint position, that is, the viewing distance can be estimated by the following procedure.

(1) In order to find two vanishing points on the left and right,
The vanishing point V1, from two sets of parallel lines parallel to the horizontal line,
Find V2.

(2) The midpoint of the line segment V1-V2 is determined, and a circle having the diameter V1-V2 is drawn around the midpoint.

(3) Convert A'B'C '(D') in which the rectangle ABCD is imprinted on the image into the plan view A {B {C} D} on the circumference obtained above.

(4) Extend line segment B {-D} to obtain line segment V1
The point F that intersects −V2 is the vanishing point of the line segment BD. That is, V1, V2, and F are line segment AB and line segment B-, respectively.
C is a point at which lines parallel to the line segment BD intersect on the screen.

(5) The angle DBC, α, is given by the line segment BC and the line segment CD.

(6) The viewpoint E is a space having a diameter V1-V2.
, And also on a horizontal circle having a circumferential angle 2α with respect to the chord F-V2, it is given as the intersection of these circles.

(7) The visual center, here H, is determined from the perpendicular drawn from the viewpoint E to the line segment V1-V2, and the viewing distance F is determined from the line segment EH.

The above description is for a case of two vanishing point images. Next, a method for estimating viewpoint position information in one vanishing point image will be described.

FIG. 10 (A) shows a method of estimating viewpoint position information using vanishing points and coordinate values input in advance with respect to a reference plane projected on a real image. is there. At this time, the known information is assumed to be the vanishing point V and the coordinates of each vertex of the square ABCD as the reference plane.

Here, for convenience, the reference plane corresponds to the upper surface of a rectangular parallelepiped whose size is known in the actual image as shown in FIG. 10B by changing the relationship with the viewpoint position. It is rectangular. Therefore, the size and height of the reference plane are known, and the size is vertical and horizontal. The height is 0 if the rectangle is on the floor, and the floor if the rectangle is above the floor. Given by height from

In FIG. 10A, the straight line SL is a horizontal line, and the square abcd has a side ab corresponding to the line segment AB and the side ab is SL.
If it is a plan view of a square ABCD in contact with
The viewpoint E and the viewing distance F can be obtained by a calculation method based on a drawing method of a perspective view described below.

1. A vanishing point V is obtained from an intersection of a pair of horizontal parallel lines AD and BC in a real image in which a rectangular ABCD is imprinted.

2. Lines perpendicular to the horizontal line SL are drawn from both ends of the front side AB in the horizontal direction on the input reference plane ABCD, and their intersection points are a and b, respectively.

3. Here, the reference plane is the upper surface (rectangle) of the rectangular parallelepiped as described above, and the lengths of the sides are input in advance. A plan view abcd of the reference plane ABCD is created corresponding to the vertices A and B.

4. Lines perpendicular to the horizontal line SL are drawn from the points at both ends of the horizontal side CD in the horizontal direction of the reference plane, and their intersection points are C 'and D', respectively.

5. From the points c and d in the plan view, C ′,
Draw a half line through D '. The intersection point of these two straight lines is the viewpoint E.

6. The viewing distance F is obtained from the distance between the viewpoint E and SL.

As described in detail above, the calculation for estimating the viewpoint position information such as the viewpoint position, the viewing distance, and the positional relationship of the object is executed by the viewpoint position information calculating section 20. When the processing is completed, three-dimensional space information is estimated from the two-dimensional photographed image using the viewpoint position information, and in the case of the photographed image of FIG. 3 or FIG. It is possible to estimate the size of the space (step 5).

This will be specifically described with reference to the actual image shown in FIG. 12 displayed on the screen of the image display device 14.
However, since it is difficult to understand the actual photographed image, it is simplified in FIG.

In the image synthesizing system of this embodiment, when the point indicated by the arrow A is designated by clicking with a mouse or the like on the actual photographed image, the point is centered in the vertical direction and from the center to the left and right vanishing points. An axially extending guide line consisting of the respective lines is displayed.

Then, the reference point in the actual photographed image, that is,
When the point at which the floor intersects with the wall or the point at which the ceiling intersects with the wall is designated with a mouse or the like, the space calculated from the interior space size information calculation unit 26 is used as shown in FIG. Can be trace-displayed with a mesh, and the three-dimensional configuration of the space reflected in the actual image can be estimated. Further, this image synthesizing system also has a guide function that allows a user to specify a point at which a floor and a wall intersect with each other by using a mouse or the like while guessing the point even when the point is not shown in the actual image.

The size of the indoor space estimated as described above is
As shown in FIG. 11 or FIG. 13, based on the three-dimensional information obtained from the actually photographed image, it is possible to confirm by inserting a mesh at intervals of, for example, 40 cm. In this drawing, although two-dimensionally displayed for convenience, a three-dimensional mesh having a size of, for example, 40 cm × 40 cm × 40 cm is actually applied.

Next, the three-dimensional conversion of the two-dimensional image in the two vanishing point images, which is performed to estimate the three-dimensional information from the photographed image, will be described in detail.

As shown in FIGS. 5 and 6, it is possible to convert a two-dimensional image into a three-dimensional image by utilizing the fact that the side of the rectangular parallelepiped is parallel to the straight line connecting the viewpoint and the vanishing point. This will be described in detail with reference to FIG. This method is described in detail in "Sketch Interface for 3D Shape Generation" by Sugishita, Saitama University, February 1994.

In FIG. 14, at points P1 and P2 and points on the projection plane,
A straight line connecting both points passes through the vanishing point V. C is the sight and E is the viewpoint. A straight line connecting the viewpoint and the vanishing point is parallel to a straight line passing through P1 'and P2' in the three-dimensional space, and P1 'exists on a straight line connecting the viewpoints E and P1, and the viewpoints E and P2 Since P2 'exists on a straight line connecting P1' and P2 ', P1', P2 '
The position (coordinates) of 2 'can be determined.

Then, the visual center C is set at the origin (0, 0, 1) of the ground coordinate system.
0, 0), and the viewpoint E is assumed to be located in the positive direction on the x-axis of the ground coordinate system, and P1 (x1, y1), P2 (x2, y2) P1 '(x1', y1) ', Z1'), P2 '(x2',
y2 ', z2') C (x0, y0), E (F, 0,0), V (xs, ys
, Zs), the coordinate values of P1 'and P2' are obtained by the following equations (6) to (12) using the parameters t and s. Here, F is a viewing distance.

X1 '= F (1-t), y1' = tY1, z1 '= tZ1 (6) t = AB, Y1 = x1-x0, Z1 = y0-y1 (7) x2' = F ( 1-s), y2 '= sY2, z2' = sZ2 (8) s = AG, Y2 = x2-x0, Z2 = y0-y2 (9) A = D / [{F (BC)} ^{2 + (CY2 -BY1) 2 +} (CZ2 -BZ1) 2] 1/2 ... (10) B = (y0 -ys -Z2) / (Z2 -Z1) ... (11) G = (BY1 + xs -x0) /Y2...(12)

In the above equation (10), D is the distance between P1 'and P2'. By giving the distance between P1 'and P2', a three-dimensional shape can be obtained from a two-dimensional shape. The obtained three-dimensional shape is a coordinate under the assumption that the visual center C is located at the origin of the ground coordinate system and the viewpoint E is located in the positive direction on the X axis. In order to obtain the coordinates of, it is necessary to perform coordinate conversion using a conversion matrix. The conversion matrix is configured using the azimuth angle α and the elevation angle β obtained earlier. If the coordinates of the first two points are obtained, the coordinates of the remaining points can be obtained based on the obtained coordinates.

Next, the case of three-dimensional conversion (reconstruction) of a two-dimensional image in the one vanishing point image will be described with reference to FIG. The suffixes 2, 3 added to A, B, C, P below
Represents two-dimensional coordinates and three-dimensional coordinates of each point, respectively.

Points A and B in the figure are points at both ends of a line segment AB serving as a size reference, and points A 'and B' are coordinates on a three-dimensional space. Point P is the intersection of a line drawn in parallel with line segment A'B 'from viewpoint E3 in three dimensions and projection plane S.

On the projection plane S, the visual center C2 (xc, yc
), Viewing distance F, point A2 (xa, ya), point B2 (xb,
yb) and point P2 (xp, yp), these are three-dimensionalized assuming that the viewpoint is orthogonal to the projection plane.

A3 (0, xa-xc, ya-yc), B
3 (0, xb-xc, yb-yc), C3 (0, 0,
0), E3 (F, 0, 0), P3 (0, xp -xc, y
p-yc)

The points A 'and B' correspond to the straight lines EA and E, respectively.
On B, if s times the length of the line segment EA is the length of EA 'and t times the length of the line segment EB is the length of EB', the following equation (13) is used. Equation (14) holds.

[0078]

(Equation 1)

Since the line segment A'B 'and the line segment EP are parallel, assuming that the length of the line segment A'B' is D, the following equation (15) is used to obtain the equation (16). Is obtained.

[0080]

(Equation 2)

The above equations (13), (14) and (16)
From the equations, A '(za', ya ', za'), B '(xb
, Yb ', zb') are the following (1
Expressions 7) and (18) are provided.

[0082]

(Equation 3)

The constant s in the equation (17) and the constant t in the equation (18) are as shown in the following equations (19) and (20), respectively.

[0084]

(Equation 4)

In this case, the length reference is the diagonal line of the input reference plane. However, the three-dimensional reconstruction can be performed on the horizontal and depth sides of the reference plane by changing the equation according to them. is there.

When the coordinates of the first two points A 'and B' are obtained, the coordinates of the remaining points can be obtained based on the obtained coordinates.

The above-described step 4 is performed by performing the calculation processing for estimating the three-dimensional space information from the two-dimensional photographed image described above in detail by the indoor space size information calculation unit 24.
The stereoscopic reconstruction as shown in FIG. 11 or FIG. 13 described above can be performed using the viewpoint position information estimated in. That is,
By using the size information input in the step 3, it is possible to trace and display the size of the space viewed from the viewpoint position with a mesh, and to provide three-dimensional information of the space projected on the real image, that is, The size of the indoor space can be estimated. Therefore, as shown in FIG. 11 or FIG. 13, it is possible to create an image in which a mesh having a larger size is inserted at a position closer to the viewpoint.

In step 5 of FIG. 2, when the estimation of the size of the indoor space is completed, the obtained three-dimensional space information is output from the information calculation unit 24 to the memory 12 and held.

Next, the light source condition is estimated using the information (step 6). The estimation of the light source information is performed by designating the position of the light source projected on the actual image displayed on the display of the image display device 14 with a mouse or the like, and then the position is described in the three-dimensional space. This is performed by estimating and calculating by the light source information calculation unit 22 using the information.

That is, since the viewpoint position information is obtained as described above, it is possible to estimate three-dimensional space information such as the position and size of the object projected in the actual image by using the viewpoint position information. Thus, it is possible to estimate the three-dimensional position of a light-emitting object such as a lighting fixture or a window in the image, that is, the light source. CG to be synthesized when the position of the light source is determined
The direction in which light illuminates the object in the image can be calculated optically.
The light source position (condition) estimated in this way is output to the memory 12 and held.

Next, in step 7, an object to be combined with the CG image, which is an object projected on the real image, is selected, and the vertex of the object is designated on the display of the image display device 14 with a mouse or the like. By inputting the actual height of the point, the size of the target object, that is, the size of the combined position is estimated and calculated by the combined position size information calculating unit 26.

This will be specifically described with reference to FIGS. 16 and 17 corresponding to FIG. When the process shifts to the synthesis position size estimation, the tip of the cursor moved by the mouse on the screen is set as an intersection, and a line that extends vertically through the intersection and
Two guidelines consisting of a line passing through the intersection to the vanishing point are displayed.

Then, the user moves the cursor to the object to be combined (here, the door of the storage shelf), designates the vertex as shown in FIG. 16, and changes the height of the point in the three-dimensional indoor space. Input, and then specify the vertex at the diagonal of the specified vertex in the same way. From the above operation, the size (length × width) of the door as the object is estimated. The above operation is similarly performed for the other doors. The size information estimated in this way is output to the memory 12 and held.

In the next step 8, the unit texture image calculation unit 28 creates a unit texture image which is the basis of a pattern, a pattern, and a shape from a real image which is a texture of a CG image. This is output to the memory 12 and stored in a predetermined texture file.

FIG. 18 conceptually shows the procedure for creating this unit texture image. FIG. 2A shows an original texture image, which corresponds to a real image that becomes a texture of the CG image. When a basic texture unit image is created from this texture image, (1) a repetitive pattern or a basic unit portion of the pattern is designated, and (2) a basic texture unit as shown in FIG. Enter the unit size information (in this case, the vertical and horizontal dimensions),
(3) The creation is completed by storing the above information and the like in the texture file in association with the corresponding texture image. A necessary type of unit texture image is created by the same operation, and is stored in the corresponding texture file.

Further, the CG image generation unit 30 uses the combined position size information read from the memory 12 and the unit texture image on the memory 12 read from the texture file specified by the information input unit 34. Also, using the already-designated viewpoint position information, light source information, and the like, together with the actual image data read from the memory 12, generation of a texture image in accordance with the actual size of the target object, and location of the texture image in the actual image Is generated (step 9). FIG. 19 shows an image of the CG image generated as described above. However, the gradation is omitted.

The generation of the CG image executed here will be described in further detail. The brightness of the surface of the CG image changes depending on the spatial orientation of the surface, the position of the viewpoint, and the position of the light source. That is, the viewpoint and the position of the light source are determined from the viewpoint position information of the real image, and the direction of the surface is determined by designating the position where the CG image is synthesized, and the shape of the shadow is obtained.

Next, the reflection coefficient is determined by designating the material of the object to be synthesized, that is, the texture, and the conditions necessary for the shadow processing of the CG image are prepared. The CG image generated by performing the shading process using such a condition is output to the image data synthesizing unit 32.

In the image data synthesizing section 32, when the CG image generated in the step 9 is input, the image data synthesizing section 32 designates the information input section 34 such as a mouse and the like. Using the coordinate values of the compositing target position input in step 7 read from the memory 12 with respect to the compositing position, perspective projection transformation is performed on the CG image in accordance with the viewpoint conditions when an actual image is generated (photographed). Then, the converted images are arranged and combined (step 10).

FIG. 20 shows a specific operation in the case where the image synthesizing system of this embodiment performs the synthesizing process in step 10 described above.
This will be described with reference to the composite image of FIG.

FIG. 20 shows a synthesis start screen using a real image corresponding to FIG. 12 as a background image. In the lower part of this screen, a plurality of types of unit texture images are displayed in a window. When the operator uses the mouse to specify the door of the storage shelf, which is the object to be synthesized, on this synthesis screen, the perimeter thereof is framed to indicate that the door has been selected.

On the same screen, when a texture to be synthesized with the object is selected from the window using a mouse, the window of the selected unit texture is similarly displayed with a border.

FIG. 21 shows four doors to be combined.
This indicates that the selection of the unit texture image of the second window from the left is displayed on the screen.

When the user instructs execution of synthesis in the state of the screen shown in FIG. 21, the CG image generating section 30 refers to the size information and the texture information of the object as described above, and
A CG image is generated, the CG image is synthesized at the designated location by the image data synthesizing unit 32, the synthesized screen shown in FIG. 22 is displayed, and the synthesis is completed.

As described in detail above, according to this embodiment, it is possible to estimate viewpoint position information and shading information even from a real image having unknown shooting conditions, so that it is possible to generate a synthetic image without a sense of discomfort. Can be. Therefore, the following specific processing can be performed by using the image composition system of this embodiment.

In general, sanitary appliances such as baths and toiletries and kitchens used in houses and the like can be provided in many combinations of different colors due to the same shape, different materials, and the like. These product catalogs are set up in a studio for each product and photographed with a camera. But,
As described above, there is usually only one kind of catalog created by photography, and in many cases, color samples are displayed for products of different colors.

Therefore, by using this system, a CG image is synthesized with one photographed real image, and a product of a different color or material can be imaged in the same manner as an actual product photographed in a studio. Can be easily expressed with uniform quality without increasing shooting costs.

In the interior simulation or the like, when the interior of a house such as furniture, curtains, and wallpaper is changed, the furniture, curtains, and interior materials to be changed are represented by CG images with respect to actual images taken of the current situation. By generating and synthesizing, the image after the change can be confirmed in advance.

Although the present invention has been specifically described above, the present invention is not limited to the above-described embodiment, and can be variously modified without departing from the gist thereof.

[0110]

As described above, according to the present invention,
A CG image of an object having substantially the same shape and a different texture as the object projected in the real image is generated, and the C
When the G image is combined with the real image, a combined image without a sense of incongruity can be easily created without any special experience or intuition.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an image composition system according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating a processing procedure of image synthesis according to the embodiment;

FIG. 3 is an explanatory diagram showing how to find a vanishing point;

FIG. 4 is an explanatory diagram showing a relationship between a projection plane and a camera in a two vanishing point image.

FIG. 5 is an explanatory diagram showing a relationship between a viewpoint and a vanishing point.

FIG. 6 is a diagram for explaining viewpoint estimation calculation;

FIG. 7 is an explanatory diagram showing a method for estimating viewpoint information.

FIG. 8 is an explanatory diagram showing a method of estimating viewpoint information from two vanishing point real images.

FIG. 9 is another explanatory diagram showing a method of estimating viewpoint information from a two-vanishing-point real image.

FIG. 10 is an explanatory diagram showing a method of estimating viewpoint information using one vanishing point image.

FIG. 11 is an explanatory diagram showing a state in which the size of a space is estimated based on three-dimensional space information.

FIG. 12 is an explanatory diagram showing an example of a photographed image.

FIG. 13 is an explanatory diagram showing a state in which the size of a space is estimated for the real image.

FIG. 14 is a diagram for explaining three-dimensional conversion of a two-dimensional image with two vanishing points;

FIG. 15 is a diagram for explaining three-dimensional conversion of a two-dimensional image with one vanishing point;

FIG. 16 is an explanatory diagram showing a state in which a vertex of an object to be combined is designated in the photographed image;

FIG. 17 is an explanatory diagram showing a state in which the opposite vertex of the object to be combined is specified in the actual image.

FIG. 18 is an explanatory diagram showing a photographed image for creating a unit texture image;

FIG. 19 is an explanatory diagram illustrating an example of a CG image for synthesis.

FIG. 20 is an explanatory diagram showing a screen on which a real image before synthesis is displayed.

FIG. 21 is an explanatory diagram showing a screen on which a photographed image during a combining operation is displayed.

FIG. 22 is an explanatory diagram showing a screen on which a photographed image after the combining operation is displayed.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 ... Image input device 12 ... Image holding memory 14 ... Image display device 16 ... Image output device 18 ... Calculation part 20 ... Viewpoint position information calculation part 22 ... Light source information calculation part 24 ... Indoor space size information calculation part 26 ... Synthesis Position / size information calculation unit 28 ... unit texture image calculation unit 30 ... CG image generation unit 32 ... image data synthesis unit 34 ... information input unit

Claims (4)

[Claims] An image synthesizing apparatus for synthesizing a real image and a CG image, means for estimating viewpoint position information from the real image, and means for estimating three-dimensional spatial information of the real image from the estimated viewpoint position information. Means for estimating the size of an object to be synthesized in a real image based on the estimated three-dimensional spatial information; and generating a CG image for synthesis based on the estimated size of the object. An image synthesizing apparatus, comprising: means; and means for performing perspective projection conversion of the generated CG image and synthesizing the actual captured image. 2. The apparatus according to claim 1, wherein the means for estimating the size of the object includes a coordinate value of a vertex of the object projected on the real image and an actual height of the point. An image synthesizing device having an arithmetic function for estimating the size of an object. 3. The apparatus according to claim 1, further comprising: means for generating a basic unit texture image from a real image serving as a texture for a CG image, and wherein said generating means includes an estimated size of the object. An image synthesizing apparatus having a function of generating a CG image for synthesis from a unit texture image based on the image quality. 4. The apparatus according to claim 1, further comprising: means for estimating light source position information from the real image, and means for performing shading processing on the CG image based on the estimated light source position information. Image synthesizing device.