JP2003323636A - Three-dimensional supplying device and method and image synthesizing device and method and user interface device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a three-dimensional model supplying device and an image synthesizing device for automatically changing a three-dimensional model according to various virtual environments corresponding to depth distribution based on stereoscopic images where the three-dimensional models are to be disposed and disposing the three- dimensional model in a virtual space and a user interface device for easily changing the three-dimensional model according to the virtual environment. <P>SOLUTION: This three-dimensional model supplying and an image synthesizing device are provided with a three-dimensional model storing means, a virtual space information acquiring means, a three-dimensional model information acquiring means, a comparing means for comparing the virtual space information with the three- dimensional model information, a means for calculating the amount of change of the three-dimensional model, and a changing means for changing the three-dimensional model information. The user interface device is provided with a virtual space roughly displaying means for roughly displaying the size of a virtual space, a three-dimensional model roughly displaying means for roughly displaying the size of the three-dimensional model, and a means for enabling a user to change the size of the three-dimensional model. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional model supplying device and method, an image synthesizing device and method, and a user interface device.

[0002]

2. Description of the Related Art In recent years, not only high-performance workstations equipped with hardware specialized for 3D graphics, such as graphics workstations, but also 3D contents are available not only on personal computers for general homes and mobile terminals for individuals. Can be displayed easily. Under such circumstances, a three-dimensional model has become easily available, and the three-dimensional model can be used for producing various contents.

Under such circumstances, the applicant of the present invention has proposed an image synthesizing apparatus for generating a stereoscopic photograph by arranging a three-dimensional model in a virtual space composed of a photographed image and its depth information. .

The outline of the image synthesizing apparatus in this proposal will be described below.

First, a stereo image at a plurality of viewpoint positions obtained by mounting a stereo adapter or the like and corresponding pixels between the images of the stereo image are detected, and a parallax map is calculated by calculating a parallax from the corresponding relationship. To do. Using the parallax map, a three-dimensional space in which a photographed image is captured is constructed as a virtual space, and a three-dimensional model (polygons formed by the vertex coordinates of the object surface and each vertex row and its surface attributes and A 2D texture image that forms the surface, which is composed of normal line information, etc.) is arranged, and the user operates the 3D model such as moving, rotating, enlarging, etc. to make it as desired in the virtual space. Deploy. At that time, when observing the final composite result as a stereoscopic image,
The range in which the 3D model can move is limited so that there is no contradiction between the sense of depth and the sense of depth of the 3D model. In this way, a multi-view image sequence is generated by synthesizing a real image and a virtual space image with different viewpoint positions from the three-dimensional model and the real image arranged in the virtual space, and the multi-view image sequence is integrated into a stereoscopic image. And print with the printing device. By observing the printed stereoscopic image through an optical member such as a lenticular plate, it is possible to finally obtain a high-quality stereoscopic image that is consistent with the sense of depth between the actual image and the three-dimensional model. Become.

[0006]

As described above, when arranging a three-dimensional model in a virtual space photographed and constructed from a photographed image, the size of the virtual space is not always constant. The angle of view at the time of shooting and the distance to the main subject all differ depending on the input image. Therefore, it is general that the size of the virtual space to be constructed also differs depending on the input data. However, generally available 3D models are usually created assuming a certain size of virtual space, and usually 3D model creator and virtual space are constructed. Since the size of the three-dimensional model is determined by making a meeting with the manufacturer in advance, the size of the three-dimensional model is eventually assumed to be various depending on the virtual space.

Therefore, according to the present application, the IP serial No. Like the image synthesizer described in 1755402,
When arranging various 3D models in a virtual space constructed from general stereoscopic image information, the 3D model may be too small to be displayed, or the 3D model may be too large to be displayed outside the image. There was a problem that it would be done.

[0008]

In order to achieve the above object, the three-dimensional model supply device according to claim 1 acquires three-dimensional model storage means for storing three-dimensional model information and virtual space information. Virtual space information acquiring means, three-dimensional model information acquiring means for acquiring three-dimensional model information, comparing means for comparing the virtual space information and the three-dimensional model information, and the comparing means, It is characterized in that it has a calculating means for calculating the amount of change when the original model needs to be changed and a changing means for changing the three-dimensional model information using the calculation result of the calculating means.

A three-dimensional model supply apparatus according to a second aspect of the present invention is the three-dimensional model supply apparatus according to the first aspect, wherein the virtual space information acquisition means is the nearest plane from the viewpoint position of the virtual space, The feature is that coordinate information determined from the farthest surface and the angle of view of the virtual camera is acquired.

A three-dimensional model supply apparatus according to a third aspect of the present invention is the three-dimensional model supply apparatus according to the first or second aspect, wherein the three-dimensional model information acquisition means is bounding box information of the three-dimensional model. It is characterized by being acquired from.

A three-dimensional model supply apparatus according to a fourth aspect of the present invention is the three-dimensional model supply apparatus according to the first or third aspect, wherein the comparison means performs comparison without using the size in the depth direction. The amount of change is calculated by the changing means.

A three-dimensional model supply apparatus according to a fifth aspect of the present invention is the three-dimensional model supply apparatus according to the first aspect, wherein the comparison means includes the configuration information acquired by the virtual space information means, It is characterized in that the three-dimensional model information acquired by the three-dimensional model information acquisition means and the case where the three-dimensional model is arranged on the nearest surface from the viewpoint position are compared with the size when projected onto a two-dimensional image.

A three-dimensional model supply apparatus according to a sixth aspect of the present invention is the three-dimensional model supply apparatus according to any one of the first to fifth aspects, wherein the comparison means includes the virtual space and the three-dimensional model. Comparing the models is performed immediately before the three-dimensional model is placed in the virtual space.

A three-dimensional model supply apparatus according to a seventh aspect of the present invention is the three-dimensional model supply apparatus according to the first or second aspect, wherein the three-dimensional model storage unit is present on a network. To do.

An image synthesizing apparatus according to an eighth aspect of the present invention is a three-dimensional model storage means for storing three-dimensional model information, a virtual space information acquisition means for acquiring virtual space information, and three-dimensional model information. A three-dimensional model information acquisition means, a comparison means for comparing the information of the virtual space and the three-dimensional model information, in the comparison means, when the change of the three-dimensional model is required It is characterized in that it has a calculating means for calculating the change amount and a changing means for changing the three-dimensional model information using the calculation result of the calculating means.

An image synthesizing apparatus according to a ninth aspect of the present invention is the image synthesizing apparatus according to the eighth aspect, wherein the virtual space information acquisition means includes the nearest plane, the farthest plane from the viewpoint position of the virtual space, and It is characterized in that coordinate information determined from the angle of view of the virtual camera is acquired.

An image synthesizing apparatus according to a tenth aspect of the present invention is the image synthesizing apparatus according to the eighth or ninth aspect,
The three-dimensional model information acquisition means acquires from the bounding box information of the three-dimensional model.

An image synthesizing apparatus according to an eleventh aspect of the present invention is the image synthesizing apparatus according to any one of the eighth to tenth aspects, wherein the comparing means performs comparison without using the size in the depth direction. Then, the change means calculates the change amount.

According to a twelfth aspect of the present invention, in the image synthesizing apparatus according to the eighth aspect, the comparing means includes the configuration information acquired by the virtual space information means and the three-dimensional model information. The three-dimensional model information acquired by the acquisition means and the case where the three-dimensional model is arranged on the closest surface from the viewpoint position are compared with the size when projected onto a two-dimensional image.

An image synthesizing apparatus according to a thirteenth aspect of the present invention is the image synthesizing apparatus according to any one of the eighth to twelfth aspects, wherein the comparing means compares the virtual space with the three-dimensional model. Is executed immediately before the three-dimensional model is placed in the virtual space.

An image synthesizing apparatus according to a fourteenth aspect of the present invention is the image synthesizing apparatus according to any one of the eighth to thirteenth aspects, wherein the three-dimensional model storage unit is present on a network. And

According to a fifteenth aspect of the present invention, there is provided a user interface device comprising: a virtual space outline display means for roughly displaying the size of the virtual space; and a three-dimensional model outline table means for roughly displaying the size of the three-dimensional model. , And means for changing the size of the three-dimensional model by the user.

A user interface device according to a sixteenth aspect of the present invention is the user interface device according to the fifteenth aspect, wherein the virtual space schematic display means displays a bounding box of the virtual space. To do.

A user interface device according to a seventeenth aspect of the present invention is the user interface device according to the fifteenth or sixteenth aspect, wherein the three-dimensional model outline display means displays a bounding box of the three-dimensional model. It is characterized by

A user interface device according to an eighteenth aspect of the present invention is the user interface device according to any one of the fifteenth to seventeenth aspects,
In the three-dimensional model outline display means, when the change amount set by the change means is too large or too small to be displayed in the display area, a message to that effect is displayed so that the change cannot be reflected. It is characterized by

A three-dimensional model supply method according to a nineteenth aspect of the present invention is a virtual space information acquisition step of acquiring virtual space information, a three-dimensional model information acquisition step of acquiring three-dimensional model information, and the virtual space. Of the information of the three-dimensional model and the comparison step, in the comparison step, if the change of the three-dimensional model is required to calculate the amount of change, the calculation of the calculation step And a changing step of changing the three-dimensional model information using the result.

A three-dimensional model supply method according to a twentieth aspect of the present invention is the three-dimensional model supply method according to the nineteenth aspect, wherein the virtual space information acquisition step is performed from the viewpoint position of the virtual space. It is characterized in that coordinate information determined from the surface, the farthest surface, and the angle of view of the virtual camera is acquired.

A three-dimensional model supply method according to a twenty-first aspect of the present invention is the three-dimensional model supply method according to the nineteenth or twentieth aspect, wherein the three-dimensional model information acquisition step is bounding of the three-dimensional model. It is characterized by being acquired from box information.

A three-dimensional model supply method according to a twenty-second aspect of the present invention is the three-dimensional model supply method according to any one of the nineteenth to twenty-first aspects, wherein the comparison step is
The comparison is performed without using the size in the depth direction, and the change amount is calculated in the changing step.

A three-dimensional model supply method according to the invention described in claim 23 is the three-dimensional model supply method according to claim 19, wherein the comparison step is performed by using the configuration information acquired in the virtual space information step. The three-dimensional model information acquired in the three-dimensional model information acquisition step and the case where the three-dimensional model is arranged on the most recent surface from the viewpoint position are compared by the size when projected onto a two-dimensional image.

A three-dimensional model supply method according to a twenty-fourth aspect of the present invention is the three-dimensional model supply method according to any one of the nineteenth to twenty-third aspects, wherein the virtual space and the virtual space are used in the comparison step. Comparing 3D models is
The method is executed immediately before the three-dimensional model is placed in the virtual space.

According to a twenty-fifth aspect of the present invention, there is provided an image synthesizing method, a virtual space information acquiring step of acquiring virtual space information, a three-dimensional model information acquiring step of acquiring three-dimensional model information, and information on the virtual space. And a comparison step of comparing the information of the three-dimensional model, in the comparison step, a calculation step of calculating the change amount when the change of the three-dimensional model is required, and a calculation result of the calculation step. And a changing step of changing the three-dimensional model information by using the changing step.

An image synthesizing method according to a twenty-sixth aspect of the present invention is the image synthesizing method according to the twenty-fifth aspect, wherein the virtual space information obtaining step includes the most recent surface and the farthest point from the viewpoint position of the virtual space. It is characterized by acquiring coordinate information determined from the angle of view of the surface and the virtual camera.

An image synthesizing method according to a twenty-seventh aspect of the present invention is the image synthesizing method of the twenty-fifth aspect or the twenty-sixth aspect, wherein the three-dimensional model information obtaining step obtains from bounding box information of the three-dimensional model. It is characterized by doing.

An image synthesizing method according to a twenty-eighth aspect of the present invention is the image synthesizing method according to any one of the twenty-fifth to twenty-seventh aspects, wherein the comparing step does not use the size in the depth direction. In comparison with the above, the change amount is calculated in the change step.

An image synthesizing method according to a twenty-ninth aspect of the present invention is the image synthesizing method according to the twenty-fifth aspect, wherein the comparing step includes the configuration information acquired in the virtual space information step and the three-dimensional image. It is characterized in that the three-dimensional model information acquired in the model information acquisition step and the case where the three-dimensional model is arranged on the most recent surface from the viewpoint position are compared in size when projected onto a two-dimensional image.

An image synthesizing method according to a thirtieth aspect of the present invention is the image synthesizing method according to any one of the twenty-fifth to twenty-ninth aspects, wherein the virtual space and the three-dimensional model are combined in the comparing step. The comparison is performed immediately before the three-dimensional model is placed in the virtual space.

[0038]

BEST MODE FOR CARRYING OUT THE INVENTION [First Embodiment] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing the configurations of a three-dimensional model supplying device and an image synthesizing device according to the embodiment of the present invention.

The stereoscopic image input device 101 is a device for inputting stereo images taken at a plurality of viewpoint positions. The image synthesizing apparatus 102 synthesizes a stereo image captured from a plurality of viewpoint positions from the stereoscopic image input apparatus 101 with a virtual space image generated from a three-dimensional model arranged in a virtual space, and creates a three-dimensional integrated image. And a general-purpose personal computer, for example. The two-dimensional display device 103 is configured by a CRT display or the like that displays an image input from the stereoscopic image input device 101 and a two-dimensional image processed / combined by the image compositing device 102.
The operation input device 104 is a pointing device for rotating, moving, enlarging, reducing, etc. the three-dimensional model displayed on the two-dimensional display device 103, and is constituted by a mouse or a joystick. The printing device 105 is configured to print the image data and the like generated and displayed by the image synthesizing device 102.

Next, the structure of the internal blocks of the image synthesizing device 102 will be described.

The stereoscopic image information generation unit 1021 is composed of a circuit for generating a parallax map or the like from a stereo image input from the stereoscopic image input device 101. The virtual space image generation unit 1022 synthesizes the image generated from the three-dimensional model arranged in the virtual space with the reference image input from the stereoscopic image input device 101 and set by the stereoscopic image information generation unit 1021, It is composed of a circuit for displaying a composite image on the three-dimensional display device 103. Three-dimensional model supply device 1023
Is composed of an auxiliary storage device for storing geometrical shape information and surface attribute information or surface texture image information, and a circuit for changing the size of the three-dimensional model according to the virtual space to be arranged. Virtual space construction unit 102
4 includes a circuit that constructs a virtual space using parameters such as a parallax map from the stereoscopic image information generation unit 1021. The collision detection unit 1025 detects whether or not the virtual space constructed by the virtual space construction unit 1024 and the three-dimensional model supplied from the three-dimensional model supply device 1023 collide according to the depth distribution of the photographed image. It is composed of a circuit. The 3D model operation unit 1026 is an operation for moving, rotating, enlarging, reducing, etc., the information input from the operation input unit 104, which is an operation input by the operator, to the 3D model in the actual virtual space. It is configured by a circuit that converts the amount into a quantity and presents the effect to the operator when the predetermined operation range is exceeded. The multi-view image sequence generation unit 1027 uses the stereoscopic image input device 101.
It is composed of a circuit that generates a real-life multi-viewpoint image sequence by using the stereo image input from, and also generates a virtual space multi-viewpoint image sequence. 3D image composition unit 1
Reference numeral 028 is composed of a circuit for synthesizing the real multi-view image sequence generated by the multi-view image sequence generation unit and the virtual space multi-view image sequence as a three-dimensional image.

Next, the structure of the internal block of the three-dimensional model supply device 1023 will be described.

Reference numeral 10231 is a three-dimensional model storage unit, which is composed of an auxiliary storage device storing a plurality of three-dimensional models. Reference numeral 10233 is an information collection unit, which is information on the virtual space configured by the virtual space construction unit 1024 based on the parallax map or the like generated by the stereoscopic image information generation unit 1021 and the three-dimensional information to be arranged in the virtual space. The circuit is configured to collect model information from the three-dimensional model storage unit 10231. The change determination unit 10234 is the information collection unit 1
Based on the information collected in 0233, it is configured by a circuit that determines whether or not the coordinate information of the three-dimensional model needs to be changed. The 10232 three-dimensional model size changing unit is configured by a circuit that actually changes the coordinate information of the three-dimensional model when the size change determining unit 10234 determines that the size change of the three-dimensional model to be arranged is necessary. ing.

Next, the processing flow of the image synthesizing apparatus and the three-dimensional model supplying apparatus of the present invention will be described with reference to FIG.

Step S201 is a stereoscopic image information generation step.
A stereo image input by the stereoscopic image input device 101 and camera parameters (baseline length between viewpoints, focal length, etc.) at the time of shooting are input, and a parallax map representing a depth distribution to a subject is generated. At this time, the left image of the stereo images is determined as the reference image. There is no problem in using the right image as the reference image, but in that case, it is necessary to recalculate the reference image for calculating the parallax of the parallax map as the right image. S202 is a virtual space construction step, which is a stereoscopic image information generation step S20.
1. Construct a virtual space from camera parameters from the parallax map generated in 1 and the stereo camera.

In the virtual space construction step S202, a virtual space corresponding to the depth distribution of the photographed image is constructed from the camera parameters input from the stereoscopic image input device and the parallax map calculated by the stereoscopic image information generation unit 101. From the viewpoint, the depth distribution of the photographed image can be calculated by the equation (1) from the relationship between the parallax and the camera base line length.

Z (x, y) = b · f / d (x, y) ---------- (1) where d (x, y) is (x, y in world coordinates.
parallax in y), b is the baseline length of the stereo camera, f is the focal length of the camera, z (x, y) is the world coordinate (x,
It represents the distance from the viewpoint in y). A virtual space corresponding to the depth distribution of a real image can be constructed by using a polygon expression that connects the vertex coordinates at appropriate intervals and the vertex coordinates.

Next, in step S203, the three-dimensional model selected by the operator is placed in the virtual space from the three-dimensional model supply device 1023 in the virtual space constructed in the virtual space construction step S202.

The flow of processing in step S203 will be described in detail with reference to the flowchart of FIG.

First, in step S301, reference information of the virtual space constructed in virtual space construction step S201 is acquired. Here, in FIG. 4, the closest surface 402 to be clipped for display from the viewpoint position 401.
And a rectangular parallelepiped 404 constituted by the farthest surface 403 and a predetermined viewing angle, that is, each vertex of the rectangular parallelepiped area 404 whose depth is the distance between the display area of the nearest surface 402 and the farthest surface 403 from the nearest surface 402. Xv, Y calculated from
Get v, Zv.

Next, in S302, reference information of the three-dimensional model to be arranged is acquired. The reference information of the three-dimensional model mentioned here is the bounding box 502 including the three-dimensional model 501 as shown in FIG.
The lengths Xm, Ym, and Zm of each side calculated from are acquired.

Next, in step S303, information X relating to the size of the virtual space acquired in step S301.
v, Yv, Zv and the information Xm, Ym, Zm regarding the size of the three-dimensional model calculated in step S302 are used for comparison. The comparison method uses the value Mm = Max [Xm, Ym, Z of the axis having the maximum value of the size of the three-dimensional model.
m] and the size Mv of the axis of the virtual space with respect to it are determined to be constant values or not. If it is not within the range of the predetermined value, it is determined that the size of the three-dimensional model needs to be changed, and the process proceeds to S304.
If the size of the three-dimensional model does not need to be changed, the process proceeds to step S306, and the three-dimensional model is arranged at a position in contact with the nearest plane 402 in the virtual space in FIG.

In step S304, the change amount of the coordinate information of the three-dimensional model is calculated. Here, the magnification S =
Calculated from Mv / Mm and used as a direct magnification. At this time, it is important to confirm whether or not Xm and Ym do not exceed Xv and Yv by the magnification S. If it exceeds, change it so that it does not exceed the value of S.

Next, the processing is moved to step S305, and a scalar multiplication is applied to each vertex coordinate value of the three-dimensional model according to the magnification calculated in step S305. That is, when a certain vertex Pv = (xp, yp, zp), the changed value is Pv.
If expressed in the homogeneous coordinate system as',

[0056]

[Equation 1] Can be calculated by At this time, generally, the texture coordinates and the arrangement of vertices in the three-dimensional model are expressions that do not depend on the size of the vertex coordinates, and therefore the information of the three-dimensional model is not changed. The three-dimensional model whose size has been changed in step S305 is placed in the virtual space in step S306.
At this time, as described above, the three-dimensional model is the fourth one in the virtual space.
It is arranged at a position in contact with the closest surface 402 in the figure.

In step S307, the collision detection unit 1025 determines whether or not the depth distribution of the photographed image and the changed three-dimensional model collide. If it is determined that there is a collision, the process proceeds to step S308, the change amount S of the three-dimensional model is decreased by a certain amount, and the three-dimensional model is recorrected again in step S305. If there is no collision, the three-dimensional model placement step S203 is ended, and the process proceeds to step S204.

As described above, the three-dimensional model can be correctly displayed in a size suitable for the virtual space.

FIG. 7 shows a user interface device for adjusting the amount of change for changing the three-dimensional model immediately before placing the three-dimensional model in the virtual space as described above.

In FIG. 6, 601 schematically shows the size of the constructed virtual space. For example, the bounding box 405 of the virtual space shown in FIG. 4 may be displayed, or the bounding box 405 may be displayed. It may be displayed two-dimensionally as projected on the XY axes. Reference numeral 602 is a simplified display of the size of the three-dimensional model to be arranged, and a bounding box of the three-dimensional model may be displayed as in the virtual space 601. At this time, if it is too large or too small for the virtual space, "
It is easy for the operator to display a message to that effect, such as displaying a message such as "3D model cannot be displayed." In such a case, OK
The button 605 may be set in a non-inputtable state. Reference numerals 603 and 604 are user interfaces for displaying the rate of enlargement / reduction with respect to the three-dimensional model and changing the rate. These may be a slider type user interface, but it is desirable that the values can be changed only within a predetermined range.

The presence of such a user interface allows the operator to check the relative size of the virtual space and the three-dimensional model while adjusting the amount of change to the three-dimensional model in an easy-to-understand manner. . Further, needless to say, the above-mentioned interface improves not only the case of arranging the three-dimensional model in the virtual space but also the fact that it can be called once after arranging, so that the convenience of the user is further improved.

Next, in S204, an image obtained by arranging the three-dimensional model in the virtual space constructed in the virtual space constructing step S202 as shown in FIG. An image and a composite image are generated, and the composite image is displayed on the two-dimensional display device 103.

In step S205, the two-dimensional image device 103 is operated via the operation input device 104 in the three-dimensional model operation step.
Select the 3D model displayed in, move, rotate,
Perform operations such as inversion and reduction. At this time, when moving the selected three-dimensional model in the depth direction, the virtual space construction step S202 in the collision detection device 1025.
It is necessary to detect a collision with a three-dimensional scene composed of the depth distribution generated in. As a method of collision detection,
When the three-dimensional scene constructed in the virtual space construction step S202 is constructed by a triangular mesh or the like,
By using the methods described in Nakamae and Nishida, "3D Computer Graphics," Shokodo (1986), "Mutual positional relationship between figures," etc., each polygon and the 3D scene that make up the 3D model. It can be detected by determining whether and does not intersect. More simply, it can be roughly executed from each vertex (8 vertices) of the bounding box (bounding box) included in the three-dimensional model and the viewpoint position of each polygon of the three-dimensional scene. Moreover,
Also in the case where a three-dimensional scene is constructed by dividing the three-dimensional space in the virtual space construction step S202, similarly, by knowing the relationship between each vertex position of the bounding box of the three-dimensional model and each box of the divided space, the collision can be prevented. It can be detected.

In the multi-view image sequence generation step of S206, a real image multi-view image sequence is arranged in the virtual space constructed in the virtual space step S202 from the reference image and the parallax map calculated in the stereoscopic image information generation step S201. A multi-viewpoint image sequence is generated by combining a virtual space multi-viewpoint image sequence generated from a virtual space generated from the three-dimensional model displayed on the two-dimensional display device 103.

S207 is a three-dimensional image integration step, in which the multi-view image sequence generated in the multi-view image sequence generation step S205 is combined into a three-dimensional image, and the printing apparatus 1
Print at 05.

S208 is a printing step, in which the underground production image synthesized in the three-dimensional image integrating step S206 is printed. Finally, by superimposing an optical member such as a lenticular plate on the printed result and observing the image, it is possible to observe a high-quality stereoscopic image without a stereoscopic discomfort between the actual image and the three-dimensional model.

In the above embodiment, in the present embodiment, the three-dimensional model storage unit is described as being configured in the three-dimensional model storage unit, but the three-dimensional model storage unit is input as an external device on the network. You may do it.

Further, the present invention is not limited to the apparatus of the above-described embodiment, and may be applied to a system composed of a plurality of devices or an apparatus composed of one device.
A storage medium storing a program code of software that realizes the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. Needless to say, it can be completed by doing.

In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the storage medium storing the program code constitutes the present invention. As a storage medium for supplying the program code, for example, a floppy disk (registered trademark), hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R / RW, magnetic tape, non-volatile memory card, ROM is used. You can In addition, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the OS or the like running on the computer executes actual processing based on the instruction of the program code. It goes without saying that a case where some or all is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program code read from the storage medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, based on the instruction of the next program code. Needless to say, this also includes the case where the CPU or the like provided in the expansion board or the expansion unit performs the expansion function to perform a part or all of the actual processing, and the processing realizes the function of the above-described embodiment. Yes.

[Second Embodiment] In the first embodiment,
As the stereo image data in which the information input from the stereo image input device 101 is captured at a plurality of different viewpoint positions, the stereo image information generation unit 1021 generates a parallax map from the relationship between corresponding pixels in the stereo image data.
In the above, the stereoscopic image information is used, but unlike this, there is also a case where a pseudo parallax map is created from one image data and the one image and the pseudo parallax map are input as the stereoscopic image information.

The configuration of the three-dimensional model supplying apparatus and the case of applying the image synthesizing apparatus of the present invention in such a case will be described in detail with reference to the drawings.

First, FIG. 8 schematically shows the outline of one image and the pseudo parallax map described above. (A) shows the case where the main subject 801 to be input exists.
In (b), outline information 802 surrounding the main subject 801 is acquired manually or by image processing. From this contour information, it is possible to generate a near view area 803 having a front side parallax and a distant view area 804 having a rear side parallax as shown in (c). This near view area 803
If a bounding box 805 that encloses the main subject 801 corresponding to is obtained, the reference size for adjustment can be determined from the bounding box 805. Therefore, compare the reference value with the size when the three-dimensional model is projected. By doing so, the size of the three-dimensional model can be adjusted.

As described above, even in the virtual space formed from the stereoscopic image information based on the pseudo parallax, the projection image obtained by projecting the three-dimensional model on the two-dimensional image and the area in which the pseudo parallax is set are used to generate the three-dimensional model. The adjustment amount for the change can be calculated and can be combined with an appropriate three-dimensional model.

[0075]

As described above, according to the three-dimensional model supply apparatus according to the inventions of claims 1 to 7, the three-dimensional model is changed to an appropriate size for the constructed virtual space. Can be supplied.

According to the image synthesizing apparatus according to the inventions of claims 8 to 14, it becomes possible to supply a three-dimensional model which is appropriately sized for a virtual space constituted by a real image, and Many 3D models are available.

According to the user interface device of the fifteenth to eighteenth aspects of the invention, when the three-dimensional model is arranged in the virtual space, the size of the three-dimensional model is checked while confirming the approximate size. It can be changed.

According to the three-dimensional model supply method according to the nineteenth to twenty-fourth aspects of the present invention, it is possible to supply a three-dimensional model having a proper size to the constructed virtual space.

According to the image synthesizing method of the twenty-fifth to thirtieth inventions, it becomes possible to supply a three-dimensional model which is appropriately sized for a virtual space constituted by a real image. Many 3D models are available.

[Brief description of drawings]

FIG. 1 is a block diagram showing configurations of a three-dimensional model supply device and an image composition device according to a first embodiment of the present invention.

FIG. 2 is a flowchart of a processing program of the image synthesizing apparatus according to the embodiment.

FIG. 3 is a flowchart of a processing program of the three-dimensional model supply device according to the embodiment.

FIG. 4 is a schematic diagram illustrating a bounding box of a virtual space and a three-dimensional model.

[Fig. 5] Fig. 5 is a diagram for describing pseudo generation of a parallax map from one image.

FIG. 6 is a diagram illustrating a user interface device of the present invention.

FIG. 7 is a real shot / CG in the image synthesizing apparatus according to the embodiment.
It is an example of a composite display.

FIG. 8 is an example of acquiring a bounding box when a parallax map is artificially created in the second embodiment.

[Explanation of symbols]

101 stereoscopic image input device 102 Image synthesizer 103 two-dimensional display device 104 operation input device 105 printing device 1023 Three-dimensional model supply device 10231 Three-dimensional model storage unit 10232 3D model change unit 10233 Information collection unit 10234 Change determination unit

Claims (30)

[Claims] 1. A three-dimensional model supply device for supplying a three-dimensional model to be arranged in a virtual space, the three-dimensional model storage means storing three-dimensional model information, and the virtual space information for acquiring the virtual space information. Acquiring means, three-dimensional model information acquiring means for acquiring three-dimensional model information, comparing means for comparing the virtual space information with the three-dimensional model information, and the comparing means for changing the three-dimensional model A three-dimensional model supply device, comprising: a calculating unit that calculates the amount of change when required, and a changing unit that changes the three-dimensional model information using the calculation result of the calculating unit. . 2. The virtual space information acquisition means acquires coordinate information determined from a view angle of a virtual camera, a nearest surface and a farthest surface from a viewpoint position of the virtual space. 3D model supply device. 3. The three-dimensional model supply device according to claim 1, wherein the three-dimensional model information acquisition unit acquires from the bounding box information of the three-dimensional model. 4. The tertiary according to claim 1, wherein the comparison means compares without using the size in the depth direction and the change amount is calculated by the change means. Original model feeder. 5. The comparison means arranges the configuration information acquired by the virtual space information means, the three-dimensional model information acquired by the three-dimensional model information acquisition means, and the three-dimensional model from the viewpoint position on the nearest surface. The three-dimensional model supply device according to claim 1, wherein the sizes are compared when projected onto a two-dimensional image. 6. The method according to claim 1, wherein the comparison means compares the virtual space with the three-dimensional model immediately before the three-dimensional model is placed in the virtual space. The three-dimensional model supply device according to any one of items. 7. The three-dimensional model supply device according to claim 1, wherein the three-dimensional model storage unit exists on a network. 8. An image synthesizing device for synthesizing a photographed image and a virtual space image to generate a stereoscopic photograph, wherein the three-dimensional model storage means stores three-dimensional model information and the virtual space information is acquired. Virtual space information acquisition means, three-dimensional model information acquisition means for acquiring three-dimensional model information, comparison means for comparing the virtual space information with information of the three-dimensional model, and the comparison means in the three-dimensional An image synthesizing device comprising: a calculating unit that calculates the amount of change when a model change is required; and a changing unit that changes the three-dimensional model information using a calculation result of the calculating unit. apparatus. 9. The virtual space information acquisition means acquires coordinate information determined from the angle of view of the closest surface, the farthest surface and the virtual camera from the viewpoint position of the virtual space. Image synthesizer. 10. The image synthesizing apparatus according to claim 8, wherein the three-dimensional model information acquisition unit acquires from the bounding box information of the three-dimensional model. 11. The image according to claim 8, wherein the comparison unit performs comparison without using the size in the depth direction, and the change unit calculates the change amount. Synthesizer. 12. The comparison means arranges the configuration information acquired by the virtual space information means, the three-dimensional model information acquired by the three-dimensional model information acquisition means, and the three-dimensional model from the viewpoint position on the nearest surface. 9. The image synthesizing apparatus according to claim 8, wherein the sizes are compared when projected onto a two-dimensional image. 13. The comparison means compares the virtual space and the three-dimensional model with each other immediately before the three-dimensional model is placed in the virtual space. The image synthesizing apparatus according to any one of items. 14. The image synthesizing apparatus according to claim 8, wherein the three-dimensional model storage unit exists on a network. 15. A user interface device for adjusting the sizes of a virtual space and a three-dimensional model, wherein a virtual space outline display means for roughly displaying the size of the virtual space, and a size of the three-dimensional model. A user interface device comprising: a three-dimensional model schematic display means for displaying; and means for a user to change the size of the three-dimensional model. 16. The user interface device according to claim 15, wherein the virtual space outline display means displays a bounding box of the virtual space. 17. The user interface device according to claim 15, wherein the three-dimensional model outline display means displays a bounding box of the three-dimensional model. 18. In the three-dimensional model outline display means, if the change amount set by the change means is too large or too small to be displayed in the display area, a message to that effect is displayed to make the change. The user interface device according to any one of claims 15 to 17, wherein the user interface device cannot be reflected. 19. A three-dimensional model supply method for supplying a three-dimensional model to be arranged in a virtual space, the virtual space information acquisition step of acquiring virtual space information, and the three-dimensional model information acquisition of acquiring three-dimensional model information. A step of comparing the information of the virtual space and the information of the three-dimensional model with a step of calculating a change amount of the three-dimensional model when a change of the three-dimensional model is required. And a changing step of changing the three-dimensional model information using the calculation result of the calculating step, the three-dimensional model supplying method. 20. The virtual space information acquisition step acquires coordinate information determined from the angle of view of the closest surface, the farthest surface, and the virtual camera from the viewpoint position of the virtual space. 3D model supply method. 21. The three-dimensional model supply method according to claim 19, wherein the three-dimensional model information acquisition step is acquired from bounding box information of a three-dimensional model. 22. The comparison step, wherein the comparison is made without using the size in the depth direction, and the change amount is calculated in the change step.
The three-dimensional model supply method according to item. 23. In the comparing step, the configuration information acquired in the virtual space information step, the three-dimensional model information acquired in the three-dimensional model information acquiring step, and the three-dimensional model are arranged on the nearest surface from the viewpoint position. 20. The three-dimensional model supplying method according to claim 19, wherein the sizes are compared when projected onto a two-dimensional image. 24. The comparison of the virtual space and the three-dimensional model in the comparing step is executed immediately before the three-dimensional model is arranged in the virtual space. The three-dimensional model supply method according to any one of items. 25. An image synthesizing method for synthesizing a real image and a virtual space image to generate a three-dimensional photograph, comprising a virtual space information obtaining step of obtaining virtual space information, and a three-dimensional image obtaining of three-dimensional model information. A model information acquisition step, a comparison step of comparing the information of the virtual space with information of the three-dimensional model, and in the comparison step, when the change of the three-dimensional model is required, the change amount is calculated. An image synthesizing method comprising: a calculating step; and a changing step of changing the three-dimensional model information using a calculation result of the calculating step. 26. The virtual space information acquisition step acquires coordinate information determined from a view angle of a virtual camera and a nearest surface from a viewpoint position of the virtual space. Image composition method. 27. The image synthesizing method according to claim 25, wherein the three-dimensional model information acquisition step is acquired from bounding box information of a three-dimensional model. 28. The comparison process according to any one of claims 25 to 27, wherein the comparison is performed without using the size in the depth direction, and the modification amount is calculated in the modification process.
The image synthesizing method described in the item. 29. In the comparison step, when the configuration information acquired in the virtual space information step, the three-dimensional model information acquired in the three-dimensional model information acquisition step, and the three-dimensional model are arranged on the nearest plane from the viewpoint position. 26. The image synthesizing method according to claim 25, wherein the sizes are compared when projected onto a two-dimensional image. 30. The comparison of the virtual space and the three-dimensional model in the comparing step is executed immediately before the three-dimensional model is arranged in the virtual space. The image synthesizing method according to any one of items.