JP2005063041A - Three-dimensional modeling apparatus, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus for mapping textures to each polygon constituting a three-dimensional model by acquiring a camera position and attitude information from a video. <P>SOLUTION: A camera position/attitude information arithmetic part 14 calculates the position and attribute of an image input apparatus 4 which has performed photographing from the continuous viewpoint moving images of a target object inputted from an image input part 12, and an image and corresponding position/attitude information associating part 15 correlates the calculated position/attitude information with each of corresponding images. Then, a rendering information arithmetic part 16 based on the position/attribute of the model reads the three-dimensional shape model of the object registered in a database 11, and the three-dimensional model based on the calculated position/attitude information of an image inputting device 4 is rendered and displayed on a display device 2. A texture projection information arithmetic part 17 maps or projects the photographed texture image on the displayed three-dimensional model. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  According to the present invention, a texture is optimally mapped to a three-dimensional shape model of a target object on the basis of position / posture information of a camera obtained based on a moving image obtained by capturing the target object or a continuous image sequence. The present invention relates to a three-dimensional modeling apparatus and method for generating a high-quality arbitrary viewpoint three-dimensional model, and a program for causing a computer to realize the function of such a three-dimensional modeling apparatus.

  In recent years, 3D model data has been effectively used in various scenes such as the field of virtual reality, the Internet, and the field of map information. In order to create three-dimensional model data, a dedicated measuring device such as a three-dimensional laser scanner is required to measure a three-dimensional shape. In such a device, a high-definition of a three-dimensional object is required. Texture information cannot be generated easily. Therefore, the texture is photographed by means other than the shape measuring means, the corresponding texture area is manually divided for each polygon of the three-dimensional shape model, and further, the texture according to the viewpoint direction at the time of photographing is further divided. A method of performing texture mapping on each polygon after correcting the distortion was used.

  In addition, as a method for simplifying the above method, Patent Document 1 uses a schematic shape model of a display target object and a plurality of images obtained by shooting the display target object from different directions having information on the shooting direction. Select the captured image that has the closest viewpoint direction to the rendering direction and the shooting direction from the multiple captured images, project the captured image from the shooting position as the viewpoint, and render the projected approximate shape. A method of performing perspective projection on a screen from the viewpoint direction to be performed is disclosed.

  Alternatively, Patent Document 2 uses a technique in which an image with the highest resolution is selected for each patch surface from a plurality of captured images from different viewpoints, and texture mapping is performed as a texture image applied to each patch surface. Furthermore, a process of executing a correction process for the difference in gray value between the images due to the mismatch of the light source conditions of each texture image, that is, between the patch surfaces is also performed. By applying image tone value correction processing between frames, texture image shift correction processing between patches, and tone smoothing correction processing at the boundary between patches as this correction processing, high-quality texture mapping is possible. As a result, it is possible to generate a high-quality three-dimensional image.

Further, as shown in Non-Patent Document 1, a target object is photographed as a moving image with respect to a three-dimensional object having a shape model, and the rendering of the three-dimensional object and the projection of the corresponding texture are performed based on the camera position / posture information. A technique for rendering a textured three-dimensional object by performing is also proposed.
JP 2001-243497 A JP 2002-24850 A IEEE Virtual Reality 2003 (March 22-26, 2003 LosAngeles California), P.61 "Augmented Virtual Environment (AVE): Dynamic Fusion of Imagery and 3D Models"

  However, in the method of Patent Document 1, it is necessary to acquire the direction of shooting by some means in advance, and it is necessary to have another apparatus for that purpose, and the complexity of shooting cannot be avoided. In addition, since each shooting direction has a certain interval, a plurality of appropriate directions for mapping may be selected. In such a case, a means for determining which is more appropriate Does not have. Furthermore, since only the direction is the criterion for selecting the optimum texture, no consideration is given to the distance to the target object, and there is a possibility of selecting an image with a blurred texture and a blurred texture. There are also challenges.

  This is also a problem that can be said with respect to the method of Patent Document 2.

  Further, in the example of Non-Patent Document 1, since the projected texture is based on the position / posture, the projected position changes as the line-of-sight direction changes, and the texture mapping position shifts. Therefore, there is a problem that the entire circumference of the object cannot be rendered.

  The present invention has been made in view of the above points, and obtains camera position / posture information by a simple means that does not require the use of a means using another measuring device other than a video, and constitutes a three-dimensional model. It is an object of the present invention to provide a three-dimensional modeling apparatus, method, and program for mapping a texture to each polygon.

In order to achieve the above object, a three-dimensional modeling apparatus according to the invention described in claim 1 comprises:
In a 3D modeling device that performs texture mapping on a 3D shape model and performs 3D modeling so that rendering can be performed from an arbitrary viewpoint,
Shape information input means for inputting shape information of the target object;
An image input means for inputting a moving image obtained by photographing the target object by moving the image input device or a continuous image sequence in time series;
Position / attitude information calculating means for calculating position / attitude information of the image input device based on a moving image or an image sequence input by the image input means;
The position / orientation information of the image input device calculated by the position / orientation information calculating means is the time series input by the image sequence constituting the moving image input by the image input means or the image input means. Association means for associating each texture image included in the continuous image sequence;
3 corresponding to each position / posture of the image input device according to the position / posture information of the image input device calculated by the position / posture information calculation unit from the shape information of the target object input by the shape information input unit. A rendering means for rendering a dimensional model;
Projection display means for projecting and displaying each texture image associated by the association means corresponding to each position / posture of the image input device on the three-dimensional model rendered by the rendering means;
It is characterized by comprising.

A three-dimensional modeling apparatus according to the invention of claim 2 is the three-dimensional modeling apparatus according to the invention of claim 1,
The position / posture information calculation means extracts at least one of feature points, characteristic line segments, and characteristic shapes in an image in the moving image or image sequence input by the image input means, and extracts these Characteristic points, line segments, and shapes are tracked, and position / posture information of the image input device is calculated based on the principle of triangulation.

A three-dimensional modeling apparatus according to the invention described in claim 3 is the three-dimensional modeling apparatus according to the invention described in claim 1 or 2,
When the texture image projected for each posture information of each position of the image input device calculated with time change moves on the three-dimensional model rendered by the rendering means, The texture image projected on the area is held, and the projected texture image is left and rendered on the area that has been projected even once with the change of the position / posture of the image input device. It is characterized by that.

A three-dimensional modeling apparatus according to a fourth aspect of the present invention is the three-dimensional modeling apparatus according to the third aspect of the present invention.
When the image region associated with each polygon of the three-dimensional model changes with the change of the image associated with the three-dimensional shape model with respect to the texture image projected along with the movement of the position / posture of the image input device, The image processing apparatus further includes a determination unit that determines whether or not to update the texture image mapped for each polygon so that the polygon and the corresponding image area are optimally associated with each other.

A three-dimensional modeling apparatus according to a fifth aspect of the present invention is the three-dimensional modeling apparatus according to the fourth aspect of the present invention.
The determination means selects, as a reference for determining whether or not to update the texture image of each polygon area, the one having a smaller angle between the line-of-sight direction of the image input device and the corresponding perpendicular direction of the polygon. Features.

The three-dimensional modeling apparatus according to the invention described in claim 6 is the three-dimensional modeling apparatus according to any one of claims 1 to 5,
The moving image or time-series continuous image sequence input by the image input means is a moving image or image sequence photographed from a plurality of directions by the image input device so as to include a common photographing region,
The position / posture information calculation means calculates the position / posture information of the image input device from each of the moving images taken from the plurality of directions or the images constituting the image sequence,
The projection display means selects a texture image suitable for rendering or a texture image more appropriate than a plurality of texture images when performing texture mapping of a common area in a moving image or image sequence captured from the plurality of directions. Generate,
It is characterized by that.

A three-dimensional modeling apparatus according to the invention described in claim 7 is the three-dimensional modeling apparatus according to the invention described in claim 6,
The projection display means has a smaller angle formed between the perpendicular direction of each polygon of the three-dimensional model in the common area and the line-of-sight direction of the image input device as a selection criterion for a texture image suitable for rendering in the common area. It is characterized by selecting.

A three-dimensional modeling apparatus according to the invention described in claim 8 is the three-dimensional modeling apparatus according to any one of claims 1 to 7,
Position and orientation information corresponding to each image of the moving image or image sequence in which the target object is captured, and an optimum texture image corresponding to each polygon of the three-dimensional model are associated with each other and registered as a database,
When performing arbitrary viewpoint rendering, it is possible to perform rendering by always associating an optimal texture image with each polygon.

A three-dimensional modeling apparatus according to the invention described in claim 9 is the three-dimensional modeling apparatus according to any one of claims 1 to 8,
Acquire a moving image or image sequence of the target object at different times,
Project and render a texture image obtained from a video or an image sequence acquired in an arbitrary time zone,
It is characterized by that.

In order to achieve the above object, a three-dimensional modeling method according to the invention of claim 10 comprises:
In the 3D modeling method of performing texture mapping on a 3D shape model and performing 3D modeling so that rendering can be performed from an arbitrary viewpoint,
Enter the shape information of the target object,
Input a moving image obtained by photographing the target object by moving the image input device or a continuous image sequence in time series,
Calculate the position / posture information of the image input device based on the input moving image or image sequence,
Associating the calculated position / posture information of the image input device with each texture image included in the image sequence constituting the input moving image or the input time-series continuous image sequence And
Rendering a three-dimensional model corresponding to each position / posture of the image input device according to the calculated position / posture information of the image input device from the shape information of the input target object,
Projecting and displaying each of the associated texture images corresponding to each position and orientation of the image input device on the rendered three-dimensional model;
It is characterized by that.

In order to achieve the above object, a program device according to claim 11 is provided.
In a program that causes a computer to perform texture mapping on a 3D shape model and perform 3D modeling so that rendering can be performed from an arbitrary viewpoint,
A function to input the shape information of the target object,
A function of inputting a moving image obtained by photographing the target object by moving the image input device or a continuous image sequence in time series;
A function for calculating position / posture information of the image input device based on the input moving image or image sequence;
Associating the calculated position / posture information of the image input device with each texture image included in the image sequence constituting the input moving image or the input time-series continuous image sequence With the ability to
A function of rendering a three-dimensional model corresponding to each position / posture of the image input device according to the calculated position / posture information of the image input device from the input shape information of the target object;
A function of projecting and displaying the associated texture images corresponding to the positions and orientations of the image input device on the rendered three-dimensional model;
It is for realizing.

  According to the present invention, the camera position / orientation information is acquired by simple means that does not require the use of means other than the video, and is optimal for each polygon constituting the three-dimensional model. It is possible to provide a three-dimensional modeling apparatus, method, and program for mapping a simple texture and rendering the entire circumference.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

[First Embodiment]
FIG. 1 is a diagram showing an overall schematic configuration of a three-dimensional modeling apparatus according to the first embodiment of the present invention.

  That is, the three-dimensional modeling apparatus includes an arithmetic device 1 such as a computer, a display device 2 such as a CRT or a liquid crystal monitor, an instruction device 3 such as a mouse, and an image input device 4 such as a camera. The calculation device 1 includes a database 11, an image input unit 12, a feature point / line / region extraction processing calculation unit 13, a camera position / posture information calculation unit 14, a position / posture information association unit 15 corresponding to an image, a model The rendering information calculation unit 16 and the texture projection information calculation unit 17 based on the position / posture of each of them are provided as basic configurations.

  Here, the database 11 registers in advance a three-dimensional shape model obtained by measuring a target object with a three-dimensional shape measuring device (not shown). The database 11 is assumed to be included in the arithmetic device 1, but may be provided in another computer connected via a network.

  The image input unit 12 receives an image captured as a moving image or a temporally continuous image sequence by the image input device 4.

  The feature point / line / region extraction processing calculation unit 13 is a point, a line segment, a feature, which is a feature that is required in the subsequent processing from within each image of the moving image or the continuous image sequence input by the image input unit 12. Recognize and extract regions.

  The camera position / posture information calculation unit 14 calculates the position / posture information of the image input device 4 for each image from the feature points / lines / regions extracted by the feature point / line / region extraction processing calculation unit 13. .

  The position / posture information associating unit 15 corresponding to the image associates the position / posture information obtained by the camera position / posture information calculation unit 14 with each image input by the image input unit 12 corresponding thereto. It is.

  The rendering information calculation unit 16 based on the position / posture of the model renders the three-dimensional shape model stored in the database 11 based on the position / posture information.

  The texture projection information calculation unit 17 performs mapping or projection of the texture on the three-dimensional shape model rendered by the rendering information calculation unit 16 based on the position / orientation of the model.

  In the three-dimensional modeling apparatus having such a configuration, the three-dimensional shape model as shown in FIG. 2A is measured in advance by the three-dimensional shape measurement apparatus and registered in the database 11 as described above. . This three-dimensional shape model includes vertex information and vertex connection information. However, any other three-dimensional shape expression format can be used. The measurement can be performed with a general three-dimensional shape measuring instrument such as a laser scanner, stereo measurement, or a light cutting method, and a three-dimensional solid model is generated.

  In the three-dimensional modeling apparatus, an operation as shown in the flowchart of FIG.

  That is, first, a continuous viewpoint moving image of the target object is taken by the image input device 4 and input by the image input unit 12 (step S11). FIG. 2B is a diagram showing the shooting scene. A moving image or a continuous image sequence is taken while moving the viewpoint with respect to the target object. Here, the description will be made assuming that the captured image is captured by the image input unit 12 in real time. However, the captured image is temporarily recorded on a recording medium, and captured later by the image input unit 12 to perform the subsequent processing. Of course, you may do.

  Next, the feature point / line / region extraction processing calculation unit 13 extracts feature points, line segments, and regions in each moving image or each image captured as a temporally continuous image sequence. (Step S12). In this feature region extraction, a characteristic region such as a density change and an edge in an image is extracted by applying various filters.

  Next, the camera position / posture information calculation unit 14 tracks the temporal change of the extracted feature region in the image, and calculates the distance to each feature region based on the principle of triangulation ( Step S13) Further, the position / orientation of the image input device 4 is calculated based on these pieces of information, and the calculated position / orientation information and the corresponding position / orientation information are associated by the position / orientation information associating unit 15 corresponding to the image. Each image is associated (step S14).

  Thereafter, the rendering information calculation unit 16 based on the model position / orientation reads out the three-dimensional shape model of the object registered in the database 11, and based on the calculated position / orientation information of the image input device 4. The three-dimensional model is rendered and displayed on the display device 2 (step S15).

  Then, the texture projection information calculation unit 17 maps or projects the captured texture image on the displayed three-dimensional model, as shown in FIG. 2C (step S16).

  With the above method, texture mapping or projection onto a three-dimensional model can be performed.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the present embodiment, in addition to the configuration of the first embodiment, the arithmetic device 1 of the three-dimensional modeling device further synthesizes images captured and input from a plurality of viewpoints to create an all-round image. A multi-viewpoint image integration calculation unit 18 (see FIG. 1).

  When shooting a target object while moving the viewpoint, the area to be shot moves simultaneously with the movement of the viewpoint. Since the area to be photographed moves, the area also moves when mapping or projecting the texture onto the three-dimensional model, and the area displayed in the mapping or projection moves with the movement. .

  That is, in FIG. 4A, when the movement of the viewpoint moves from the viewpoint 51 → the viewpoint 52 → the viewpoint 53, the area to be mapped or projected at the viewpoint 52 moves. The texture of the area 61 that has been used is lost, and further, in texture mapping or projection at the viewpoint 53, the texture of the area 62 is lost. However, even in areas where texture mapping or projection is no longer performed as the viewpoint moves, the texture area mapped or projected in that area is retained because the texture was mapped or projected in the previous viewpoint state. However, even if there is a viewpoint movement of 52 and 53, the areas of 61 and 62 keep the texture at the previous viewpoint 51, so that the texture mapping or projected area gradually increases. become.

  FIG. 3B is an operation flowchart of the three-dimensional modeling apparatus according to the second embodiment. However, since the process of step S11 thru | or step S16 is the same as that of the said 1st Embodiment, in FIG.3 (B), only step S16 and subsequent steps are shown.

  That is, after performing texture mapping or projection by the method described above (step S16), the multi-viewpoint image integration calculation unit 18 moves the rendering area according to the viewpoint change (step S21), and the next viewpoint A region where rendering is no longer performed is detected (step S22), and the texture region rendered at the previous viewpoint is mapped or projected to the region where the detected rendering is no longer performed (step S22). S23).

  By doing in this way, as shown in FIG.2 (D), it becomes possible to perform the texture mapping of the perimeter image.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. The present embodiment relates to an optimum texture generation processing method for areas overlapping in front and back texture mapping or projection areas in viewpoint movement.

  In the present embodiment, the multiple viewpoint image integration calculation unit 18 in the calculation device 1 of the three-dimensional modeling apparatus is more optimal than the multiple viewpoint image in the texture projection information calculation unit 17 for each polygon constituting the three-dimensional model. To select or synthesize a texture. Further, the computing device 1 associates the optimum texture and polygon selected or synthesized by the multi-viewpoint image integration computing unit 18 and registers them in the database 11 with the optimum texture associating unit 19 (see FIG. 1). have.

  FIG. 3C is an operation flowchart of the three-dimensional modeling apparatus according to the third embodiment. However, since the processing from step S11 to step S16 is the same as that in the first embodiment, only step S16 and subsequent steps are shown in FIG.

  That is, after texture mapping or projection is performed by the above-described method (step S16), the rendering information calculation unit 16 based on the position / orientation of the model further performs processing for changing the rendering area as the viewpoint moves. This is performed (step S31). For example, as shown in FIG. 4B, when the movement of the viewpoint changes from the viewpoint 51 → 52 → 53, the area rendered at the position of the viewpoint 51 and the area rendered at the position of the viewpoint 52 The area 63 is an area that can be rendered at both viewpoint positions. The same can be said for the overlapping area 64 in the viewpoint 52 and the viewpoint 53.

  Therefore, the multi-viewpoint image integration calculation unit 18 detects such overlapping regions 63 and 64 (step S32), and performs a process of selecting or synthesizing the optimum texture in the detected overlapping region (step S33). Normally, mapping or projection is performed using the texture at the viewpoint where rendering is performed, but depending on conditions such as the direction of the polygon representing the three-dimensional shape, the viewpoint position before and after the texture at that viewpoint position In some cases, using the texture in can improve the image quality. That is, for example, the texture closer to the viewpoint direction with respect to the direction perpendicular to the polygon is less distorted and is more likely to improve the image quality. As a condition, not only an angle in the viewpoint direction but also an arbitrary condition such as a distance can be handled. Furthermore, it is also possible to improve the image quality by newly synthesizing textures by obtaining an average for each pixel of both textures for texture regions that are mapped or projected with respect to overlapping regions.

  The texture optimally selected or synthesized for each polygon in this way is mapped or projected onto the three-dimensional model by the texture projection information calculation unit 17 (step S34).

  Further, the texture optimally selected or synthesized for each polygon is registered in the database 11 together with the polygon by the optimum texture associating unit 19 (step S35), and the generated three-dimensional object is stored. When displaying, by referring to the database 11, it is possible to use textures that are mapped or projected on each polygon as preprocessed (step S36), and perform viewpoint movement in real time. It becomes possible.

[Fourth Embodiment]
Next, a third embodiment of the present invention will be described.

  FIG. 2C shows a method of mapping or projecting images taken from a plurality of directions from each direction. As a configuration for displaying texture mapping or projection information from a plurality of viewpoints in the display device 2, a display as shown in FIG. 5 is performed.

  In the present embodiment, an object is to capture a moving image from a plurality of viewpoints so as to have overlapping regions, and to generate a three-dimensional object over a wide range and further around the entire circumference. Three-dimensional modeling similar to the above-described method is performed on an image from each direction, but an optimum texture is selected for overlapping regions. If the texture at each viewpoint is created according to the third embodiment with respect to the overlapping area between the captured images, it is optimized and has the degree of freedom of selection by the number of viewpoint directions. However, even in this case, the optimum texture can be selected or synthesized by using the same technique as that used in the third embodiment for optimizing the texture of the overlapping region in the viewpoint movement. Or, compare all the textures of the overlapping areas in each viewpoint movement with the textures of the overlapping areas in each viewpoint direction in parallel and select the optimal texture, or use all these textures to create a pixel. It is also possible to generate an optimum texture by performing optimization by performing synthesis by taking an average value for each.

  When the present embodiment is displayed on the display device 2, for example, a screen configuration as shown in FIG. 5 can be set, and texture mapping is performed in the front display area 21 displayed as an active window on the display screen. Alternatively, a projected 3D image rendering window 21A is displayed, and a 3D object can be displayed at an arbitrary viewpoint in the window 21A. In addition, sub-windows 21B-1 to 21B-n are provided so that texture mapping or projected images at respective viewpoints and raw images at the time of photographing can be displayed. Further, a control panel 21C is also provided, and control such as viewpoint movement control of a three-dimensional image, switching of display images in each sub-window 21B-1 to 21B-n, and the like can be performed. However, the viewpoint can be changed by operating the pointing device 3 in the three-dimensional image rendering window 21A.

[Fifth Embodiment]
Next, as a fifth embodiment of the present invention, when rendering is performed at an arbitrary viewpoint, a method for generating an optimized texture for each viewpoint is generated for each polygon constituting a three-dimensional object. Will be described with reference to FIG.

  Here, it is a method of creating a database by associating an optimum texture for each viewpoint with respect to each component such as a polygon constituting the three-dimensional shape model. When the viewpoint changes, for example, the orientation of each polygon changes, so that the optimum texture for mapping or projection may differ depending on the viewpoint direction for rendering. For this reason, by associating an optimal texture for each viewpoint (polygon) for each component (polygon) and creating a database, the optimal texture is applied to each viewpoint when rendering a 3D model. High-quality three-dimensional rendering is performed.

[Sixth Embodiment]
Next, a sixth embodiment of the present invention will be described. This embodiment is characterized in that a plurality of moving images in different time zones are taken as moving images of the same direction viewpoint.

  This embodiment will also be described with reference to FIG. Although the three-dimensional model can be generated at a certain time t1 by the method described in the first to fifth embodiments, a similar method is acquired in a plurality of different time zones (time t2,..., Tn). It is also possible to apply to one preset three-dimensional shape model using the texture image. Furthermore, it is also possible to use a plurality of three-dimensional shape models and associate textures of a plurality of time zones with respect to time zones. For example, it is possible to capture textures of morning, noon, and night on the same day, and to capture textures for spring, summer, autumn, and winter as seasons. After calculating optimum textures at the respective polygons and viewpoint positions for the different time zones t1, t2,..., Tn shown in FIG. It is possible to obtain an optimal high-quality three-dimensional image from an arbitrary viewpoint.

  Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention.

  For example, the above functions can be realized by supplying a software program for realizing the functions of the above embodiments to a computer and executing the programs by the computer.

(Appendix)
The invention having the following configuration can be extracted from the specific embodiment.

(1) In a three-dimensional modeling apparatus that performs texture mapping on a three-dimensional shape model and performs three-dimensional modeling so that rendering can be performed at an arbitrary viewpoint,
Shape information input means for inputting shape information of the target object;
An image input means for inputting a moving image obtained by photographing the target object by moving the image input device (that is, moving the viewpoint) or a continuous image sequence in time series;
Position / attitude information calculating means for calculating position / attitude information of the image input device based on a moving image or an image sequence input by the image input means;
The position / orientation information of the image input device calculated by the position / orientation information calculating means is the time series input by the image sequence constituting the moving image input by the image input means or the image input means. Association means for associating each texture image included in the continuous image sequence;
3 corresponding to each position / posture of the image input device according to the position / posture information of the image input device calculated by the position / posture information calculation unit from the shape information of the target object input by the shape information input unit. A rendering means for rendering a dimensional model;
Projection display means for projecting and displaying each texture image associated by the association means corresponding to each position / posture of the image input device on the three-dimensional model rendered by the rendering means;
A three-dimensional modeling apparatus comprising:

(Corresponding embodiment)
The first to sixth embodiments correspond to the embodiment relating to the three-dimensional modeling apparatus described in (1). Here, the shape information input unit corresponds to the database 11, the image input unit corresponds to the image input unit 12, and the position / posture information calculation unit corresponds to the camera position / posture information calculation unit 14. The association means corresponds to the position / posture information association unit 15 corresponding to the image, the rendering means corresponds to the rendering information calculation unit 16 based on the position / posture of the model, and the projection display means corresponds to the texture. This corresponds to the projection information calculation unit 17.
(Function and effect)
According to the three-dimensional modeling apparatus described in (1), the position / posture information of the image input apparatus can be calculated by moving image or continuous image acquisition while moving the viewpoint of the image input apparatus. Since it is not necessary to use the position / posture measuring unit of the apparatus, a simple system can be configured.

  (2) The position / posture information calculation unit extracts at least one of a feature point, a characteristic line segment, and a characteristic shape in an image in the moving image or image sequence input by the image input unit, The extracted characteristic points, line segments, and shapes are tracked, and the position / posture information of the image input device is calculated based on the principle of triangulation, and the three-dimensional modeling described in (1) apparatus.

(Corresponding embodiment)
The embodiment relating to the three-dimensional modeling apparatus described in (2) corresponds to the first embodiment.
(Function and effect)
According to the three-dimensional modeling apparatus described in (2), since the image input apparatus position / posture information can be calculated by moving image or continuous image acquisition while moving the viewpoint by the image input apparatus, other image input apparatuses Since it is not necessary to use position / posture measuring means, a simple system can be configured. In order to calculate the position and orientation, corresponding points between different viewpoints are required, but since arbitrary information such as feature points, line segments, and shapes can be used in the image, it is robust and accurate. It becomes possible to calculate better position / posture information.

  (3) When a texture image projected for each posture information of each position of the image input device calculated with time change moves on the three-dimensional model rendered by the rendering means, On the other hand, the texture image projected on the area is held, and the projected texture image is left and rendered on the area that has been projected even once along with the change in the position and orientation of the image input device. The three-dimensional modeling apparatus according to (1) or (2), wherein:

(Corresponding embodiment)
The embodiment related to the three-dimensional modeling apparatus described in (3) corresponds to the second embodiment.
(Function and effect)
According to the three-dimensional modeling apparatus described in (3), when texture mapping is performed together with the viewpoint movement, the rendered area is also moved along with the viewpoint movement, and textures are not mapped in areas outside the rendering area. By detecting the movement of the area and storing the texture that has been mapped up to that area before rendering, the texture mapped area increases and the rendering is performed. The possible viewpoint position increases.

  (4) When the image area associated with each polygon of the three-dimensional model changes with the change of the image associated with the three-dimensional shape model with respect to the texture image projected along with the movement of the position / posture of the image input device. (3) further comprising a judging means for judging whether or not to update the texture image mapped for each polygon so that each polygon and the corresponding image area are optimally associated with each other (3). ) 3D modeling apparatus.

(Corresponding embodiment)
The third embodiment corresponds to the embodiment relating to the three-dimensional modeling apparatus described in (4). Here, the determination means corresponds to the multi-viewpoint image integration calculation unit 18.
(Function and effect)
According to the three-dimensional modeling apparatus described in (4), the texture mapping area moves as the viewpoint moves, and overlapping areas exist at a plurality of viewpoints. Usually, the texture at the corresponding viewpoint is present. However, there is a possibility that the image quality will be higher if different textures are used due to problems such as the angle between the line-of-sight direction and the direction of the polygon normal or shooting conditions. By comparing textures of a plurality of viewpoints to be rendered and selecting a texture that can be optimally rendered for each polygon, a high-quality three-dimensional image can be generated.

  (5) The determination means selects a smaller angle between the line-of-sight direction of the image input device and the perpendicular direction of the corresponding polygon as a reference for determining whether to update the texture image of each polygon area. The three-dimensional modeling apparatus according to (4), characterized in that:

(Corresponding embodiment)
The embodiment related to the three-dimensional modeling apparatus described in (5) corresponds to the third embodiment.
(Function and effect)
According to the three-dimensional modeling apparatus described in (5), the angle between the perpendicular line of the polygon and the line-of-sight direction of the image input device is referred to as the criterion for selecting the optimum texture in (4), and the smaller one is selected. This makes it possible to select an image close to the front, so that distortion of the image during texture mapping is minimized, so that an image closer to reality can be selected, and a high-quality three-dimensional image is generated. It becomes possible to do.

(6) The moving image or the time-series continuous image sequence input by the image input means is a moving image or image sequence photographed from a plurality of directions by the image input device so as to include a common photographing region. Yes,
The position / posture information calculation means calculates the position / posture information of the image input device from each of the moving images taken from the plurality of directions or the images constituting the image sequence,
The projection display means selects a texture image suitable for rendering or a texture image more appropriate than a plurality of texture images when performing texture mapping of a common area in a moving image or image sequence captured from the plurality of directions. Generate,
(3) The three-dimensional modeling apparatus according to any one of (1) to (5).

(Corresponding embodiment)
The embodiment related to the three-dimensional modeling apparatus described in (6) corresponds to the third embodiment.
(Function and effect)
According to the three-dimensional modeling apparatus described in (6), it is necessary to photograph from a plurality of viewpoints in order to generate a full-circle three-dimensional image of the target three-dimensional model. Shoots to have overlapping areas, and uses the optimal texture for the overlapping areas, or generates the optimal texture by combining multiple textures corresponding to the overlapping areas In addition, by performing texture mapping, a higher-quality three-dimensional image can be generated.

  (7) The projection display means uses, as a selection criterion for a texture image suitable for rendering in the common area, an angle formed between the perpendicular direction of each polygon of the three-dimensional model in the area and the line-of-sight direction of the image input device. The three-dimensional modeling apparatus according to (6), wherein the smaller one is selected.

(Corresponding embodiment)
The embodiment related to the three-dimensional modeling apparatus described in (7) corresponds to the third embodiment.
(Function and effect)
According to the three-dimensional modeling apparatus described in (7), it is necessary to photograph from a plurality of viewpoints in order to generate a full-circle three-dimensional image of the target three-dimensional model. Shoots to have overlapping areas, and uses the optimal texture for the overlapping areas, or generates the optimal texture by combining multiple textures corresponding to the overlapping areas In addition, by performing texture mapping, a higher-quality three-dimensional image can be generated. The optimum texture selection condition is to use the angle formed by the line-of-sight information of the image input device and the direction of the normal of the polygon as a criterion, and by adopting the one with the smaller angle, an image close to what was photographed from the front is used. Since it can be selected and the amount of distortion can be reduced, a three-dimensional image closer to a real image can be generated.

(8) The position and orientation information corresponding to each image of the moving image or image sequence obtained by photographing the target object and the optimum texture image corresponding to each polygon of the three-dimensional model are associated with each other and registered as a database. Every
When performing arbitrary viewpoint rendering, rendering can be performed by always matching the optimum texture image for each polygon.
(3) The three-dimensional modeling apparatus according to any one of (1) to (7).

(Corresponding embodiment)
The fifth embodiment corresponds to the embodiment relating to the three-dimensional modeling apparatus described in (8).
(Function and effect)
Since the process of selecting or generating an optimal texture for each polygon is a costly process, it may not be suitable for real-time rendering. For this reason, in the three-dimensional modeling apparatus described in (8), after selecting an optimal texture, it is associated with the corresponding polygon, registered in the database, and stored in the database for next use. By making the reference, rendering in real time becomes possible, and a three-dimensional image that is easier to use can be generated.

(9) Acquiring moving images or image sequences of the target object in different time zones,
Project and render a texture image obtained from a video or an image sequence acquired in an arbitrary time zone,
The three-dimensional modeling apparatus according to any one of (1) to (8), wherein

(Corresponding embodiment)
The embodiment related to the three-dimensional modeling apparatus described in (9) corresponds to the sixth embodiment.
(Function and effect)
The three-dimensional shape model does not change, but the texture may change with time or season. Therefore, in the three-dimensional modeling apparatus described in (9), the user can capture a texture in different time zones and further create a database of information obtained by performing an optimum texture association process with each polygon. It becomes possible to perform rendering of a three-dimensional image in an arbitrary time zone, and the use range of the generated three-dimensional image becomes wide.

(10) In a 3D modeling method for performing texture mapping on a 3D shape model and performing 3D modeling so that rendering can be performed at an arbitrary viewpoint,
Enter the shape information of the target object,
Input a moving image obtained by photographing the target object by moving the image input device or a continuous image sequence in time series,
Calculate the position / posture information of the image input device based on the input moving image or image sequence,
Associating the calculated position / posture information of the image input device with each texture image included in the image sequence constituting the input moving image or the input time-series continuous image sequence And
Rendering a three-dimensional model corresponding to each position / posture of the image input device according to the calculated position / posture information of the image input device from the shape information of the input target object,
Projecting and displaying each of the associated texture images corresponding to each position and orientation of the image input device on the rendered three-dimensional model;
A three-dimensional modeling method characterized by this.

(Corresponding embodiment)
The first to sixth embodiments correspond to the embodiment relating to the three-dimensional modeling method described in (10).
(Function and effect)
According to the three-dimensional modeling method described in (10), the position / posture information of the image input device can be calculated by acquiring a moving image or a continuous image while moving the viewpoint by the image input device. Since it is not necessary to use the position / posture measuring unit of the apparatus, a simple system can be configured.

(11) In a program that causes a computer to perform texture mapping on a three-dimensional shape model and perform three-dimensional modeling so that rendering can be performed from an arbitrary viewpoint,
A function to input the shape information of the target object,
A function of inputting a moving image obtained by photographing the target object by moving the image input device or a continuous image sequence in time series;
A function for calculating position / posture information of the image input device based on the input moving image or image sequence;
Associating the calculated position / posture information of the image input device with each texture image included in the image sequence constituting the input moving image or the input time-series continuous image sequence With the ability to
A function of rendering a three-dimensional model corresponding to each position / posture of the image input device according to the calculated position / posture information of the image input device from the input shape information of the target object;
A function of projecting and displaying the associated texture images corresponding to the positions and orientations of the image input device on the rendered three-dimensional model;
A program to realize

(Corresponding embodiment)
Embodiments relating to the program described in (11) correspond to the first to sixth embodiments.
(Function and effect)
According to the program described in (11), the position / posture information of the image input device can be calculated by moving image or continuous image acquisition while moving the viewpoint by the image input device. -Since it is not necessary to use posture measuring means, a simple system can be configured.

It is a figure which shows the whole schematic structure of the three-dimensional modeling apparatus which concerns on the 1st thru | or 3rd embodiment of this invention. (A) is a diagram showing a three-dimensional shape model of the target object, (B) is a diagram showing a shooting scene of a continuous viewpoint moving image of the target object by the image input device, and (C) is a texture image on the three-dimensional model. (D) is a figure for demonstrating the texture mapping of a perimeter image. (A) is a diagram showing an operation flowchart of the three-dimensional modeling apparatus according to the first embodiment, (B) is a diagram showing an operation flowchart of the three-dimensional modeling apparatus according to the second embodiment of the present invention, (C) These are figures which show the operation | movement flowchart of the three-dimensional modeling apparatus which concerns on 3rd Embodiment of this invention. (A) is a figure for demonstrating that the area | region image | photographed moves with the movement of a viewpoint at the time of imaging | photography while moving a viewpoint of a target object, (B) can be rendered by several viewpoint positions. It is a figure for demonstrating an overlap area | region. It is a figure which shows the example of the display screen structure for displaying the texture mapping or projection information from the several viewpoint in 4th Embodiment of this invention. A method of associating an optimum texture with each viewpoint for each component such as a polygon constituting a three-dimensional shape model in the fifth embodiment of the present invention and creating a database, and a difference in the sixth embodiment of the present invention It is a figure which shows the relationship of the optimal texture in each viewpoint and each time about each component for demonstrating the method of making a database about a time slot | zone.

Explanation of symbols

    DESCRIPTION OF SYMBOLS 1 ... Arithmetic unit, 2 ... Display device, 3 ... Instruction device, 4 ... Image input device, 11 ... Database, 12 ... Image input unit, 13 ... Feature point / line / region extraction processing calculation unit, 14 ... Camera position / posture Information computing unit 15. Position / attitude information associating unit corresponding to image 16. Rendering information computing unit based on model position and orientation 17. Texture projection information computing unit 18. Multiple viewpoint image integration computing unit 19 ... Polygon and optimum texture associating portion 21. Previous display area 21 A 3D image rendering window 21 B- 1 to 21 Bn Sub-window 21 C Control panel 51 to 53 View point 61 62 62 Area 63 , 64 ... Overlapping area.

Claims (11)

In a 3D modeling device that performs texture mapping on a 3D shape model and performs 3D modeling so that rendering can be performed from an arbitrary viewpoint,
Shape information input means for inputting shape information of the target object;
An image input means for inputting a moving image obtained by photographing the target object by moving the image input device or a continuous image sequence in time series;
Position / attitude information calculating means for calculating position / attitude information of the image input device based on a moving image or an image sequence input by the image input means;
The position / orientation information of the image input device calculated by the position / orientation information calculating means is the time series input by the image sequence constituting the moving image input by the image input means or the image input means. Association means for associating each texture image included in the continuous image sequence;
3 corresponding to each position / posture of the image input device according to the position / posture information of the image input device calculated by the position / posture information calculation unit from the shape information of the target object input by the shape information input unit. A rendering means for rendering a dimensional model;
Projection display means for projecting and displaying each texture image associated by the association means corresponding to each position / posture of the image input device on the three-dimensional model rendered by the rendering means;
A three-dimensional modeling apparatus comprising:   The position / posture information calculation means extracts at least one of feature points, characteristic line segments, and characteristic shapes in an image in the moving image or image sequence input by the image input means, and extracts these The three-dimensional modeling apparatus according to claim 1, wherein characteristic point, line segment, and shape are tracked, and position / posture information of the image input apparatus is calculated based on a principle of triangulation.   When the texture image projected for each posture information of each position of the image input device calculated with time change moves on the three-dimensional model rendered by the rendering means, The texture image projected on the area is held, and the projected texture image is left and rendered on the area that has been projected even once with the change of the position / posture of the image input device. The three-dimensional modeling apparatus according to claim 1 or 2, characterized in that   When the image region associated with each polygon of the three-dimensional model changes with the change of the image associated with the three-dimensional shape model with respect to the texture image projected along with the movement of the position / posture of the image input device, 4. The apparatus according to claim 3, further comprising a determination unit that determines whether or not to update a texture image mapped for each polygon so that the polygon and the corresponding image region are optimally associated with each other. 3D modeling equipment.   The determination means selects, as a reference for determining whether or not to update the texture image of each polygon area, the one having a smaller angle between the line-of-sight direction of the image input device and the corresponding perpendicular direction of the polygon. The three-dimensional modeling apparatus according to claim 4, wherein the three-dimensional modeling apparatus is characterized. The moving image or time-series continuous image sequence input by the image input means is a moving image or image sequence photographed from a plurality of directions by the image input device so as to include a common photographing region,
The position / posture information calculation means calculates the position / posture information of the image input device from each of the moving images taken from the plurality of directions or images constituting the image sequence,
The projection display means selects a texture image suitable for rendering or a texture image more appropriate than a plurality of texture images when performing texture mapping of a common area in a moving image or image sequence captured from the plurality of directions. Generate,
The three-dimensional modeling apparatus according to any one of claims 1 to 5, wherein   The projection display means has a smaller angle formed between the perpendicular direction of each polygon of the three-dimensional model in the common area and the line-of-sight direction of the image input device as a selection criterion for a texture image suitable for rendering in the common area. The three-dimensional modeling apparatus according to claim 6, wherein the three-dimensional modeling apparatus is selected. Position and orientation information corresponding to each image of the moving image or image sequence in which the target object is captured, and an optimum texture image corresponding to each polygon of the three-dimensional model are associated with each other and registered as a database,
When performing arbitrary viewpoint rendering, rendering can be performed by always matching the optimum texture image for each polygon.
The three-dimensional modeling apparatus according to any one of claims 1 to 7, Acquire a moving image or image sequence of the target object at different times,
Project and render a texture image obtained from a video or an image sequence acquired in an arbitrary time zone,
The three-dimensional modeling apparatus according to any one of claims 1 to 8, In the 3D modeling method of performing texture mapping on a 3D shape model and performing 3D modeling so that rendering can be performed from an arbitrary viewpoint,
Enter the shape information of the target object,
Input a moving image obtained by photographing the target object by moving the image input device or a continuous image sequence in time series,
Calculate the position / posture information of the image input device based on the input moving image or image sequence,
Associating the calculated position / posture information of the image input device with each texture image included in the image sequence constituting the input moving image or the input time-series continuous image sequence And
Rendering a three-dimensional model corresponding to each position / posture of the image input device according to the calculated position / posture information of the image input device from the shape information of the input target object,
Projecting and displaying each of the associated texture images corresponding to each position and orientation of the image input device on the rendered three-dimensional model;
A three-dimensional modeling method characterized by this. In a program that causes a computer to perform texture mapping on a 3D shape model and perform 3D modeling so that rendering can be performed from an arbitrary viewpoint,
A function to input the shape information of the target object,
A function of inputting a moving image obtained by photographing the target object by moving the image input device or a continuous image sequence in time series;
A function for calculating position / posture information of the image input device based on the input moving image or image sequence;
Associating the calculated position / posture information of the image input device with each texture image included in the image sequence constituting the input moving image or the input time-series continuous image sequence With the ability to
A function of rendering a three-dimensional model corresponding to each position / posture of the image input device according to the calculated position / posture information of the image input device from the input shape information of the target object;
A function of projecting and displaying the associated texture images corresponding to the positions and orientations of the image input device on the rendered three-dimensional model;
A program to realize

Images