JPH0997344A - Method and system for texture generation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a texture of high quality by correctly positioning and composting plural two-dimensional images. SOLUTION: In the texture generation for three-dimensional computer graphics which combines a texture and a three-dimensional shape model together and displays a body on a screen in three dimensions, a 1st image G1 showing a body viewed from a 1st viewpoint and the three-dimensional shape model are positioned to show the body viewed from a 2nd viewpoint, and a 2nd image G3 including visual information on an overlap part in the body that part of the 1st image corresponds to and the three-dimensional shape model are positioned; and partial images G10 and G20 corresponding to a polygon Q4 corresponding to the overlap part are extracted from the 1st and 2nd images according to the positioning result of the overlap part of the three-dimensional model about the polygon Q4, and the 1st partial image G10 and 2nd partial image G20 are positioned so that corresponding pixels overlap each other by performing pattern matching in pixel unit.

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to a texture generation method and a texture generation system for three-dimensional computer graphics.

[0002] Three-dimensional computer graphics is a technique for stereoscopically displaying a real object or a fictitious object on a screen based on its three-dimensional shape model, and an object viewed from an arbitrary viewpoint can be displayed in real time. Therefore, it is applied to landscape simulations and presentations. Usually, an object is represented by a three-dimensional shape model and a texture that represents a pattern or a texture.

[0003]

2. Description of the Related Art In order to more realistically display an existing object, a texture (image data for mapping) to be attached to a three-dimensional shape model may be generated by using a photographed image of the object.

In order to obtain realism even when the object is viewed from any position around it on the screen, at least a real image taken from the front side and a real image taken from the back side of the object are required. Is. For a large object such as a building, there is a situation in which, for example, the entire front image cannot be fit in one photograph due to restrictions on the shooting position. That is, it is necessary to divide the front image into a plurality of parts and photograph them.

In the conventional texture generation system, when a texture is generated using a plurality of photographed images, each photographed image is individually positioned with respect to the three-dimensional shape model, and each photographed image is registered based on the positioning information. It has been configured to determine the relative position and perform image composition (Japanese Patent Laid-Open No. Hei.
138784). The positioning with respect to the three-dimensional shape model is performed by a system operator in a specific position (for example, a position corresponding to a vertex of the three-dimensional shape model) in a state where the three-dimensional shape model and the real image are displayed on the same screen. Was done by specifying.

[0006]

Conventionally, when synthesizing a plurality of photographed images, the relative position of each photographed image is determined on the basis of the three-dimensional shape model. If there was a deviation (modeling error) from the structure, an error occurred in the alignment of the photographed image. Therefore, there is a problem that the images are disturbed at the overlapping portions of the real shot images, and the realism of the texture is impaired.

The present invention has been made in view of this problem, and a plurality of two-dimensional images are correctly aligned and synthesized,
It is intended to generate high quality textures.

[0008]

[Means for Solving the Problems] Each of a plurality of images is set to three.
The part corresponding to a specific surface on the object (partial image) from each image by positioning with respect to the three-dimensional shape model
Can be extracted as a target of synthesis. By performing pattern matching in which the pixels having the same pixel value are associated with each other with respect to the plurality of partial images extracted from each image, the partial images can be correctly aligned.

The method according to the first aspect of the present invention is a texture generation method for three-dimensional computer graphics in which an object is stereoscopically displayed on a screen by combining a texture and a three-dimensional shape model in a polygon format. A first image showing the object viewed from a first viewpoint and a three-dimensional shape model of the object are positioned to show the object viewed from a second viewpoint and a part of the first image of the object is The second image including the visual information of the corresponding overlapping portion and the three-dimensional shape model are positioned, and the first image and the second image based on the positioning result for the polygon corresponding to the overlapping portion in the three-dimensional model. A partial image corresponding to the polygon is extracted from both of the images, and the partial image extracted from the first image and the partial image extracted from the second image are pixel by pixel. By aligning to the corresponding pixels overlap together by performing pattern matching, and generates a texture pasted on the polygon.

According to a second aspect of the present invention, when extracting partial images corresponding to the polygons from both the first image and the second image, the two images viewed from the same viewpoint based on the inverse perspective transformation calculation are used. It is extracted as a three-dimensional image.

According to a third aspect of the present invention, there is provided a texture generation system for three-dimensional computer graphics, which has a function of generating a texture to be attached to a three-dimensional shape model of an object based on a real image of the object. And a means for positioning the real photographed image with respect to the three-dimensional shape model, and when a plurality of the real photographed images are required to generate a texture to be attached to one polygon of the three-dimensional shape model, Perform pattern matching for each pixel based on the positioning result,
It has means for aligning the plurality of real shot images so that corresponding pixels overlap each other, and means for synthesizing the plurality of aligned real shot images to generate the texture.

A system according to a fourth aspect of the present invention has a function of correcting the three-dimensional shape model in accordance with a result of alignment of the plurality of real shot images.

[0013]

1 is a block diagram of a texture generation system 1 according to the present invention. The texture generation system 1 is a CPU 1 that is responsible for system control and image data processing.
0, a keyboard 21, a mouse 22, a display 30 capable of displaying gradation in pixel units, a memory device 40 including a medium represented by a hard disk, and an image reader 5.
0. The CPU 10 incorporates software for texture generation. Mouse 2
Instead of 2, another pointing device such as a trackball may be used. Instead of the image reader 50, a video camera may be used to input the image information of the object to the CPU 10. CP images from storage media such as optical disks
It is also possible to supply to U10.

FIG. 2 is a functional block diagram of the CPU 10. The CPU 10 includes an image input unit 11 that captures image data in pixel units output from the image reader 50, a shape data input unit 12 that reads a three-dimensional shape model from the memory device 40, a positional relationship between the image and the three-dimensional shape model, and a plurality of positions. A positioning unit 13 that specifies the positional relationship between images, an image combining unit 14 that combines a plurality of images to generate a texture (data D14), a texture output unit 15 that controls output of the generated texture, and a positioning unit 13. It has a shape data output unit 16 that controls the output of the three-dimensional shape model corrected by. The output of the texture and the 3D geometric model means storage in the memory device 40 or transfer to another system (for example, a graphic system).

The positioning unit 13 includes a manual positioning unit 131 that performs a process according to an operator's instruction operation, and an automatic positioning unit 132 that performs a process for optimizing positioning.
It is composed of

FIG. 3 is a flowchart showing the operation of the manual positioning section 131, and FIG. 4 is a flowchart showing the operation of the automatic positioning section 132. The manual positioning unit 131 is
The one or more images captured by the image input unit 11 and the three-dimensional shape model obtained by the shape data input unit 12 are displayed on the screen of the display 30 (# 11).

If there is an instruction to modify the 3D geometric model from the operator (# 12), the 3D geometric model is modified (# 13), and the modified 3D geometric model is displayed (#).
14). The operator repeats the correction instruction operation as necessary. When the operator performs the correction end operation (# 1
5), the manual positioning unit 131 generates positioning data D131 indicating the positional relationship between the image and the three-dimensional shape model (# 16).

There are two modes for generating the positioning data D131. One is to display a table that associates the singular points (for example, the vertices of polygons) of the three-dimensional shape model with the coordinates of the pixels of the image, and moves the cursor of the mouse 22 to the pixel that the operator thinks corresponds to the singular point. The cursor coordinates displayed at that time are written in the table as the coordinates of pixels by the operator. That is, the operator inputs a numerical value. The other one is to display a 3D geometric model in a wire frame, for example, and display it on an image. When the operator finishes the correction of the wire frame, the pixel that overlaps with the vertex of the wire frame on the screen is displayed. This is a form in which coordinates are calculated. That is, the positioning data D131 is generated based on the result of the alignment on the screen 31 by the operator. The operator modifies (moves / rotates / enlarges / reduces) the wire frame so that it matches the image as much as possible.
In the texture generation system 1, one of these forms can be selected by an environment setting operation.

As shown in FIG. 4, the processing performed by the automatic positioning unit 132 is unique to the present invention relating to the processing for improving the positioning accuracy of the image with respect to the three-dimensional shape model (# 21 to # 25) and the image synthesis. Processing (# 26 to # 32).

The automatic positioning unit 132 takes in the positioning data D131 generated by the manual positioning unit 131 as temporary positioning information (# 21), analyzes the three-dimensional shape model and the image, and acquires information necessary for positioning. (#
22, 23). For example, ridge line data is extracted from the three-dimensional shape model, and the corresponding boundary line of the ridge line is extracted from the image. The boundary line can be obtained by paying attention to the pixels near the ridge line using the positioning data D131 acquired in advance, and applying the straight line that best matches the distribution of the data value of the pixel using the least square method or the like. .

The positioning data is reset based on the analysis result (# 24). For example, the intersections of the plurality of boundary lines extracted from the image by the above-described procedure are associated with the vertices (the intersections of the ridges) of the three-dimensional shape model. All the images read by the image reader 50 are positioned with respect to the three-dimensional shape model (# 25).

When the positioning with respect to the three-dimensional shape model is completed, it is checked whether or not there are surfaces corresponding to a plurality of images in the three-dimensional shape model (# 26). If the answer is yes, the positions of a plurality of images are aligned by pattern matching in pixel units (# 28). The data D13 indicating the result of the alignment is used for the texture generation in the image synthesis unit 14.

By aligning a plurality of images so that pixels corresponding to each other overlap with each other, a deviation between the images and the three-dimensional shape model may become apparent. In that case, the three-dimensional shape model is modified (# 2
9, 30). If it is necessary to correct the positioning data along with the modification of the three-dimensional shape model, the positioning data is also modified (# 31, 32).

The procedure of texture generation by combining a plurality of images will be specifically described below. FIG. 5 is a diagram showing a relationship between an image and an object. Here, the substantially rectangular parallelepiped building O shown in FIG. 5A is an object to be displayed by three-dimensional computer graphics. A texture of the appearance of the horizontally long surface S1 hatched in FIG. 5B is generated using the two images G1 and G2 shown in FIGS. 5C and 5D.
The image G1 is a photograph of the building O taken from the position VP1 on the right side toward the surface S1, and the image G2 is a photograph of the building O taken from the position VP2 on the left side toward the surface S1. A portion S1 between two broken lines in FIG. 5B on the surface S1
The appearance of a is shown in both images G1 and G2.

FIG. 6 is a diagram showing a display example of the display 30, and shows the contents of the screen 31 at the stage when step # 11 of FIG. 3 is completed. Screen 3
1 has an area for displaying icons and messages (not shown).

The image reader 50 is provided in the lower half of the screen 31.
The four images G1 to 4 read by are arranged. Images G3 and G4 are also photographs of the building O.
On the left side of the upper half of the screen 31, a rectangular wire frame model is displayed as the current three-dimensional shape model M of the building O.

When the operator designates the positioning mode, an area (window) for positioning in the screen 31.
Aw is set. In the area Aw, the image to be positioned and the three-dimensional shape model M are displayed in an overlapping manner. However, when the positioning data is generated by numerical input as described above, a table for numerical input is displayed.

FIG. 7 is a diagram showing an example of image positioning with respect to the three-dimensional shape model M, and FIG.
It is a schematic diagram of a positional relationship between a three-dimensional shape model M and an image G1. In the example of FIG. 7A, the image G1 and the three-dimensional shape model M are displayed in the area Aw. Actually, the three-dimensional shape model M is displayed in an appropriate color so that it can be easily distinguished from the image G1. Further, in order to facilitate understanding of the arrangement state of the three-dimensional shape model M, the operator can display the wire frame model of the sphere as a guide polygon.

The image of the three-dimensional shape model M in FIG.
A region A of the three-dimensional shape model M arranged in the global coordinate space (also referred to as the world coordinate space) as shown in FIG.
It is a projected image on w (screen 31). The operator adjusts the arrangement state (position, inclination) of the three-dimensional shape model M in the global coordinate space and deforms the three-dimensional shape model M (changes the relative ratio of width, height, and depth). As shown in FIG. 7B, each side of the three-dimensional shape model M and the ridgeline of the building O in the image G1 are made to match as much as possible. As described above, the three-dimensional shape model M based on the information of the image G1.
The operation of correcting the is the operation of positioning the image G1 and the three-dimensional shape model M.

Here, the important point is that in the image G1, the building O
Since the appearance information of a part of the surface S1 of FIG.
As shown in (B), in the state where the three-dimensional shape model M and the building O in the image G1 are matched, a part Ma of the three-dimensional shape model M is protruding from the image G1. That is, in the horizontal direction of the surface S1 of the building O, the length of the three-dimensional geometric model M does not always correspond to the actual length of the building O.

The manual positioning section 131 generates positioning data D131 according to the positioning operation by the operator. FIG. 9 is a diagram showing a representation example of the three-dimensional shape model M, and FIG. 10 is a diagram showing a representation example of the positioning data D131.

The three-dimensional shape model M is represented by six polygons of numbers 0-5. In FIG. 9 and FIG. 10, the polygon numbers are indicated by parenthesized numbers for convenience. Each polygon is a quadrangle, and four vertices correspond to each. Each vertex is represented by coordinates (x, y, z) in a local coordinate system (also called a body coordinate system) which is an orthogonal coordinate system for defining the three-dimensional shape model M.

The positioning data D131 associates the vertices of the three-dimensional shape model M on a polygon-by-polygon basis with the global coordinates (X, Y) of the pixels overlapping each vertex on the screen 31. In the example of FIG. 9, the polygons with the numbers (1) and (4) are associated with the image G1. The polygon with the number (4) (hereinafter, referred to as polygon Q4) is the surface S1 of the building O.
Corresponding to.

The operator can select the image G1 as in the case of the image G1.
Position 2 with respect to the three-dimensional shape model M. Image G
2 also lacks the appearance information of part of the surface S1 of the building O (see FIG. 5), and therefore the surface S1 can be obtained using only the information of the image G2.
Unable to determine the horizontal length of.

FIG. 11 is a schematic diagram of texture generation by image synthesis. The automatic positioning unit 132 is shown in FIG.
Image G positioned with respect to polygon Q4 as
1, G2 to partial image G1 corresponding to polygon Q4
0 and G20 are extracted [FIG. 11 (B)]. At this time, the partial images G10 and G20 whose viewpoint is the position in front of the polygon Q4 are extracted by using the calculation method of perspective transformation.

The extraction procedure is as follows. The focal length f of the images G1 and G2 is obtained by image analysis. When the image data has a shooting condition as an attribute of the images G1 and G2, the focal length f can be obtained by referring to the shooting condition. Each vertex Pi of the polygon Q4
(I is the vertex number) global coordinates (X _Pi , Y _Pi ,
Z _Pi ). A circumscribed rectangle of the three-dimensional shape model M in the global coordinate system is obtained. The partial images G10 and G20 to be combined are set as similar shapes of circumscribing rectangles.

If the vertical length (vertical) of the set partial images G10 and G20 is h and the horizontal length (horizontal) is w, the coordinates (Xs, Ys) of the partial images G10 and G20 are set.
The value of each directional component of is within the range of 0 <Xs <w and 0 <ys <h. The coordinates (Xi, Yi) of the pixels of the images G1 and G2 to be applied as the pixel of the coordinates (Xs, Ys) are: Xi = f × (Xs × Ux + Ys × Vx + Ox) / (Xs
× Uz + Ys × Vz + Oz) Yi = f × (Xs × Uy + Ys × Vy + Oy) / (Xs
× Uz + Ys × Vz + Oz). However, Ox, Oy, Oz correspond to the coordinates of the upper left point of the circumscribed rectangle, and Ux, Uy, Uz are the values obtained by subtracting the coordinates of the upper left point from the upper right point of the circumscribed rectangle, Vx, Vy,
Vz is a value obtained by subtracting the coordinates of the upper left point from the lower left point of the circumscribed rectangle. When Xi and Yi are out of the images G1 and G2, the data value of the corresponding pixel in the partial images G10 and G20 is indefinite.

In FIG. 11A, the polygon Q4 is 4
It is represented by one vertex P0, P4, P6, P2. When the two partial images G10 and G20 are combined based on the positioning data D131 for this polygon Q4, FIG.
A part r in the horizontal direction (horizontal direction in the figure) such as 1 (D)
The image is disturbed by. This is the image G as described above.
This is because part of the dimensions of the three-dimensional shape model M (the distance between the vertices P0 and P4) has not been determined due to the loss of the appearance information of 1 and G2.

The automatic positioning unit 132 performs pattern matching in pixel units on the extracted partial images G10 and G20, and correctly aligns the partial images G10 and G20 so that the pixels corresponding to the same part of the building O overlap. (FIG. 11C). Since the viewpoints are aligned when extracting the partial images G10 and G20, high-speed alignment can be performed by pattern matching of the two-dimensional image.
For pattern matching, positioning data D13
Focusing on the overlapping portion of the images G1 and G2 based on 1, the overall difference in brightness between the images G1 and G2 is obtained. And
By correcting the pixel value according to the difference, the image G1,
After making the overall brightness between G2 uniform,
The pixel values of are compared.

The image synthesizing unit 14 generates one texture D14 to be attached to the polygon Q4 based on the alignment data D13 between images showing the alignment result.
When the images G1 and G2 are combined, the average value of the pixel values of the images G1 and G2 is set as the pixel value of the combined image. Image G1,
Higher-quality texture D14 can be obtained by performing appropriate weighting in the averaging in accordance with the exposure condition of G2 shooting.

[0041]

According to the first to fourth aspects of the present invention,
A plurality of two-dimensional images can be properly aligned and combined, and a high-quality texture can be generated.

According to the invention of claim 2, it is possible to facilitate the alignment of a plurality of two-dimensional images. According to the invention of claim 4, it is possible to represent an object without a sense of discomfort by using a texture and a three-dimensional shape model matching the texture.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a texture generation system according to the present invention.

FIG. 2 is a functional block diagram of a CPU.

FIG. 3 is a flowchart showing an operation of a manual positioning unit.

FIG. 4 is a flowchart showing an operation of an automatic positioning unit.

FIG. 5 is a diagram showing a relationship between an image and an object.

FIG. 6 is a diagram showing a display example of a display.

FIG. 7 is a diagram showing an example of image positioning with respect to a three-dimensional shape model.

FIG. 8 is a schematic diagram of a positional relationship between a three-dimensional shape model and an image in a three-dimensional space.

FIG. 9 is a diagram showing a representation example of a three-dimensional shape model.

FIG. 10 is a diagram showing an example of representation of positioning data.

FIG. 11 is a schematic diagram of texture generation by image synthesis.

[Explanation of symbols]

1 Texture Generation System 13 Positioning Unit (Means for Positioning) 14 Image Compositing Unit (Means for Generating Texture) 132 Automatic Positioning Unit (Means for Positioning Real Images) D14 Texture Data (Texture) D131 Positioning Data (Positioning Result) G1 image (first image) G2 image (second image) G10, G20 partial image M three-dimensional shape model O object Q4 polygon S1a overlapping part VP1 position (first viewpoint) VP2 position (second viewpoint)

Claims (4)

[Claims] 1. A texture generation method for three-dimensional computer graphics, in which a texture and a three-dimensional shape model in a polygon format are combined to stereoscopically display an object on a screen, the method being viewed from a first viewpoint. Positioning a first image showing an object and a three-dimensional shape model of the object, showing the object from a second viewpoint, and visualizing an overlapping portion corresponding to a part of the first image of the object. A second image including information and the three-dimensional shape model are positioned, and based on the positioning result with respect to the polygon corresponding to the overlapping portion in the three-dimensional model, the polygon is calculated from both the first image and the second image. And a partial image extracted from the first image and a partial image extracted from the second image are subjected to pattern matching in pixel units. The texture generation method is characterized in that a texture to be attached to the polygon is generated by performing the above-mentioned process and aligning corresponding pixels so as to overlap each other. 2. When extracting partial images corresponding to the polygon from both the first image and the second image, they are extracted as two-dimensional images viewed from the same viewpoint based on inverse perspective transformation calculation. 1. The texture generation method described in 1. 3. A texture generation system for three-dimensional computer graphics, which has a function of generating a texture to be attached to a three-dimensional shape model of an object based on a photographed image of the object, A means for positioning with respect to the three-dimensional shape model; and a case where a plurality of photographed images are required to generate a texture to be attached to one polygon of the three-dimensional shape model,
Pattern matching is performed on a pixel-by-pixel basis based on the positioning result for the polygon, and the means for aligning the plurality of real shot images so that the corresponding pixels overlap each other, and the plurality of real shot images that have been aligned are combined. And a means for generating the texture, and a texture generation system. 4. The texture generation system according to claim 3, further comprising a function of correcting the three-dimensional shape model according to a result of alignment of the plurality of real images.