JP2003337953A - Apparatus and method for image processing, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus and a method capable of generating a three- dimensional image of high picture quality according to image data from a plurality of viewpoint positions. <P>SOLUTION: Texture images are selected by patch planes on the basis of a picture quality evaluated value to which data on distances between the patch planes and respective viewpoints and data on directions of the viewpoints to the patch planes are applied and matching processing by end-point movement is performed based upon data on pixel value errors between the texture images at path boundary parts; and pixel values of a texture image in a viewpoint direction wherein it faces a patch plane among adjacent texture images is heavily weighted and pixel values at the patch boundary part are calculated. Further, pixel values in the patch plane are calculated based upon the pixel values of a batch border line by applying a weight coefficient which is in inversely proportional to the distance from the patch boundary line. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, an image processing method, and a computer program relating to a texture image pasting (texture mapping) technique for a three-dimensional shape model. More specifically, the patch surface as the texture image pasting surface in the three-dimensional shape model and the texture image pasting process that corrects the error in the mapping relationship between the texture images produce a realistic three-dimensional image with less unnaturalness. The present invention relates to an image processing device and an image processing method capable of generating the image.

[0002]

BACKGROUND OF THE INVENTION The use of three-dimensional shapes in computer space is increasing rapidly. Therefore, one problem is to efficiently import high-quality three-dimensional shapes into a computer. When capturing the 3D shape of a real object in a computer, the most widely used at present is a non-contact 3D that measures the 3D shape by an active method such as the optical cutting method, the moire method, or the coded slit method. Digitizers (eg Ref. [1] Kazumasa Watanabe, “The latest trends of non-contact 3D digitizers and image-based modelers for capturing solid images”, NIKKEI COMPUTER
GRAPHICS, pp.46-65, Aug. 1999.).

For example, a camera is fixed, an object to be measured is placed on a rotary table, a partial shape is measured while rotating the object, and the finally obtained three-dimensional shape is integrated.
Some products can capture a color image at the same time as a three-dimensional shape, and texture mapping can be performed by pasting this captured color image onto a three-dimensional shape surface generated as a texture image.

At this time, among the images picked up from a plurality of directions, the image that best captures the patch surface of each three-dimensionally shaped patch surface, that is, the image that faces the patch surface most directly. The quality of the final mapping result can be enhanced by selecting the image taken by the existing camera. Regarding selection of a captured image by a camera directly facing the patch surface, for example, see [2] “Hir
oyuki Sato, MasaharuShimanuki, Takao Akatsuka, "Te
xture Mapped 3D Face Modeling Method on Network us
ing Java, ・ Proceedings of SPIE, vol.3460, pp.609-6
17, 1998. ”.

On the other hand, although the accuracy of the obtained three-dimensional shape data is low, a special device is not required, and the three-dimensional shape of the object can be restored simply by preparing a plurality of still images and video images. Possible image-based approaches have recently received attention for various applications. Among them, the method of simultaneously recovering the three-dimensional shape and the camera motion from the time series image is described in T
There is a factorization method introduced by Omasi-Kanade et al.

The factorization method is described in, for example, reference [3] "Conrad J. Poelman, Takeo Kanade," Parap.
erspective Factorization Method for Shape and Moti
on Recovery ”, Technical Report CMU-CS-93-219, 199
3. ”or Reference [4]“ Takeo Kanade, Toshihiko Morita, Conrad Paulman, “Reconstruction of Object Shape and Camera Motion by Factorization Method”, IEICE Transactions, vol.J76-DI
I, no.8, pp.1497-1505, 1993. ”.

This factorization method is a method of sequentially tracking feature points such as the corners of an object and then using factorization of a matrix to restore the motion of a camera and the three-dimensional shape of an object. In this method, a multi-viewpoint image sequence is used in which a relative motion is generated between the object and the camera,
When mapping an image taken under such conditions, a problem arises when the image selection method described above is used.

That is, although there is an image taken from a distance close to the patch surface, the image taken from a long distance may be selected because the camera direction is slightly not directly facing the patch surface. Will end up. Also, when the tracking result of the feature points used as the mapping relationship (texture coordinates) between the three-dimensional shape surface and the image contains an error or the illumination condition between the images is not constant, the patch surface of the different image is mapped. There is a discontinuity between them, and finally 3
The quality of D-CG will deteriorate.

As shown in FIG. 1, when a three-dimensional image of the object 101 is to be generated, for example, the object 101 is placed on a turntable 102 and the camera 103 and the object 1 are placed.
The distance D between 01 is fixed, and the object 101 is rotated and imaged.

In the case of this photographing processing, it can be said that the image photographed by the camera most directly facing the patch surface captures the texture on that surface most. That is, when selecting an image for texture mapping (texturing), the direction of the normal vector of the patch surface and the shooting direction of the camera may be simply compared.

However, under the imaging conditions assumed here, the distance between the object to be measured and the camera changes arbitrarily, and at the same time, the attitude of the camera also rotates arbitrarily. Therefore, the optimum image may not be determined only by the shooting direction of the camera.

As shown in FIG. 2, the photographed images of the cameras 1 and 201 from the photographing viewpoint 1 which is the most confronting to the patch surface P 203 of interest having the center of gravity 204 but is far away are three-dimensionally shaped. Should be selected as an image to be attached to the data, that is, as a texture mapping image, or if the shooting direction is slightly smaller than the shooting viewpoint 1, the patch plane P,
Although not directly facing 203, the patch surfaces P, 203
It is not easy to determine whether the photographed images of the cameras 2 and 202 from the photographing viewpoint 2 having a short distance to should be selected as the texture mapping image.

Further, when the illumination conditions are not constant among the images, when textures are cut out from a plurality of images, an unnatural seam is generated between patch surfaces on which different images are mapped, and the finally obtained cubic is obtained. Original image data (3D C
The quality of G) deteriorates.

As a method of preventing such quality deterioration, it is possible to model the shape, texture, and optical characteristics of the entire scene to be displayed and correct these images. This method is described in, for example, reference [5] “Yizhou Y
u, Paul Debevec, Jitendra Malik, Tim Hawkins, “In
verse Global Illumination: Recovering ReflectanceM
odels of Real Scenes form Photographs ”, ACM SIGGRA
PH ・ 9 Proceedings, pp.215-224, 1999. ”, Ref. [6]
“Paul Debevec,“ Rendering Synthetic Objects into
Real Scenes: Bridging Traditional and Image-based
Graphics withGlobal Illumination and High Dynamic
Range Photography ”, ACM SIGGRAPH / 8 Proceedings,
pp.189-198, 1998. ”and reference [7]“ Imari Sato, Yoichi Sato, Katsushi Ikeuchi, “Superimposing virtual objects on real images considering optical consistency”, 3rd Intelligence Media Symposium, pp.23-32, 1997. ”.

However, when considering shooting outdoors,
It can be said that it is impossible to model the entire scene. A texture mapping method using a plurality of real images has been proposed so far, for example, in a document [8].
“Paul Debevec, Camillo Taylor, Jitendra Malik,
“Modeling and Rendering Architecture from Photogr
aphs: A hybrid geometry- and image-based approac
h ”, ACM SIGGRAPH ・ 6 Proceedings, pp.11-20, 199
6. ”, or reference [9]“ Paul Debevec, Yizhou Y
u, George Borshukov, “Efficient View-Dependent Im
age-Based Rendering with Projective Texture-Mappin
g ”, 9th Eurographics workshop on rendering, pp.105
-116, 1998. ”

The above problems are avoided by photographing the object under the conditions that the relationship between the illumination and the object is fixed (the illumination conditions match), the image resolution is the same, and accurate camera calibration is performed. ing.

[0017]

However, in general, when photographing an outdoor landscape or a building using a digital camera, a DV cam, or the like, it is almost impossible to match the lighting conditions, and a plurality of photographed images (textures) are used. It is difficult to perform high quality texture mapping using images.

In view of the above problems in the present situation, the present invention sets a mapping optimum image selection reference when the distance between the object and the camera and the camera direction change arbitrarily, and on the shared ridge line of the patch surface. An image processing apparatus and method having a texture coordinate correction method that pays attention to the distribution of pixel values of pixels, and a pixel value (density) correction method between pixels on a patch surface onto which images captured under different positions and lighting conditions are mapped. To do.

The image processing apparatus and the image processing method of the present invention are optimal images of texture mapping images when the distance between the camera and the three-dimensional image generation target imaged by the camera and the camera direction change arbitrarily. Based on selection criteria setting, texture coordinate correction processing that focuses on the distribution of pixel values on shared ridges of patch surfaces, and pixel value correction processing between patch surfaces on which images captured at different positions and lighting conditions are mapped Therefore, it is an object of the present invention to provide an image processing apparatus and an image processing method that can make a difference between images due to a mismatch of light source conditions inconspicuous and generate higher quality three-dimensional image data.

[0020]

The present invention has been made in consideration of the above problems, and the first aspect thereof is to generate a three-dimensional image by pasting a texture image on a three-dimensional shape model. In the image processing device, from the captured images of a plurality of viewpoints with respect to a three-dimensional image generation target, for each patch surface forming a three-dimensional shape model, distance data between the patch surface and each viewpoint, and direction data with respect to the patch surface of each viewpoint. An image processing apparatus characterized by having a texture image selection processing means for executing selection processing of an optimum texture image based on an image quality evaluation value to which is applied.

Further, in one embodiment of the image processing apparatus of the present invention, the texture image selection processing means selects one viewpoint image that most directly faces the patch surface forming the three-dimensional shape model, and the selected image is selected. Letting θ be an angle formed by a directional vector V from the viewpoint to the center of gravity of the patch surface and a normal vector n of the patch surface, a captured image from the viewpoint within a region of θ ± Δθ with the center of gravity as the center. A texture image candidate image for the patch surface is selected, and an optimum texture image selection process based on the image quality evaluation value is executed from the candidate images.

Further, in one embodiment of the image processing apparatus of the present invention, the evaluation value is a distance vector between the patch surface and the viewpoint, and a direction vector from the viewpoint to the center of gravity of the patch surface: V, and the patch. It is characterized in that it is a value calculated by an arithmetic expression to which an angle θ formed by a surface normal vector: n is applied.

Further, in one embodiment of the image processing apparatus of the present invention, the arithmetic expression is a calculation expression of the evaluation value: V shown below,

[Equation 7] However, D is the distance between the center of gravity of the patch surface and the camera that has captured the image to be evaluated, and Dmax is a plurality of Ds that are candidate images.
Is the maximum distance value of θ, θ is the direction vector of the camera that has captured the image to be evaluated: V, and the normal vector n of the patch surface P of interest is the angle: θ, w _D is the image quality for distance: D Degradation evaluation value: change rate of E: | dE / dD |, w _θ is
Change rate of image quality deterioration evaluation value: E with respect to angle: θ: | dE
It is characterized in that / dθ |.

Further, a second aspect of the present invention is that in an image processing apparatus for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, a patch in which images from different viewpoints are connected as a texture image. There is a matching processing means for executing a matching process of pixel values at the boundary portion, and the matching processing means has texture images at patch boundary portions having texture images from different viewpoints as adjacent texture images in the constituent data of the three-dimensional image. Pixel value error data between
Among the calculated pixel value error data, a selection process of selecting a patch boundary part having the maximum pixel value error data, and a matching process by moving an end point of a texture image corresponding to the selected patch boundary part are executed to obtain the pixel value error data. In the image processing apparatus, the selection process and the matching process are repeatedly executed until is less than or equal to a predetermined threshold.

Furthermore, a third aspect of the present invention is a patch in which images from different viewpoints are connected as texture images in an image processing apparatus for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model. There is a pixel value correction processing means for executing a pixel value correction processing at the boundary portion, and the pixel value correction processing means, in the adjacent texture image at the patch boundary portion, for the patch surface,
Image processing characterized in that the pixel value of the texture image in the front facing direction is heavily weighted, and the pixel value at the patch boundary portion is calculated based on each pixel value of the adjacent texture image. On the device.

Further, in one embodiment of the image processing apparatus of the present invention, the pixel value correction processing means is a correction pixel value: P _{i -new} calculation formula shown below,

[Equation 8] However, λ = (cos θn) / (cos θn + cos θm), and θn and θm are images Image _n and Im, respectively.
The pixel value at the patch boundary portion is calculated according to the angle formed by the direction vector V of the camera capturing the image _m and the normal vector n of the patch surface P of interest.

Further, in one embodiment of the image processing apparatus of the present invention, the pixel value correction processing means further corrects pixel values in a patch plane having a patch boundary obtained by connecting images from different viewpoints as a texture image. The pixel value of the point: Q in the patch surface is the distance from the point: Q with respect to the pixel value of the patch boundary line.
It is characterized in that a weighting coefficient inversely proportional to dn is applied and a process of calculating based on the pixel value of the patch boundary line is executed.

Further, in one embodiment of the image processing apparatus of the present invention, the pixel value correction processing means further corrects pixel values in a patch plane having a patch boundary obtained by joining images from different viewpoints as a texture image. The pixel value correction processing means is configured to execute processing, and the pixel value of a point: Q in a patch plane formed by a triangular region surrounded by three shared ridge lines: L _i , L _j , and L _k is Corrected pixel value shown in: Q _new calculation formula,

[Equation 9] However, Q _old is a pixel value before correction of point: Q, and Q _new is
Point: Pixel value after correction of Q, P _i-old , P _i-new : Pixel value before and after correction on shared line L _i P _j-old , P _j-new : shared line L _j Above uncorrected and corrected pixel values P _k-old , P _k-new : uncorrected and corrected pixel values on the shared line L _k , and w _i , w _j , w _k : points in the patch Q and each ridge line L _i , L _j ,
L _k on points P _i, P _j, the distance d _i between the P _k, d _j, the weighting factor is inversely proportional to d _k I, J, _K: the ridge line L _i, L _j, the number of pixels on the L _k According to a certain feature, the pixel value of the point: Q in the patch plane is calculated.

Further, a fourth aspect of the present invention is, in an image processing apparatus for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, from a plurality of viewpoint photographed images of a three-dimensional image generation object. , The selection process of the optimal texture image is performed based on the image quality evaluation value to which the distance data between the patch surface and each viewpoint and the direction data with respect to the patch surface of each viewpoint are applied for each patch surface forming the three-dimensional shape model. And the pixel value error data between the texture images at the patch boundary portion having the texture images from different viewpoints as the adjacent texture images in the constituent data of the three-dimensional image. Selection process for selecting the patch boundary part having the largest pixel value error data in the middle, and the selected patch boundary part Matching processing means for executing matching processing by moving the end points of the corresponding texture image and repeatedly executing the selection processing and the matching processing until the pixel value error data becomes equal to or less than a predetermined threshold value, and an adjacent texture at the patch boundary portion. In the image, with respect to the patch surface, the pixel value of the texture image in the facing viewpoint direction is heavily weighted, and the pixel value at the patch boundary portion is calculated based on each pixel value of the adjacent texture image. , The point in the patch plane: Q, and the pixel value of the patch boundary line
Distance: dn, a pixel value correction processing unit that applies a weighting coefficient that is inversely proportional to dn, and performs a process of calculating based on the pixel value of the patch boundary line.

Further, a fifth aspect of the present invention is an image processing method for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, in which images taken from a plurality of viewpoints with respect to a three-dimensional image generation object are taken. , The selection process of the optimal texture image is performed based on the image quality evaluation value to which the distance data between the patch surface and each viewpoint and the direction data with respect to the patch surface of each viewpoint are applied for each patch surface forming the three-dimensional shape model. The image processing method is characterized by including a texture image selection processing step.

Further, in one embodiment of the image processing method of the present invention, the texture image selection processing step comprises:
Most directly facing the patch surface that makes up the three-dimensional shape model 1
One viewpoint image is selected, and an angle between a direction vector: V from the viewpoint of the selected image toward the center of gravity of the patch surface and a normal vector: n of the patch surface is θ, and θ ± is centered on the center of gravity. A captured image from a viewpoint within the region of Δθ is set as a candidate image of a texture image for the patch surface, and a step of performing a selection process of an optimal texture image based on the image quality evaluation value from the candidate image is included. Characterize.

Furthermore, in one embodiment of the image processing method of the present invention, the evaluation value is a distance vector between the patch surface and the viewpoint, and a direction vector from the viewpoint to the center of gravity of the patch surface: V and the patch. It is characterized in that it is a value calculated by an arithmetic expression to which an angle θ formed by a surface normal vector: n is applied.

Further, in one embodiment of the image processing method of the present invention, the arithmetic expression is a calculation expression of the evaluation value: V shown below,

[Equation 10] However, D is the distance between the center of gravity of the patch surface and the camera that has captured the image to be evaluated, and Dmax is a plurality of Ds that are candidate images.
Is the maximum distance value of θ, θ is the direction vector of the camera that has captured the image to be evaluated: V, and the normal vector n of the patch surface P of interest is the angle: θ, w _D is the image quality for distance: D Degradation evaluation value: change rate of E: | dE / dD |, w _θ is
Change rate of image quality deterioration evaluation value: E with respect to angle: θ: | dE
/ Dθ |, where

Further, the sixth aspect of the present invention is an image processing method for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, which is a patch in which images from different viewpoints are connected as a texture image. There is a matching processing step for executing the matching processing of the pixel value at the boundary portion, and the matching processing step has a texture image at the patch boundary portion having texture images from different viewpoints as adjacent texture images in the constituent data of the three-dimensional image. Selection processing of calculating pixel value error data between the selected pixel value error data and selecting a patch boundary portion having the largest pixel value error data among the calculated pixel value error data, and matching by moving the end points of the texture image corresponding to the selected patch boundary portion Processing is performed, and the pixel value error data is determined in advance. Until a value below in the image processing method characterized by comprising the step of repeatedly executing the selection process and the matching process.

Further, a seventh aspect of the present invention is a patch in which images from different viewpoints are connected as texture images in an image processing method for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model. There is a pixel value correction processing step for performing a pixel value correction processing at the boundary portion, and the pixel value correction processing step is performed in the adjacent texture image at the patch boundary portion, and the texture in the viewpoint direction facing directly the patch surface. In the image processing method, a step of heavily weighting the pixel value of the image and calculating the pixel value at the patch boundary portion based on each pixel value of the adjacent texture image.

Further, in one embodiment of the image processing method of the present invention, the pixel value correction processing step includes the following corrected pixel value: P _{i -new} calculation formula,

[Equation 11] However, λ = (cos θn) / (cos θn + cos θm), and θn and θm are images Image _n and Im, respectively.
The pixel value at the patch boundary portion is calculated according to the angle formed by the direction vector V of the camera capturing the image _m and the normal vector n of the patch surface P of interest.

Further, in one embodiment of the image processing method of the present invention, the pixel value correction processing step further comprises:
Including a step of performing a pixel value correction process in a patch plane having a patch boundary obtained by joining images from different viewpoints as a texture image, the pixel value of the point: Q in the patch plane is set to the pixel value of the patch boundary line. On the other hand, it is characterized in that a weighting coefficient inversely proportional to the distance from the point: Q: dn is applied to perform the calculation based on the pixel value of the patch boundary line.

Further, in an embodiment of the image processing method of the present invention, the pixel value correction processing step further comprises:
Three shared edges: L _i , including a step of performing pixel value correction processing within a patch plane having patch boundaries obtained by joining images from different viewpoints as texture images
Points in the patch plane consisting of a triangular area surrounded by L _j and L _k :
The pixel value of Q is the following corrected pixel value: Q _new calculation formula,

[Equation 12] However, Q _old is a pixel value before correction of point: Q, and Q _new is
Point: Pixel value after correction of Q, P _i-old , P _i-new : Pixel value before and after correction on shared line L _i P _j-old , P _j-new : shared line L _j Above uncorrected and corrected pixel values P _k-old , P _k-new : uncorrected and corrected pixel values on the shared line L _k , and w _i , w _j , w _k : points in the patch Q and each ridge line L _i , L _j ,
L _k on points P _i, P _j, the distance d _i between the P _k, d _j, the weighting factor is inversely proportional to d _k I, J, _K: the ridge line L _i, L _j, the number of pixels on the L _k The pixel value of the point: Q in the patch plane is calculated according to

Further, an eighth aspect of the present invention is, in an image processing method for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, from a plurality of viewpoint photographed images of a three-dimensional image generation object. , The selection process of the optimal texture image is performed based on the image quality evaluation value to which the distance data between the patch surface and each viewpoint and the direction data with respect to the patch surface of each viewpoint are applied for each patch surface forming the three-dimensional shape model. Texture image selection processing step, and pixel value error data between the texture images at the patch boundary portion having texture images from different viewpoints as adjacent texture images in the three-dimensional image configuration data, and the calculated pixel value error data Selection processing for selecting the patch boundary portion having the largest pixel value error data in the middle, and the selected patch boundary A matching process step of executing the matching process by moving the end points of the texture image corresponding to the part and repeatedly executing the selection process and the matching process until the pixel value error data becomes equal to or less than a predetermined threshold value; In the adjacent texture image, the patch surface is heavily weighted to the pixel value of the texture image in the facing viewpoint direction with respect to the patch surface, and the pixel value at the patch boundary portion is calculated based on each pixel value of the adjacent texture image. In addition, the pixel value of the point: Q in the patch plane is applied to the pixel value of the patch boundary line by applying a weighting coefficient inversely proportional to the distance from the point: Q: dn to obtain the pixel value of the patch boundary line. And a pixel value correction processing step for performing a calculation based on the image processing method.

Further, a ninth aspect of the present invention is a computer program for executing image processing for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, which is a three-dimensional image generation object. Optimum based on the image quality evaluation value obtained by applying the distance data between the patch surface and each viewpoint and the direction data to the patch surface of each viewpoint from the captured images of multiple viewpoints for each patch surface forming the three-dimensional shape model A texture image selection processing step for executing texture image selection processing and calculating pixel value error data between texture images at a patch boundary having texture images from different viewpoints as adjacent texture images in the three-dimensional image constituent data. , Of the calculated pixel value error data, select the patch boundary having the maximum pixel value error data A matching process that executes the selection process and the matching process by moving the end points of the texture image corresponding to the selected patch boundary, and repeatedly executes the selection process and the matching process until the pixel value error data becomes equal to or less than a predetermined threshold value. Step, in the adjacent texture image in the patch boundary portion, with respect to the patch surface, a large weight is given to the pixel value of the texture image in the facing viewpoint direction, and the patch is calculated based on each pixel value of the adjacent texture image. The pixel value at the boundary is calculated, and the pixel value at the point: Q in the patch plane is calculated from the point: Q with respect to the pixel value at the patch boundary line:
A pixel value correction processing step of applying a weighting coefficient that is inversely proportional to dn and performing a processing of calculating based on the pixel value of the patch boundary line.

[0041]

In the image processing apparatus and method of the present invention,
Image quality obtained by applying the distance data between the patch surface and each viewpoint and the direction data to the patch surface of each viewpoint from the captured images of the three-dimensional image generation target from multiple viewpoints for each patch surface forming the three-dimensional shape model It is a configuration that executes the selection process of the optimum texture image based on the evaluation value, it is possible to easily select the optimum image with the least deterioration of image quality for each patch surface, and high-quality 3D image data Is realized.

Further, in the image processing apparatus and method of the present invention, pixel value error data between texture images at a patch boundary having texture images from different viewpoints as adjacent texture images in the three-dimensional image constituent data is calculated. Then, among the calculated pixel value error data, a selection process of selecting a patch boundary part having the maximum pixel value error data, and a matching process by moving the end points of the texture image corresponding to the selected patch boundary part are executed to obtain the pixel value. Until the error data becomes less than or equal to a predetermined threshold value, the selection process and the matching process are configured to execute the matching process repeatedly, so even at the patch surface boundary where images from different viewpoints are pasted as texture images, It is possible to generate high-quality 3D image data without image displacement. That.

Further, in the image processing apparatus and method of the present invention, in the adjacent texture images at the patch boundary portion, the pixel values of the texture image in the viewpoint direction directly facing the patch surface are heavily weighted, The pixel value at the patch boundary portion is calculated based on each pixel value of the adjacent texture images, and the pixel value of the point: Q in the patch plane is calculated from the point: Q with respect to the pixel value of the patch boundary line. Distance: dn, a weighting factor inversely proportional to the applied value is applied, and a pixel value correction process is performed to perform a process of calculating based on the pixel value of the patch boundary line. Therefore, images are taken under different shooting conditions. Even in a configuration in which images from different viewpoints are pasted as texture images of adjacent patch surfaces, the images between the patch surfaces are smoothly corrected, and high image quality without discomfort Generating three-dimensional image data is realized.

The computer program of the present invention is, for example, a storage medium or communication medium provided in a computer-readable format for a general-purpose computer system capable of executing various program codes, such as a CD or FD. , MO, etc., or a computer program that can be provided by a communication medium such as a network. By providing such a program in a computer-readable format, processing according to the program is realized on the computer system.

Further objects, features and advantages of the present invention are as follows.
A more detailed description will be made clear based on embodiments of the present invention described below and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to a device in which each configuration is provided in the same housing.

[0046]

BEST MODE FOR CARRYING OUT THE INVENTION An image processing apparatus and an image processing method according to the present invention will be described below in detail with reference to the drawings.

The image processing apparatus and the image processing method of the present invention are optimum images of texture mapping images when the distance between the camera and the three-dimensional image generation target imaged by the camera and the camera direction change arbitrarily. Setting of selection criteria, correction of texture coordinates paying attention to the distribution of pixel values on the shared ridges of the patch surface, density correction between patch surfaces on which images photographed at different positions and lighting conditions are mapped,
It includes each of these processes.

The texture mapping method of the present invention is described in, for example, document [3] "Conrad J. Poelman, Takeo Kan.
ade, “Paraperspective Factorization Method for S
hape and Motion Recovery ”, Technical Report CMU-CS
-93-219, 1993. ”or Reference [4]“ Takeo Kanade,
Toshihiko Morita, Conrad Paulman, “Reconstruction of Object Shape and Camera Motion by Factorization Method”, IEICE Transactions, vol.J76-D-II, no.8, pp.1497-1505, 1993. ” The present invention is applicable to a configuration using an image sequence photographed by a free camera work in which the camera pose and the distance are arbitrarily changed, as assumed by the factorization method described in A description will be given of a configuration using an image sequence photographed by such a free camerawork in which the camera posture and the distance are arbitrarily changed. However, the present invention is obtained from a plurality of images photographed from various arbitrary viewpoints. It is also applicable to the created 3D object.

[(1) Optimal Texture Image Selection Processing] First, the optimal texture mapping is performed when the distance between the camera to be measured for which a three-dimensional image is to be generated and the camera changes, and at the same time the orientation of the camera also rotates arbitrarily. The image selection process will be described. As described in the section of the related art, when the distance between the measurement object and the camera changes and the posture of the camera also rotates at the same time, it is difficult to select the optimum texture mapping image.

With reference to FIG. 3, the distance D from the photographing viewpoint to the measurement object and the angle θ with respect to the measurement object will be described.

As shown in FIG. 3, when there are a plurality of photographing viewpoints with respect to the patch surface P, 303 of interest where the three-dimensional image generation target is present, the distance between the cameras 1, 301 (photographing viewpoint 1) and the measurement target. Is D1, and the distance between the cameras 2 and 302 (imaging viewpoint 2) and the measurement target is D2.

Further, assuming that the normal vector of the patch surface P, 303 of interest is n, the vector from the camera 1, 301 (photographing viewpoint 1) to the center of gravity g, 304 of the patch surface of interest P, 303 is V1, and the camera 2, 302 is A vector from (shooting viewpoint 2) toward the center of gravity g304 of the patch surface P, 303 of interest is V
When 2, the cos θ1 for the angle θ1 formed by the normal vector n and the vector V1 is cos θ1 = (V
1 · n) / | V1 || n | and cos θ2 about the angle θ2 formed by the normal vector: n and the vector V2
Is cos θ2 = (V2 · n) / | V2 || n |.

In the example shown in FIG. 3, since θ1 <θ2, V1 · n> V2 · n, and if only the image directly facing the patch surface is selected, the cameras 1,301
It becomes an image of (shooting viewpoint 1). However, on the other hand, the distance between the cameras 1 and 301 (shooting viewpoint 1) and the measurement target is D
1, the distance between the camera 2, 302 (shooting viewpoint 2) and the object to be measured is D2, D1> D2, and if a camera with a short distance is selected, the camera 2,302
The image is (photographing viewpoint 2).

In this way, the image to be attached to the three-dimensional shape data, that is, the image to be selected as the texture mapping image is taken by the cameras 1 and 301 from the photographing viewpoint 1 that is most facing but is far away. Or, whether to select the captured image of the camera 3,202 from the shooting viewpoint 2 that is not directly facing the patch surfaces P and 303 from the shooting viewpoint 1 but is closer to the patch surfaces P and 303 Is difficult to judge.

In FIG. 4, the distance between the patch surface and the photographing viewpoint: D
And the angle between the normal vector of the patch surface and the vector from the shooting point of view to the center of gravity of the patch surface: θ, the image quality degradation evaluation value as the degradation level of the captured image corresponding to each value change:
The correspondence diagram with E is shown. Image quality deterioration evaluation value: E is E = 0
Indicates that the error rate is minimum, and E = 1.0 indicates that the error rate is high. Here, the deterioration degree of the image quality is calculated based on the mapping result when the initial value of the visual distance = D1 and the image rotation angle θ = 0 [deg], and as shown in (Equation 1) below, It is a value E (D, θ) calculated using the mean square error with the difference for each pixel as a scale.

[0056]

[Equation 13] … (Equation 1)

In the above (formula 1), I _base (i, j)
Is the initial value = D1, the image rotation angle θ = 0 [deg]
I _target (i, j) is based on the reference image, and I _target (i, j) changes the visual distance: D from the initial value: D1 to a distance 5 times the initial value: D1: D5, and at the same time, the vertical axis. This is a projection image that is rotated about θ = 0 to 85 [deg] about the center and maps these images to a plane of the same size as the reference image.

Although the image (N pixels × M pixels) used here is a color image, the deterioration degree: E was calculated using only the luminance value in order to simplify the examination. As the interpolation method for mapping the texture onto the plane, the bilinear interpolation method was used for both expansion and contraction. Therefore, when θ is constant, D
E (D, θ) increases linearly with the increase of
It was found that when θ was constant, E (D, θ) increased exponentially as θ increased. As shown in the instruction unit 401 of FIG. 4, when θ is constant, the distance (Distance
The image quality deterioration evaluation value: E shows a linear increase with an increase in e), and when D is constant as shown in the instruction unit 402 of FIG.
When θ increases, E (D, θ) increases exponentially.

Next, the evaluation value E calculated in this way
Use (D, θ) to introduce the optimal image selection criteria.

First, as shown in FIG. 5, the direction vector of the camera 503 most directly facing the patch planes P, 502 of interest on the three-dimensional image generation object 501, the vector V shown in FIG. When the angle formed by the normal vector n of the patch planes P, 502 that are present is n, then θ ±
A captured image acquired by another camera 504 within the range of Δθ is selected, and these captured images including the captured image of the camera 503 most directly facing the patch surface P, 502 are set as texture candidate images.

At this time, the value of Δθ is E with respect to the change of θ.
It is determined based on the magnitude of change in (D, θ) | dE (D, θ) / dθ |. First, based on the image quality deterioration evaluation value described above with reference to FIG. 4, the change rate of the image quality deterioration evaluation value: E with respect to the angle: θ is checked.

The camera 503 which is most directly facing the target patch plane P, 502 in the object 501 for generating a three-dimensional image.
Direction vector: V and the normal vector: n of the patch surface P, 502 of interest have a large angle θ, in this case,
Only the images of the patch surfaces P and 502 taken from an oblique direction with a large θ are held. As is clear from the data of FIG. 4, in the region where θ is large, the rate of change of the image quality deterioration evaluation value: E with respect to the change of the angle: θ, that is, | dE
The value of (D, θ) / dθ | becomes large.

Therefore, in the region where θ is large, the distance:
Even if D is close, the image from the viewpoint within the range of θ ± Δθ has a large deviation from the front facing direction with respect to the patch surface, and further the angle deviation causes a sharp increase in the degree of image quality deterioration. Therefore, it is necessary to focus on the direction of the camera and narrow down the candidates. That is, it is necessary to set Δθ small.

Further, the direction vector V of the camera 503 most directly facing the target patch plane P, 502 in the three-dimensional image generation target 501 and the target patch plane P, 502.
When the angle formed by the normal vector: n is θ is small, in this case, an image of the patch surfaces P and 502 taken from the front side where θ is small is held. As is clear from the data in FIG. 4, in the region where θ is small, the change rate of the image quality deterioration evaluation value: E with respect to the change of the angle: θ, that is, | d
The value of E (D, θ) / dθ | is small.

Thus, in the region where θ is small, θ ± Δθ
The image from the viewpoint within the range of (3) has a small deviation from the front facing direction with respect to the patch surface, and the degree of image quality deterioration is small even when the angle is deviated. Therefore, it is necessary to consider the camera distance and the direction at the same time.

As described above, the three-dimensional image generation target object 50
1, the direction vector of the camera 503 most directly facing the target patch surface P, 502: V, the target patch surface P,
It is necessary to set the optimum value of Δθ according to the value of the angle formed by the normal vector 502 of n: n: θ, and the value of Δθ is determined by the following equation (equation 2).

[0067]

[Equation 14] … (Equation 2)

In the above equation, | dE (D, θ) / dθ |
θ is an angle formed by the direction vector: V of the camera 503 most directly facing the target patch plane P, 502 in the three-dimensional image generation target 501 and the normal vector: n of the target patch plane P, 502: θ Image deterioration evaluation value: θ of E
Is the rate of change with respect to. | DE (D, θ) / dθ | ma
x and | dE (D, θ) / dθ | min are the maximum and minimum values of the change rate of the image quality deterioration evaluation value: E for various θ values, and for example, values obtained from simulation results should be applied. You can In the above equation, Δθmax
Applies the setting value determined according to the setting condition of the camera. For example, as the value of Δθmax, 45 [deg], which is the angle at which the value of E (D, θ) starts to rapidly increase, is adopted.

In the above (formula 2), the value of Δθ is set, and for example, in the photographing environment shown in FIG. 5, the camera 503 which is most directly facing the target patch plane P, 502 in the three-dimensional image generation target 501. And a captured image from a viewpoint within a range of θ ± Δθ, which is a region determined based on the value of Δθ set in the above (Formula 2),
The captured images from these multiple viewpoints are used as the patch surface P of interest,
Let 502 be a candidate image for texture mapping.

Next, the optimum texture mapping image is selected from these candidate images. The selection of the optimum image from the candidate images is determined based on the evaluation value: V calculated by the following (formula 3). The image having the maximum evaluation value V is set as the optimum texture image for the patch surfaces P and 502 of interest.

[0071]

[Equation 15] … (Equation 3)

Applying the above (formula 3), the evaluation value: V for each viewpoint position specified by the camera in the area defined by Δθ obtained by the above (formula 2)
Ask for. For example, in the shooting environment shown in FIG. 5, the patch planes P, 5 of interest in the three-dimensional image generation object 501 are selected.
02, the plurality of cameras 5 within the range of θ ± Δθ, which is the region determined based on the captured image of the camera 503 that is most directly facing 02 and the value of Δθ set in (Equation 2).
The evaluation values of all the captured images of 04 are calculated.

In Equation 3, D is the distance between the center of gravity of the patch surface and the camera that has photographed the image to be evaluated, and Dmax is the maximum distance value of the plurality of D images that are candidate images. Note that FIG.
As described with reference to the image quality deterioration evaluation value data of E, since the result that E (D, θ) linearly increases with the increase of the viewing distance D, D in the above (Formula 3) is obtained. And Dm
As ax, a value normalized between 0 and 1 can be used. In Expression 3, θ of cos θ is an angle θ formed by the direction vector V of the camera that has captured the image to be evaluated and the normal vector n of the patch surface P of interest.

Further, w _D and w _θ in the above (formula 3) are the change rates of the image quality deterioration evaluation value: E with respect to the distance: D, which correspond to the distance: D and the angle: θ in the candidate image and each image. : |
dE / dD | and the image quality deterioration evaluation value: E for the angle: θ
Change rate of | dE / dθ |.

As can be understood from the graph shown in FIG. 4 and the above (formula 3), the larger the angle: θ, the greater the weight of the angle change, and the evaluation value that emphasizes the camera direction is calculated. In this case, the weight of the distance change becomes large, and the evaluation value that emphasizes the distance of the camera is calculated.

For example, as shown in FIG. 5, a photographed image of the camera 503 which directly faces the target patch planes P and 502 of the three-dimensional image generation target 501 and Δθ set in the above (formula 2). When there are images captured by the three cameras 504 in the range of θ ± Δθ, which is a region determined based on the value of, the distance: D, the angle: θ of each camera are calculated based on (Equation 3) above. Based on the four evaluation values V1
To V4 are calculated, and the image of the camera corresponding to Vx that has the maximum evaluation value from these evaluation values V1 to V4 is selected as the optimum texture image corresponding to the patch surface P, 502 of interest.

By using the texture image selection method described above, it is possible to easily select the optimum image for each patch surface in the three-dimensional image generation object from a plurality of photographed images.

As a procedure for creating a patch surface and mapping a texture image, first, associating data between images by tracing the feature points in each frame image of the multi-viewpoint image series, that is, images captured by a plurality of cameras described above. Then, the three-dimensional shape of the object is constructed based on these correspondence data to create the patch surface. Next, mapping is performed on each patch surface using the tracking result of the feature points in each frame as texture coordinates at the time of texture mapping. By these processes, the texture image selected by the above-described texture image selection method can be mapped to each patch surface.

[(2) Correction Processing at Shared Boundary Line of Adjacent Patches] However, the tracking result of the feature points in each frame image of the multi-view image series contains an error, which may affect the accuracy of mapping. . Therefore, in order to eliminate the discontinuity on the ridge line due to the texture coordinate shift, that is, the image shift at the boundary line of each patch surface, pay attention only to the shared ridge line and the pixel value on that ridge line. The process of performing the alignment by performing the matching will be described below.

With reference to FIG. 6, an example of the correction processing when the feature point position shift on the shared boundary line of adjacent patches occurs will be described. As shown in FIG. 6A, the pasted images, that is, the texture images for the adjacent patch surfaces 551 and 552 are different captured images Image _n 561 and Im.
When trying to apply age _n 562, a difference occurs in the grayscale distribution on the boundary line between the adjacent patch surfaces 551 and 552, that is, the shared edge line. For example, as shown in FIG. 6B, captured images Image _n 561 and Image _n 56
A shift occurs in the grayscale distribution of 2. The graph shown in FIG. 6B shows that the luminance value I (n, n of the i-th pixel from the end point on the shared edge of the images Image _n 561 and Image _n 562 has
It is the figure which showed i) and I (m, i).

In order to correct the deviation of the pixel value on the shared edge line, the end point (feature point) position of the adjacent patch is relatively moved so that the error value E of the following equation (Equation 4) is minimized. As the error value E used here, the sum of squares of the errors of the brightness values obtained from the RGB components is used.

[0082]

[Equation 16] … (Equation 4)

Here, I (n, i) and I (m, i) are
Image on each shared edge Image _n, Image_mEdge of
It is the brightness value of the i-th pixel from the point, and pay attention to L.
It is the length of the ridgeline. End point (feature point) position of adjacent patch
Calculated by the above (Equation 4)
Error value: The relative position at which E is minimized is set.

However, when a certain shared edge line is corrected by the above method, the pixel values on other shared edge lines connected to the end point also change. Therefore, when it is executed as a process for the entire three-dimensional image data composed of a large number of patch planes, the local density error is minimized by focusing on a certain shared ridge line, and the overall density associated with the local correction is reduced. It is necessary to minimize changes in various density errors at the same time.

Therefore, in order to minimize the error on all patch boundary lines forming a certain three-dimensional image, the problem is solved as an energy minimization problem. Specifically, attention is paid only to the maximum value of the error values on all the boundary lines, and processing for minimizing this is performed.

The procedure of this correction processing will be described below. FIG. 7 shows a flow for explaining the correction processing procedure by the processing of paying attention to the maximum value of the error values at all the patch boundary lines forming the three-dimensional image and minimizing it.

Each step of the patch surface shared ridge line correction processing shown in FIG. 7 will be described. First, initial setting is executed in step S101. The initial settings are t = 1 and Max {E (x, t-1)} = 0.

“T” is a numerical value indicating the number of times the error value E is calculated in this flow, and the initial value is 1.
“MAX {E (x, t−1)}” is the number of ridge lines between patch surfaces in which different images are selected among all patch boundary lines forming the three-dimensional image to be corrected. x ”indicates the maximum value among the error values: E calculated using the above (Equation 4) for these x ridge lines. (T-1) indicates that it is the (t-1) th calculation process. The initial value of “MAX {E (x, t−1)}” is 0.

In step S102, an error value {E (x, t) on all boundary lines between patch planes for which different images are selected among all patch boundary lines forming the three-dimensional image to be corrected. | X: the number of edges}.

In step S103, step S102
The maximum error value: Max {E (x, t)} is selected from the error value {E (x, t) | x: the number of edges} calculated in step 1, and the maximum error value for the previous (t−1) th time is calculated. In the process, a difference: D _E from the acquired previous maximum error value: Max {E (x, t−1)} is calculated according to the following formula. D _E = Max {E (x, t)}-Max {E (x, t-
1)} is calculated.

In step S104, step S103
It is determined whether or not the difference calculated in step D: D _E is smaller than a predetermined threshold value. If Yes, end the process,
The end point correction process at this point is the final correction result.
D _E <threshold value means that maximum error value: Max {E
It shows that (x, t)} has converged, and that a large change does not occur even if a new end point correction is performed.

If D _E <threshold value is not satisfied in step S104, end point correction is executed in the following steps. First, in step S105, the previous time (t-1)
In the maximum error value calculation process of the second time, the acquired previous maximum error value: Max {E (x, t-1)} is changed to this time (t).
The maximum error value acquired in the maximum error value calculation process for the second time is updated to Max {E (x, t)}, and t = t +
Perform update of 1.

Further, among all the patch boundary lines forming the three-dimensional image to be corrected acquired in step S102 this time, the error value {E on all the boundary lines between the patch surfaces for which different images are selected {E (X, t) | x:
The number of edges} is rearranged (sorted) in descending order of the error value, and the shared edge L _E-MAX having the maximum error is determined.

Next, in step S106, with respect to the shared edge L _E-MAX having the maximum error, the end point is corrected so that the error value E calculated based on the above (Equation 4) is minimized by moving the end point. To execute.

Further, the process returns to step S102, and the process of step S102 and subsequent steps is repeatedly executed on the corrected three-dimensional image data based on the updated t.

By repeating these processes, the process is terminated when the determination in step S104, that is, when D _E <threshold becomes Yes, and the end point correction process at this time is regarded as the final correction result. To do. As described above, D _E <threshold indicates that the maximum error value: Max {E (x, t)} has converged, and that a large change does not occur even if a new end point correction is performed. .

As described above, the process ends when the maximum error value converges. By this end point correction processing, the shift between patches is corrected in texture mapping, and the image quality of three-dimensional image data can be improved.

[(3) Pixel Value Correction Processing on Shared Boundary Line (Shared Edge) and in Patch Surface] However, even if the above-mentioned end point correction is performed, if there is a difference in illumination condition between images of adjacent patches. Causes an overall discontinuity in density values between adjacent patches. In order to eliminate this discontinuity, it is necessary to correct the common boundary line and the density value in the patch plane.
The process of correcting the shared boundary line and the density value in the patch plane will be described below.

First, the pixel value on the shared ridge line is replaced with the average value of the pixel values weighted by the cosine of the angle formed by the camera direction and the normal vector of the patch surface. In FIG. 8, when different images Image _n 611 and Image _m 612 are pasted images for the adjacent patch surfaces 621 and 622, the pixel value P after correction of the point P _i on the shared ridge line
_i-new is expressed by the following equation (P _i ), using pixel values of corresponding points of two different images, that is, a pixel value based on the image Image _n 611: P _i , and a pixel value based on the image Image _m 612: P _i ′. It is calculated by Equation 5).

[0100]

[Equation 17] … (Equation 5)

However, λ = (cos θn) / (cos θn + cos θm) Is.

Θn and θm are angles formed by the camera direction vector when the images Image _n and Image _m are taken and the normal vector g of the patch surface, respectively.

After the correction processing on the shared boundary line of the adjacent patches described above, the pixel value correction processing in the patch plane is executed. The pixel value correction processing within the patch plane is performed by the difference between the original pixel value on the ridge and the corrected pixel value dP _i = P _i-new −P
_i-old is executed as density correction in the patch plane to which the method of propagating in the patch plane with the distance from the edge line as the weight is applied.

On the patch surface set as the triangular area,
As shown in FIG. 9, one patch surface 651 has three adjacent patch surfaces 652, 653, 654. Therefore, it is necessary to determine the pixel value correction of the point without the patch surface 651: Q based on the three adjacent patch surfaces 652, 653, 654.

In FIG. 9, the correction pixel value Qnew at the point Q inside the patch surface is set by the following equation (Equation 6).

[0106]

[Equation 18] … (Equation 6)

However, Q_oldIs a point: pixel before correction of Q
Value, Q_newIs the pixel value after correction of the point: Q, P_i-old, P_i-new: Shared line L_iBefore correction and after correction
Pixel value of P_j-old, P_j-new: Shared line L_jBefore correction and after correction
Pixel value of P_k-old, P_k-new: Shared line L_kBefore correction and after correction
Pixel value of And w_i, W_j, W_k: Point Q in the patch and each ridge line L_i, L_j，
L_kUpper point P_i, P_j, P _kDistance d_i, D_j, D_kContrary to
Example weighting factor I, J, K: Each ridge line L_i, L_j, L_kNumber of pixels above Is.

The pixel value correction according to the above (formula 6) is performed by paying attention to the shared ridge lines of all patch surfaces forming the three-dimensional image data. By obtaining the correction pixel value of each patch surface according to the above (formula 6) and setting the correction pixel value as a new pixel value, it is possible to eliminate the unnatural difference in the pixel value from the adjacent patch surface. Therefore, even when the illumination condition at the time of shooting is unknown, the discontinuity that occurs between patch surfaces can be eliminated, and a natural texture mapping result can be generated.

FIG. 10 shows a flowchart for explaining the procedure of the pixel value correction processing on the shared boundary line and the patch surface described above. Each step will be described.

In step S201, first, a shared ridge line having different viewpoint images is selected for the adjacent patch, and the boundary of the adjacent patch surfaces obtained from the different images by applying the previous formula (Equation 5) to the selected shared ridge line. Line (shared ridge)
Perform the upper pixel value correction. That is, the corrected pixel value P _i-new of the point P _i on the shared ridge line is set to the pixel value of the corresponding point of two different images, that is, the pixel value based on the image Image _n : P _i , and the image Image _m. Pixel value based on:
Equation 5 is applied using P _i ′.

In step S202, it is determined whether the pixel value correction has been completed for all shared ridge lines having different viewpoint images in adjacent patches. If not completed, step S201 is continuously executed.

When all the pixel value corrections on the boundary line (shared ridge line) of the adjacent patch surfaces are completed, then in step S202, the above-mentioned (formula 6) is applied to apply three pixel values to three adjacent patch surfaces. Shared edge _lines : pixel values before and after correction on L _i , L _j , and L _k : P _{i -old} , P _i-new ,
The pixel value Q _new in the patch is calculated based on P _j-old , P _j-new , P _k-old , and P _k-new .

In step S204, it is determined whether the pixel value correction in all patches has been completed. If not completed, step S203 is continuously executed. When the pixel value correction in all patches is completed, the processing ends.

As described above, in the texture mapping process for joining images from different viewpoints in units of patch planes to generate a three-dimensional image, pixel value correction on the boundary line (shared ridgeline) of the patch (Equation 5) is performed. ) And the pixel value correction in the patch based on (Equation 6), it is possible to eliminate the unnatural difference in pixel value between the adjacent patch surface and Even when the illumination condition of is unknown, the discontinuity that occurs between patch surfaces can be eliminated, and a natural texture mapping result can be generated.

(1) Optimal texture mapping image selection processing in the case where the distance between the measurement object and the camera changes and at the same time the orientation of the camera also arbitrarily rotates, (2) correction processing in the shared boundary line of adjacent patches ( 3) The pixel value correction processing on the shared boundary line (shared edge) and in the patch plane has been described. Next, the effects of the above-described processing according to the present invention will be described with reference to actual examples.

First, referring to FIGS. 11 and 12, (1) the effect of the optimum texture mapping image selection processing when the distance between the object to be measured and the camera changes, and at the same time the orientation of the camera also rotates arbitrarily. Explain.

A model for generating three-dimensional image data by texture mapping, that is, 4 as a photographing target.
Vis in a folding screen type three-dimensional shape consisting of three triangular patch surfaces
The reference images in the Ion Texture database were texture-mapped from the front to each surface and used.

Simultaneously changing the camera direction and the position of this three-dimensional object, a simulation image series, 100 sheets in total, are generated in the computer, and the optimum image is selected for each patch surface from the generated simulation image series. , A new texture mapping process was executed.

FIG. 11 shows the relationship between the shooting camera position and direction of the images forming the simulation image series and the normal direction of the patch surface of the shooting target. Cameras 1 to 1 shown in FIG.
The camera 5 shows a part of the shooting camera position and direction of the images forming the simulation image sequence. The simulation image series is camera 1 to camera 5 in FIG.
As shown in, a plurality of image data having different distances: D and angles: θ with respect to the imaging target and different (D, θ) are acquired as a simulation image series.

From these image series, the process of selecting the optimum image for each patch surface is executed. In FIG. 12 (a),
An image obtained as a result of selecting an optimum texture image only by comparing the normal vector direction of the patch surface and the camera direction and executing texture mapping based on the selected texture image is shown.

FIG. 12B shows an image selection process according to the above-mentioned (1) optimal texture mapping image selection process,
That is, an image of ± θ is selected as a texture image candidate image from the image most directly facing the patch surface, and both distance: D and angle: θ are considered for each candidate image based on (Equation 3) described above. The texture mapping image generated by calculating the evaluated value: V and executing the process of selecting the image having the largest evaluated value: V from the candidate images as the texture image is shown in.

In the case of the image shown in FIG. 12A in which the optimum texture image is selected only based on the comparison between the normal vector direction of the patch plane and the camera direction, the camera direction is farthest directly facing the patch plane. It can be seen that the image picked up from is selected, and the mapping result 701 for the patch surface on the right side is blurred toward it. On the other hand, when the image selection by this method is performed, an appropriate image considering the trade-off between the viewing distance and the direction is selected for each surface, and the mapping result 702 with less blurring and distortion is obtained. You can see that it has been obtained.

Next, an example of the effect of each of the above-described processing of (2) correction processing on the shared boundary line of adjacent patches and (3) pixel value correction processing on the shared boundary line (shared ridge line) and in the patch plane will be described. Show and explain.

In order to see the stability of the present method against changes in lighting conditions and tracking errors of feature points, an image series in which the overall lighting conditions are significantly changed is prepared, and the standard deviation is 2 pixels in the tracking result of feature points. An experiment was performed by adding noise based on a considerable Gaussian distribution. Regarding the texture mapping image in this case, (a) uncorrected (before correction), (b) only density correction, (c) density correction is performed after correction of texture coordinates, and the texture mapping image as the three different types of processing results Each is shown in FIGS. 13 (a), 13 (b) and 13 (c).

FIG. 14 shows the distribution of brightness values on the arrow line 703 in FIG. 14 is the same as FIG.
(A) uncorrected (before correction), (b) only density correction,
(C) Density correction is executed after correction of texture coordinates
The pixel value of the pixel corresponding to the line 703 of the image is shown. The horizontal axis represents the pixels (300 pixels) of each image corresponding to the line 703, and the vertical axis represents the pixel value (luminance value) normalized data of each pixel.

The line in FIG. 14C shows a pixel value with less blur compared to the other lines, and the present method, that is, (2) the correction processing in the shared boundary line of the adjacent patches, and (3) ) It is possible to confirm that the density can be corrected so that the texture is connected more smoothly by executing each of the pixel value correction processing on the shared boundary line (shared edge) and in the patch surface as compared with the case of only the density correction. .

The above-described image processing of the present invention is not limited to the 3D shape by the factorization method described in the embodiment, and other various methods, for example, the 3D shape measuring method by irradiating laser light and triangulation. Even for a 3D shape measured by applying the above, the optimum texture mapping becomes possible by using the multi-viewpoint image observed together with the 3D shape measurement.

[System Configuration] A system configuration example for executing the above-mentioned image processing will be described. FIG. 15 is a block diagram showing the configuration of an embodiment of the image processing apparatus of the present invention.
The image processing apparatus according to this embodiment processes images captured by a camera or multi-viewpoint image data stored in a recording medium such as a DVD or a CD, or is distributed via a data communication network such as the Internet. The viewpoint image data is stored in a writable storage medium such as a DVD, a CD, a hard disk, and the processing of these image data is executed.

The structure of the image processing apparatus 750 shown in FIG. 15 will be described. CPU (Central processing Unit) 7
56 is various application programs and OS (Op
(1) Optimal texture mapping image selection processing, (2) correction processing on a shared boundary line between adjacent patches, and (3) shared boundary line (shared). Pixel value correction processing on the ridge line and in the patch plane is executed. Further, control of data read processing from the storage means or display processing is executed according to the command input from each input device.

The memory 757 is a ROM (Read-Only-Memo).
ry), a RAM (Random Access Memory), and the like, and a storage area for storing a program executed by the CPU 756 or fixed data as a calculation parameter, a program executed in the processing of the CPU 756, and a parameter appropriately changed in the program processing. , Used as a work area. The recording medium 758 is DV
The recording medium is a D, a hard disk, a CD, or the like, and stores multi-view image data to be processed, for example.

Further, the image processing apparatus 750 has a network interface 752 which functions as an interface with a communication network, receives multi-viewpoint image data via the network, and the received data is a DVD, a hard disk or the like. It is stored in the recording medium 758.

The data processing command from the user or the processing request command relating to the display image data on the display 732 is executed by the mouse 737 and the keyboard 73.
6, input from various input devices of the controller 738 via the input interface 753. In addition, from the AV data input device such as the video camera 733 and the microphone 734,
The data input via the V interface 754 is
Based on the processing request command, each processing described above is executed and stored in the recording medium 758 such as a DVD or a hard disk, or displayed on the display 732.

The image data stored in the recording medium 758 such as a DVD or a hard disk is stored in the frame memory 761 on a frame-by-frame basis, and then the D / A converter 762.
After the conversion processing via the display, it is displayed on the display 732. On the other hand, the audio data is also stored in the audio buffer 763, converted through the D / A converter 764, and then reproduced and output by the speaker 735.

Next, in the system shown in FIG. 15, there is shown a block diagram for explaining a functional structure for executing the above-mentioned texture mapping processing mainly executed by the CPU 756.

As described above, the image processing apparatus of the present invention comprises (1) optimum texture mapping image selection processing,
(2) Correction processing on the shared boundary line of adjacent patches,
(3) Pixel value correction processing on the shared boundary line (shared edge) and in the patch plane is executed. The multi-viewpoint image that is the target for executing each of these processes is the input / output interface 801.
Is stored in the memory 802 via, and the three-dimensional shape data generation unit 803 generates three-dimensional shape data composed of a plurality of patch surfaces.

The texture mapping processing for the three-dimensional shape data is executed by the texture mapping processing unit 810. Texture mapping processing unit 810
Has a texture image selection processing unit 811, a matching processing unit 812, and a pixel value correction processing unit 813.

The texture image selection processing unit 811 selects the above-mentioned (1) optimum texture mapping image selection processing, that is, selects an image of ± θ from the image most directly facing the patch surface as a texture image candidate image, and selects each candidate image. For, based on (Equation 3) above, distance: D, angle:
An evaluation value: V in which both θ are taken into consideration is calculated, and an image having the largest evaluation value: V is selected from the candidate images as a texture image.

The matching processing unit 812 receives the processing result of the texture image selection processing unit 811, extracts two groups of patches which connect images from different viewpoints based on the received image data, and extracts these patches. The correction processing (matching processing) of the boundary line (shared edge line), that is, the error minimization processing by the end point movement described above with reference to the flow of FIG. 7 and (Equation 4) is executed.

The pixel value correction processing unit 813 receives the image after the matching processing by the end point movement in the matching processing unit 812, and regarding the received image, the boundary line (shared ridge line) of the patch based on the above-mentioned (Equation 5). Pixel value correction is executed, and further, pixel value correction in the patch is executed based on (Equation 6) described above.

Each of these processes, that is, (1) optimum texture mapping image selection process, (2) correction process on the shared boundary line of adjacent patches, (3) pixels on the shared boundary line (shared edge line) and in the patch plane The image for which the value correction processing has been executed is output and displayed on the image display device 822 via the input / output interface 821, or the storage device 823.
Stored in.

The present invention has been described in detail above with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the scope of the present invention. That is, the present invention has been disclosed in the form of exemplification, and should not be limitedly interpreted. In order to determine the gist of the present invention, the section of the claims described at the beginning should be taken into consideration.

The series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When executing the processing by software, the program recording the processing sequence is installed in the memory in the computer incorporated in the dedicated hardware and executed, or the program is stored in a general-purpose computer capable of executing various processing. It can be installed and run.

For example, the program can be recorded in advance in a hard disk or a ROM (Read Only Memory) as a recording medium. Alternatively, the program may be a flexible disk or a CD-ROM (Compact Disc Read Only).
Memory), MO (Magneto optical) disk, DVD (Dig
Ital Versatile Disc), magnetic disk, semiconductor memory, or other removable recording medium can be temporarily (or permanently) stored (recorded). Such a removable recording medium can be provided as so-called package software.

The program is installed in the computer from the removable recording medium as described above, is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as LAN (Local Area Network) or the Internet. Then, the computer can receive the program thus transferred and install it in a recording medium such as a built-in hard disk.

The various processes described in the specification are
The processing may be executed not only in time series according to the description, but also in parallel or individually according to the processing capability of the device that executes the processing or the need. Further, the system in the present specification is a logical set configuration of a plurality of devices,
The devices of the respective configurations are not limited to being in the same housing.

[0146]

As described above, according to the image processing apparatus and method of the present invention, from the captured images of the three-dimensional image generation target object from a plurality of viewpoints, for each patch surface constituting the three-dimensional shape model, The configuration is such that the optimum texture image selection processing is executed based on the image quality evaluation value to which the distance data between the patch surface and each viewpoint and the direction data for the patch surface of each viewpoint are applied. It is possible to easily select an optimum image with a low degree, and it is possible to generate high-quality three-dimensional image data.

Further, according to the image processing apparatus and method of the present invention, the pixel value error data between the texture images at the patch boundary portion having the texture images from different viewpoints as the adjacent texture images in the three-dimensional image constituent data is obtained. A selection process of selecting the patch boundary portion having the maximum pixel value error data among the calculated pixel value error data,
And a matching process is performed by moving the end points of the texture image corresponding to the selected patch boundary portion, and the matching process is repeatedly executed until the pixel value error data becomes equal to or less than a predetermined threshold value. Since the configuration is adopted, even at a patch surface boundary where images from different viewpoints are pasted as texture images, image displacement does not occur, and high-quality three-dimensional image data generation is realized.

Furthermore, according to the image processing apparatus and method of the present invention, in the adjacent texture images at the patch boundary portion, the pixel values of the texture image in the front facing direction with respect to the patch surface are heavily weighted, The pixel value at the patch boundary portion is calculated based on each pixel value of the adjacent texture image, and the pixel value of the point: Q in the patch plane is set to the point: Q with respect to the pixel value of the patch boundary line. Distance: dn, a weighting coefficient inversely proportional to dn is applied to perform pixel value correction processing for performing calculation based on the pixel value of the patch boundary line. Therefore, shooting is performed under different shooting conditions. Even in a configuration in which images from different viewpoints are pasted as texture images of adjacent patch surfaces, the images between the patch surfaces are smoothly corrected, and high image quality without discomfort Generating three-dimensional image data is realized.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a configuration example of a multi-view image capturing system.

FIG. 2 is a diagram illustrating a correspondence between a shooting viewpoint and a patch surface in a multi-view image shooting system.

FIG. 3 is a diagram illustrating correspondence between a shooting viewpoint and a patch surface and texture image selection in the multi-view image shooting system.

FIG. 4 is a viewpoint distance and angle, and an image quality deterioration evaluation value: E
It is a figure which shows correspondence with.

FIG. 5 is a diagram illustrating a texture image selection process in the configuration of the present invention.

FIG. 6 is a diagram illustrating a matching process in the configuration of the present invention.

FIG. 7 is a flowchart illustrating a matching process in the configuration of the present invention.

FIG. 8 is a diagram illustrating a pixel value correction process at a patch surface boundary portion in the configuration of the present invention.

FIG. 9 is a diagram illustrating a pixel value correction process inside a patch surface in the configuration of the present invention.

FIG. 10 is a flowchart illustrating a pixel value correction process in the configuration of the present invention.

FIG. 11 is a diagram illustrating a configuration example of a multi-view image sequence in the configuration of the present invention.

FIG. 12 is a diagram illustrating an effect when the configuration of the present invention is applied.

FIG. 13 is a diagram illustrating an effect when the configuration of the present invention is applied.

FIG. 14 is a diagram illustrating an effect when the configuration of the present invention is applied.

FIG. 15 is a block diagram showing a configuration example of an image processing apparatus of the present invention.

FIG. 16 is a functional block diagram showing a functional configuration of the image processing apparatus of the present invention.

[Explanation of symbols]

101 Target 102 rotating table 103 camera 201,202 camera 203 Notable patch surface 204 center of gravity 301,302 camera 303 Featured patch surface 304 center of gravity 501 object 502 patch surface 503, 504 camera 551,552 patch surface 561,562 Photograph image image 611,612 image 621,622 patch surface 651-654 patch surface 732 display 733 video camera 734 microphone 735 speaker 736 keyboard 737 mouse 738 controller 750 image processing device 751 codec 752 network interface 753 input interface 754 AV interface 756 CPU 757 memory 758 recording media 761 frame memory 762 D / A converter 763 audio buffer 764 D / A converter 801 I / O interface 802 memory 803 Three-dimensional shape data generation unit 810 Texture Mapping Processing Unit 811 Texture image selection processing unit 812 Matching processing unit 813 Pixel value correction processing unit 821 I / O interface 822 display device 823 storage device

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Hiroyuki Sato             27-5 Nishikoyama, Tsukidate-cho, Kurihara-gun, Miyagi Prefecture F term (reference) 5B057 BA02 BA26 CA01 CA08 CA12                       CA17 CB01 CB08 CB13 CB16                       CC01 CE08 CE11 CE16 CH08                       CH11 CH12 CH18 DA07 DA16                       DB02 DB06 DB09 DC02 DC08                       DC09                 5B080 CA01 FA02 GA22                 5C061 AA06 AA21 AB04 AB08 AB17                 5L096 AA09 FA41 FA66 FA67

Claims (21)

[Claims] 1. An image processing apparatus for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, comprising a patch forming a three-dimensional shape model from a plurality of viewpoint photographed images of a three-dimensional image generation object. A texture image selection processing unit that performs selection processing of an optimum texture image based on an image quality evaluation value to which the patch surface and distance data between viewpoints and direction data with respect to the patch surface of each viewpoint are applied for each surface An image processing device characterized by: 2. The texture image selection processing means 1 which is most directly opposed to a patch surface forming a three-dimensional shape model.
One viewpoint image is selected, and an angle between a direction vector: V from the viewpoint of the selected image toward the center of gravity of the patch surface and a normal vector: n of the patch surface is θ, and θ ± is centered on the center of gravity. An image captured from a viewpoint within the region of Δθ is set as a candidate image of a texture image for the patch surface, and a process of selecting an optimal texture image based on the image quality evaluation value is executed from the candidate images. The image processing apparatus according to claim 1, which is characterized in that. 3. The evaluation value has a distance D between the patch surface and the viewpoint, and an angle formed by a direction vector V from the viewpoint to the center of gravity of the patch surface and a normal vector n of the patch surface: The image processing apparatus according to claim 1, wherein the value is a value calculated by an arithmetic expression to which θ is applied. 4. The calculation formula is a calculation formula of the following evaluation value: V, and Where D is the distance between the center of gravity of the patch surface and the camera that has captured the image to be evaluated, Dmax is the maximum distance value of the multiple Ds that are candidate images, and θ is the direction vector of the camera that has captured the image to be evaluated. : V and the normal vector of the patch surface P of interest:
The angle made by n: θ, w _D is the change rate of the image quality deterioration evaluation value: E with respect to the distance: D: | dE / dD |, w _θ is the angle: θ
Image degradation evaluation value for: Change rate of E: | dE / dθ |
The image processing apparatus according to claim 3, wherein: 5. An image processing device for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, wherein a pixel value matching process at a patch boundary part where images from different viewpoints are connected as a texture image is performed. The matching processing unit has a matching processing unit to execute, and the matching processing unit calculates pixel value error data between texture images at a patch boundary having texture images from different viewpoints as adjacent texture images in the three-dimensional image constituent data. The pixel value error data, the selection process of selecting the patch boundary having the maximum pixel value error data, and the matching process by moving the end points of the texture image corresponding to the selected patch boundary, Until the data becomes below a predetermined threshold, the selection process The image processing apparatus characterized in that is configured to execute repeatedly the fine matching process. 6. An image processing apparatus for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, wherein pixel value correction processing is executed at a patch boundary portion where images from different viewpoints are stitched together as a texture image. Pixel value correction processing means is provided, and the pixel value correction processing means performs a large weighting on the pixel value of the texture image in the facing front direction with respect to the patch surface in the adjacent texture image at the patch boundary portion. The image processing apparatus is configured to calculate a pixel value at the patch boundary portion based on each pixel value of the adjacent texture image. 7. The pixel value correction processing means comprises a corrected pixel value: P _i-new calculation formula, However, λ = (cos θn) / (cos θn + cos θm), and θn and θm are images Image _n and Im, respectively.
The pixel value at the patch boundary portion is calculated according to an angle formed by a direction vector V of the camera capturing the image _m and a normal vector n of the patch surface P of interest. Item 6. The image processing device according to item 6. 8. The pixel value correction processing means is further configured to execute pixel value correction processing within a patch plane having a patch boundary obtained by joining images from different viewpoints as a texture image. A process of calculating the pixel value of the point: Q based on the pixel value of the patch boundary line by applying a weighting coefficient inversely proportional to the distance from the point: Q: dn to the pixel value of the patch boundary line The image processing apparatus according to claim 6, wherein the image processing apparatus is configured to execute. 9. The pixel value correction processing means is further configured to execute pixel value correction processing within a patch plane having a patch boundary in which images from different viewpoints are connected as a texture image. The processing means calculates the pixel value of a point: Q in a patch plane formed by a triangular region surrounded by three shared ridge lines: L _i , L _j , and L _k : a corrected pixel value: Q _new calculation formula, Number 3] However, Q _old is the pixel value before correction of point: Q, Q _new is the pixel value after correction of point: Q, P _i-old , P _i-new : Before correction on shared line L _i and the pixel value P _j-old after correction, P _j-new: shared lines L _j on the uncorrected and the corrected pixel value _{P k-old, P k-} new: uncorrected and corrected on a shared line L _k The pixel values after that, w _i , w _j , w _k : point Q in the patch and each edge L _i , L _j ,
L _k on points P _i, P _j, the distance d _i between the P _k, d _j, the weighting factor is inversely proportional to d _k I, J, _K: the ridge line L _i, L _j, the number of pixels on the L _k The image processing apparatus according to claim 6 or 7, wherein a pixel value of a point: Q on the patch surface is calculated according to the following. 10. An image processing apparatus for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, comprising a patch forming a three-dimensional shape model from a plurality of viewpoint photographed images of a three-dimensional image generation object. Texture image selection processing means for executing optimum texture image selection processing based on the image quality evaluation value obtained by applying, for each surface, distance data between the patch surface and each viewpoint, and direction data with respect to the patch surface of each viewpoint; Among the constituent data of the original image, the pixel value error data between the texture images at the patch boundary having the texture images from different viewpoints as the adjacent texture images is calculated, and the maximum pixel value error data is calculated from the calculated pixel value error data. Selection processing to select the patch boundary part that has, and the texture image corresponding to the selected patch boundary part Matching processing means that executes matching processing by point movement and repeatedly executes the selection processing and matching processing until the pixel value error data becomes equal to or less than a predetermined threshold value; On the other hand, the pixel value of the texture image in the facing viewpoint direction is heavily weighted, and the pixel value in the patch boundary portion is calculated based on each pixel value of the adjacent texture image, and A process of calculating the pixel value of the point: Q based on the pixel value of the patch boundary line by applying a weighting coefficient inversely proportional to the distance from the point: Q: dn to the pixel value of the patch boundary line An image processing apparatus comprising: a pixel value correction processing unit that executes the image data. 11. An image processing method for generating a three-dimensional image by attaching a texture image to the three-dimensional shape model, comprising a patch forming a three-dimensional shape model from a plurality of viewpoint photographed images of a three-dimensional image generation object. For each surface, a texture image selection processing step is executed for executing the selection processing of the optimum texture image based on the image quality evaluation value to which the distance data between the patch surface and each viewpoint and the direction data with respect to the patch surface of each viewpoint are applied. An image processing method characterized by: 12. The texture image selection processing step is the step 1 which is most directly opposed to a patch surface forming a three-dimensional shape model.
One viewpoint image is selected, and an angle between a direction vector: V from the viewpoint of the selected image toward the center of gravity of the patch surface and a normal vector: n of the patch surface is θ, and θ ± is centered on the center of gravity. A captured image from a viewpoint within the region of Δθ is set as a candidate image of a texture image for the patch surface, and a step of performing a selection process of an optimal texture image based on the image quality evaluation value from the candidate image is included. The image processing method according to claim 11, which is characterized in that. 13. The evaluation value, wherein the distance between the patch surface and the viewpoint is D, and the angle between the direction vector from the viewpoint to the center of gravity of the patch surface: V and the normal vector of the patch surface: n: The value is a value calculated by an arithmetic expression to which θ is applied.
The image processing method described in. 14. The calculation formula is a calculation formula of the following evaluation value: V, and However, D is the distance between the center of gravity of the patch surface and the camera that has captured the image to be evaluated, and Dmax is a plurality of Ds that are candidate images.
Is the maximum distance value of θ, θ is the direction vector of the camera that has captured the image to be evaluated: V, and the normal vector n of the patch surface P of interest is the angle: θ, w _D is the image quality for distance: D Degradation evaluation value: change rate of E: | dE / dD |, w _θ is
Change rate of image quality deterioration evaluation value: E with respect to angle: θ: | dE
The image processing method according to claim 13, wherein / dθ | 15. An image processing method for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, which comprises matching processing of pixel values in a patch boundary part where images from different viewpoints are connected as a texture image. A matching process step to be executed, wherein the matching process step calculates pixel value error data between texture images at a patch boundary portion having texture images from different viewpoints as adjacent texture images in the three-dimensional image constituent data. The pixel value error data, the selection process of selecting the patch boundary having the maximum pixel value error data, and the matching process by moving the end points of the texture image corresponding to the selected patch boundary, Until the data falls below a predetermined threshold, Image processing method characterized by comprising the step of repeatedly executing the-option processing and matching processing. 16. An image processing method for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, wherein pixel value correction processing is executed at a patch boundary portion where images from different viewpoints are stitched together as a texture image. Pixel value correction processing step is performed, and in the pixel value correction processing step, in the adjacent texture image at the patch boundary portion, a large weight is given to the pixel value of the texture image in the facing front direction with respect to the patch surface. An image processing method comprising the step of calculating a pixel value at the patch boundary portion based on each pixel value of the adjacent texture image. 17. The pixel value correction processing step comprises the following correction pixel value: P _{i -new} calculation formula, However, λ = (cos θn) / (cos θn + cos θm), and θn and θm are images Image _n and Im, respectively.
The pixel value at the patch boundary portion is calculated in accordance with an angle formed by the direction vector of the camera capturing the image _m : V and the normal vector of the patch surface P of interest: n. The described image processing method. 18. The pixel value correction processing step further includes a step of executing pixel value correction processing within a patch plane having a patch boundary in which images from different viewpoints are joined as a texture image. A process of calculating the pixel value of the point: Q based on the pixel value of the patch boundary line by applying a weighting coefficient inversely proportional to the distance from the point: Q: dn to the pixel value of the patch boundary line The image processing method according to claim 16, which is executed. 19. The pixel value correction processing step further includes a step of performing pixel value correction processing within a patch plane having a patch boundary in which images from different viewpoints are connected as a texture image, and three shared ridge lines are provided. : Point in the patch plane consisting of a triangular area surrounded by L _i , L _j , and L _k : pixel value of Q, corrected pixel value shown below: Q _new calculation formula, However, Q _old is the pixel value before correction of point: Q, Q _new is the pixel value after correction of point: Q, P _i-old , P _i-new : Before correction on shared line L _i and the pixel value P _j-old after correction, P _j-new: shared lines L _j on the uncorrected and the corrected pixel value _{P k-old, P k-} new: uncorrected and corrected on a shared line L _k The pixel values after that, w _i , w _j , w _k : point Q in the patch and each edge L _i , L _j ,
L _k on points P _i, P _j, the distance d _i between the P _k, d _j, the weighting factor is inversely proportional to d _k I, J, _K: the ridge line L _i, L _j, the number of pixels on the L _k The image processing method according to claim 16 or 17, wherein a pixel value of a point: Q in the patch plane is calculated according to the following. 20. An image processing method for generating a three-dimensional image by pasting a texture image on the three-dimensional shape model, comprising a patch forming a three-dimensional shape model from a plurality of viewpoint photographed images of a three-dimensional image generation object. A texture image selection processing step for executing the selection processing of the optimum texture image based on the image quality evaluation value to which the distance data between the patch surface and each viewpoint and the direction data with respect to the patch surface of each viewpoint are applied for each surface; Among the constituent data of the original image, the pixel value error data between the texture images at the patch boundary having the texture images from different viewpoints as the adjacent texture images is calculated, and the maximum pixel value error data is calculated from the calculated pixel value error data. Selection process to select the patch boundary part that has, and the texture image corresponding to the selected patch boundary part A matching process step of repeatedly performing the selecting process and the matching process until the pixel value error data becomes equal to or less than a predetermined threshold value by performing the matching process by moving the end points of the patch; A large weight is given to the pixel value of the texture image in the facing viewpoint direction with respect to the surface, and the pixel value at the patch boundary portion is calculated based on each pixel value of the adjacent texture image. Point: Q, the pixel value of the patch boundary line is calculated based on the pixel value of the patch boundary line by applying a weighting factor inversely proportional to the distance: dn from the point: Q to the pixel value of the patch boundary line. An image processing method, comprising: a pixel value correction processing step for executing. 21. A computer program for executing image processing for generating a three-dimensional image by pasting a texture image on a three-dimensional shape model, comprising: A texture for executing the selection process of the optimum texture image based on the image quality evaluation value to which the distance data between the patch surface and each viewpoint and the direction data with respect to the patch surface of each viewpoint are applied for each patch surface forming the original shape model. Image selection processing step, in the configuration data of the three-dimensional image, the pixel value error data between the texture images in the patch boundary portion having the texture images from different viewpoints as adjacent texture images is calculated, and in the calculated pixel value error data, The selection process for selecting the patch boundary having the maximum pixel value error data, and the selection A matching process step of executing a matching process by moving the end points of the texture image corresponding to the boundary part, and repeatedly executing the selection process and the matching process until the pixel value error data becomes equal to or less than a predetermined threshold value; In the adjacent texture image in, for the patch surface, a large weight is given to the pixel value of the texture image in the facing front direction, based on each pixel value of the adjacent texture image, the pixel value in the patch boundary portion. The pixel value of the patch boundary line is calculated by applying a weighting factor inversely proportional to the pixel value of the point: Q in the patch plane to the pixel value of the patch boundary line and the distance from the point: Q: dn. A pixel value correction processing step for executing a calculation based on the value; Beam.