WO2006061999A1 - Image conversion method, device, and program, texture mapping method, device, and program, and server-client system - Google Patents

Abstract

A plurality of parameters constituting a predetermined illumination equation giving a luminance to each of pixels in an image are acquired (S10). For each of the parameters, a homogeneous area formed by pixels having similar values of the parameter is identified (S20). For each parameter, according to the content of image conversion, the parameter conversion processing is performed for each of the identified homogenous areas (S30). By using each of the parameters after the conversion, luminance of each of pixels in a second image is obtained (S40).

Description

 Specification

 Image conversion method, apparatus and program, texture mapping method, apparatus and program, and server client system

 Technical field

 The present invention relates to an image processing technique, and more particularly to a technique for realizing image conversion such as enlargement or reduction, image compression, and texture mapping.

 Background art

 Arbitrary image devices can be connected by the digital network of image devices and networks, and the degree of freedom of image exchange is increasing. In addition, an environment has been established in which users can freely handle images without being restricted by differences in systems. For example, users can output images taken with a digital still camera to a printer, publish them on a network, or view them on a home TV.

 [0003] Conversely, the system side needs to support various image formats, and image format conversion is naturally required to be sophisticated. For example, image size conversion frequently occurs, and an up converter (a conversion device that increases the number of images and lines) and a down converter (a device that reduces the number of pixels and lines) are required. For example, when printing on A4 paper (297 mm x 210 mm) at a resolution of 600 dpi, a document that requires 7128 pixels x 5040 lines is required. The resolution of many digital still cameras is below this, so an upconverter is required. Become. On the other hand, since the final output form of the image published on the network is not decided, it is necessary to convert it to the corresponding image size every time the output device is decided. Also, digital terrestrial services have been started for home televisions, and conventional standard television images and HD (High Definition) television images are mixed, and image size conversion is frequently used.

[0004] The importance of scalability in image compression increases as the image sizes vary. Scalability refers to the ability to extract standard TV image data in some cases and HD TV image data in other cases from a single bit 'stream, and the degree of freedom to extract various image sizes. When scalability is secured, transmission is performed for each image format. Less transmission capacity is needed without preparing a route.

 [0005] Image conversion such as image enlargement and image reduction is also frequently used in texture mapping in computer graphics (patterns and patterns appearing on the surface of a subject are collectively referred to as “texture”). Texture mapping is a technique that expresses the pattern and texture of an object surface by attaching a 2D image to the surface of the 3D object formed in the computer. In order to paste a 2D image in accordance with the direction of the surface of the 3D object, it is necessary to perform processing such as enlargement, reduction, deformation, and rotation on the 2D image (see Non-Patent Document 1).

 [0006] Conventionally, processes such as image enlargement, image reduction, and image compression utilize differences in luminance values among a plurality of pixels.

 In other words, in image enlargement, in order to newly generate image data that does not exist at the time of sampling, luminance values are interpolated by a bilinear method, a bicubic method, or the like (see Non-Patent Document 1). . Since interpolation can generate only intermediate values of sampling data, sharpness such as edges tends to deteriorate. Therefore, a technique is disclosed in which an interpolated image is used as an initial enlarged image, and thereafter, an edge portion is extracted to emphasize only the edge (see Non-Patent Document 2 and Non-Patent Document 3). In particular, Non-Patent Document 3 devised a technique for selectively performing edge enhancement according to the sharpness of an edge by introducing a multi-resolution expression and a Lipschitz index.

 [0008] In image reduction, a part of the pixels is deleted. However, if pixels located at a distant position before image reduction are adjacent to each other, continuity is disturbed and moire occurs. Therefore, a general method is to apply a low-pass filter before deleting a part of the pixel to smooth the luminance change, and then delete a part of the pixel.

 [0009] Further, in image compression, the property that the correlation of luminance values between adjacent pixels is high is used. In order to express the correlation of luminance values, the spatial frequency component is decomposed into orthogonal components. Discrete cosine transform is generally used for orthogonal transformation, and energy concentrates on the low frequency term from the high correlation of luminance values between adjacent pixels, so the high frequency term is deleted and image information is compressed (Non-patent Document) (See 4).

 Patent Document 1: JP 2005-149390 A

Non-Patent Document 1: Shinya Araya, “Clear 3D Computer Graphics”, Kyoritsu Shuppan, pp 144—145, 25 September 2003,

 Patent Document 2: H. Greenspan, CH Anderson, “Image enhanced by non-linear extrapolation in frequect space”, SPIE Vol. 2182 Image and Video Processing II, 1994 Non-Patent Document 3: Nakashige et al., “Multi-scale Image Resolution on Luminance Gradient Planes, ”IEICE Transactions D—Π Vol. J81 -D-II No. 10 pp. 2249— 2258 Oct 1998

 Non-Patent Document 4: Multimedia Communication Study Group, “Point Illustrated Broadband + Mono Standard MPEG Textbook”, ASCII, pp. 25-29, February 11, 2003

 Non-Patent Document 5: Image Processing Handbook Editorial Committee, “Image Processing Handbook”, Shogodo, pp. 393, June 1987 80

 Non-Patent Document 6: Shinji Umeyama, “Separation of Diffuse Z Specular Reflection Components from Object Appearance Using Multiple Observations and Stochastic Independence through Polarization Filters”, Image Recognition 'Understanding Symposium 2002, pp 1-469-pp. 1-476, 2002

 Disclosure of the invention

 Problems to be solved by the invention

[0010] However, the conventional technique has the following problems.

 [0011] As described above, when image conversion such as image enlargement, image reduction, and image compression is performed using a difference in luminance between pixels, image quality is improved by image conversion in which separation of edge components and noise is difficult. There is a problem that the possibility of deterioration is high.

 That is, in the image enlargement, since the edge portion of the initial enlarged image blurred by the interpolation is emphasized, noise is also enhanced together with the edge portion, which may cause image deterioration. In addition, in image compression, noise reduces the correlation between adjacent pixels and causes compression efficiency to decrease. Furthermore, it was said that edge enhancement of interpolated images during image enlargement and smoothing during image reduction are empirical methods, and no clear noise countermeasures are taken, so image quality after image conversion cannot be guaranteed. There are also problems.

In view of the above problems, an object of the present invention is to make the image quality more stable in image conversion, image compression, and texture mapping by making it less susceptible to noise than in the past. Means for solving the problem

 [0014] According to the present invention, as an image conversion method, for each pixel of the first image, a plurality of parameters constituting a predetermined illumination equation that gives luminance are acquired, and the value of the parameter is determined for each parameter. Homogeneous regions with similar pixel power are specified, and for each parameter, the parameter conversion processing is performed for each specified homogeneous region according to the content of the image conversion, and each parameter after the conversion processing is used. The brightness of each pixel of the second image is obtained.

 [0015] According to the present invention, a plurality of parameters constituting an illumination equation that gives luminance are respectively acquired for the first image to be subjected to image conversion. The parameters referred to here are, for example, the optical characteristics of the subject, environmental conditions, the surface normal of the subject, and the like. Then, a homogeneous region is specified for each meter, and the parameter conversion process is performed for each specified homogeneous region according to the content of the image conversion. The luminance of each pixel of the second image after image conversion is obtained using each parameter after conversion processing.

 That is, the luminance is decomposed into illumination equation parameters, and image conversion is performed using the correlation between pixels for each parameter. The illumination equation parameters are highly independent, such as surface normals and optical properties. For this reason, when processing is performed for each parameter, it is easier to grasp the peculiarities of noise than when processing is performed using the luminance given as the integral value of the norameter. Furthermore, the optical characteristics can be decomposed into diffuse reflection components and specular reflection components, which are highly independent factors, so that the peculiarities of noise can be emphasized. In addition, the homogeneous region is specified based on the similarity of the illumination equation parameters, which are physical characteristics of the subject, so that it is defined with physical support. Since image conversion is performed for each homogeneous region for each homogeneous region, the edge portion is stored as a boundary condition between the homogeneous regions. Therefore, it is possible to realize image conversion with stable image quality while preserving the sharpness of edges and texture. Also, the problem of noise mixing that does not require the direct detection of edges as in the prior art does not occur.

[0017] In the image conversion method of the present invention, when image enlargement is performed, a process for increasing the density of the parameters may be performed as the conversion process for each parameter. As described above, in the present invention, the homogeneous region is defined with physical support. this Therefore, compared to the conventional empirical technique for edge enhancement of the initial enlarged image obtained by interpolation, the present invention for increasing the density of parameters for each homogeneous region is an objective one. The image quality can be further stabilized.

 [0018] Further, when performing image reduction in the image conversion method of the present invention, as a conversion process of each parameter, a process for reducing the density of the parameter may be performed. As in the case of image enlargement, the present invention for reducing the parameters for each homogeneous region is objective compared to the empirical method using a low-pass filter as in the prior art. The image quality can be made more stable.

 [0019] Further, according to the present invention, as an image compression method, for each pixel of an image, a plurality of parameters constituting a predetermined illumination equation that gives luminance are respectively acquired, and the value of the parameter is similar for each parameter. A homogenous region having a pixel power is specified, and for each parameter, compression coding of the parameter is performed for each specified homogenous region.

 [0020] According to the present invention, a plurality of parameters constituting an illumination equation for giving brightness are acquired for each image to be compressed. Then, a homogeneous region is specified for each parameter, and the parameter is compressed and encoded for each specified homogeneous region. In the homogeneous area, the correlation between neighboring pixels is high with respect to the illumination equation parameter, so that the compression efficiency can be improved over the image compression based on the luminance value. The edge part is saved as a boundary condition between homogeneous regions. Therefore, it is possible to realize image compression with high compression efficiency while preserving the sharpness of the edges and the texture.

 [0021] Further, the present invention provides a texture mapping method in which a preprocessing for pasting a texture image to an object of a three-dimensional CG model is performed, and brightness is given to each pixel of the texture image pasted to the object. A plurality of parameters constituting the lighting equation are obtained, a homogeneous region consisting of pixels with similar parameter values is identified for each parameter, and a predetermined image conversion is performed for each parameter. Depending on the content, the parameter conversion processing is performed for each identified homogeneous region, and the luminance of each pixel of the object image is obtained using each parameter after the conversion processing.

[0022] According to the present invention, as in the image conversion method described above, texture mapping with stable image quality can be realized while preserving edge sharpness and texture. The invention's effect

 [0023] According to the present invention, since the conversion process is performed for each homogeneous region for each illumination equation parameter constituting the luminance value, image conversion and texture with stable image quality while preserving edge sharpness and texture. Mapping can be realized. In addition, it is possible to achieve high compression efficiency and image compression while preserving the sharpness of edges and texture.

 Brief Description of Drawings

 FIG. 1 is a flowchart showing an image conversion method according to the first embodiment of the present invention.

 FIG. 2 is a schematic diagram showing the relationship between luminance and illumination equation parameters.

 [FIG. 3] FIG. 3 is a conceptual diagram showing a geometric condition which is a premise of an illumination equation.

 FIG. 4 is a diagram for explaining an example of a surface normal vector measurement method.

 FIG. 5 is a diagram for explaining an example of a technique for separating diffuse reflection and specular reflection.

[FIG. 6] FIG. 6 is a diagram for explaining a method of acquiring illumination equation parameters with reference to learning data.

 FIG. 7 is a diagram showing a pattern for determining a homogeneous region.

 FIG. 8 is a diagram showing an example of a unit area scanning method.

 FIG. 9 is a diagram showing an example of noise removal.

 [FIG. 10] FIG. 10 is a diagram showing processing for increasing the density of parameters for image enlargement.

 [FIG. 11] FIG. 11 is a diagram showing processing for reducing parameters for image reduction.

 FIG. 12 is a conceptual diagram showing parameter conversion processing for image compression in the second embodiment of the present invention.

 FIG. 13 is a diagram for explaining a third embodiment of the present invention, and shows a flow of a rendering process.

 FIG. 14 is a diagram illustrating a first configuration example that implements the present invention, and illustrates a configuration using a non-sonal computer.

FIG. 15 is a second configuration example for realizing the present invention, and is a diagram showing a configuration using a server client system. FIG. 16 is a third configuration example for realizing the present invention, and is an example of a configuration for performing image conversion according to the present invention in photographing with a camera.

 FIG. 17 is a diagram showing the relationship between the position of the light source and the image taken with the wide-angle lens.

 FIG. 18 is a diagram showing a third configuration example for realizing the present invention, showing a configuration using a folding mobile phone.

 Explanation of symbols

[0025] S10 first step

 S20 second step

 S30 3rd step

 S40 4th step

 AA1 to AA3, AB1 to AB3, AC1 to AC3, AD1 to AD3, AE1, AE2, AF1, AF2

, AG1, AG2 Homogeneous area

 TIA, TIB texture image

 OA, OB object

 205 Parameter operation section

 207 Image generator

 301 server

 302 clients

 501 Image feature analysis unit

 502 Image feature vector database

 503 Lighting equation parameter database

 BEST MODE FOR CARRYING OUT THE INVENTION

[0026] In the first aspect of the present invention, as a method of performing a predetermined image conversion on the first image and generating a second image, a predetermined value that gives luminance to each pixel of the first image is provided. A first step of acquiring each of a plurality of parameters constituting the illumination equation, a second step of identifying a homogeneous region composed of pixels having similar parameter values for each of the parameters, For each parameter, according to the content of the predetermined image conversion, the parameter conversion process is performed for each homogeneous region specified in the second step. A third step and a fourth step for obtaining the luminance of each pixel of the second image using each parameter after the conversion process in the third step are provided.

[0027] In the second aspect of the present invention, the predetermined image conversion is image enlargement, and the conversion process in the third step is a process for increasing the density of the parameter. I will provide a.

 [0028] In the third aspect of the present invention, the predetermined image conversion is image reduction, and the conversion process in the third step is a process for reducing the density of the parameter. Provide a method.

 [0029] In a fourth aspect of the present invention, there is provided the image conversion method according to the first aspect, wherein the acquisition of the plurality of parameters in the first step is performed by measuring subject force or estimating the first image force. .

 [0030] In a fifth aspect of the present invention, there is provided the image conversion method according to the first aspect, wherein, in the second step, the degree of similarity is evaluated using variances of values of the parameters in a plurality of pixels.

 [0031] In a sixth aspect of the present invention, there is provided the image conversion method according to the first aspect, wherein the second step includes a process of performing noise removal in the specified homogeneous region.

 [0032] In the seventh aspect of the present invention, as a texture mapping method, a preprocessing step of pasting a texture image onto an object of a three-dimensional CG model, and a luminance for each pixel of the texture image pasted on the object A first step of acquiring each of a plurality of parameters constituting a predetermined illumination equation for giving a predetermined illumination equation, and a second step of specifying a homogeneous region having a pixel force with a similar value of the parameter for each of the parameters, and A third step of performing a conversion process of the parameter for each homogeneous region identified in the second step according to the content of predetermined image conversion for each parameter; and the third step And a fourth step of obtaining the luminance of each pixel of the image of the object using each parameter after the conversion processing in To provide things.

[0033] In the eighth aspect of the present invention, as a device that performs predetermined image conversion on the first image and generates a second image, a predetermined value that gives brightness to each pixel of the first image is provided. A parameter acquisition unit for acquiring a plurality of parameters constituting the illumination equation; For each parameter, a homogeneous region specifying unit that specifies a homogeneous region having a similar pixel value and also having a pixel power, and for each of the parameters, the homogeneous region specifying unit according to the content of the predetermined image conversion For each homogenous region identified by the parameter conversion unit that performs the conversion process of the parameter, and the luminance for obtaining the luminance of each pixel of the second image using each parameter after the conversion process by the parameter conversion unit Provided with a calculator.

 [0034] In the ninth aspect of the present invention, as a texture mapping device, a pre-processing unit that pastes a texture image on an object of a three-dimensional CG model, and brightness for each pixel of the texture image pasted on the object A parameter acquisition unit that acquires each of a plurality of parameters constituting a given illumination equation to be given, and a homogeneous region specifying unit that specifies a homogeneous region composed of pixels having similar parameter values for each parameter! A parameter conversion unit that performs conversion processing of the parameter for each homogeneous region specified by the homogeneous region specification unit according to the content of predetermined image conversion for each parameter, and conversion processing by the parameter conversion unit A luminance calculation unit that obtains the luminance of each pixel of the image of the object by using each parameter later. To provide things.

 [0035] In the tenth aspect of the present invention, the server client system that performs image conversion includes the server having the parameter acquisition unit, the homogeneous region specifying unit, and the parameter conversion unit of the eighth aspect, and the luminance calculation unit of the eighth aspect. And the client provides the server with an instruction to change the contents of the image.

[0036] In the eleventh aspect of the present invention, each pixel of the first image is used as a program for causing a computer to execute a method of performing a predetermined image conversion on the first image and generating a second image. For each of the parameters, a homogeneous region composed of pixels having similar values for each of the plurality of parameters constituting a predetermined illumination equation that gives brightness. A second step of identifying a parameter, and a third parameter conversion process for each of the parameters for each homogeneous region identified in the second step according to the content of the predetermined image conversion. There is provided a program for causing a computer to execute a step and a fourth step of obtaining the luminance of each pixel of the second image using each parameter after the conversion processing in the third step. [0037] In the twelfth aspect of the present invention, as a texture mapping program, a pre-processing step of pasting a texture image onto an object of a three-dimensional CG model, and brightness for each pixel of the texture image pasted onto the object A first step of acquiring a plurality of parameters constituting a given illumination equation to be given, and a second step of specifying a homogeneous region composed of pixels having similar parameter values for each of the parameters; In each of the parameters, a third step of performing conversion processing of the parameter for each homogeneous region specified in the second step according to the content of predetermined image conversion, and in the third step A fourth step of determining the luminance of each pixel of the image of the object using each parameter after the conversion process; It provides that causes a computer to execute.

 Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0039] (First embodiment)

 FIG. 1 is a flowchart showing an image conversion method according to the first embodiment of the present invention. Note that the image conversion method according to the present embodiment can be realized by causing a computer to execute a program for realizing the method.

In the present embodiment, for example, the equations shown in (Equation 1) and (Equation 2) are used as the illumination equations that give the luminance, and the homogeneous region is specified for each of a plurality of parameters constituting this equation. Then, for each homogeneous region, the parameter conversion process is performed to realize the predetermined image conversion.

[Number 1] [Equation 2]

4m ² cos ⁴ β

g ² = n ² + c ² 1 1

 c = (H)

[0041] where la is the brightness of the ambient light, a is the reflectance of the ambient light, Ii is the brightness of the illumination, the vector N is the surface normal vector, the vector L is the light source vector indicating the light source direction, and d co is the light source The solid angle, is the bidirectional reflectance of the diffuse reflection component, ps is the bidirectional reflectance of the specular reflection component, F λ is the Fresnel coefficient, m is the microfacet distribution, and n is the refractive index. Kd is the diffuse reflection component ratio, ks is the specular reflection component ratio, and kd + ks = l. Vector H is a half vector located between light source vector L and viewpoint vector V, β is the angle between surface normal vector Ν and viewpoint vector V, and light source vector L, surface normal vector N, viewpoint vector V Force can also be calculated.

 FIG. 2 is a schematic diagram showing the relationship between luminance and illumination equation parameters. In the figure, (a) is a graph showing the luminance distribution of the image shown in (b), and (c) to (f) are the bi-directional reflectance pd of the diffuse reflection component and the specular reflection component of the illumination equation parameters. 4 is a graph showing the distribution of bidirectional reflectance ps, diffuse reflection component ratio kd, and surface normal vector N, respectively. In the graphs of Fig. 2 (a) and (c) to (f), the horizontal axis is the spatial position, and the vertical axis is the brightness or the value of each parameter.

 [0043] In the image of FIG. 2 (b), there are four types of objects XI to X4. Object XI is a luminance distribution that brightens from left to right, object X2 is a regular, random luminance distribution, object X3 is a luminance distribution with no and illite in the center, and object X4 is in all space positions. It has an isoluminance distribution.

[0044] In the range of the object XI, the bidirectional reflectance pd of the diffuse reflection component, the diffuse reflection component ratio kd, and the surface normal vector N force have homogeneous regions (AAl, AC1, AD1), respectively. Only the bidirectional reflectance ps of the surface reflection component changes. This change in pS causes a change in brightness. In the range of object X2, the bidirectional reflectance pd of the diffuse reflection component, the bidirectional reflectance ps of the specular reflection component, and the surface normal vector N force have homogeneous regions (AA2, AB1, AD2), respectively, and the diffuse reflection component ratio kd Only has changed. The diffuse reflection component ratio kd has a random change with no regularity, and the brightness also changes randomly to form a fine texture.

 [0045] In the range of object X3, the bidirectional reflectance pd of the diffuse reflection component, the bidirectional reflectance / 0 s of the specular reflection component, and the diffuse reflection component ratio kd have homogeneous regions (AA2, AB2, AC2). Only the surface normal vector N changes. This N change force causes a change in brightness. In the range of object X4, each parameter; 0 (1, s, kd, N has the same homogeneous area (AA3, AB3, AC3, AD3), so the luminance value is constant.The diffuse reflection component ratio kd The diffuse reflectance component is mainly high (Fig. 2 (e)), and the bidirectional reflectance pd of the diffuse reflection component is low! (Fig. 2 (c)), so the luminance value in the range of the object X4 is low! ,.

 [0046] In the conventional image enlargement processing shown in Non-Patent Documents 2 and 3, the brightness change force as shown in FIG. 2 (a) is also detected and emphasized. In this case, there is a problem that edge extraction from luminance change is difficult to separate from noise, and noise is also enhanced by edge enhancement.

 [0047] (Equation 1) Force If the parameter force constituting the illumination equation changes as much as the component force, the luminance changes. Therefore, it can be understood that edge detection is more stable for each parameter than for luminance change. In the present embodiment, since different homogeneous regions are adjacent to each other, the edge can be obtained more stably for parameters for which the homogeneous region is obtained more stably. Therefore, by converting each parameter for each homogeneous region, image conversion can be executed while preserving the sharpness and texture of the edges.

 [0048] Returning to the flow of FIG. 1, initial setting is performed in step SOO. Here, the first image to be subjected to image conversion is acquired, and the threshold value THEPR for homogeneous region determination, the threshold value THMEPR for homogeneous region merge determination, and the threshold value THN for noise determination are set. How to use these threshold values will be described later.

[0049] In step S10 as the first step, for each pixel of the first image, a predetermined A plurality of parameters constituting the illumination equation are respectively acquired. Here, the illumination equations of (Equation 1) and (Equation 2) described above are used. Here, the ambient light luminance Ia, the ambient light reflectance pa, the light source luminance Ii, the light source vector L, and the solid angle d co of the light source are called environmental conditions, and the bidirectional reflectance pd of the diffuse reflection component, specular reflection The bidirectional reflectance ps of the component, the diffuse reflection component ratio kd, and the specular reflection component ratio ks are called optical characteristics. These give the luminance value Iv of the reflected light in the direction of the viewpoint according to the illumination equation shown in (Equation 1).

 FIG. 3 is a conceptual diagram showing the geometric conditions assumed by (Equation 1). As shown in Fig. 3, light from the light source is incident on the current point of interest P on the object surface SF with irradiance Ii (NL) d ω, diffuse reflection component is kd pd, specular reflection component Is reflected by ks ps. Ambient light is light that enters the current attention point P on the object surface SF from the surroundings by multiple reflections, etc., and hits the bias component of the luminance Iv in the viewing direction (vector V).

 [0051] It should be noted that the types of illumination equations and parameters shown here are merely examples, and the present invention does not limit the structure of illumination equations and the types of parameters, and these are arbitrary.

 [0052] Each parameter of (Equation 1) can be obtained by measurement from a subject or estimation from a given captured image. For example, the surface normal vector N can be measured by a range finder or the like using the principle of triangulation (see, for example, Non-Patent Document 5). The principle of triangulation is that the triangle is uniquely determined when the angles of one side and both ends of the triangle are determined. As shown in Fig. 4, two points A separated by a known distance 1 , B is the angle at which point P is viewed as α and β, respectively. The coordinate values (x, y) of point Ρ are given by

[Equation 3]

[0053] As a technique for separating diffuse reflection and specular reflection, for example, Non-Patent Document 6 discloses a technique that utilizes the property that a specular reflection component is polarized. When light is reflected on the surface of an object, the electric field component parallel to the light incident / reflecting surface and the electric field component perpendicular to the light reflection surface are usually Since the channel coefficients are different, the reflected light is polarized. For this reason, the specular reflection component is generally polarized, but diffuse reflection is irregular reflection, and thus has no polarization. Therefore, as shown in FIG. 5, when the reflected light RR is observed through the polarizing filter PF, the intensity of the transmitted light RRP is the intensity of the component parallel to the polarization axis PFA of the polarizing filter PF in the reflected light RR. Therefore, when the specular reflection component from the object surface SF is observed while rotating the polarizing filter PF, the intensity of the transmitted light RRP is the angle between the polarizing axis PFA of the polarizing filter PF and the polarizing plane SP P of the specular reflection. It changes according to φ and is given by the following equation.

 Picture

I people ψ, = _d +-(F _v φ ',) + F _P (θ \)-((θ',)-F _p (θ ',)) cos 2ψ)

[0054] where Ld is the luminance of the diffuse reflection component, Ls is the luminance of the specular reflection component, Θ'i is the incident angle of the light on the minute reflection surface, FP is the Fresnel coefficient of the parallel electric field component to the insulator, and FV is It is the Fresnel coefficient of the vertical electric field component for the insulator.

 [0055] On the other hand, as a method for estimating the captured image force of each parameter, for example, the correspondence relationship between the spatial response characteristic and the illumination equation parameter is learned in advance, and the learning data is referred to when the parameter is acquired. The method to do is effective. For example, as shown in FIG. 6, the relationship between the image feature vector and the illumination equation parameter is learned in advance, and an image feature vector database 502 and an illumination equation parameter database 503 are prepared. The input image の as the first image is converted into an input image feature vector IINFV by the image feature analysis processing 501. In the image feature analysis processing 501, the spatial response characteristic is obtained by, for example, wavelet transformation. The image feature vector database 502 selects the image feature vector closest to the input image feature vector IINFV, and outputs the input image feature vector number IINFVN. The illumination equation parameter database 503 receives the input image feature vector number IINFVN and outputs the illumination equation parameter corresponding to this as the input image illumination equation parameter IINLEP. By using this method, it is possible to obtain all the parameters of a predetermined illumination equation.

[0056] It should be noted that the present invention can be applied to any method that does not limit the method of measuring and estimating the parameters of the illumination equation. For example, the surface normal vector N is It is possible to estimate by using the generalized inverse matrix (Equation 9), which can obtain 3 or more image powers with different light source directions by the Kustero method (RJWoodham, "Photometric method for determining surface orientation" rrom multiple images, Optical Engineenn g 19, pp.l39-144 1980).

 [Equation 8]

 v-Lx

[Equation 9] x = (L'L)-'ϋν

[0057] Here, the vector x is a vector obtained by collecting the surface normal vector p dN having the reflectance pd in the length for the number of times of photographing, and the matrix L is a light source matrix for collecting the number of light source vectors L for the number of times of photographing, and the vector V is a vector in which the luminance values Iv of reflected light in multiple viewpoint directions are collected for the number of times of shooting. However, the object surface is assumed to be a uniform diffuse surface (Lambertian surface), and the light source is assumed to be a point light source at infinity. In addition to the method shown in Fig. 5, the method of separating diffuse reflection and specular reflection is different from the method shown in Fig. 5, for example, in the distribution shape of diffuse reflection components and specular reflection components distributed in a three-dimensional color space formed from RGB signals. (S. Tominaga, N. Tanaka, “Estimating reflection parameters from a single color image, ΙΕΕϋ computer Graphics and Applications, vol. 20, Issue 5, pp. 58—66, 2000)” is there.

[0058] Next, in step S20 as the second step, for each parameter, a homogeneous region having a similar pixel value and a similar pixel value is specified. Here, the similarity of the meters is evaluated by the variance of the parameters in a plurality of pixel areas. When this variance value is smaller than the homogeneous region determination threshold THEPR set in step S00, the region of the plurality of pixels is determined to be a homogeneous region, and on the other hand, it is greater than or equal to the homogeneous region determination threshold THEPR. When it is determined that the region is not homogeneous. In this case, it is presumed that all the pixels in the area are different from each other or that different homogeneous areas are included. In any case, since it is considered that the edge is included, in order to preserve the sharpness and texture of the edge, it is not included in the homogeneous area. [0059] For example, in the case of surface normal vectors, the angle difference between the vectors becomes smaller when they are homogeneous. Therefore, the homogeneous region determination threshold THEPR is set to 0.5 degrees, for example, and when the variance value is smaller than 0.5 degrees, it is determined as a homogeneous area, and when it is greater than or equal to 0.5 degrees, it is determined as heterogeneous. The The diffuse reflection component ratio kd is a ratio and takes a value from 0 to 1. Therefore, the homogeneous region determination threshold THEPR is set to, for example, 0.01. When the variance value is smaller than 0.01, it is determined as a homogeneous region, and when the variance value is greater than or equal to 0.01, it is determined as a heterogeneous region.

 The setting of the area of the plurality of pixels for determining the similarity of the parameters is arbitrary, but here, a unit area having 5 pixels in the vertical direction and 5 pixels in the horizontal direction is used (S21). In this case, if 28 types of determinations from P01 to P28 as shown in FIG. 7 are performed, homogeneous regions can be extracted for all pattern shapes in the unit region UA. However, (1) all the pixels included in the homogeneous area are adjacent to each other, and (2) the central pixel of the unit area UA must be included. Judgment is made in two stages for all 28 patterns. First, in the central area CA of 3 × 3, it is determined whether or not three gray pixels out of nine pixels are homogeneous. Next, with respect to the pattern determined to be homogeneous, it is determined whether or not the pattern is homogeneous including the hatched pixels outside the central area CA. When multiple patterns are determined as homogeneous regions, the sum of them is taken as the homogeneous region.

 [0061] By such processing, a homogeneous region can be recognized for each specified unit region.

 (S22). When a homogeneous region is newly recognized (Yes in S22), the homogeneous region data that should follow the new homogeneous region is updated (S23). Steps S21 to S23 are repeatedly executed until the determination is completed for all unit areas (S24). As shown in Fig. 8, if the unit area UA of 5 pixels x 5 pixels is scanned so that one line overlaps horizontally and vertically, the homogeneous regions generated in the unit area UA are joined together, and the image It can be expanded to the whole.

[0062] Next, in step S25, the similarity between a plurality of homogeneous regions recognized in adjacent unit regions is evaluated, and similar homogeneous regions are merged. The method of evaluating the similarity of homogeneous regions is arbitrary. For example, an average value of parameter values may be obtained for each unit region, and a determination may be made using the difference value of the average values. In other words, if the difference value is smaller than the homogeneous region merge determination threshold THMEPR set in step SO 0, the homogeneous region is identical. To merge.

 Next, in step S26, it is determined whether or not there is noise in the homogeneous region. For example, this determination is based on the average value of the parameter values of all pixels in the homogeneous region, and the difference between the parameter value of a pixel and this average value is larger than the noise determination threshold TH N set in step SOO. Sometimes this is determined as noise. When the surface normal vector is noise, the difference from the average value of the vector angles becomes large. Therefore, the noise determination threshold THN is set to 30 degrees, for example, and when the difference from the average value is larger than 30 degrees, it is determined as noise. Regarding the diffuse reflection component ratio kd, the noise determination threshold THN is set to 0.2, for example, and when the difference from the average value is larger than 0.2, it is determined to be noise.

 [0064] When it is determined that the homogeneous region includes noise (Yes in S26), the noise in the homogeneous region is removed in step S27! Fig. 9 shows an example of noise removal. Gray pixels are homogeneous regions, and PI and P2 are pixels that are determined to be noise. For example, the average value of the parameter values of the pixels included in the homogeneous region is obtained from the 8 pixels around the pixel determined to be noise, and this is replaced with noise. In the case of pixel P1, since all the surrounding 8 pixels belong to the homogeneous region, replace with the average value of the parameter values of all the surrounding 8 pixels. On the other hand, in the case of pixel P2, two of the eight surrounding pixels belong to the homogeneous region, so they are replaced with the average value of these two pixels. Note that the noise removal method described here is merely an example, and any method may be used.

 [0065] As a result of step S20, pixels that do not fall within the homogeneous region form edges.

 [0066] Then, in step S30 as the third step, for each parameter, the parameter is changed for each homogeneous region identified in step S20 according to the content of the predetermined image conversion. Perform the conversion process.

FIG. 10 is a conceptual diagram showing processing when image enlargement is performed as image conversion. As shown in Fig. 10, when enlarging the image, the parameters are made dense within the homogeneous region. Figure 10 (a) shows the distribution of the parameters before conversion. The homogeneous region AE1 where the average parameter value is P1 is adjacent to the homogeneous region AE2 where the average parameter value is P2. Then, the luminance difference edge of the pixels SI and S 2 located at the boundary between the homogeneous regions AE1 and AE2 is formed. In order to enlarge the image of Fig. 10 (a), for example, by a factor of 2, it is shown in Fig. 10 (b). In this way, a white circle pixel may be inserted between each black circle pixel. The parameter value of the white circle pixel is, for example, the parameter value of the adjacent black circle pixel. In addition, a new pixel S3 may be generated by copying either parameter value as it is between the pixels SI and S2. In FIG. 10 (b), the parameter value of the pixel S1 is copied to the pixel S3, and the luminance difference between the pixels S2 and S3 is made to match the luminance difference between the pixels SI and S2 in FIG. 10 (a). This preserves the edges.

 [0068] It should be noted that all parts that are not homogeneous regions should be treated as edges. For example, in the part sandwiched between homogeneous regions AC1 and AC2 in Fig. 2 (e), there are 10 sections between pixels, but all of these are edges one by one. Capture. For the densification method, the parameter values of adjacent pixels may be copied in the same manner as the pixel S3 in FIG. For example, you can copy from the pixel on the left side of the position where you want to increase the density, copy from the right side, or switch between the left side and the right side every other section.

 FIG. 11 is a conceptual diagram showing a process when image reduction is performed as image conversion. As shown in Fig. 11, when the image is reduced, the parameters are reduced in the homogeneous region. Although the density reduction method is arbitrary, in Fig. 11, the average value of the parameter values of the surrounding pixels is used as an example. Figure 11 (a) shows the distribution of the parameters before conversion. The homogeneous region AF1 where the average parameter value is P1 is adjacent to the homogeneous region AF2 where the average parameter value is P2. Then, the luminance difference edge of the pixels S6 and S7 located at the boundary between the homogeneous regions AF1 and AF2 is formed. The distribution shown in Fig. 11 (a) is reduced to 1Z2, for example, to generate the distribution shown in Fig. 11 (b). In the homogeneous region AF1, the average value of the parameter values in the pixel group SG1 is set as the parameter value of the pixel S4, and the average value of the parameter values in the pixel group SG2 is set as the parameter value of the pixel S5, thereby realizing a reduction in density. At this time, by partially overlapping the pixel group SG1 and the pixel group SG2, the change in the parameter value of the reduced image is smoothed. The luminance difference between the pixels S6 and S7 which are the edges in FIG. 11A is stored as the luminance difference between the pixels S7 and S8 in FIG. That is, the parameter value of pixel S8 is copied from pixel S6.

[0070] Then, in step S40 as the fourth step, the luminance of each pixel of the second image after the predetermined image conversion using each parameter after the conversion processing in step S30 Ask for. At this time, if each parameter is given to the lighting equation of (Equation 1), the reflected light intensity Iv is calculated for each pixel.

 As described above, according to the present embodiment, the luminance is decomposed into illumination equation parameters, and image conversion such as image enlargement or image reduction is performed using the correlation between pixels for each parameter. That is, since image conversion is executed for each parameter for each homogeneous region, the edge portion is stored as a boundary condition between the homogeneous regions. In addition, since the homogeneous region is specified based on the similarity of the illumination equation parameters that are physical characteristics of the subject, it is determined with physical support. Therefore, image conversion with stable image quality can be realized while preserving the sharpness of the edge and the texture.

 [0072] It should be noted that a parameter acquisition unit that executes step S10, a homogeneous region specifying unit that executes step S20, a parameter conversion unit that executes step S30, and a luminance calculation unit that executes step S40 are provided. An image conversion apparatus may be configured.

 [0073] <Other examples of lighting equations>

 The illumination equation used in the present invention is not limited to that shown in the present embodiment. For example, the following may be used.

 [Equation 5]

Ι _ν = Ι _ν _ _α + Ι ΐ) άω- _Ρά

[Equation 6]

_{Ν ν} -Ι _{ν α} + IJV .L) dm. P

[Equation 7]

[0074] (Equation 5) is for a diffusely reflecting object, and has 6 parameters. Where Iv, a represents the light intensity in the direction of the peripheral force line of sight. (Equation 6) does not distinguish between diffuse reflection and specular reflection, and there are five parameters. (Equation 7) does not consider the reflectivity, and there are two parameters. However, Iv, i represents the pixel power of interest and the light intensity in the line-of-sight direction.

[0075] (Second Embodiment) In the second embodiment of the present invention, image compression is performed as the predetermined image conversion. The basic processing flow is the same as in the first embodiment. In step S30, processing for compression-encoding each parameter is performed for image compression. In this case, normally, the compressed image data is transferred or recorded without executing step S40. When reproducing an image, each parameter is decoded and the luminance of each pixel is calculated. Similarly to the first embodiment, the image conversion method according to the present embodiment can also be realized by causing a computer to execute a program for realizing the method. Further, an image compression apparatus including a parameter acquisition unit that executes step S10, a homogeneous region specifying unit that executes step S20, and a parameter compression unit that executes step S30 may be configured.

 FIG. 12 is a conceptual diagram showing parameter conversion processing in the present embodiment. In Fig. 12 (a), the white circles represent the parameter values of the pixels belonging to the homogeneous regions AG1 to AG3, and the hatched circles represent the parameter values of the pixels not belonging to the homogeneous region. As shown in Fig. 12 (a), the parameter values are almost equal in the homogeneous regions AG1 to AG3, and therefore the amount of information related to the parameter values is almost integrated into the average value. Therefore, in each homogeneous region AG1 to AG3, the average value of the parameter value and the difference between the parameter value and the average value of each pixel are encoded, and a small amount of code is assigned to the difference. To do. As a result, it is possible to perform compression coding of the meter value with a small amount of code without impairing the image quality.

 [0077] For example, as shown in the code sequence in Fig. 12 (b), for the homogeneous region AG1, first, the code type TP1 is declared (here, "difference from the average value"), and then the average The value Dl and the difference D2 from the average value at each pixel are followed by D2, followed by the separator signal SG1. Instead of attaching the separation signal SG1, a special code may be assigned as the code type so that the separation can be recognized. If the difference D2 is so small that it can be ignored, apply the run-length code.

[0078] For pixels that do not belong to the homogeneous region, the value of the meter changes irregularly, so compression of the amount of data cannot be expected even if it is encoded by the difference from the average value. So, even «JP EG (Joint Photographic Experts uroup) and MPEG (Moving Picture Experts uroup) It is sufficient to use the orthogonal transformation adopted in the above. That is, as the sign key type TP2

Declare “orthogonal transformation” and sign the frequency coefficient D3 in order of the first frequency term. In addition

If the homogeneous region occupies most of the range in the image, it is not a problem to code the parameter values of pixels that do not belong to the homogeneous region as they are.

[0079] After the delimiter signal SG2, the homogeneous regions AG2 and AG3 declare "difference from the average value" as the encoding types TP3 and TP4 as in the homogeneous region AG1.

 As described above, according to the present embodiment, a higher correlation than the luminance value can be expected by decomposing the luminance value into parameters constituting the luminance value and obtaining the correlation with the neighboring pixels, thus improving the compression efficiency. Can be made. In addition, since compression coding is performed for each homogeneous region, it is possible to achieve a higher compression ratio than the luminance value base while preserving sharpness and texture.

 [0081] (Third embodiment)

 In the third embodiment of the present invention, the image conversion method as described above is applied to texture mapping in computer graph status.

 FIG. 13 is a flowchart showing the main flow of the rendering process. The rendering process is a process of converting a three-dimensional model generated in a computer into two-dimensional image data in computer graphics (see, for example, pp. 79 of Non-Patent Document 1). As shown in Figure 13, the rendering process is the main step S101, coordinate transformation S102, hidden surface removal S 103, shading and shadowing S 104, texture mapping S105, and viewport transformation S 106 .

 First, in step S101, when the viewpoint VA and the light source LS are set, the appearance is determined.

 Next, in step S102, the objects managed in the local coordinate system are grouped into a regular coordinate system, and in step S103, the hidden surface portion that cannot be seen from the viewpoint is deleted. Then, in step S104, the light source LS force is also calculated as how light strikes the objects OA and OB, and a shade and a shadow are generated.

In step S 105, texture mapping is performed to generate textures TA and TB for the objects OA and OB. The texture is generally acquired as image data. The texture image TIA is deformed according to the shape of the object OA and is synthesized on the object OA. Similarly, the texture image TIB is matched to the shape of the object OB. Then transform it and compose it on the object OB.

 In this embodiment, the image conversion as described above is applied in this texture mapping. That is, first, pre-processing for pasting the texture images TIA and TIB to the objects OA and OB of the 3D CG model is performed. Then, processing is performed according to the flow of FIG. In step S10, each of the texture images TIA and TIB pasted to the objects OA and OB using the optical parameters of the two-dimensional texture images TIA and TIB and the surface normal vectors of the objects OA and OB. The parameter is acquired for the pixel. The subsequent processing is the same as in the first embodiment. Note that the texture mapping method according to the present embodiment can also be realized by causing a computer to execute a program for realizing the method. In addition, a preprocessing unit that performs the above-described preprocessing, a parameter acquisition unit that executes step S10, a homogeneous region specifying unit that executes step S20, a parameter conversion unit that executes step S30, and a step A texture mapping device including a luminance calculation unit that executes S40 may be configured.

 [0086] Finally, in step S106, viewport conversion is performed to generate a two-dimensional image having an image size matching the displayed screen SCN or window WND.

 Here, the rendering processing needs to be executed because the viewpoint and the position of the light source change, and the rendering processing is frequently repeated in an interactive system such as a game device. In texture mapping, texture data to be pasted on the object surface is usually prepared as an image. Therefore, whenever the viewpoint or light source changes, it is necessary to convert the texture data by enlarging, reducing, rotating, or changing colors.

 Therefore, if image conversion is performed for each parameter as in the present embodiment, texture mapping according to various viewpoints and light source settings can be realized while preserving the texture. In particular, when the position of the light source is changed, it is difficult to calculate the texture change on the basis of the luminance value. Therefore, the method that can directly control the light source vector as in this embodiment is superior to the conventional method in principle. It can be said that there is.

 Hereinafter, a configuration example for realizing the present invention will be illustrated.

 [0090] (First configuration example)

FIG. 14 is a diagram showing a first configuration example, which is related to the present invention using a personal computer. 2 is an example of a configuration for performing image conversion. The resolution of the camera 101 is lower than the resolution of the display 102. In order to take full advantage of the display capability of the display 102, an enlarged image is created by an image conversion program loaded in the main memory 103. The low resolution image captured by the camera 101 is recorded in the image memory 104. An image feature vector database 502 and an illumination equation parameter database 503 as shown in FIG. 6 are prepared in advance in the external storage device 105, and the image conversion program capability of the main memory 103 can be referred to.

 The processing by the image conversion program is the same as in the first embodiment, and a homogeneous region is determined for each illumination equation parameter, and densification is performed in the homogeneous region. That is, a low-resolution image in the image memory 104 is read via the memory bus 106, enlarged in accordance with the resolution of the display 102, and transferred again to the video memory 107 via the memory bus 106. The enlarged image transferred to the video memory 107 is displayed on the display 102.

 It should be noted that the present invention can take various configurations other than those constrained by the configuration of FIG. For example, the illumination equation parameters may be measured directly from the subject by a measuring instrument. In this case, the image feature vector database 502 and the illumination equation parameter database 503 of the external storage device 105 are not necessary. Also, acquiring low resolution images from the network 108 does not help. It is also possible to store the texture data in the external storage device 105 and execute the texture mapping shown in the third embodiment in the main memory 103.

 [0093] If the resolution of the camera 101 is higher than the resolution of the display 102, the image conversion program loaded in the main memory 103 may perform image reduction as shown in the first embodiment. . In this case, the image compression may be performed according to the second embodiment, and the illumination equation parameters are data-compressed and the network 108 isotropic force can be transmitted.

 [0094] As the camera 101, any type of imaging device such as a camera-equipped mobile phone, a digital still camera, or a video movie, can be applied. Furthermore, the present invention can be realized in a playback device that plays back pre-recorded video.

[0095] (Second configuration example) FIG. 15 is a diagram showing a second configuration example, which is an example of a configuration for performing image conversion according to the present invention using a server client system. The resolution of the camera 201 is lower than the resolution of the display 202. In order to make the best use of the display capability of the display 202, image enlargement is executed in the server-client system. Similarly to FIG. 6, the server 301 includes an image feature analysis unit 501, an image feature vector database 502, and an illumination equation parameter database 503. The server 301 calculates the input image repulsive illumination equation parameter IINL EP and sets the parameter operation unit 205. Output to. This operation corresponds to step S10 in the flow of FIG. The image feature analysis unit 501, the image feature vector database 502, and the illumination equation parameter database 503 constitute a parameter acquisition unit.

 On the other hand, an image conversion instruction is passed from the image conversion instruction unit 203 of the client 302 to the parameter operation instruction unit 204 of the server 301 as an image conversion instruction signal ICIS. The norometer operation instruction unit 204 replaces the content of the image conversion by the image conversion instruction signal ICIS with the operation content of the illumination parameter, and outputs it to the parameter operation unit 205 as the parameter operation instruction signal LEPS. In accordance with the image conversion method described in the first embodiment, the parameter operation unit 205 operates the illumination equation parameter IINLEP to perform image enlargement or image compression, and generates a new parameter value IOUTLEP. This operation corresponds to steps S20 and S30 in the flow of FIG. The meter operation unit 205 corresponds to the homogeneous region specifying unit and the parameter converting unit.

 With this operation, the server 301 can provide the client 302 with the new parameter value IOUTLEP according to the image conversion instruction from the client 302 via the network 206. In the client 302 that has received the new parameter value IOUTLEP, an image generation unit 207 as a luminance calculation unit generates an enlarged image and supplies it to the display 202. This operation corresponds to step S40 in the flow of FIG.

Note that the present invention is not limited to the configuration of FIG. 15, and when the resolution of the camera 201 is higher than the resolution of the display 202, the parameter operation unit 205 is as shown in the first embodiment. Image reduction may be performed. Further, if the parameter operation unit 205 operates as an encoding device according to the second embodiment, and the image generation unit 207 operates as a decoding device. , Compressed data can be distributed to the network 206.

 [0099] It should be noted that the combination of image devices and the position of each means on the system (whether belonging to the server 301, the force belonging to the client 302 or the like, etc.) are arbitrary. As the camera 201, any type of imaging device such as a mobile phone with camera, a digital still camera, or a video movie camera can be applied. Furthermore, the present invention can also be realized in a playback apparatus that plays back pre-recorded video.

 [0100] (Third configuration example)

 FIG. 16 is a diagram showing a third configuration example, which is an example of a configuration for performing image conversion according to the present invention in photographing with a camera.

 [0101] The camera 401 includes a wide-angle lens 402, and can, for example, capture a wide field of view with an angle of view of 180 degrees at a time. The light source 403 can be photographed by attaching the wide-angle lens 402 facing upward. A wide-angle xyz three-dimensional coordinate system with the optical axis of the wide-angle lens 402 as the z-axis, the horizontal direction of the wide-angle image sensor 404 inside the camera 401 as the X-axis, and the vertical direction of the wide-angle image sensor 404 as the y-axis The focal position of lens 402 is determined as the coordinate origin, and the light source vector L is obtained.

 FIG. 17A shows the relationship between the position of the light source 403 and the wide-angle image 405 taken by the wide-angle lens 402. Consider the case where the position of the light source 403 moves along the curve LT. The light source 403 moved from the position PS1 on the curve LT to the position PS5 is recorded from the position PXI on the straight line ST of the wide-angle image 405 to the position PX5. When the light source 403 is located at the position PS2, a method for obtaining the light source vector L2 will be described in which the angle formed by the straight line ST and the x axis is 0, and the angle formed by the straight line ST and the light source vector L2 is φ.

 [0103] Figure 17 (b) is a view of the wide-angle image 405 of Figure 17 (a) from the z-axis direction. The distance between position PX1 and coordinate origin O is d, and the distance between position PX2 and coordinate origin O is r. And The position PX1 is φ = 0 and the coordinate origin Ο is φ = 90 degrees. Since the light source position and the position on the wide-angle image are linearly distributed, the angle φ of the position ΡΧ2 is given by

 [Equation 10]

2d

[0104] Here, the pixel positions at position ΡΧ1, position ΡΧ2, and coordinate origin Ο on the wide-angle image are ( x, y), (x, y), (χ, y), the distance d between the position PX1 and the coordinate origin O is

 LI LI L2 L2 O O

[Equation 11] + (Ίι-y ₀ ) ² and the distance r between the position PX2 and the coordinate origin O is

[Equation 12] r = ^ x _L2 -x ₀ ) + [y _L2 -y ₀ f

 Figure 17 (c) shows a triangle obtained by subtracting the intersection line LT with the light source vector L2 from the position PX2 in the z-axis direction. If the length of the intersection line LT is z, the following equation is obtained.

 L2

 [Equation 13] rn

 r tan

 2 If the light source vector L2 is defined by a unit vector of length 1,

[Equation 14]

It becomes.

 The subject is photographed by the subject photographing lens 406 and the subject imaging element 407, and the first image output from the subject photographing element 407 is converted into the second image by the image conversion unit 408. The image conversion unit 408 executes, for example, image enlargement according to the flowchart in FIG. 1, image compression according to FIG. Although there is no limitation on the coordinate system used for image conversion, it is preferable to use the xyz three-dimensional coordinate system of the subject imaging element 407 because the image conversion is performed on the output of the subject imaging element 407. Therefore, the light source vector (Expression 14) expressed in the xyz three-dimensional coordinate system of the wide-angle imaging element 404 is converted into the xyz three-dimensional coordinate system of the subject imaging element 407. The transformation of the coordinate system can be realized by transformation of the coordinate axes. Kool (X, y, z

light, ught light Is a vector in which the x-axis of the xyz three-dimensional coordinate system of the wide-angle imaging device 404 is represented by the xyz three-dimensional coordinate system of the subject imaging device 407: Kutor (X, y, z) is

 The X axis of the xyz three-dimensional coordinate system of the image sensor 404 for wide angle, light, light, x light, x light is a vector expressed in the xyz three-dimensional coordinate system of the image sensor 404 for wide angle. If the y-axis and z-axis are defined in the same way as the X-axis, the vector of each axis is related to the 3 X 3 matrix M as follows.

 [Equation 15],

 NO-- M

¹ - ^z This is connected to the matrix M! /

 [Equation 16]

 M y y y

 It will be no. By applying this matrix M to (Equation 14), the light source vector L is converted from the xyz three-dimensional coordinate system of the wide-angle imaging element 404 to the xyz three-dimensional coordinate system of the subject imaging element 407.

 [0107] Since the light source is often located above the camera 401, for example, if the wide-angle lens 402 having an angle of view of 180 degrees is used, the light source 403 can be photographed. If 403 cannot be captured by the angle of view of the wide-angle lens 402, the direction of the camera 401 is changed and the light source 403 is captured by the angle of view. Therefore, since it is necessary to measure the change in the orientation of the camera 401, the camera 401 has a built-in 3D attitude sensor 409 (consisting of an acceleration sensor, etc.) to measure the 3D motion of the xyz 3D coordinate axis of the wide-angle imaging element 404. If it is acquired from the dimensional attitude sensor 409 and coordinate-transformed in the same way as (Equation 16),

[0108] As another method of changing the direction of the camera, a configuration of a foldable mobile phone is also effective. In other words, as shown in FIG. 18, the mobile phone 601 includes a far-end camera 602 (a camera that captures a subject in front of the user of the mobile phone 601) and a local camera 603 (a user of the mobile phone 601). The other camera 602 opens the folded display 604 Change the direction greatly. That is, as shown in (a), when the opening angle DAG of the display unit 604 is small, the top of the mobile phone 601 is captured. As shown in (c), when the opening angle DAG of the display unit 604 is large, the mobile phone As shown in (b), when the opening angle DAG of the display unit 604 is large in the middle, as shown in (b), the middle direction between the top of the mobile phone 601 and the front of the user of the mobile phone 601 is caught. . Therefore, the opening angle DAG of the display unit 604 is detected by the angle sensor 606 provided in the hinge 605, and the direction of the counterpart camera 602 is calculated. The xyz three-dimensional coordinate system, for example, uses the focal position of its own camera 603 as the coordinate origin, and the relationship between the focal position of the other camera 602 determined by the structure of the mobile phone 601 and The captured images can be managed in the same xyz 3D coordinate system. It is obvious that the own camera 602 can also be used for photographing the light source. As described above, among the parameters of the illumination equation shown in FIG. 3, the light source vector can be calculated.

 [0109] Further, the camera 401 includes a polarizing filter, so that the reflected light from the object incident on the subject photographing lens 408 is diffused and reflected by the method described in (Equation 4) and Fig. 5, for example. It can be separated into reflection components. If the diffuse reflection component is used, the surface normal vector N can be calculated by the photometric stereo method described in (Equation 9). As described in (Equation 8), the photometric stereo method requires three or more images with different light source orientations. Therefore, if the light source 403 is movable, (Equation 8) can be obtained by setting three or more positions of the light source 403 and performing photographing each time. Also, as the subject moves, the positional relationship between the light source and the subject changes, and as a result, the direction of the light source changes. Therefore, it is possible to obtain (Equation 8) by tracking a specific point of the subject and shooting three or more times. On the other hand, the specular reflection component corresponds to ks ps in (Equation 1), and when the light source vector L and the surface normal vector N are known, the unknown parameters included in (Equation 2) include the specular reflection component ratio ks and the Fresnel coefficient. F, Mic mouth facet distribution m and refractive index n. These parameters can be obtained by a method using a plurality of sample data fittings using the least square method or a method using a measuring instrument such as a refractometer.

[0110] It is obvious that the surface normal vector N can be measured by using a range finder in addition to the configuration of FIG. [0111] As described above, the present invention can be executed in a wide variety of video devices such as widely used personal computers, server client systems, mobile phones with cameras, digital still cameras, video movie cameras, and televisions. No special equipment, operation or management is required. It should be noted that the device connection form and the internal configuration of the device, such as mounting on dedicated hardware or a combination of software and hardware, are not constrained.

 Industrial applicability

 [0112] In the present invention, it is possible to realize image conversion with stable image quality while preserving the sharpness and texture of edges, so that, for example, video entertainment for recording scenes in front of eyes such as sports, sightseeing, and commemorative photography. Can be used in various fields. In the field of culture and art, it can be used to provide a highly flexible digital eve system that is not restricted by the subject or shooting location.

Claims

The scope of the claims

 [1] A method of performing a predetermined image conversion on a first image to generate a second image, wherein a plurality of pixels that form a predetermined illumination equation that gives luminance is provided for each pixel of the first image. A first step of obtaining a number parameter, respectively,

 A second step of identifying, for each of the parameters, a homogeneous region that also has a pixel power with similar parameter values;

 For each of the parameters, a third step for performing the parameter conversion process for each homogeneous region specified in the second step according to the content of the predetermined image conversion, and the third step. And a fourth step of calculating the luminance of each pixel of the second image using each parameter after the conversion processing in

 An image conversion method characterized by that.

[2] In claim 1,

 The predetermined image conversion is image enlargement;

 The image conversion method according to claim 3, wherein the conversion process in the third step is a process for increasing the density of the parameter.

[3] In claim 1,

 The predetermined image conversion is image reduction;

 The image conversion method according to claim 3, wherein the conversion process in the third step is a process for reducing the density of the parameter.

[4] In claim 1,

 The acquisition of the plurality of parameters in the first step is performed by measuring subject force or estimating from the first image.

 An image conversion method characterized by that.

[5] In claim 1,

 In the second step, the degree of similarity is evaluated using the variance of the parameter values in a plurality of pixels.

An image conversion method characterized by that.

[6] In claim 1,

 The image conversion method according to claim 2, wherein the second step includes a process of removing noise in the identified homogeneous region.

[7] A pre-processing step of pasting a texture image onto an object of a three-dimensional CG model, and a plurality of parameters constituting a predetermined illumination equation that gives brightness for each pixel of the texture image pasted on the object , Respectively, the first step to get,

 A second step of identifying, for each of the parameters, a homogeneous region that also has a pixel power with similar parameter values;

 For each of the parameters, according to the content of the predetermined image conversion, a third step of performing the parameter conversion process for each homogeneous region specified in the second step, and a step of the third step And a fourth step of determining the luminance of each pixel of the image of the object using each parameter after the conversion process.

 A texture mapping method characterized by that.

[8] An apparatus for performing a predetermined image conversion on the first image to generate a second image, and for each pixel of the first image, a compound that forms a predetermined illumination equation that gives luminance. A parameter acquisition unit for acquiring a number of parameters, and

 For each of the parameters, a homogeneous region identifying unit that identifies a homogeneous region that also has a pixel force with similar parameter values;

 For each parameter, a parameter conversion unit that performs conversion processing of the parameter for each homogeneous region specified by the homogeneous region specification unit according to the content of the predetermined image conversion;

 A luminance calculation unit that obtains the luminance of each pixel of the second image using each parameter after the conversion processing by the parameter conversion unit.

 An image conversion apparatus characterized by that.

[9] A pre-processing unit that pastes texture images onto 3D CG model objects,

For each pixel of the texture image pasted on the object, a plurality of parameters constituting a predetermined illumination equation giving brightness are respectively acquired. Data acquisition unit;

 For each of the parameters, a homogeneous region identifying unit that identifies a homogeneous region that also has a pixel force with similar parameter values;

 A parameter conversion unit that performs a conversion process of the parameter for each homogeneous region identified by the homogeneous region identification unit according to the content of predetermined image conversion for each parameter;

 A luminance calculation unit that calculates the luminance of each pixel of the image of the object using each parameter after the conversion processing by the parameter conversion unit.

 A texture mapping apparatus characterized by that.

 [10] A server client system for image conversion,

 The parameter acquisition unit according to claim 8, a homogeneous region identification unit, and a parameter conversion unit

 A client having the luminance calculation unit of claim 8,

 The client instructs the server to change the image

 Server client system characterized by that.

[11] A program for causing a computer to execute a method of performing predetermined image conversion on a first image and generating a second image,

 A first step of acquiring, for each pixel of the first image, a plurality of parameters constituting a predetermined illumination equation that gives brightness;

 A second step of identifying, for each of the parameters, a homogeneous region that also has a pixel power with similar parameter values;

 For each of the parameters, a third step for performing the parameter conversion process for each homogeneous region specified in the second step according to the content of the predetermined image conversion, and the third step. And a fourth step of determining the luminance of each pixel of the second image using each parameter after the conversion processing in

 An image conversion program to be executed by a computer.

[12] A pre-processing step to paste the texture image onto the object of the 3D CG model, A first step of obtaining, for each pixel of the texture image pasted on the object, a plurality of parameters constituting a predetermined illumination equation that gives brightness;

 A second step of identifying, for each of the parameters, a homogeneous region that also has a pixel power with similar parameter values;

 For each of the parameters, according to the content of the predetermined image conversion, a third step of performing the parameter conversion process for each homogeneous region specified in the second step, and a step of the third step A fourth step of determining the luminance of each pixel of the image of the object using each parameter after the conversion process;

A texture mapping program to be executed by a computer.