WO2007108041A1 - Video image converting method, video image converting device, server client system, mobile apparatus, and program - Google Patents

Abstract

A video image characteristic analyzing unit (101) carries out a video image characteristic analysis of an input video image (IIN) and outputs a video image characteristic vector (IINFV). A parameter output unit (10) stores a plurality of video image characteristic vectors and a plurality of parameter values for an illumination equation corresponding to the video image characteristic vectors and outputs an initial parameter value (IINLEP) corresponding to the video image characteristic vector (IINFV). A parameter operation setting unit (106) determines contents of a parameter operation in accordance with an instructed video image conversion, and a parameter operation unit (104) operates the initial parameter value (IINLEP) in accordance with an instruction of the parameter operation setting unit (106) to obtain a new parameter value (IOUTLEP). A video image generating unit (107) generates an output video image (IOUT) based on the new parameter value (IOUTLEP).

Description

 Specification

 Image conversion method, image conversion apparatus, server client system, portable device, and program

 Technical field

 [0001] The present invention relates to an image processing technique, and more particularly to a technique for realizing image conversion such as image enlargement, illumination conversion, and viewpoint conversion.

 Background art

 [0002] Image devices and network digital connections enable easy connection between different types of image devices, increasing the degree of freedom of image exchange. For example, users can freely access images taken with a digital still camera without being restricted by differences in the system, such as outputting to a printer, publishing on a network, or viewing on a home TV. An environment that can handle images has been improved.

 On the other hand, in order to realize such an environment, the system side is required to support various image formats, and advanced image format conversion is required. For example, an up-converter (a conversion device that increases the number of pixels and lines) and a down-converter (a conversion device that reduces the number of pixels and lines) are required to cope with frequently occurring image size conversion. For example, when printing on A4 paper (297mm x 210mm) at 600dpi resolution, the power required for 7128 pixel x 5040 line data is lower than this, so an upconverter is required. . Also, images published on the network require conversion to the corresponding image size every time an output device is determined. With regard to home TV, since digital terrestrial services have been started, conventional standard TV and HD (High Definition) TV are mixed, so image size conversion is frequently performed.

[0004] In order to enlarge an image, it is necessary to newly create powerful image data that does not exist at the time of acquisition, but various methods have already been proposed. For example, methods using interpolation such as bilinear method and no-cubic method are common (Non-patent Document 1). However, when interpolation is used, only intermediate values of sampling data can be generated. The sharpness of edges and the like tends to deteriorate, resulting in a blurred image. Therefore, a technique is disclosed in which an interpolated image is used as an initial enlarged image, and then an edge portion is extracted to emphasize only the edge (Patent Document 1, Non-Patent Document 2). However, along with the emphasis of the edge part, which makes it difficult to separate the edge part from the noise, the noise tends to be emphasized and the image quality tends to deteriorate.

[0005] Therefore, there is a learning method as a method for enlarging an image while suppressing image quality deterioration. That is, a high-resolution image corresponding to the enlarged image is taken in advance by a high-definition camera or the like, and a low-resolution image is created from the high-resolution image. Low-resolution images are usually generated by sub-sampling through a low-pass filter. Many pairs of such low-resolution images and high-resolution images are prepared, and the relationship is learned as an image enlargement method. Therefore, in the learning method, the above-described enhancement processing does not exist, and therefore, it is possible to realize an image enlargement with relatively little image quality deterioration.

[0006] As an example of a learning method, for example, a technique of performing learning using a statistical method is disclosed assuming that the relationship between luminance values with adjacent pixels is determined by a Markov process (Non-patent Document 3). In addition, a technique has been disclosed in which a feature vector is obtained for each pixel in a conversion pair from low resolution to high resolution, and an enlarged image is generated from the degree of coincidence with the feature vector of the input pixel and the consistency with the surroundings ( Non-patent document 4).

 [0007] In addition, the learning method is used not only for image enlargement but also for conversion of the illumination direction, for example (Non-Patent Document 5). In Non-Patent Document 5, a plurality of objects with different textures (such as surface irregularities and patterns) are illuminated from different directions, learning data is created, and the illumination direction is changed while maintaining the texture. Techniques to do this are disclosed.

 Patent Document 1: U.S. Pat.No. 5,717,789 (Figure 5)

 Non-Patent Document 1: Shinya Araya, “Clear 3D Computer Graphics”, Kyoritsu Shuppan, 2 September 25, 003, pp. 144-145

 Non-Patent Document 2: Makoto Nakashizuka, “High-resolution image in multi-scale luminance gradient plane”, IEICE Transactions D—Π Vol. J81 -D-II No. 10 pp. 2249— 2258, 1998 October

Non-Special Reference 3: Freeman et al., “Learnmg Low—Level Vision J, International Journal of Computer Vision 40 (1), pp. 25-47, 2000 Non-Patent Document 4: Hertzmann et al., “Image Analogies”, SIGGRAPH 2001 Proceedings, pp. 327-340, 2001

 Non-Patent Document 5: Malik et al., "Representing and Recognizing the Visual Appearance of Materials using Three-dimensional TextonsJ, International al Journal of Computer Vision 43 (1), pp. 29-44, 2001

 Disclosure of the invention

 Problems to be solved by the invention

[0008] However, the conventional technique has the following problems.

 [0009] In the learning method as described above, since the enlarged image is selected from the group of images used for learning, there is a problem that the enlargement method depends on the learning data. Similar problems also occur in other image conversions, such as conversion of the illumination direction that can be achieved only by image enlargement.

 [0010] In addition, since it is necessary to prepare a large number of low-resolution images and high-resolution images, there is also a problem that it takes too much manpower for pre-processing for learning. Furthermore, since it is necessary to create the image power actually captured when learning the image data at the time of learning, there is a possibility that the image data is naturally biased, which is not preferable for performing image conversion with a high degree of freedom.

 In view of the above problems, an object of the present invention is to increase the degree of freedom of image conversion in the image conversion using the learning method as compared with the conventional technique.

 Means for solving the problem

 [0012] In the present invention, image feature analysis is performed on the first image, and the relationship between the image feature and the illumination equation parameter is referred to from the image feature of the first image, and the corresponding illumination equation parameter is determined. Is obtained as the original parameter value. Also, the operation details of the illumination equation parameters are determined according to the instructed image conversion. Then, the original parameter value is manipulated according to this parameter manipulation content to obtain a new parameter value. Based on this new parameter value, a second image after image conversion is generated.

According to the present invention, the value of the illumination equation parameter corresponding to the image feature of the first image is acquired as the original parameter value, and the original parameter value is operated according to the operation content corresponding to the instructed image conversion. As a result, a new parameter value is obtained. The second image is then generated from this new meter value. That is, image conversion is Since it is realized by the number conversion, it is possible to perform image conversion with a higher degree of freedom than in the past without being restricted by the image data at the time of learning. For example, in the case of image enlargement, it is only necessary to increase the density of the surface normal vector representing the shape information of the object among the parameters of the illumination equation. In this case, an arbitrary enlargement magnification can be set. In the case of conversion of the illumination direction, the illumination vector representing the illumination direction may be changed. In addition, conversion of the viewing direction can be easily realized by manipulating the illumination equation parameters. In the prior art, learning images are required for each type of image conversion. On the other hand, since the present invention performs image conversion by manipulating the parameters of the illumination equation, the number of learning images can be reduced.

 [0014] In the present invention, in the preprocessing for learning the relationship between the image feature and the illumination equation parameter, the value of the illumination equation parameter is set, the set parameter value force also generates a learning image, and the learning image It is preferable to store the image features obtained from the image feature analysis in the database in association with the original parameter values.

 [0015] With this, in the pre-processing, the learning image can be generated by a computer using the illumination equation, so that shooting using the real object is not required for generating the learning image. Therefore, the processing becomes simple and various learning images can be easily prepared.

 The invention's effect

 [0016] According to the present invention, since image conversion is realized by parameter conversion of the illumination equation, image conversion with a high degree of freedom is possible. In addition, the number of learning images can be reduced. Furthermore, various learning images can be easily prepared in the preprocessing.

 Brief Description of Drawings

 FIG. 1 is a block diagram showing an image conversion apparatus according to a first embodiment of the present invention.

 FIG. 2 is a diagram for explaining the geometric conditions and optical conditions of the illumination equation.

 FIG. 3 is a flowchart showing an image conversion method according to the first embodiment of the present invention.

 FIG. 4 is a diagram showing image feature analysis using wavelet transform.

FIG. 5 is a diagram showing an example of parameter operations for performing image conversion. FIG. 6 is a diagram showing an example of an edge portion formed by overlapping two objects.

 [FIG. 7] FIG. 7 is a diagram showing an example of a method for generating and canceling an error of a specular reflection component due to mixing of materials.

 [FIG. 8] FIG. 8 is a diagram showing an example of a method of error generation and error cancellation of diffuse reflection components due to mixing of materials.

 FIG. 9 is a diagram showing an example of a surface normal vector interpolation method for double enlargement.

FIG. 10 is a diagram showing an example of a method for generalizing (Equation 2) with respect to the image enlargement ratio.

FIG. 11 is a diagram showing an example of a method for generalizing (Equation 2) with respect to the image enlargement ratio. [12] Fig. 12 is a diagram showing illumination conversion as a second example of parameter operation for image conversion.

[13] FIG. 13 is a diagram showing the relationship between the position of the illumination and the illumination vector.

 [FIG. 14] FIG. 14 is a diagram showing the conversion of the diffuse reflection ratio of a specific material as a third example of the parameter operation for image conversion.

 FIG. 15 is a diagram illustrating an example of an interface for adjusting the diffuse reflection component ratio of the image conversion instruction unit 105.

 [FIG. 16] FIG. 16 is a diagram for converting the material of a specific pixel as a fourth example of the parameter operation for performing image conversion.

 FIG. 17 is a diagram showing an example of a material change interface that the image conversion instruction unit 105 has.

 FIG. 18 is a diagram showing image reduction as a fifth example of parameter operation for image conversion.

[19] FIG. 19 is a diagram showing a method of learning the relationship between image features and illumination equation parameters.

[20] FIG. 20 is a diagram for explaining the geometric condition and optical condition of the illumination equation.

FIG. 21 is a diagram showing another method for acquiring parameters according to image characteristics.

[22] FIG. 22 is a diagram showing a configuration using a personal computer as a first configuration example for realizing the present invention.

23] FIG. 23 shows a second configuration example for realizing the present invention, which is a server client system. It is a figure which shows the structure using this.

 FIG. 24 is a third configuration example for realizing the present invention, and shows a configuration using a camera-equipped mobile phone and a television set.

 Explanation of symbols

[0018] 10 parameter output section

 33 servers

 34 clients

 41 Mobile phone with camera (mobile device)

 45 Camera

 101 Image feature analysis unit

 102 Image feature vector database

 103 Lighting equation parameter database

 104 Parameter operation section

 105 Image conversion instruction section

 106 Parameter operation setting section

 107 Image generator

 ΠΝ Input image (first image)

 IINFV input image feature vector (first image feature vector)

 IINFVN input image feature vector number

 IINLEP original parameter value

 ICIS image conversion instruction signal

 LEPS parameter operation instruction signal

 IOUTLEP New parameter value

 IOUT output image (second image)

 LEP1 first parameter value

 IL learning image

 BEST MODE FOR CARRYING OUT THE INVENTION

In the first aspect of the present invention, the first image is converted into the second image by the first image. The image of the first image obtained in the first step is referred to by referring to the relationship between the image feature and the illumination equation parameter in the first step of performing image feature analysis. From the features, a second step of acquiring the value of the illumination equation parameter corresponding to this as an original parameter value, a third step of determining the operation content of the illumination equation parameter according to the instructed image conversion, and the original The parameter value is manipulated according to the operation content determined in the third step, and a fourth step for obtaining a new parameter value, and a fifth step for generating the second image based on the new parameter value. With a step.

 [0020] According to a second aspect of the present invention, there is provided the image conversion method according to the first aspect, wherein the image feature analysis in the first step is performed using a spatial frequency analysis.

 [0021] In a third aspect of the present invention, a pre-processing for learning the relationship between the image feature and the illumination equation parameter is provided, and the pre-processing sets the first parameter value as the value of the illumination equation parameter; A step of generating a learning image from the first parameter value; and a step of performing image feature analysis substantially equivalent to the first step for the learning image. Provided is a first mode image conversion method for storing in a database in association with a first parameter value.

 [0022] In the fourth aspect of the present invention, there is provided the image conversion method according to the third aspect, wherein the first parameter value is set assuming an illumination equation parameter when the first image is taken. The

 [0023] In the fifth aspect of the present invention, there is provided the image conversion method according to the first aspect, wherein the illumination equation represents a luminance in a viewpoint direction by a sum of a diffuse reflection component, a specular reflection component, and an ambient light component. To do.

 [0024] In the sixth aspect of the present invention, the illumination equation parameters include a surface normal vector, an illumination vector, a ratio of a diffuse reflection component to a specular reflection component, a reflectance of the diffuse reflection component, and a specular reflection component. An image conversion method according to a first aspect including at least one of reflectances is provided.

[0025] In the seventh aspect of the present invention, when the instructed image conversion is image enlargement, the third step includes a surface normal vector, an illumination vector, a diffuse reflection component, and a specular surface as the operation content of the illumination equation parameter. Ratio with reflection component, reflectance of diffuse reflection component, and specular reflection Provided is an image conversion method according to a first aspect which defines at least one densification of component reflectances.

 In the eighth aspect of the present invention, the relationship between the image feature and the illumination equation parameter is represented by a plurality of image feature vectors and a plurality of parameter values respectively associated with the image feature vectors, The second step includes selecting a predetermined number of image feature vectors similar to the first image feature vector representing the image features of the first image from the plurality of image feature vectors; and A step of obtaining a distance between each of the image feature vectors and the first image feature vector, and a parameter value corresponding to each of the predetermined number of image feature vectors obtained with respect to the image feature vector. And a step of calculating the original parameter value. The first aspect of the image conversion method is provided.

 [0027] In the ninth aspect of the present invention, an image feature analysis is performed on the input image, an image feature analysis unit that outputs a first image feature vector representing the image feature of the input image, and a plurality of image feature analysis units. And a plurality of parameter values corresponding to each of the image feature vectors of the illumination equation, and the original parameter value corresponding to the first image feature vector is received by receiving the first image feature vector. A parameter output unit to output, a parameter operation setting unit for determining the operation content of the illumination equation parameter according to the instructed image conversion, and the original parameter value output from the parameter output unit by the parameter operation setting unit. Operate according to the defined operation details and obtain a new parameter value based on the new parameter value output from the parameter operation unit and the parameter operation unit. To provide an image conversion apparatus and an image generation section that generates an image.

 In the tenth aspect of the present invention, the parameter output unit includes an image feature vector database that stores the plurality of image feature vectors and an illumination equation parameter database that stores the plurality of parameter values. An image conversion device according to the ninth aspect is provided.

[0029] In an eleventh aspect of the present invention, as a server client system that performs image conversion, a server having an image feature analysis unit, a parameter output unit, a meter operation setting unit, and a parameter operation unit according to the ninth aspect; A client having an image generation unit of The client provides the server to instruct the contents of image conversion.

 [0030] In a twelfth aspect of the present invention, an image feature analysis unit that performs image feature analysis on an image photographed by the camera and outputs a first image feature vector representing the image feature; An image feature vector database that stores a plurality of image feature vectors together with numbers, identifies an image feature vector similar to the first image feature vector, and outputs the number, and the image feature vector database Provide a portable device that transmits the number output from.

 [0031] In the thirteenth aspect of the present invention, as a program for causing a computer to execute a method of converting a first image into a second image, a first image analysis is performed on the first image. Referring to the relationship between the step and the image feature and the illumination equation parameter, the value of the illumination equation parameter corresponding to the image feature of the first image obtained in the first step is obtained as the original parameter value. A second step of determining the operation content of the illumination equation parameter according to the instructed image conversion, and operating the original parameter value according to the operation content determined in the third step; A fourth step of obtaining a new parameter value and a fifth step of generating the second image based on the new parameter value are executed on the computer. To provide a shall.

 Hereinafter, embodiments of the present invention will be described with reference to the drawings.

 [0033] (First embodiment)

FIG. 1 is a block diagram showing an image conversion apparatus according to the first embodiment of the present invention. In FIG. 1, an image feature analysis unit 101 performs an image feature analysis on an input image 、 and generates an input image feature vector IINFV. The image feature vector database 102 stores a plurality of image feature vectors, and the illumination equation parameter database 103 is associated with each image feature vector stored in the image feature vector database 102 for a predetermined illumination equation. Multiple meter values are stored. That is, the relationship between image features and lighting equation parameters is prepared. Then, the image feature vector database 102 and the illumination equation parameter database 103 output the value of the illumination equation parameter corresponding to the input image feature vector IINFV as the original parameter value IINLEP. Image feature vector database 102 and lighting equation parameter data The database 103 constitutes the parameter output unit 10.

The image conversion instruction unit 105 outputs, for example, the contents of image conversion instructed by an external force as an image conversion instruction signal ICIS. The parameter operation setting unit 106 determines the operation content of the illumination equation parameter according to the image conversion instructed by the image conversion instruction signal ICI S, and outputs this as the parameter operation instruction signal LEPS. The parameter operation unit 104 operates the original parameter value ΠΝ LEP according to the operation content instructed by the parameter operation instruction signal LEPS, and generates a new parameter value IOUTLEP. The image generation unit 107 calculates an illumination equation using the new parameter value IOUTLEP and generates an output image IOUT.

That is, the input image と し て as the first image is converted into an output image IOUT as the second image by the parameter conversion of the illumination equation.

[0036] Here, assuming the geometric condition and optical condition as shown in Fig. 2, the following illumination equation is used.

[Equation 1] = PJa + N (N · L) ^k + PS)

[0037] where Iv is the luminance in the viewpoint direction (viewpoint vector V), la is the luminance of the ambient light, a is the reflectance of the ambient light, Ii is the luminance of the illumination, vector N is the surface normal vector, and vector L is Dco is the solid angle of illumination, pd is the reflectance of the diffuse reflection component, is the reflectance of the specular reflection component, kd and ks are the ratio of the diffuse reflection component and the specular reflection component, kd + ks = l relationship. The viewpoint vector V coincides with the optical axis of the camera CAM and starts from the point of interest P on the object surface SF. Ambient light is light that enters the current attention point P on the object surface SF through multiple reflections and enters from the periphery, and corresponds to the bias component of the luminance Iv in the viewing direction (vector V). Irradiance at attention point P

 [Number 1A]

The light is incident from the illumination, and the diffuse reflection component is reflected by kd pd and the specular reflection component is reflected by ks ps. [0038] In the illumination equation parameter database 103 in FIG. 1, the surface normal vector N, illumination vector L, diffuse reflection component ratio kd, diffuse reflection component reflectance pd, specular reflection component reflectance ps, environment Seven types of light intensity Ia and ambient light reflectance pa are set. Note that the definitions of the illumination equations and the types of parameters according to the present invention are not limited to those shown here, and any illumination equations and parameters can be applied.

 FIG. 3 is a flowchart showing an operation of the image conversion apparatus of FIG. 1, ie, an image conversion method according to the present embodiment. Note that the image conversion method according to the present embodiment can be realized by causing a computer to execute a program for realizing the method.

 [0040] First, in the preprocessing SOO, the relationship between image features and illumination equation parameters is learned. Details of this preprocessing SOO will be described later. Here, it is assumed that an image feature vector database 102 and an illumination equation parameter database 103 as shown in FIG.

 [0041] In step S1, the image feature analysis unit 101 performs image feature analysis on the input image と し て as the first image. The image feature analysis here is performed using spatial frequency analysis such as wavelet transform as shown in Fig. 4, for example. At this time, the image features are represented by multiple resolution expressions. In the example shown in Fig. 4, the wavelet transform outputs HL, LH, HH, and LL are obtained for each of the n scalings, and these are combined for each layer, thereby obtaining the (3n + l) -dimensional vector as the first one. It is obtained as an input image feature vector IINFV as an image feature vector. Since the image feature vector IINFV is required for each pixel, the LL image is made the same size in each scale. Of course, the method of analyzing and expressing image features in the present invention is not limited to this, and any method can be applied.

[0042] Next, in step S2, the image feature vector database 102 and the illumination equation parameter database 103 obtained by learning in advance are referred to. The input image feature vector IINFV obtained in step S1 is used to determine the corresponding illumination equation parameter. The value is obtained as the original parameter value IINLEP. Here, first, the image feature vector database 102 selects the input image feature vector IINFV from the q stored image feature vectors. Recently, an image feature vector is selected, and the selected image feature vector number is output as an input image feature vector number IINFVN. The illumination equation parameter database 103 receives the input image feature vector number IINFVN, reads the corresponding parameter value, and outputs it as the original parameter value IINLEP.

 [0043] Next, in step S3, the parameter operation setting unit 106 determines the operation content of the illumination equation parameter in accordance with the instructed image conversion. In step S4, the parameter operation unit 104 force operates the original parameter value IINLEP obtained in step S2 in accordance with the operation content determined in step S3, and obtains a new parameter value IOUTLEP.

 [0044] Fig. 5 is a diagram showing an example of parameter operation, in which the original parameter value IINLEP and the new parameter value IOUTLEP are written and arranged for each pixel for one line. The seven parameters described above are defined for each pixel, and three of the ambient light luminance Ia, the ambient light reflectance pa, and the illumination vector L are common to each pixel. Since the diffuse reflection component ratio kd, the diffuse reflection component reflectance p d and the specular reflection component reflectance p s depend on the material of the object, a suffix indicating the type of material is attached. For the surface normal vector N of the original parameter value IINLEP, the first subscript indicates the type of material, and the second subscript indicates the difference in pixels within the same material.

Now, it is assumed that image conversion “image enlargement with double enlargement ratio” is instructed by the image conversion instruction signal ICIS. At this time, the parameter operation setting unit 106 replaces “image enlargement with double enlargement ratio” and “image conversion” with “double density of surface normal vector N” t parameter operation instruction. The signal LEPS is given to the parameter operation unit 104.

[0046] In accordance with the parameter operation instruction signal LEPS, the norameter operation unit 104 doubles the surface normal vector N twice. In other words, the number of pixels in the original parameter value IINLEP u force The number of pixels in the new parameter value IOUTLEP is 2u. Since parameters that depend on the material of the object do not depend on the resolution, the original parameter value IINLEP can be passed directly to the new parameter value IOUTLE P. Specifically, the diffuse reflection component ratio kd, The new parameter value IOUTLEP of the specular reflection component reflectivity ps is the same as the original parameter value IINLEP. On the other hand, since the normal vector N is a resolution-dependent parameter, the surface normal vector N of the new parameter value IOUTLEP represents the difference in pixels after densification. Therefore, the third subscript is memorized. At this time, the boundary of the material (for example, between pixel 2 and pixel 3 in the original parameter value IINLEP) is as shown in FIG. Since it is highly possible that an edge portion is caused by overlapping, it is desirable to maintain this boundary condition even after densification. Specifically, pixel 4 and pixel 5 of the new parameter value IOUTLEP are matched with pixel 2 and pixel 3 of the original parameter value IINLEP, respectively. That is, to maintain the boundary conditions,

 N = N and N = N

 1,2 1,2,2 2,1 2,1,1

And

 However, the border of the image sensor of the camera rarely matches the boundary of the two objects. In the case of Fig. 6, material 1 is mixed in part of pixel 3. This error can be offset by the normal vector N of IOUTLEP. That is, as shown in Fig. 7 (a), when the specular reflection component reflectivity P of material 1 is higher than the specular reflection component reflectivity p of material 2, the pixels 3 s, l s, 2 of IINLEP

Contains an error that increases the brightness, so the normal vector N is shown in Fig. 7 (b).

 The brightness is reduced by moving 2,1,1 away from the specular direction to cancel the error. On the other hand, if the specular reflection component reflectance P of material 1 is lower than the specular reflection component reflectance p of material 2, IINLEP image s, l s, 2

Element 3 contains an error that causes the brightness to become darker, so the normal vector is moved as shown in Fig. 7 (b) to Fig. 7 (a). On the other hand, the intensity of diffuse reflection is

[Number 1A]

As shown in, the brightness is determined by the inner product (N'L) of the normal vector 照明 and the illumination vector L, so it is not related to the direction of the line-of-sight vector V. Therefore, as shown in Fig. 8 (a), if the diffuse reflection component reflectivity P of material 1 is higher than the diffuse reflection component reflectivity p of material 2, then d, l d, 2 of IINLEP

Since pixel 3 contains an error that increases the brightness, as shown in Fig. 8 (b), the normal vector N is moved away from the illumination vector L and the brightness is reduced to cancel the error. Material 1 diffusion

2,1,1

If the reflection component reflectivity P is lower than the diffuse reflection component reflectivity p of material 2, then IINLEP d, l d, 2

Since pixel 3 contains an error that causes the brightness to become dark, the normal vector is moved as shown in Fig. 8 (b) to Fig. 8 (a). The balance between specular reflection and diffuse reflection is the diffuse reflection component ratio k. Therefore, control is possible.

 [0048] As shown in FIG. 6, the curved surface shape CSF included in the object 601 is discretized into a normal vector to form a polygonal line shape LF, which causes a quantization error. Therefore, for example, the pixel 3 of IINLEP is originally the surface 604 indicated by the force normal vector N having the intersecting edge 603 of the object 601 and the object 602 and the surface 605 indicated by the normal vector N.

 2,1 2,2

 Sharpness will deteriorate like Di 606. Therefore, the direction of normal vector N or N

 2,1 2,2 Adjust to cover the sharpness of edge 606. Therefore, the normal vector control has the effect of edge enhancement in addition to the correction of the luminance error due to the material mixing described in FIGS. 7 and 8, and the normal vector is determined by comprehensively considering the two effects. Control. An example of a normal vector control method is a method of referring to a database generated in advance by case learning. That is, normal vector data with a number of pixels twice different is measured using a plurality of sample subjects to obtain low-density normal vector data and high-density normal vector data. The relationship between low-density normal vector data and high-density normal vector data is stored in a database, and when edge enhancement is performed, high-density normal vector data is obtained from low-density normal vector data.

 [0049] Normal vectors N that are not located at material boundaries within the same material, for example, N and N

 1,1,2 1,2,1

As shown in Fig. 9, assuming that the surface shape is changing smoothly, it can be calculated by the following interpolation. For example, for N,

 1,1,2

 [Equation 2]

N =-N, | + -N

 "4 4. That is, the normal vector Ν of pixel 2 of IOUTLEP is the highest in IINLEP.

 1,1,2

 The normal vector N and N force located in the vicinity are also interpolated. The interpolation weights 3Z4 and 1Z4 are I

 1,1 1,2

 N to N distance and N to N measured in OUTLEP sub-pixel units

 The distance force of 1,1,2 1,1 1,1,2 1,2 is also calculated. In Fig. 9, the space density is doubled, so if the size of the IINLEP pixel is 1, the size of the IOUTLEP sub-pixel is 1Z4. Therefore, N

 The distance to 1,1 force N is 1Z4, the distance to N force N is 3Z4, and 1Z4 goes to N

, 2 1,1 1,1,2 1,2 1,2 and 3Z4 is the weight for N, and (Equation 2) is obtained. [0050] A method of generalizing (Equation 2) regarding the image magnification will be described with reference to FIGS. 10 and 11. FIG. FIG. 10 shows a case where IINLEP pixel u is enlarged five times and five pixels u, 1, u, 2, u, 3, u, 4, u, 5 are generated in IOUTLEP. The general rule of the image enlargement ratio is that the number of pixels of the IO UTLEP is arbitrary. Here, a method of generating α pixels in the UTLEP, where the image enlargement ratio is α, will be described. Therefore, although FIG. 10 is shown as α = 5, the following explanation is a derivation of an interpolation formula when the image enlargement ratio is arbitrary, and is a generalization of (Equation 2). In the interpolation calculation of (Equation 2), pixel 1 and pixel 2 were selected as neighboring pixels in IINL EP for pixel 2 of IOUTLEP. This is generalized in Figs. 10 and 11.

 [0051] Let IOUTLEP pixels to be calculated by interpolation calculation be images u, k (k = l, 2,..., Α). In FIG. 10, since it is illustrated as a = 5, the image u, k corresponds to five pixels of pixels u, 1, u, 2, u, 3, u, 4, u, 5. The normal vector in the vicinity of IINLEP of normal vector N of IOUTLEP pixels u and k is the combination of normal vectors N and N t, u, k t, u—l t, u

 The force is a combination of “normal vectors N and N”. IOUTLEP pixels u, k are II t t + l

 Since it is always included in NLEP pixel u, the normal vector N of IINLEP pixel u is always selected.

 t, u

 It is released. The second neighbor normal vector is IINLEP pixel u—l, u + 1 normal vector N t

, N, but this selection depends on whether IOUTLEP pixels u and k are IINLEP images.

, U-1 t, u + l

 It is determined by whether it is closer to the element u−l or pixel u + 1. In other words, as in the example of FIG. 10, when a / 2≥k, the pixels u and k of IOUTLEP are close to the pixel u-1 of IINLEP, so “normal vectors N and N” are selected. On the other hand, as in the example in Fig. 11, if α Ζ2 k, IOUTL t, u— 1 t, u

 Since the pixels u and k of EP are close to the pixel u + 1 of IINLEP, “normal vectors N and N” are selected.

 t t + l is selected.

[0052] The weight of the interpolation calculation is calculated by the size of the IOUTLEP sub-pixel, where the size of the IINLEP pixel is 1, as in FIG. One pixel of IINLEP is densified to α pixels by TL UTLEP according to the image magnification factor α. Therefore, the size of the IOUTLEP sub-pixel is ΐΖ (2 α). The boundary between IINLEP pixel u-l and pixel u is defined as boundary 801, and the boundary between IINLEP pixel u and pixel u + 1 is defined as boundary 802. Figure 10 Z2≥k, a = 5, and k = 2)), the distance between normal vector N and boundary 801 is the size of IINLEP sub-pixel That is 1Z2. The distance between boundary 801 and normal vector N is

 t, u, k

Since there are (2k−1) subpixels, (2k−1) Ζ (2 _α ). Furthermore, the distance between the normal line Ν and the normal vector Ν is the distance between the boundary 801 and the normal vector Ν, and the boundary 80 t, u, kt, ut, u

 It can be calculated by subtracting the distance between 1 and the normal vector N,

 t, u, k [a- (2k- 1)] Ζ (2α). From the above, the weight for the normal vector N is [α— (2k- 1)

 t, u_l] Z (2 a), normal vector Ν

 The weights for t and u are [lZ2 + (2k−l) Z (2Q;)], and the normal vector N is

 t, u, k

 It can be calculated by interpolation.

 [Equation 3]

N ,, _u , _k =

 ― A ~ 2k + l— a + 2k-\ —, a ^.

<^ N _IU k N _t , _u Νι, „, where —> k

 2a 2a 2

On the other hand, in the case of 11 (illustrated as a / 2 <k, a = 5, k = 5), the distance between the boundary 802 and the normal vector N is the size of the subpixel of IINLEP, and 1Z2 It becomes. Also, the normal solid t, U + l

 The distance between the line N and the normal vector N is (2k— 1— α) subpixels of IOUTLEP t, u t, u, k

 Since it is minutes, (2k-1−α) / (2α). Furthermore, the boundary 802 and the normal vector Ν

 The t, u, k distance is the distance force between boundary 802 and normal vector N. Normal vector N and normal vector N

 It can be calculated by subtracting the distances t, u t, u t, u, k, and it becomes [a-(2k-l-a)] / (2a). From the above, the weight for the normal vector N is [lZ2 + {Q ;-( 2k-l Q;

 t, U)} Z (2Q ；)], normal vector N

 The weight for t, U + l is (2k— 1 a) Ζ (2 α), and the normal vector 補 間 is interpolated as

 t, u, k

 It can be calculated by

 Picture

,

 The

FIG. 12 is a diagram showing illumination conversion as a second example of the parameter operation. The original parameter value IINLEP and the new parameter value IOUTLEP are written for each pixel for one line. The above seven parameters are defined for each pixel, and the ambient light intensity Ia and the ambient light reflectance. The pa and illumination vector L are common to each pixel. Since the diffuse reflection component ratio kd, the diffuse reflection component reflectance pd, and the specular reflection component reflectance ps depend on the material of the object, a suffix indicating the type of material is attached. For the surface normal vector N of the original parameter value IINLEP, the first subscript indicates the type of material, and the second subscript indicates the difference in pixels within the same material.

[0055] Now, as shown in FIG. 13, the image conversion instruction unit 105 receives the image conversion instruction signal ICIS to change the illumination position from (X, y, z) (ILLT1) to (x, y, z) (ILLT2). `` Convert to ''!

 LW1 LW2

 Assume that image conversion is instructed. In (Equation 1)

 [Number 1A]

As can be seen, the illuminance at the point of interest 上 の on the small surface PS of the subject surface increases as the angle between the surface normal vector Ν and the illumination vector L decreases, and the surface normal vector Ν and the illumination vector The larger the angle of L, the lower. Therefore, changing the position of the illumination changes the illuminance at the point of interest Ρ. In addition, as the relationship between the illumination vector L and the viewpoint vector V changes, the intensity of the specular reflection also changes.As explained in Fig. 7, the specular reflection intensity increases as the viewpoint vector approaches the regular reflection position of the illumination vector. Therefore, if the viewpoint vector moves away from the specular reflection position of the illumination vector, the specular reflection intensity decreases. In this way, the amount of light reflected to the viewpoint vector V changes by changing the position of the illumination, and the image recorded in the camera CAM changes.

[0056] The parameter operation setting unit 106 converts “the position of the illumination into (x, y, z) force (x, y, z).

 LW1 LW2

The image conversion instruction “” is replaced with a parameter operation “convert illumination vector L1 to illumination vector L2”, and given to parameter operation section 104 as parameter operation instruction signal LEPS. Here, the illumination vector L1 is given by (X−X, y−y, ζ−ζ) as a difference between the attention point P vector OW and the illumination position vector LW1. The illumination vector L2 is

 P LW1 P LW1 P LW1

 The difference between the eye point Ρvector OW and the illumination position vector LW2 (X-X

 P LW2, y —y

 P LW2, z —z

 P

 ).

 LW2

[0057] The meter operation unit 104 converts the illumination vector L1 into the illumination vector L2 in accordance with the parameter operation instruction signal LEPS. Note that the illumination vector L is at all positions on the subject. When the same parallel light assumption is used, if the illumination vector is calculated at the point of interest P, this can be applied to all pixels. On the other hand, in the case of a point light source where the illumination vector L is emitted from one point in all directions, the illumination vector L changes depending on the position of the attention point P. Therefore, the attention vector OW is replaced for each pixel. Therefore, it is necessary to calculate the illumination vector L.

 [0058] Fig. 14 shows the third example of parameter operation, where the diffuse reflection component ratio kd of a specific material is converted. The original parameter value IINLEP and the new parameter value IOUTLEP are written for each pixel for each line. Lined up. The seven parameters described above are defined for each pixel, and three of the ambient light luminance Ia, the ambient light reflectance pa, and the illumination vector L are common to each pixel. Since the diffuse reflection component ratio kd, the diffuse reflection component reflectance p d, and the specular reflection component reflectance p s depend on the material of the object, a suffix indicating the type of material is attached. Also for the surface normal vector N of the original parameter value IINLEP, the first subscript indicates the type of material, and the second subscript indicates the difference in pixels within the same material.

 FIG. 15 shows an interface 1501 for adjusting the diffuse reflection component ratio kd of the image conversion instruction unit 105. The operator 1502 inputs a conversion instruction to the image conversion instruction unit 105 using an input device 1504 such as a mouse while viewing the adjustment interface 1501 displayed on the display 1503. The operator 1502 moves the cursor 1506 to a material to change the diffuse reflection component ratio kd in the pre-conversion image 1505, and picks up the diffuse reflection component ratio kd of the material of the cursor 1506 by clicking the input device 1504 or the like. If this is pixel 1 of IINLEP in Fig. 14, k is picked up.

 d, l

 Respond. In FIG. 15, k = 0.25, and the position of 0.25 of slide no 1507 is shown.

 d, l

 A slider 1508 is displayed on the screen. As shown in Fig. 14, since IINLEP pixel 2 is made of the same material as pixel 1, its diffuse reflection component ratio k is also subject to conversion. That is, the same

 d, l

 The diffuse reflection component ratio kd force of all pixels belonging to the object 1509 in FIG.

 [0060] Subsequently, the operator 1502 moves the slider 1508 to the 0.75 position of the slide bar 1507.

Move to (shown as slide bar 1510) and change diffuse reflection component ratio kd to 0.75. The diffuse reflection component of object 1509 is increased, and in this example, it is shown in converted image 1511. As shown, object 1509 has been reduced in brightness and darkened. The change result of the diffuse reflection component ratio kd is displayed on the converted image 1511 as an output image IOUT by the method described later, and the operator 1502 evaluates the adjustment result of the diffuse reflection component ratio kd based on his own image production intention. If the adjustment is not enough, continue to move the slider 1510, check the converted image 1511 and repeat the series of operations. The conversion contents of the diffuse reflection component ratio kd set in the image conversion instruction unit 105 are set to “change the diffuse reflection component ratio k of material 1 to k ′” by the image conversion instruction signal ICIS, and parameter operation is performed. Setting part 1

 d, l d, l

 Reported to 06. As shown in (Equation 1), kd and ks, which represent the ratio of the diffuse reflection component and the specular reflection component, have the relationship kd + ks = l. So, if k is changed to k ', k'

 d, l d, l s, l is 1−k. The meter operation setting unit 106 reads “diffuse reflection component ratio of material 1 k d, l

 Change the image conversion instruction to “k ′” to “change the diffuse reflection component ratio k of material 1 to k′d, ld, ld, ld, l, and change the specular reflection component ratio k to l−k ′. Put it in the parameter operation

 s, l d, l

 Instead, it is given to the parameter operation unit 104 as a parameter operation instruction signal LEPS.

The parameter operation unit 104 sets k ′ to the diffuse reflection component ratio kd in (Equation 1) and sets the specular reflection component ratio ks to 1 k ′ in accordance with the parameter operation instruction signal LEPS. Picture

 d, l d, l

 The generation unit 107 calculates the illumination equation (Equation 1) using IOUTLEP in FIG. 14, and the result is the output image IOUT. In the example of FIG. 15, the diffuse reflection component ratio kd of a specific material is converted, but it is also possible to multiply the diffuse reflection component ratio kd of all the pixels by α (or a positive arbitrary constant). In this case, it is not necessary to specify the material with the cursor 1506, and the diffuse reflection component ratio kd is multiplied by α. If the diffuse reflection component ratio kd exceeds 1 as a result of multiplication by α, clipping is performed so that the constraint condition of kd + ks = l is satisfied.

[0062] Fig. 16 is a diagram of converting the material of a specific pixel as a fourth example of parameter operation. The original parameter value IINLEP and the new parameter value IOUTLEP are written and arranged for each pixel for one line. The seven parameters described above are defined for each pixel, and three of the ambient light luminance la, the ambient light reflectance pa, and the illumination vector L are common to each pixel. Since the diffuse reflection component ratio kd, diffuse reflection component reflectance pd, and specular reflection component reflectance ps depend on the material of the object, a suffix indicating the type of material is attached. Also, for the surface normal vector N of the original parameter value IINLE P, the first subscript indicates the type of material and the second subscript is Shows the difference in pixels within the same material.

 FIG. 17 shows a material change interface 1701 included in the image conversion instruction unit 105.

 The operator 1502 inputs a conversion instruction to the image conversion instruction unit 105 using the input device 1504 while viewing the material change interface 1 701 displayed on the display 1503. The material database 1702 stores material images and parameters. In FIG. 17, the images and parameters of material A to material D are read from the material database 1702 and displayed. The operator 1502 moves the cursor 1506 to the pixel whose material is to be changed in the pre-conversion image 1505, and determines the conversion target pixel by clicking the input device 1504 or the like. Since the parameters specific to the material are the diffuse reflection component ratio kd, the diffuse reflection component reflectance p d, and the specular reflection component reflectance p s, these three parameters are to be converted. Specifically, assuming that the position force of the cursor 1506 in FIG. 17 is the pixel 2 of IINLEP in FIG. 16, the (k p p) of the pixel 2 is the conversion target. The operator 1502 then d, l d, l s, l

 Specify the material of the conversion destination. In the example of FIG. 17, the radio button 1703 of material D is checked, and the pixel 2 of IOUTLEP is determined as (k p p) = (0.9.0.3.0.8). Material d, 3 d, 3 s, 3

 The quality D is a material having a lower luminance than the object 1509 and is converted into a material and a texture. The above operation is performed by the image conversion instructing unit 105, and “diffuse reflection component ratio kd, diffuse reflection component reflectance pd, specular reflection component reflectance ps of IOUTLEP pixel 2 is set to 0.9, 0.3, 0. .8 ”is transmitted to the parameter operation setting unit 106 by the image conversion instruction signal ICIS. In response to the parameter operation instruction signal LEPS, the parameter operation setting unit 106 sets the diffuse reflection component ratio kd, diffuse reflection component reflectance pd, and specular reflection component reflectance P s of IOUTLEP pixel 2 to 0.9, 0. The instruction “3, 0.8” is given to the parameter operation unit 104.

 [0064] In accordance with the parameter operation instruction signal LEPS, the meter operation unit 104 converts the diffuse reflection component ratio k of pixel 2 to the diffuse reflection component ratio k (= 0.9), and the diffuse reflection component reflectivities pd, ld. , 3

 Is converted to diffuse reflectance P (= 0.3), and specular reflectance p is converted to specular reflection d, ld, 3 s, l reflectance / o (= 0.8). To do. Then, the image generation unit 107 performs s, 3

To obtain the output image IOUT. In the above example, only one pixel is targeted for conversion. However, multiple pixels can be specified together, or an area can be specified on the image, and converted to a specific material. It is also possible. To extract another material from the material database 1702, use the “next material list” button 1704, and to register a new material in the material database 1702, use the “new registration” button 1705.

 FIG. 18 is a diagram showing image reduction as a fifth example of the parameter operation. The original parameter value IINLEP and the new parameter value IOUTLEP are written for each pixel for one line. The seven parameters described above are defined for each pixel, and three of the ambient light luminance Ia, the ambient light reflectance pa, and the illumination vector L are common to each pixel. Since the diffuse reflection component ratio kd, the diffuse reflection component reflectance p d, and the specular reflection component reflectance p s depend on the material of the object, a suffix indicating the type of material is attached. For the surface normal vector N of the original parameter value IINLEP, the first subscript indicates the type of material, and the second subscript indicates the difference in pixels within the same material.

 Assume that image conversion “reducing an image to 1/3” is instructed by the image conversion instruction signal ICIS. At this time, the parameter operation setting unit 106 replaces the image conversion instruction “reduce the image to 1Z3” with the parameter operation “obtains an average value for every three pixels of IINLEP and sets it as one pixel of IOUTL EP”. The parameter operation instruction signal LEPS is given to the parameter operation unit 104.

 [0067] In accordance with the parameter operation instruction signal LEPS, the parameter operation unit 104 obtains an average value for every three pixels of IINLEP and sets it as one pixel of IOUTLEP. For example, the surface normal vector of pixel 1 of IOUTLEP is the surface normal vector N of pixel 1, pixel 2, and pixel 3 of IINLEP.

 1,

, N, N (N + N + N) Z3. Similarly, enlargement of IOUTLEP pixel 1

1 1,2 2,1 1,1 1,2 2,1

 The ratio of diffuse reflection components is the ratio of diffuse reflection components of IINLEP pixel 1, pixel 2, and pixel 3 k d, l

, K and k (2k + k) Z3. Anti-d, l d, 2 d, l d, 2 of diffuse reflection component of pixel 1 of IOUTLEP

 The emissivity is the average (2 P + p) Z3 of the reflectances p, ρ, pd, ld, ld, 2 of the diffuse reflection components of pixels 1, 2 and 3 of IINLEP. The reflectance of the specular reflection component of pixel 1 of IOUTLEP is d, l d, 2

The average of the reflectances p and _{P 3} ρ of the specular reflection components of pixel 1, pixel 2, and pixel 3 of IINLEP (s, ls, ls, 2

 2 p + p) Z3.

 s, l s, 2

Note that the present invention does not limit the image reduction method, and any method can be applied. For example, low-pass filter is applied to each parameter image of IINLEP, A sub-sampling method can be used.

 [0069] The present invention does not limit the method of operating the parameters of the illumination equation. Therefore, it is possible to manipulate any parameter of the lighting scheme by any method, and any parameter can be manipulated by any method, as in the example of the parameter manipulation method described with reference to FIGS. The geometrical characteristics such as the surface normal vector are illustrated one-dimensionally in FIG. 2 and the like for convenience of explanation, but are originally three-dimensional information, and the present invention is not limited to one dimension. The interpolation methods shown in (Equation 2), (Equation 3), and (Equation 4) are also examples, and any method can be applied to increase the spatial density. In FIG. 5, the image enlargement is explained by increasing the density of the surface normal vector N. However, this is an example, and the present invention does not limit the illumination equation parameter to be operated and the operation method thereof. Spatial densification can be achieved using any combination of parameters, such as increasing the density kd or increasing the density of the surface normal vector N and diffuse reflection component ratio kd. Also, the illumination parameters to be subjected to image conversion are arbitrary, and the combination thereof is also arbitrary. 5 to 18 show one line in the image, but if this is applied in the vertical and horizontal directions, it can be expanded into two-dimensional image processing.

 [0070] Then, in step S5, the image generation unit 107 generates an output image IOUT as a second image based on the new parameter value IOUTLEP obtained in step S4.

 Here, details of the preprocessing SOO will be described with reference to FIG. Here, the image feature vector is associated with the illumination equation parameters by using the image created from the illumination equation for learning the image feature vector.

[0072] Specifically, first, a first parameter value LEP1 (number 1) is set as the value of the illumination equation parameter. Then, using the first parameter value LEP1, the calculation of (Equation 1) is performed to generate a learning image IL. Then, the generated learning image IL is subjected to image feature analysis substantially equivalent to step S1 described above to obtain an image feature vector ILFV. This image feature vector ILFV is stored in number 1 of the image feature vector database 102. As a result, the image feature vector ILFV and the first parameter value LEP1 are associated with each other and stored in the databases 102 and 103. By repeatedly executing such processing, an image feature vector database 102 and an illumination equation parameter database 103 as shown in FIG. 1 are generated. In FIG. 19, the image generation unit 107 generates an image. The image feature analysis unit 101 performs image feature analysis, but image generation and image feature analysis may be performed by other means in the pre-processing S00.

 As described above, according to the present embodiment, the illumination equation parameter IINLEP suitable for the input image ΠΝ is selected from the image feature vector IINFV of the input image ΠΝ. By manipulating this parameter IINLEP, various output images IOUT can be generated. Therefore, image conversion with a high degree of freedom is possible without being limited to image data at the time of learning.

 [0074] Further, since image conversion is performed by manipulating the illumination equation parameter IINLEP, it is not necessary to prepare a large number of learning images. Therefore, the number of learning images can be reduced. In the preprocessing, the learning image IL can be generated by a computer using an illumination equation, so that it is not necessary to shoot with the real object for generating the learning image. Therefore, the processing is simplified and various learning images can be easily prepared. As the image feature ILFV of the learning image IL is closer to the image feature IINFV of the input image ΠΝ, an appropriate illumination equation parameter can be acquired. Therefore, when generating the learning image IL, it is desirable to set the illumination equation parameters assuming the conditions when the input image ΠΝ was taken. For example, when the shooting location of the input image 撮 影 can be limited, and as a result, the illumination position can be limited, the illumination scale L uses the data when the input image ΠΝ was taken.

 In the present embodiment, image enlargement has been described as an example of image conversion. However, the present invention is not limited to this, and parameter operations can be similarly performed for other image conversions. For example, if you want to change the illumination direction, you can change the illumination vector L, and if you want to change the ratio of the diffuse reflection component to the specular reflection component, change the diffuse reflection component ratio kd.

[0076] Further, since the specular reflection component reflectance ps is defined by bidirectional reflectance, it changes according to the line-of-sight direction. Therefore, for example, if the Cook-Torrance model given by (Equation 5) is introduced, the line-of-sight vector V, roughness coefficient m, and Fresnel coefficient F can be added to the illumination equation parameters. This enables image conversion such as changing the viewpoint direction and changing the surface roughness. [Equation 5]

g ² = n ² + c ^2- \

 C = (L- H)

Here, as shown in FIG. 20, vector H is an intermediate vector between viewpoint vector V and illumination vector L, and β represents an angle between intermediate vector 中間 and surface normal vector Ν. m is a coefficient representing the roughness of the surface of the object. When m is small, the angle | 8 is small, that is, the surface normal vector N shows strong reflection, and when m is large, the angle | 8 is large, that is, the surface. The reflection distribution also spreads away from the normal vector N. G is the geometric attenuation factor, and represents the effect of shading due to the unevenness of the object surface. n is a refractive index.

 Thus, in the present invention, the illumination equation can be arbitrarily defined and is not limited to (Equation 1) or (Equation 5).

In this embodiment, the illumination equation parameter corresponding to the image feature vector closest to the image feature vector IINFV is acquired as the original parameter value IINLEP. However, the method for acquiring the original parameter value IINLEP is as follows. It is not limited to. For example, as shown in FIG. That is, first, a predetermined number (three in FIG. 21) of image feature vectors similar to the input image feature vector IINFV are selected. For each selected image feature vector, the distance from the input image feature vector IINFV is obtained, and a weighting coefficient IINFVWF corresponding to this distance is determined. The parameter values respectively corresponding to the three selected image feature vectors are weighted and added using the weighting coefficient IINFVWF, and the result is output as the original parameter value IINLEP. Instead of weighted calorie calculation, parameter values corresponding to the selected predetermined number of image feature vectors, respectively. I can't help simply by taking the average.

 [0080] Note that the relationship between the image feature and the illumination equation parameter may be prepared by any means that does not necessarily have to be learned in advance.

 Hereinafter, a configuration example for realizing the present invention will be exemplified.

 [0082] (First configuration example)

 FIG. 22 is a diagram showing a first configuration example, which is an example of a configuration for performing image conversion according to the present invention using a personal computer. In order to take full advantage of the display capability of the display 22 whose resolution is lower than that of the display 22, an enlarged image is created by an image conversion program loaded into the main memory 23. The low resolution image captured by the camera 21 is recorded in the image memory 24. In the external recording device 25, an image feature vector database 102 and an illumination equation parameter database 103 are prepared in advance and can be referred to by an image conversion program module in the main memory 23. The operation of the image conversion program, the contents of the image feature vector database 102 and the illumination equation parameter database 103, the creation method, and the like are as described in the first embodiment. The image conversion program in the main memory 23 reads the low-resolution image in the image memory 24 via the memory bus 26, converts it into a high-resolution image in accordance with the resolution of the display 22, and then the video memory via the memory bus 26 again. Forward to 27. The high resolution image transferred to the video memory 27 can be observed on the display 22.

 It should be noted that the present invention can take various configurations that are not restricted by the configuration of FIG. For example, it is possible to acquire the external storage device power connected to another personal computer via the network 28, even if the image feature vector database 102 and the illumination equation parameter database 103 are acquired. The low resolution image may be acquired via the network 28.

 [0084] (Second configuration example)

FIG. 23 is a diagram showing a second configuration example, which is an example of a configuration for performing image conversion according to the present invention using a server client system. The resolution of the camera 31 is lower than that of the display 32. In order to make the best use of the display capability of the display 32, image conversion is executed in the server client system. In the server 33, the first embodiment Similarly to the above, the image feature analysis unit 101, the image feature vector database 102, and the illumination equation equation parameter database 103 also calculate the original parameter value IINLEP for the input image repulsion. The image feature vector database 102 and the illumination equation parameter database 103 constitute a parameter output unit 10. On the other hand, an image conversion instruction (image enlargement in this example) is passed from the image conversion instruction unit 105 of the client 34 to the parameter operation setting unit 106 of the server 33 as an image conversion instruction signal ICIS. The parameter operation setting unit 106 replaces the content of the image conversion by the image conversion instruction signal ICIS with the operation content of the illumination equation parameter, and outputs it to the parameter operation unit 104 as the parameter operation instruction signal LEPS. The parameter operation unit 104 operates the original parameter value IINLEP to generate a new parameter value IOUTLEP.

 With this operation, the server 33 can provide the client 34 with the new parameter value IOUTLEP according to the image conversion instruction from the client 34 via the network 35. In the client 34 that has received the new parameter value IOUTLEP, the image generation unit 107 generates an enlarged image and supplies it to the display 32.

 It should be noted that the present invention is not limited to the configuration shown in FIG. 23, the combination of image devices, the position of each means on the system (the force belonging to the force client 34 belonging to the server 33, or other Whether it belongs or not) is arbitrary.

 [0087] (Third configuration example)

 FIG. 24 is a diagram showing a third configuration example, which is an example of a configuration for performing image processing according to the present invention using a camera-equipped mobile phone and a television. A camera-equipped mobile phone 41 as a portable device can send image data to the television 44 via the network 42 or the memory card 43. The camera 45 of the camera-equipped mobile phone 41 has a resolution lower than that of the television 44. In order to take full advantage of the display capability of the television 44, the image is converted by the image conversion device according to the present invention installed in the internal circuit of the television 44. Perform magnification.

[0088] Considering the usage fee, the smaller the amount of data sent from the camera-equipped mobile phone 41 to the television 44, the more user power is welcomed. Therefore, it is desirable that the data transmitted through the network 42 be the input image feature vector number IINFVN. Also, in order to minimize the damage caused by eavesdropping on the network 42, the number itself has no special meaning. It is desirable to send the collection vector number IINFVN. That is, the camera-equipped mobile phone 41 has the image feature vector database 102, and the television 44 has the illumination equation parameter database, so that the desired image conversion can be realized only when both are provided. As a result, usage fees can be kept low, and damage from eavesdropping can be minimized. In FIG. 24, the camera-equipped mobile phone 41 performs image feature analysis on the camera 45 and the image 撮 影 taken by the camera 45 and outputs the first image feature vector IINFV. The image feature vector database that identifies the image feature vector similar to the first image feature vector IINFV and outputs its number IINFVN. 102 is shown.

 It should be noted that the present invention can take various configurations other than the configuration shown in FIG. For example, the camera-equipped mobile phone 41 may be a digital still camera or a video movie camera.

 [0090] As described above, the present invention can be executed on a wide variety of video devices such as personal computers, server client systems, camera-equipped mobile phones, digital still cameras, video movie cameras, and televisions that are widely used. No special equipment, operation or management is required. It should be noted that the device connection form and the internal configuration of the device, such as mounting on dedicated hardware or a combination of software and hardware, are not constrained.

 Industrial applicability

[0091] In the present invention, various image conversions such as enlargement / reduction, illumination conversion, viewpoint conversion, and change of the ratio of the diffuse Z specular reflection component can be freely performed. For example, sports, sightseeing, commemorative photography, etc. It can be used in the field of video entertainment that records these scenes as video. In the field of culture and art, it can be used to provide a highly flexible digital archiving system that is not limited by subject or shooting location.

Claims

The scope of the claims

 [1] A method for converting a first image into a second image,

 A first step of performing image feature analysis on the first image;

 A second parameter is obtained by referring to the relationship between the image feature and the illumination equation parameter, and obtaining the value of the illumination equation parameter corresponding to the image feature of the first image obtained in the first step as an original parameter value. And the steps

 A third step for determining the operation content of the illumination equation parameter according to the instructed image conversion;

 A fourth step of operating the original parameter value in accordance with the operation content determined in the third step to obtain a new parameter value;

 And a fifth step of generating the second image based on the new parameter value.

 An image conversion method characterized by that.

[2] In claim 1,

 The image feature analysis in the first step is performed using spatial frequency analysis.

 An image conversion method characterized by that.

[3] In claim 1,

 With preprocessing to learn the relationship between image features and lighting equation parameters,

 The pretreatment includes

 Setting a first parameter value as a value of the illumination equation parameter; generating a learning image from the first parameter value;

 A step of performing an image feature analysis substantially equivalent to the first step for the learning image,

 The obtained image feature is stored in the database in association with the first parameter value.

 An image conversion method characterized by that.

[4] In claim 3, The first parameter value is set assuming an illumination equation parameter when the first image is taken.

 An image conversion method characterized by that.

 [5] In claim 1,

 The illumination equation represents the luminance in the viewing direction by the sum of the diffuse reflection component, the specular reflection component, and the ambient light component.

 An image conversion method characterized by that.

[6] In claim 1,

 The illumination equation parameter includes at least one of a surface normal vector, an illumination vector, a ratio between a diffuse reflection component and a specular reflection component, a reflectance of a diffuse reflection component, and a reflectance of a specular reflection component.

 An image conversion method characterized by that.

[7] In claim 1,

 In the third step, when the instructed image conversion is image enlargement, the operation content of the illumination equation parameter includes the surface normal vector, the illumination vector, the ratio between the diffuse reflection component and the specular reflection component, and the diffuse reflection component. It defines at least one densification among the reflectance of the component and the reflectance of the specular reflection component.

 An image conversion method characterized by that.

[8] In claim 1,

 The relationship between the image feature and the illumination equation parameter is represented by a plurality of image feature vectors and a plurality of parameter values respectively associated with each image feature vector.

 Selecting a predetermined number of image feature vectors similar to the first image feature vector representing the image features of the first image from the plurality of image feature vectors; Then, obtaining each distance from the first image feature vector,

Parameter values respectively corresponding to the predetermined number of image feature vectors are assigned to the image feature vectors. And calculating the original parameter value by weighting and adding to the collection vector according to the distance obtained in the step

 An image conversion method characterized by that.

 [9] An image feature analysis unit that performs image feature analysis on the input image and outputs a first image feature vector representing the image feature of the input image;

 It stores a plurality of image feature vectors and a plurality of parameter values corresponding to each of the image feature vectors of the illumination equation, receives the first image feature vector, and corresponds to the original parameter value A parameter output unit for outputting

 A parameter operation setting unit that determines the operation content of the illumination equation parameter according to the instructed image conversion;

 Parameter output unit force A parameter operation unit for operating the output original parameter value according to the operation content determined by the parameter operation setting unit and obtaining a new parameter value;

 An image generation unit for generating an output image based on the new parameter value output from the parameter operation unit.

 An image conversion apparatus characterized by that.

[10] In claim 9,

 The parameter output unit includes:

 An image feature vector database for storing the plurality of image feature vectors; and an illumination equation parameter database for storing the plurality of parameter values.

 An image conversion apparatus characterized by that.

[11] A server client system for image conversion,

 A server having the image feature analysis unit, the parameter output unit, the parameter operation setting unit, and the parameter operation unit according to claim 9;

 A client having the image generation unit of claim 9,

 The client instructs the contents of image conversion to the server

Server client system characterized by that.

[12] Camera and

 An image feature analysis unit that performs image feature analysis on an image captured by the camera and outputs a first image feature vector representing the image feature;

 A plurality of image feature vectors are stored together with numbers, an image feature vector similar to the first image feature vector is specified, and an image feature vector database for outputting the numbers is provided.

 The number output from the image feature vector database is transmitted.

 A portable device characterized by that.

[13] A program for causing a computer to execute a method of converting a first image into a second image,

 A first step of performing image feature analysis on the first image;

 A second parameter is obtained by referring to the relationship between the image feature and the illumination equation parameter, and obtaining the value of the illumination equation parameter corresponding to the image feature of the first image obtained in the first step as an original parameter value. And the steps

 A third step for determining the operation content of the illumination equation parameter according to the instructed image conversion;

 A fourth step of operating the original parameter value in accordance with the operation content determined in the third step to obtain a new parameter value;

 A program for causing a computer to execute a fifth step of generating the second image based on the new parameter value.