JPH11265440A - Image compositing method for optional light source position and display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To synthesize an image against an optional light source position only from the data that can be easily acquired with no man power nor any special mechanical equipment required by extracting the information on the surface direction and the color of a photographing object from plural sheets of original images which are photographed at different positions of a light source and composting an image from those extracted information. SOLUTION: The images of the same object are photographed at different positions of a light source. The parameters serving as the coefficients which derive the original color and tilt, etc., of the surface of the object from the color changes of those photographed images occurred to the positions of the light source, i.e., the scattering changes of the light reflected on the surface of the object and also derive the colors which are actually obseved at the light source positions. Thus, an image is composited against an optional light source position based on those parameters. In this system, a processor serves as a CPU or an PMU, for example, and executes a parameter extraction process 300 and an image compositing process 400. An input device serves as a mouse or a touch panel, etc., for example, and receives the input of a direction vector 120 of a light source position for the image to be composited.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to image synthesis for performing a calculation on a plurality of images to generate a new image.

[0002]

2. Description of the Related Art Conventionally, as a method of obtaining an image obtained by irradiating a real three-dimensional object with light from an arbitrary position, a method described in Masayuki Nakajima and Masanobu Yamamoto, "Graphics and Vision-Digital Video Processing-(published in 1996) (Ohm Co., Ltd.) p.9
As shown in 7-p.104, three of the feature points on the object surface
Generally, a 3D geometric model is constructed by measuring the dimensional coordinates, pasting a texture image of the object surface on a polyhedron with those points as vertices, and then rendering by ray tracing, etc. with the light source position given Is known.

[0003]

The conventional method described in the preceding section has a difficulty in that the measurement of the three-dimensional coordinates of the surface of the object requires humans and special equipment. Also, rendering taking into account the light source position of a 3D object requires a lot of computation,
Due to the long processing time required for image synthesis, there is also difficulty in performing processing in real time.

[0004] The object of the present invention is to obtain 3D data from only easily available data such as an image of an object taken by a camera or the like.
An object of the present invention is to provide a method for synthesizing an image at an arbitrary light source position without requiring any manual operation or special equipment as in the case of constructing a three-dimensional geometric model.

[0005]

Means for Solving the Problems To solve the above-mentioned problems, the following method is adopted. As a simple synthesis method, it is conceivable to take a weighted average for each pixel of the image data in which the light source is placed on the left and right, but since the relationship between the light source position and the synthesized image is not clear because it is not based on a model, the light source position It is hard to say that it reflects the change correctly. Masayuki Nakajima and Masanobu Yamamoto, `` Graphics and Vision-Digital Image Processing-(Published in 1996, Ohm Co., Ltd.) p.4
As shown in 0-p.43, in general, the original color and normal of the object surface and the direction of the light source and viewpoint from the object surface,
There is a certain relationship between the actually observed colors. Therefore, in the present invention, an image for an arbitrary light source position is synthesized using this relationship.

In Japanese Patent Application Laid-Open No. 1-143127, a method for measuring the surface of an object by mounting a plurality of electron detectors on an electron microscope and measuring the scattering in each direction of the electrons emitted from a certain direction and reflected on the surface of the object is measured. I want a line. In the same manner, in the present invention, images of the same object with light sources placed at a plurality of different positions are taken, and the change in their color with respect to the light source position, that is, the change in scattering of light reflected on the object surface, Move the light source freely by measuring parameters such as the original color and inclination of the surface, which are the coefficients for deriving the color actually observed from the light source position, and using this to synthesize an image for any light source position Provided is an image synthesizing method capable of observing an object while causing it to move.

[0007]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a first embodiment of the present invention will be described in detail. FIG. 1 is a diagram showing a main processing procedure of the present invention and its input / output data. FIG. 2 is a diagram illustrating a hardware system configuration according to the first embodiment. FIG. 3 is a flowchart showing a processing procedure of the parameter extraction processing 300. FIG. 4 illustrates an input original image group and an output composite image.

As shown in FIG. 2, the hardware system configuration of the first embodiment includes a display device 201, a processing device 202, an input device 203, a storage device 204, and an external storage device 205, and a bus 206 serving as a data transfer path. Connect to and configure. Display device 201
Is, for example, a CRT display, and displays the composite image 140, instructions to the user, and the like. The processing device 202 is, for example, a CPU, an MPU, or the like, and executes a parameter extraction process 300 and an image synthesis process 400. The input device 203 is, for example, a mouse or a touch panel, and receives an input of a direction vector 120 of a light source position in an image to be combined. Storage device 2
Reference numeral 04 denotes, for example, a memory or the like, which is a light source movement parameter data 130, a light source direction matrix L 311 used for the composite image 140 and the parameter extraction processing 300, and a parameter calculation matrix C 3.
Store 12 temporarily. The external storage device 205 is, for example, an HDD
Such as a storage device such as an FD or a CD-ROM, and an external storage device of another computer connected to a network or the like.
0, a program 211 storing a parameter extraction process 300 and an image synthesis process 400 is stored. The program 211 is read out to the processing device 202 when executing the process.

FIG. 1 shows a processing procedure of the present embodiment.
It is described using. The input data of the present invention includes a plurality of original images IM [m] 101 (1 ≦ m) of the same size, which are photographed while the positional relationship between the camera and the target object is relatively fixed and only the light source position with respect to them is changed. ≦ M) original image group 100 (each original image I
In the set of M [m] 101, points at the same position when viewed from the upper left corner of each image indicate the same position on the surface of the target object), and the light source from the center of the object in each original image IM [m] 101. Vector of length 1 indicating the three-dimensional direction to the position LV [m] = [LX
A light source direction vector file 110 storing [m], LY [m], LZ [m]] 111, and a length 1 vector lv = [lx, ly, lz indicating the three-dimensional direction of the light source position in the image to be synthesized ] 120.
Here, the constant M indicates that the number of images in the original image group 100 is M. Hereinafter, the parameters expressed in uppercase letters, including those with subscripts, represent constants, and the parameters expressed in lowercase letters represent variables. Each original image IM [m] 101 is a digital image. A digital image is obtained by dividing an image into small discrete points called pixels, and expressing the luminance value of each pixel by a discrete value. Each pixel in the digital image is represented by a two-dimensional vector [x, y] (1 ≦ x ≦ X, 1 ≦
y ≦ Y). Here, the constants X and Y are expressed as follows:
Pixel, the vertical axis represents the size of the Y pixel. Pixel [x, y]
Indicates the xth pixel from the left and the yth pixel from the top in the image.

The parameter extraction process 300 is a process for obtaining, from the original image group 100 and the light source direction vector file 110, light source movement parameter data 130 used for synthesizing an image at an arbitrary light source position.

The light source moving parameter data 130 is a four-dimensional vector k used for synthesizing an image at an arbitrary light source position corresponding to each pixel [x, y] of the original image group 100.
It is an array of Y rows and X columns having [x, y] = [kdx [x, y], kdy [x, y], kdz [x, y], ka [x, y]] as elements.

The image synthesizing process 400 synthesizes an image corresponding to the light source position indicated by the vector 120 from the light source moving parameter data 130 and the vector lv [lx, ly, lz] 120 indicating the light source direction in the synthesized image, and generates a synthesized image 140 Is output.

A synthetic image 140, which is output data, is a digital image having X pixels on the horizontal axis and Y pixels on the vertical axis, similarly to each original image IM [m] 101.

Hereinafter, the parameter extraction process 300 will be described with reference to FIG.
This will be described in detail. In processing step 301, from the light source direction vector file 110 on the external storage device 205,
Unit vector representing the light source direction for each original image IM [m] 101
LV [m] = [LX [m], LY [m], LZ [m]] 111 are sequentially read, and a light source direction matrix L 311 of M rows and 4 columns shown in Expression 1 is constructed and written to the storage device 204.

[0015]

(Equation 1)

In processing step 302, light source direction matrix L
311 is read from the storage device 204 and the operation shown in Expression 2 is performed to obtain a parameter calculation matrix C 312 of 4 rows and M columns, and is written to the storage device 204.

[0017]

(Equation 2)

Here, LT is a matrix of 4 rows and M columns in which the elements at row r and column c of L are arranged at row c and column r (this is called the transposed matrix of L). The product of an M-column matrix LT and an M-by-4 matrix L (4 rows
INV (LT · L) represents an inverse matrix of a matrix (LT · L) having the same number of rows and columns. The inverse matrix of the matrix A having the same number of rows and columns is the matrix B in which A · B = I (I is a matrix in which A · I = A for an arbitrary matrix A).

In processing step 303, the parameter calculation matrix C 312 is read from the storage device 204, and the original image group 100 is read from the external storage device 205.
A vector IV [x, y] = [I [1] [x, y], I [2] [x, y] whose element is the luminance I [m] [x, y] observed in IM [m] 100 ],…, I [m] [x,
y]], and a parameter vector k [x, y] = [kdx [x, y], kdy [x, y], kdz [x,
y], ka [x, y]] are calculated by the equation (3), and k [x, y] is stored in the light source movement parameter data 130.

[0020]

(Equation 3)

Here, k [x, y] and IV [x, y] are column vectors (one-column matrix). Hereinafter, when performing an operation related to a matrix and a vector, the vector is a column vector. This operation is repeated for all the pixels [x, y] (1 ≦ x ≦ X, 1 ≦ y ≦ Y).

The basis of these calculation processes from processing steps 301 to 303 is shown as follows.

In this embodiment, it is assumed that specular reflection does not occur on the surface of the object, and the normal vector of the surface of the object, the coefficient of color by diffuse reflection light, and the luminance of color by background light are used as parameters. The light source is considered to be a parallel light source at a sufficiently far distance from the object, and the direction to the light source does not change at any position on the surface of the object. The direction of the light source is the direction from the object center to the light source position.

When there is no specular reflection component and there is background light, the luminance value i [x, y] at each pixel [x, y] is represented by kd [x, y] being the coefficient of the diffuse reflection component and ka [x , y] is the luminance value from the background light, θ
Is the angle between the normal direction of the object surface and the direction from the center of the object to the light source.

[0025]

(Equation 4)

Cos θ is a vector of length 1 indicating the light source direction
lv = [lx, ly, lz] and a vector of length 1 n [x,
y] = [nx [x, y], ny [x, y], nz [x, y]]
Equation 5 is obtained.

[0027]

(Equation 5)

In equation 5, a vector [kdx [x, y] kdy [x, y] kd whose direction is newly equal to n [x, y] and whose size is equal to kd [x, y]
When z [x, y]] is introduced, it becomes Equation 6.

[0029]

(Equation 6)

When the parameter vector k [x, y] introduced in the processing step 303 is used for Expression 6, Expression 7 is obtained.

[0031]

(Equation 7)

Formula 7 is summarized as shown in Formula 8 for all original images IM [m] 101.

[0033]

(Equation 8)

Using the vector IV [x, y] and the matrix L used in the processing steps 302 and 303, they can be summarized into the following equation (9).

[0035]

(Equation 9)

Equation 9 is a system of linear equations relating to the parameter of k [x, y]. By solving this equation, an image for an arbitrary light source position can be synthesized. If the luminance does not exactly follow the model shown in Equation 4, there may be no solution to this equation. Therefore, using the least squares method, the luminance value i [x, y] in the original image and the calculation result lv · k [x,
The parameter value is determined so that the error from [y] is minimized.

To apply the least-squares method to a system of equations in matrix representation, multiply the transposed matrix of the coefficient matrix,
What is necessary is just to find and solve the inverse matrix of the coefficient matrix related to the parameter vector. Therefore, first, as shown in Equation 10, multiply the both sides of Equation 9 by the transposed matrix LT of L,

[0038]

(Equation 10)

Next, the inverse matrix INV (L
T ・ L), as shown in Equation 11,

[0040]

[Equation 11]

## EQU1 ##

From equation 11 (equal to equation 3), the vector IV [x, y]
And the matrix C, a parameter vector k [x, y] is obtained.

Hereinafter, the image synthesizing process 400 will be described in detail. In the image synthesizing process 400, a length 1 vector lv [lx, ly, lz] 120 indicating the direction from the object to the light source position in the synthesized image is read from the input device 203 and stored in the light source moving parameter data 130. Parameter vector k
Using [x, y], the luminance i at the pixel [x, y] of the composite image 140
[x, y] is obtained as shown in Expression 12, and written to the storage device 204.

[0044]

(Equation 12)

This is repeated for all pixels [x, y] (1 ≦ x ≦ X, 1 ≦ y ≦ Y) to obtain a composite image 140.

In this embodiment, the case where each pixel of each original image 101 holds only luminance, that is, the case where the image is a single color has been described. Parameter extraction processing for each color 30
By performing 0 and the image synthesis processing 400, the image synthesis processing can be performed. At this time, by assigning different weights to each primary color, it is possible to irradiate light of a color different from the light source at the time of shooting.

When only original image groups photographed under certain conditions, such as when the light source position at the time of photographing is on a horizontal plane or a vertical plane passing through the center of the target object, is used, parameter calculation is performed in advance based on the conditions. By formulating the matrix C 312 and performing only the processing step 303 in the parameter extraction processing 300, the speed of the parameter extraction processing 300 can be improved.

Further, when many image synthesizing processes are performed on the same original image group, the parameter data 130 obtained in the parameter extracting process 300 is temporarily output to the external storage device 205 as an intermediate file, and the intermediate After loading the file into the storage device 204, the image combining process 400
The processing speed can be improved by performing only the processing.

In this embodiment, it is assumed that the light source is sufficiently far from the object and does not change its direction to the light source at any position on the object surface.
By correcting the directions of the light source direction vectors 111 and 120 according to the positions of the pixels and executing the parameter extraction processing 300 and the image synthesis processing 400, it is possible to perform image synthesis in consideration of the distance from the object to the light source.

Hereinafter, a second embodiment of the present invention will be described in detail. In the configuration of the second embodiment, as shown in FIG. 5, a target object 501 is
A camera 503 is mounted on a tripod 504 and fixed within a range where 1 can capture an image, and a plurality of light source groups 505 are arranged and fixed around a target object 501. The shutter of the camera 503 is linked with the power switch of the light source group 505, and each time the shutter is pressed, one or more light sources in the light source group 505 are turned on in a predetermined order, and an image is taken. As a result, a plurality of images different from the target object 501 only in the position of the light source are obtained.

In the present embodiment, the operation of the power switch of the light source group 505 is linked to the shutter of the camera 503, but the operation of the power switch can also be performed manually before each image is captured.

In the present embodiment, the target object 501 and the camera 503 are fixed, and some of the plurality of light sources arranged around are lit. However, one or more light source groups are arranged and fixed. The target object 501 and the camera 503 are fixed on the same turntable or the like, and the turntable or the like is moved to perform sequential shooting. Can be obtained.

[0053]

As described above, according to the present invention, it is possible to arbitrarily convert a plurality of images of the same object photographed at different light source positions without using a method which requires a large amount of calculation such as rendering. Can be synthesized with respect to the light source positions. Further, it is possible to easily photograph a plurality of images of the same object photographed at different light source positions, which are necessary for synthesizing an image at an arbitrary light source position.

[Brief description of the drawings]

FIG. 1 is an exemplary view showing a processing procedure and input / output data according to a first embodiment;

FIG. 2 is a configuration diagram of a hardware system according to the first embodiment.

FIG. 3 is a flowchart illustrating a processing procedure of parameter extraction processing according to the first embodiment;

FIG. 4 is an example of an input / output image used in the first embodiment.

FIG. 5 is a configuration diagram of a second embodiment.

[Explanation of symbols]

100: Original image group, 110: Light source direction vector file, 130
... light source moving parameter data, 300 ... parameter extraction processing, 400 ... image synthesis processing.

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Nobuo Kotatsu 890 Kashimada, Saiyuki-ku, Kawasaki-shi, Kanagawa 12 Information Systems Division, Hitachi, Ltd.

Claims (6)

[Claims] In a method of synthesizing an image at an arbitrary light source position, information on the direction and color of the surface of an object to be photographed is extracted from a plurality of original images photographed by changing the position of the light source, and the information is extracted. An image synthesis method for synthesizing an image from an image. 2. The method according to claim 1, wherein the information on the direction and color of the surface of the object to be imaged to be extracted is a normal to the surface of the object,
An image synthesizing method characterized by using a coefficient of diffuse reflection and luminance by background light. 3. An image according to claim 1, wherein information relating to the orientation and color of the surface of the object to be photographed is extracted using a least squares method so that an error between the original image and the composite image is minimized. Synthesis method. 4. A means for executing a processing step of extracting information on the surface of the object to be photographed from the plurality of original images, a processing step of generating the composite image, and displaying the generated composite image. An image display device comprising means. 5. The positional relationship between a camera and a target object is fixed,
An image photographing apparatus, comprising: a plurality of light sources arranged around the periphery; and means for illuminating one or more arbitrary light sources by switching a switch to perform photographing. 6. An image photographing apparatus comprising: means for arranging and lighting one or more light sources, and moving the camera while relatively fixing the positional relationship between the camera and the target object to perform photographing. apparatus.