JP5059503B2 - Image composition apparatus, image composition method, and image composition program - Google Patents

Description

  The present invention relates to an image synthesizing apparatus, method, and program for synthesizing a moving image of a person captured separately and a background moving image.

  Accompanying advances in image synthesis technology, in television shooting and the like, a person image and a background image are captured separately, and only the person area of the person image is extracted with a specific threshold for the person image and extracted. A technique has been put to practical use in which a person area and the background image are combined with a chroma key, and the person is combined as if the person is in the background imaging location. However, in this technique, since the shadow appearing on the surface of the person does not appear according to the illumination condition of the background image, there is one problem that the image becomes unnatural.

  As a technique for improving such a problem, for example, a technique described in Patent Document 1 has been proposed. This technology separates the surface color of the target object into diffuse reflection components and specular reflection components, calculates their reliability, and then applies the reflection model to restore the normal direction in each image, An image in which the illumination direction, that is, the shadow is corrected by the illumination direction given by the normal direction and the reflection model is generated.

JP-A-5-233826

  However, since the technique described in Patent Document 1 estimates the color of an object by statistical processing on a grouped area in an image, it is expected that the color is uniform over a wide area such as a car. It works well for objects that can be done, but cannot work well for objects that change color in fine cycles like human skin and that affect the impression of the viewer It was.

  The present invention has been made in view of the above-described conventional problems, and is capable of synthesizing a separately captured person and background with a natural moving image similar to that captured at the same place. Another object is to provide an image composition method and an image composition program.

The present invention is an image composition device for compositing a synthesized image of a person's face synthesized based on an actual moving image of a person with an actual moving image of a background captured separately from the actual moving image of the person. ,
Person actual moving image input / storage means for inputting or storing the actual moving image of the person;
Background actual moving image input / storage means for inputting or storing the actual moving image of the background;
Light environment actual moving image input / storage means for inputting or storing the light environment actual moving image captured together with the background actual moving image;
Surface reflection characteristic information input / storage means for inputting or storing the surface reflection characteristic information of the person's face composed of three-dimensional shape information, normal information and light reflection characteristic information of the person's face;
Person image extraction processing means for extracting a person image from each frame image of the person's actual moving image stored in the person actual moving image input / storage means;
Based on the extracted person image and the three-dimensional shape information of the face input or stored by the surface reflection characteristic information input / storage means, the position information of the face image of the person and the specific part of the face of the person And tracking processing means for obtaining face tracking information comprising the face direction information;
Light environment information processing for obtaining light environment information corresponding to each frame image of the background actual moving image based on each frame image of the light environment actual moving image input or stored by the light environment actual moving image input / storage means Means,
Synthetic face image information processing means for determining shadow information and surface reflection information of the face based on the face surface reflection characteristic information, the face tracking information, and the light environment information;
Skin information processing means for obtaining skin information comprising color component information and shadow component information from each frame image of the face image of the person;
Face image synthesis processing means for synthesizing the synthesized image of the face based on the skin information, the shadow information of the face, and the surface reflection information of the face;
By providing an image composition device comprising a person background composition moving image composition processing means for composing a person background composition moving image based on the face composite image, the person image, and the background actual moving image, The goal has been achieved.

In addition, the present invention provides an actual moving image of a person, a background actual moving image captured separately from the actual moving image of the person, an optical environment actual moving image captured with the background actual moving image, and A data input / storage processing step for inputting or storing the surface reflection characteristic information of the person's face composed of the three-dimensional shape information, normal information and light reflection characteristic information of the person's face;
An optical environment information processing step for obtaining optical environment information according to each frame image of the background actual moving image, based on each frame image of the optical environment actual moving image;
A person image extraction processing step of extracting a person image from each frame image of the actual moving image of the person;
A tracking processing step for obtaining tracking information of the face image of the person and the face of the person based on the three-dimensional shape information of the person image and the face;
A synthetic face image information processing step for obtaining shadow information and surface reflection information of the face based on the surface reflection characteristic information of the face, the tracking information of the face, and the light environment information;
Skin information processing step for obtaining skin information consisting of color component information and shadow component information from the face image of the person,
A face image synthesis processing step of synthesizing the synthesized image of the face based on the skin information, the shadow information of the face, and the surface reflection information of the face;
There is provided an image composition method including a person background composition moving image composition processing step for composing a person background composition moving image based on the face composite image, the person image, and the background actual motion image. .

  In addition, the present invention provides an actual moving image of a person, an actual moving image of a background captured separately from the actual moving image of the person, and an actual moving image of the light environment of the background captured together with the actual moving image of the background. A data input / storage processing function for inputting or storing the surface reflection characteristic information of the person's face composed of the image and the three-dimensional shape information, normal line information and light reflection characteristic information of the person's face into the storage means; Based on the person image extraction processing function for extracting a person image from each frame image of the actual moving image and the three-dimensional shape information of the person image and the face, the person face image and the person face tracking information are obtained. A tracking processing function, a light environment information processing function for obtaining light environment information corresponding to each frame image of the background actual moving image stored based on each frame image of the light environment actual moving image, and the face surface Reflection characteristics information Based on the face tracking information and the light environment information, it is composed of a synthetic face image information processing function for obtaining shadow information and surface reflection information of the face, and color component information and shadow component information from the face image of the person. Skin information processing function for obtaining skin information, face image synthesis processing function for synthesizing the synthesized image of the face based on the skin information, shadow information of the face, and surface reflection information of the face, and synthesis of the face An image composition program for realizing a person background composition moving image composition processing function for composing a person background composition moving image based on an image, the person image, and the background actual moving image is provided.

  According to the present invention, since the face image and background image of a person captured at different locations can be combined under substantially the same illumination conditions as the background image, the resulting combined moving image is obtained by Therefore, it is possible to synthesize a moving image as if the person and the background were naturally captured at the same shooting location.

  Hereinafter, an image composition device of the present invention will be described based on a preferred embodiment thereof.

  FIG. 1 shows an embodiment of an image composition apparatus of the present invention. The image synthesizing apparatus according to the present embodiment uses a person image including a synthesized image of a person's face synthesized based on a person's actual moving image (hereinafter also referred to as a person's actual moving image) as the person actual moving image. Is synthesized with a background actual moving image (hereinafter also referred to as a background actual moving image).

  As shown in FIG. 1, the image synthesizing apparatus 1 includes a person actual moving image input / storage means 11 for inputting or storing the person actual moving image, and a background actual moving image input / storage for inputting or storing the background actual moving image. Storage means 12; light environment actual moving image input / storage means 13 for inputting or storing the light environment actual moving image captured together with the background actual moving image; and three-dimensional shape information and method of the person's face The surface reflection characteristic information input / storage means 14 for inputting or storing the surface reflection characteristic information of the person's face composed of line information and light reflection characteristic information, and the person actual moving image input / storage means 11 The person image extraction processing means 21 for extracting the person image (person area image) from each frame image of the person actual moving image, and the extracted person image and surface reflection characteristic information input / storage means 14 have input or stored them. The face A tracking processing means 22 for obtaining face tracking information comprising the face image (face area image) of the person and the position information and the direction information of the face of the person based on the three-dimensional shape information; Light obtained by approximating a light source to a point light source according to each frame image of the background actual moving image based on each frame image of the light environment actual moving image input or stored by the light environment actual moving image input / storage means 13 Light environment information processing means 23 for obtaining environment information, and synthetic face image information for obtaining shadow information and surface reflection information of the face based on the surface reflection characteristic information of the face, the tracking information of the face, and the light environment information Processing means 24; skin information processing means 25 for obtaining skin information comprising color component information and shadow component information from the face image of the person; and obtaining the high-frequency shadow component information from the shadow component information; Information, the shadow component information processing means 26 for synthesizing the composite image of the face based on the high-frequency shadow component information, the shadow information of the face and the surface reflection information, and a facial image composition processing means 27; Based on the person image and the background actual moving image, a person background synthesized moving image synthesizing processing unit 28 for synthesizing a person background synthesized moving image, and a moving image display processing unit 29 for displaying the person background synthesized moving image. It has.

Hereinafter, the image composition device 1 of the present embodiment will be described in more detail.
As shown in FIG. 2, the image synthesizing apparatus 1 according to the present embodiment including the above-described units 11 to 14 and 21 to 29 includes a central processing unit (hereinafter referred to as “CPU”) 10A in a computer 10 and basic software (OS ) And an image composition program (hereinafter also simply referred to as a program) 100 that operates together with the image composition program.

  The program 100 inputs or stores the actual moving images and the surface reflection characteristic information in the input / storage units 11 to 14, respectively, in order to cause the computer 10 to execute the units 11 to 14 and 21 to 29. Based on the data input / storage processing function 101 to be performed, the person image extraction processing function 102 for extracting the person image from each frame image of the person actual moving image, and the three-dimensional shape information of the person image and the face Based on each frame image of the actual moving image of the light environment, the light source is approximated to a point light source based on each frame image of the actual moving image of the light environment based on the tracking processing function 103 for obtaining the face image and the tracking information of the face The light environment information processing function 104 for obtaining the light environment information, the face reflection information on the face, the tracking information on the face, and the shadow of the face based on the light environment information A combined face image information processing function 105 for obtaining information and surface reflection information, a skin information processing function 106 for obtaining the skin information composed of color component information and shadow component information from the face image of the person, and a high frequency from the shadow component information A facial image that synthesizes the composite image of the face based on the shadow component information processing function 107 for obtaining shadow component information and the skin information, the high-frequency shadow component information, the shadow information of the face, and the surface reflection information of the face A composition processing function 108; a person background composition moving image composition processing function 109 for composing a person background composition moving image based on the composite image of the face, the person image, and the background actual moving image; and a person background composition moving image. Is a program that realizes a moving image display processing function 110 for displaying the image on the display means.

  The image composition device 1 includes a central processing unit (CPU) 10A, a main memory (RAM) 10B, an image processing processor (GPU) 10C, a video memory (VRAM) 10D, a chip set 10E, a file storage device (HDD) 10F, a display A computer 10 provided with a device (monitor) 10G, a surface reflection characteristic information input device 10H, an input device 10I, and a bus 10J connecting them, a human actual moving image input device (video camera) 10K, and a background actual moving image It is composed of an input device (video camera) 10L and an optical environment actual image input device (video camera) 10M. As in this embodiment, the image composition apparatus 1 may be configured by a general-purpose computer including a general-purpose CPU, or may be configured as a computer dedicated to image composition processing including a dedicated CPU.

The person actual moving image input device 10K, the background actual moving image input device 10L, and the light environment actual moving image input device 10M are each configured by imaging means such as a video camera, and the actual moving images used for image synthesis are received from each device. It can be acquired in real time, or converted to a moving image file that can be stored in the file storage device 10F by using the computer 10 or a computer prepared separately, and data used for image composition is converted into the file storage device 10F. It is also possible to acquire from the above, and it is also possible to combine these acquisition methods.
In the present embodiment, as described below, a moving image is acquired in real time from the human actual moving image input device 10K, and the background actual moving image input device 10L, the light environment actual moving image input device 10M, and the surface reflection characteristics are acquired. From the information input device 10H, an actual moving image is stored in advance in the file storage device 10F as a predetermined moving image file, and image synthesis is performed using these data. Such a combination of data acquisition methods is preferable in realizing real-time processing because it can reduce the processing load on the computer. The real-time processing as in the present embodiment uses an image processing description language (for example, OpenGL (registered trademark of US Silicon Graphics) or DirectX (registered trademark of US Microsoft)) and uses the function of the GPU 10C. Implementation is possible.

The CPU 10A includes a predetermined arithmetic processing circuit that interprets the program 100 stored in the main memory 10B and executes the command content. The main memory 10B is a storage device that stores the basic software OS, the program 100, processing data used by the CPU 10A and the GPU 10C, and the like. In this embodiment, since a memory (not shown) is provided for arithmetic processing of the GPU 10C, processing data in the GPU 10C is stored in this memory. The GPU 10C is an arithmetic processing device that includes a dedicated arithmetic processing circuit that performs image processing including synthesis of a final moving image instead of the CPU 10A. The video memory 10D is a memory that stores an image that is finally displayed as a result of image processing performed by the GPU 10C. The chip set 10E is a plurality of integrated circuits (LSIs) in which control circuits are integrated, and is interposed between the CPU 10A and peripheral devices connected thereto. The file storage device 10F is an auxiliary storage device that stores information necessary for image processing in a predetermined file format. The display device 10G is a display device that displays an image processed by the GPU 10C via the video memory 10D. The person actual moving image input device 10K is a device that inputs an actual moving image of a person to be combined in real time. The background actual moving image input device 10L, the light environment actual moving image input device 10M, and the surface reflection characteristic information input device 10H input each moving image and information to be captured and stored in the file storage device 10F in a predetermined file format. It is a device to do. The input device 10I is a device that performs various input operations by an operator. The bus 10J connects each circuit / device in a communicable manner according to each communication protocol.
In the image composition device 1 of the present embodiment, by cooperating with the basic software OS and the program 100, the CPU 10A and / or the GPU 10C mainly function as the processing units 21 to 28, and the person actual moving image input device 10K and The file storage device 10F mainly functions as each of the input / storage units 11-14.
Various file formats such as MPEG format, Quick Time format, and AVI format can be adopted as the file format for saving the moving image in the storage device, but the file format is based on reversible conversion that does not cause deterioration in image quality. Among them, the uncompressed AVI format that is generally used and excellent in versatility is more preferable.

  The program 100 cooperates with the CPU 10A, the main memory 10B, the file storage device 10F, the person actual moving image input device 10K, the surface reflection characteristic information input device 10H, and the like, and the data input / storage processing function 101 performs the person actual moving image. The person actual moving image from the input device 10K, the background actual moving image, the background light environment moving image, and the surface reflection characteristic information of the person's face from the file storage device 10F are stored in the data storage unit on the GPU 10C, respectively. To make it happen.

  The program 100 cooperates with the CPU 10A, the main memory 10B, the GPU 10C, and the like to realize the person image extraction processing function 102 to extract the person image from each frame image of the person actual moving image.

  The program 100 cooperates with the CPU 10A, the main memory 10B, the GPU 10C, the file storage device 10F, and the like, and performs the tracking processing function 103 on the basis of the person image and the three-dimensional shape information of the face. This makes it possible to obtain face tracking information.

  The program 100 cooperates with the CPU 10A, the main memory 10B, the GPU 10C, the file storage device 10F, and the like, and performs each of the background actual moving images based on each frame image of the light environment actual moving images by the light environment information processing function 104. According to the frame image, it is possible to obtain light environment information in which a light source is approximated to a point light source.

  The program 100 cooperates with the CPU 10A, the main memory 10B, the GPU 10C, the file storage device 10F, and the like, and by the synthetic face image information processing function 105, the face surface reflection characteristic information, the face tracking information, and the light environment information Based on the above, it is possible to obtain the shadow information and the surface reflection information of the face.

  The program 100 cooperates with the CPU 10A, the main memory 10B, the GPU 10C, the file storage device 10F, and the like, and uses the skin information processing function 106 to obtain skin information including the color component information and the shadow component information from the face image of the person. We are realizing what we want.

  The program 100 cooperates with the CPU 10A, the main memory 10B, the GPU 10C, the file storage device 10F, and the like to obtain the high-frequency shadow component information from the shadow component information by the shadow component information processing function 107.

  The program 100 cooperates with the CPU 10A, the main memory 10B, the GPU 10C, the file storage device 10F, and the like, and uses the face image synthesis processing function 108 to perform the skin information, the high-frequency shadow component information, the face shadow information, and the face surface. Based on the reflection information, the composite image of the face is synthesized.

  The program 100 cooperates with the CPU 10A, the main memory 10B, the GPU 10C, the file storage device 10F, and the like from the synthesized image of the face, the person image, and the background actual moving image by the person background synthesized moving image synthesis processing function 109. The composition of the person background composition moving image is realized.

The program 100 cooperates with the CPU 10A, the main memory 10B, the GPU 10C, the video memory 10D, the file storage device 10F, and the display device 10G to display the person background synthesized moving image on the display device 10G by the moving image display processing function 110. To make it happen.
When the program 100 is installed and used in a general-purpose computer system as in the present embodiment, it is provided by being recorded on a recording medium together with the program installer. Further, when it is incorporated in a dedicated computer dedicated to image composition processing, it is recorded in a nonvolatile memory and provided.

Next, the image composition method of the present invention will be described based on a preferred embodiment.
As shown in FIGS. 3 to 7, the image composition method of the present embodiment is a background actual moving image captured separately from a human actual moving image, and a light environment actual moving image of the background captured together with the background actual moving image. A data input / storage processing step S1 for inputting and storing the surface reflection characteristic information of the person's face composed of the image and the three-dimensional shape information, normal line information and light reflection characteristic information of the person's face; Based on each frame image of the moving image, a light environment information processing step S2 for obtaining light environment information corresponding to each frame image of the background actual moving image, and extracting a person image from each frame image of the person actual moving image Person image extraction processing step S3, tracking processing step S4 for obtaining tracking information of the person's face image and the person's face based on the person image and the three-dimensional shape information of the face, Based on the skin information processing step S5 for obtaining skin information consisting of color component information and shadow component information from the frame image, and on the face surface reflection characteristic information, the face tracking information, and the light environment information, A combined face image information processing step S6 for obtaining shadow information and face surface reflection information, and a face image for combining the face composite image based on the skin information, the face shadow information, and the face surface reflection information. A composition processing step S7 and a person background composition moving image composition processing step S8 for composing a person background composition moving image based on the face composition image, the person image, and the background actual moving image are provided.

Hereinafter, the image composition method of the present embodiment will be described in detail based on the image composition method using the image composition device 1.
In the image composition apparatus 1, as shown in FIG. 3, first, in the data input / storage processing step S1, the CPU 10A uses the data input / storage processing function 101 in the program 100 to separate from the target person's actual moving image. The captured background actual moving image, the background light environment actual moving image captured together with the background actual moving image, and the person's face three-dimensional shape information, normal information, and light reflection characteristic information The surface reflection characteristic information of the human face is obtained from the background actual moving image input device 10L, the light environment actual moving image input device 10M, and the surface reflection characteristic information input device 10H, and then the file storage device 10F in a predetermined file format. In advance.

  The background actual moving image and the light environment actual moving image input by the background actual moving image input device 10L and the light environment actual moving image force device 10M are acquired as follows.

  As shown in FIG. 8, a background imaging video camera V <b> 1 that captures an actual moving image of the background and a video camera V <b> 2 that captures an actual moving image of the background light environment have the same optical axis X of their lenses. Install them at an interval in front and back so that they are positioned in a straight line. Between these video cameras V1 and V2, a sphere S having a mirror-finished surface is installed so that the optical axis X passes through the center. The camera V2 is focused so that the image displayed on the sphere S is the clearest. The cameras V1 and V2 and the sphere S are fixed so that their relative positions do not change during imaging. The actual background moving image may be captured in a stationary state at a fixed point or may be captured while moving. The video camera V1 is preferably the same model as a camera that captures a human actual moving image later. The video cameras V1 and V2 may be either analog or digital video cameras as long as color (RGB) imaging is possible. However, in consideration of later data handling, the digital video cameras preferable.

  In the optical environment actual moving image captured by the video camera V2, a large sphere S is imaged in the center and the background is also imaged in the vicinity thereof. However, what is used for the optical environment information processing described later is the portion of the sphere S. It is an image. In the captured image, the surrounding landscape is distorted and projected on the sphere S. Since specular reflection (reflection where the incident angle and the reflection angle are completely equal) occurs on the surface of the sphere S, the image of the portion of the sphere S shows how much light has entered from each corresponding direction. Yes. Since the shape of the sphere S can be expressed completely geometrically, it can be specified completely from which direction the light entering the video camera V2 through each part of the sphere S comes.

  In the present embodiment, the background actual moving image and the light environment actual moving image are stored in advance in the file storage device 10F as moving image files, so that the video cameras V1 and V2 may not be connected to the computer 10. These video cameras can be used as the background actual moving image input device 10L and the light environment actual moving image input device 10M, and the actual moving images can be taken in from them and image synthesis can be performed by real time processing. In this case, these video cameras V1 and V2 are connected to the computer 10 via an interface such as IEEE1394. Also, each moving image acquired from the video cameras V1 and V2 needs to be synchronized. Although there is no particular limitation on the synchronization method, a simple method is, for example, a method in which images obtained by shining a strobe before and after imaging are acquired and synchronized so that the time when the strobe shines is aligned.

  In the image composition device 1, in the light environment information processing step S <b> 2 (see FIG. 3), the CPU 10 </ b> A obtains light environment information using the light environment information processing function 104 in the program 100. In the present embodiment, for each frame image of the light environment actual moving image stored in the file storage device 10F, coordinate conversion is performed so that the elevation angle / azimuth angle is a two-dimensional (x, y) coordinate, Light environment information in which the light source is approximated to a point light source is obtained according to each frame image of the background actual moving image. There is no particular limitation on the method for extracting the region of the sphere S from the background actual moving image. In the present embodiment, the image of the circumference formed by photographing in advance under the condition that the boundary of the sphere S is clearly visible and using the commercially available image processing software from one frame image obtained at that time. This is done by acquiring the position inside. If the relative position between the video camera V2 and the sphere S does not change, the position occupied by the sphere S in the background actual moving image also does not change, so the area of the sphere S can be extracted by the circumference. .

  In this embodiment, this light environment information is obtained from the Median cut algorithm for light probe sampling. In Proceedings of SIGGRAPH 2005 Conference Abstracts and Applications (Poster), ACM Press / ACM SIGGRAPH, Computer. Based on Graphics Proceedings, Annual Conference Series, ACM.)

First, the number of point light sources used for approximation is set. The number of point light sources is 2 ⁿ (2, 4, 8, 16, 32, 64,... Based on the fineness of the structure of the actual moving image of the light environment and the gloss width of the light reflection characteristics of the face to be synthesized. ) In advance. And as shown in FIG. 9, CPU10A performs the following process 1-the process 4 according to the number of the set light sources.
Process 1: The entire image is put on the list as a single rectangular area. That is, in the program, an array in which the number of elements is variable and each element represents an arbitrary rectangular area in the image is prepared, and the entire image is assigned to the first element. Specifically, when the image width is W and the image height is H when counted by the number of pixels, the coordinates of each pixel in the image are expressed by (x, y), and the upper left of the entire image An arbitrary rectangular area is assigned to each element of the array by taking the coordinate system so that is (1, 1) and the lower right is (W, H). If the upper left coordinate of an arbitrary rectangular area is (x1, y1) and the lower right coordinate is (x2, y2), this rectangular area is specified by a sequence of four elements (x1, y1, x2, y2). Therefore, it is only necessary to store four number sequences in each element. In order to put the entire image in the list as the only rectangular area, according to the above-described method, an array having only one element may be prepared and the only element may be (1, 1, W, H).
Process 2: Each rectangular area on the list is divided (segmented) into two by a boundary line parallel to the short side of the area. The division by the boundary line is performed so that the light amounts of the divided areas are equal. In the present embodiment, the actual light environment moving image is obtained with three channels of RGB, and the amount of light Y at each pixel is the BT. 709, and is obtained by the following equation (1) in relation to RGB.
Y = 0.2125R + 0.7154G + 0.0721B (1)
Process 3: The above process 2 is repeated n times according to the number of point light sources, and as a result, the original image is divided into 2 ⁿ regions.
Process 4: The center of gravity of the light in each region is obtained, and it is approximated that there is a point light source corresponding to the total amount of light in the region. Here, specifically, the center of gravity of light is expressed as Yc (y, x) where the light quantity of the coordinate (x, y) in the coordinate system is expressed as xc = Σ (x × Y (X, y)
) / ΣY (x, y) and yc = Σ (y × Y (x, y)) / ΣY (x, y). Here, Σ represents the sum total over all pixels in the region.
When the light environment actual moving image is acquired in advance and stored as a moving image file as in the present embodiment, this processing can be completed in advance. Thereby, it is possible to reduce a calculation load on the computer when processing is performed.

  The face surface reflection characteristic information includes face three-dimensional shape information, normal line information, and light reflection characteristic information (glossy light reflection characteristic information). The function representing the light reflection characteristic information is called a bidirectional reflection function (BRDF), which is a function representing how much light is reflected in which direction when light is irradiated from a certain direction. And a complicated shape depending on the emission direction. However, perceptually, it is known that gloss can be expressed almost by intensity and width, so in practice, the value of each parameter when the gloss component of BRDF is approximated by a model including two parameters of intensity and width. Can be used as the light reflection characteristic information, and in this embodiment, the intensity and width of the gloss are also used as the light reflection characteristic information. In the present embodiment, the surface reflection characteristic information is calculated based on data measured by a surface reflection characteristic measurement apparatus (hereinafter referred to as a measurement apparatus) A described below.

  As shown in FIG. 10, the measuring apparatus A of this embodiment includes a face light reflection characteristic information measuring means 1A, a face three-dimensional shape information measuring means 1B, and a face normal information measuring means 1C. is doing.

  The light reflection characteristic information measuring means 1A is suspended from a movable light source 2 that irradiates the face F with linear light, and a movable light source 2 while moving along the orbit. The light reflection characteristics of the face F are obtained based on the imaging unit 3 arranged to capture the state in which the face F is irradiated with light and the image data of the state of the face F captured by the imaging unit 3. And information processing means 4.

  The three-dimensional shape information measuring unit 1B includes a projecting unit 5 that projects an optical pattern composed of repeated bright and dark on the face F. In the present embodiment, the imaging unit 3 is disposed so as to capture the state in which the optical pattern is projected onto the face F. The information processing unit 4 is provided so as to obtain the three-dimensional shape information of the face F based on the image data of the face F in a state where the optical pattern imaged by the imaging unit 3 is projected.

  The normal information measuring unit 1C includes a plurality of fixed light sources 6 that are fixed at a fixed position and irradiate light on the face. In the present embodiment, the image pickup means 3 is arranged so as to pick up an image of a state in which the measurement object is irradiated with light from each of the fixed light sources. The information processing means 4 is provided so as to obtain the normal information of the face based on the image data in a state where the face F is irradiated with light from the fixed light source 6 imaged by the imaging means 3.

<Light reflection characteristic information measuring means 1A>
The measuring apparatus moves the position of the movable light source 2, turns on and off each fixed light source 6, projects the optical pattern by the projection unit 5 and switches the optical pattern, and captures the face in each state by the imaging unit 3. Control means 7 is provided for controlling the taking of image data of each of the captured states into the information processing means 4. In the present embodiment, the information processing means 4 and the control means 7 are configured by a computer system (hereinafter also simply referred to as a computer) 8 that functions as the information processing means 4 and the control means 7.

  The measuring apparatus includes a frame 9 that forms a space that can be accommodated while a person to be measured sits down, and that is provided with a movable stage serving as a trajectory of the movable light source 2 and each fixed light source 6.

  The light source of the movable light source 2 is not particularly limited as long as it can irradiate the measurement target with light having an intensity that can be imaged by the imaging means 3 from an arbitrary position on the orbit. A light-emitting diode array, a neon tube, etc. are preferable. Furthermore, when considering the axis of the line light source as the center, it is excellent in the axial coverage of the amount of radiated light, and considering that the thin form can be easily obtained, the neon tube is preferable. The length of the light source of the movable light source 2 only needs to be long enough up and down including the measurement object.

  The trajectory of the movable light source 2 is set according to the outer shape of the measurement target, the shape of the light source, and the form of the movable stage that moves the light source. In the present embodiment, the trajectory of the movable light source 2 is two substantially V-shaped trajectories that are bilaterally symmetrical with respect to the measurement target and open with respect to the measurement target when the measurement apparatus is viewed in plan. As the movable stage, one that can be moved in a straight line is easily available, and the movable light source 2 is moved around the measurement target by combining the two to surround the measurement target. . By moving the movable light source along such a substantially V-shaped trajectory, it is possible to obtain gloss information from a portion where gloss cannot be observed when it is simply moved linearly toward the measurement object. The trajectory of the movable light source 2 may be a curve or a substantially circular shape.

  In the present embodiment, the movable light source 2 includes two light sources, a light source 2A that moves to the left of the measurement target and a light source 2B that moves to the right of the measurement target. At the start of measurement, these light sources are in the left rear and right rear fixed positions as viewed from the measurement object. The light intensity of the neon tube is not very stable immediately after lighting, and the fluctuation increases immediately after the light source is turned on, but the fluctuation of the light source intensity strongly affects the obtained light reflection characteristic information. For this reason, these light sources are turned on when the apparatus is activated, and are placed at fixed positions when not in use. Further, shielding plates 21A and 21B are installed in a direction in which the measurement object is expected from this fixed position, so that the measurement object is not illuminated even when the light source is in the fixed position.

  The light sources 2A and 2B are attached to a carriage (not shown) that moves along the movable stage. The carriage is electrically connected to the control means 7, and the driving means is controlled by the control means 7, so that the carriage stops or moves according to a predetermined stationary position of the light sources 2 </ b> A and 2 </ b> B.

  From the viewpoint of improving the accuracy of the light reflection characteristics, the interval between the stationary positions for irradiating the measurement target of the light sources 2A and 2B is preferably as short as possible. As described above, in the case where it is difficult to remain stationary for a long time, such as when the measurement target is a person, the determination is made in consideration of the measurement time in practice. The upper limit of the interval between the stationary positions is determined as follows. When attention is paid to the appearance of an arbitrary part to be measured from the imaging means, when the light source is moved, there is a phenomenon that a dark thing becomes bright and becomes dark again in the vicinity of an angle condition where gloss is recognized. The upper limit of the interval between the stationary positions is preferably set within a range that is sufficiently shorter than the width of the brightened area. In the present embodiment, the interval between the stationary positions is 1 cm on the movable stage.

  The imaging means 3 is fixed to a substantially front face of the face so as to face the face to be measured. The imaging means 3 is not particularly limited as long as it can capture an image of a face as a measurement target in each state with a predetermined angle of view and has sufficient spatial resolution. It is preferable to be able to capture images with different exposures, and it is preferable to be able to capture both moving images and still images. The control means 7 includes a charge coupled device (CCD) and a complementary oxide semiconductor (CMOS) device that can control the capturing of the captured image and the captured image into the computer 8 as image data (electronic data). A video camera is preferred. Even if the movable light source 2 is moved continuously, the light source is stationary when the imaging means is fast enough to assume that the light source hardly moves during a series of imaging times with different exposures. You may move it without doing. Also, when the dynamic range for the light intensity of the imaging means is wide enough to correspond to areas with high glossiness, and the resolution for the light intensity is fine enough to represent areas with low glossiness For this, it is not necessary to capture images with different exposures, and it is sufficient to capture images with a single exposure.

  The imaging means 3 uses a pixel value that is linear with respect to the amount of incident light so that later data processing is simple, and can capture images with different exposures to accommodate a large range of reflected light. It has a function.

<Three-dimensional shape information measuring means 1B>
The projection means 5 is an apparatus that projects an optical pattern composed of repeated white and black (bright and dark) horizontal stripes on the face. The projection unit 5 is not particularly limited in type as long as the projection unit 5 can project light of an intensity that can be imaged by the imaging unit 3 from the projection position onto the measurement target. However, the projection unit 5 can be easily obtained at low cost. Considering that the pattern can be easily set, a commercially available projector that can be connected to a computer is preferable. Further, it is preferable that the projection unit 5 can automatically change the optical pattern to be projected by the control unit 7 as in this embodiment.

  As for the arrangement of the projection means 5, the direction of the imaging means 3 and the direction of the projection means 5 with respect to the measurement object are different as much as possible within a range in which the projection pattern is not projected in the region to be measured of the measurement object and the shadow does not occur. It is desirable that Considering that the shadow of the nose is unlikely to occur and that the projection pattern is evenly projected on the left and right, it is preferable that the nose shadow be installed below the imaging means as viewed from the measurement target.

The control means 7 automatically switches the light / dark width in the optical pattern. The finer the repetition of light and darkness in the optical pattern, the more desirable it is to improve the accuracy of the obtained shape, but there is a limit depending on the resolution of the projection means and the resolution of the imaging means. Considering the purpose, the repetition width of the finest pattern projected onto the measurement object is preferably 20 mm or less, and more preferably 5 mm or less. The resolution of the projection means and the imaging means needs to have sufficient performance to realize this. Specifically, for example, the projection pattern starts with a space divided into two vertically and sequentially increases the number of divisions twice, and finally projects 10 patterns up to 2 ¹⁰ (= 1024) divisions. .

<Normal information measuring means 1C>
A point light source is used as the light source of the fixed light source 6. The light source of the fixed light source 6 is not particularly limited as long as it can irradiate the measurement target with light having an intensity that can be imaged by the imaging means 3 from the fixed position, but the intensity is stabilized immediately after lighting. In view of the excellent durability, a light emitting diode (LED) is preferable.

  Since the normal is obtained by performing fitting from a plurality of relationships between the direction of the fixed light source 6 and the amount of light entering the imaging device 3 at that time, the larger the number of fixed light sources 6 is, the more preferable. It is preferable that they are arranged evenly when viewed from above. On the other hand, in order to shorten the measurement time, it is preferable that the number of light sources is small. Considering efficient data acquisition with a small number, a ring shape is formed so as to surround the measurement object in front of the measurement object. It is preferable to arrange. In the present embodiment, nine fixed light sources 6 are sufficient to ensure the accuracy required for the surface reflection characteristics.

  The fixed light source 6 includes a relay circuit (not shown) to which each light source is connected. The relay circuit is connected to the control means 7, and when the control means 7 controls the ON / OFF switch of the relay circuit, each light source is turned on / off.

  In the measuring apparatus, polarizers 3A and 6A are arranged between the face to be measured and the imaging means 3 and each fixed light source 6, respectively. Here, these polarizers are arranged such that the polarization planes of the polarizer on the light source side and the polarizer on the imaging means side cross each other. Here, the planes of polarization intersecting each other means that the amount of light emitted from the light source, reflected by the measurement object, and incident on the imaging means is minimized. It can be considered that the orientation of the polarizer on the imaging means side is fixed and the light source side polarizer intersects with the orientation of the light source side polarizer satisfying the condition that the image obtained from the imaging means is the darkest while rotating the polarizer on the light source side. it can. When the light source and the imaging means are in substantially the same direction with respect to the measurement target, the respective polarizers are orthogonal to each other. Under this condition, since the gloss is reflected only once on the surface, the direction of the glossy polarized light reflected by the measurement object through the polarizer on the light source side is maintained, and on the imaging means side that is orthogonal to it. It cannot pass through the polarizer. In this embodiment, in order to simplify a series of operations, imaging is performed while the polarizer on the imaging unit side is arranged at the time of other imaging, but a polarizer is arranged for each light source used for other imaging. In this case, the light that enters the imaging means at that time is hardly polarized, so that it can be considered that the other imaging is not affected.

  As described above, the information processing means 4 and the control means 7 are constituted by the computer 8 that functions as each of these means. The computer 8 includes a hardware provided with a central processing unit (CPU), a main storage device, an auxiliary storage device, an input device, and an output device, a basic software and a basic surface and a basic surface. Software having a program for calculating reflection characteristics is provided. In the computer 8, the CPU interprets the programs held in the main storage device and executes the contents of the instructions, and causes the computer 8 to function as the information processing means 4 and the control means 7 according to the following steps.

<Measurement method>
Next, a method for measuring the surface reflection characteristics of the face by the measurement apparatus will be described with reference to FIGS.
First, in step S11 (see FIG. 11), under the control of the control means 7, the projection means 5 projects the horizontal stripe optical pattern consisting of the above-described black and white repetition onto the face of the measurement subject. The period of the optical pattern is continuously changed by the control means 7, and the projection state is continuously imaged by the imaging means 3. After imaging the projection state of all the optical patterns, the projection of the optical pattern onto the face by the projection unit 5 is stopped.

  The projected image data of the face on which the optical pattern is projected is taken into the information processing means 4 under the control of the control means 7.

  Next, in step S12 (see FIG. 11), each fixed light source 6 is turned on and off once in a predetermined order under the control of the control means 7, and a fixed light source light irradiation image of the face by each fixed light source 6 is obtained. The image is picked up by the image pickup means 3. When the lighting and extinguishing of each fixed light source 6 is completed once, the lighting and extinguishing of the light source 6 are completed.

  The fixed light source irradiation image of the face by each fixed light source 6 is taken into the information processing means 4 under the control of the control means 7 as electronic data (fixed light source light irradiation image data, hereinafter also referred to as fixed light source image data). .

  Next, in step S13 (see FIG. 11), the light source 2A of the movable light source 2 moves from a fixed position on the left rear side of the face to the apex side of the substantially V-shaped trajectory under the control of the control means 7, The face is stationary at a predetermined position, and the moving light source image of the face is picked up by the image pickup means 3 under two types of exposure conditions. When the light irradiation and photographing by the light source 2A are finished, the light source 2A returns to the left rear fixed position, and the light is blocked by the shielding plate 21A. Next, the light source 2B is first moved to the apex side of the substantially V-shaped track by the control of the control means 7, and then moved from the apex side of the V-shaped track to the right rear side of the face. Then, the movable light irradiation image of the face is picked up by the image pickup means 3 under two types of exposure conditions. When the light irradiation and photographing by the light source 2B are finished, the light source 2B returns to a fixed position, and the light is blocked by the shielding plate 21B.

  The movable light source light irradiation image of the face by the light sources 2A and 2B is taken into the information processing means 4 under the control of the control means 7 as electronic data (movable light source light irradiation image data, hereinafter also referred to as movable light source image data). .

  Based on the projection image data, fixed light source light irradiation image data, and movable light source light irradiation image data captured by the information processing means 4 as described above, the surface light reflection characteristics of the face are obtained in the information processing means 4 as follows. It is done.

  That is, in step S14 (see FIG. 11), the information processing means 4 calculates the three-dimensional shape information of the face (coordinate information in the three-dimensional reference coordinates) based on the projection image data. There are a light cutting method, an image encoder method, and the like as a method for obtaining the three-dimensional shape information by projecting the optical pattern onto the measurement target. However, since the configuration of the apparatus and the analysis program can be made relatively simple, A spatial coding method for calculating three-dimensional shape information is preferred.

  In the measurement apparatus of the present embodiment, the three-dimensional shape information in the present embodiment is the spatial coding method (Valkenburg, RJ, et al., 1998, Accurate 3D measurement using a structured light system. Image and Vision Computing 16, 2, 99-110).

The projection means 5 projects a pattern in which the upper half is dark and the lower half is bright for the first time. Of the measurement object that can be seen from the imaging means, the dark area belongs to the upper half space, and the bright area belongs to the lower half space. When the second half is divided into the upper half and the lower half, and a dark-bright-dark-bright optical pattern is projected in order from the top, the dark area in both the first and second times is the upper quarter. It is possible to narrow down the space to which each position of the measurement object belongs, such as the area that is dark at the first time and brighter at the second time, such as the second quarter from the top.
Specifically, the shape is obtained in the following manner. As shown in FIG. 12, based on a series of captured images, the state of each position of the measurement target is represented by a binary number. The first place is 0 when the focused position in the image captured when the first pattern is projected is dark, and 1 when it is bright. The tens place is 0 when the focused position in the image captured when the second pattern is projected is dark, and 1 when it is bright. In the following, the image when the third pattern is projected is associated with the hundreds, and the image when the fourth pattern is projected is associated with the thousands. If this is continued, one binary number corresponds to each minute surface to be measured projected on the imaging means 3. Each binary number obtained as a result corresponds to one when the space is cut thinly horizontally. On the other hand, when a minute surface to be measured appears in a pixel of an imaged image, the minute surface belongs to somewhere on a straight line corresponding to that pixel extending from the imaging device. When the thinly cut space described above is approximated to a plane, the spatial position of the minute surface to be measured is obtained as the intersection of the plane and the straight line from the position of the pixel corresponding to the obtained binary number. be able to.

  Next, in step S15 (see FIG. 11), the information processing means 4 calculates facial normal information based on the fixed light source image data and the three-dimensional shape information. Examples of the method for obtaining the normal line of the measurement target include a method for obtaining from the inclination of each surface of the three-dimensional shape. Considering the measurement accuracy, it is desirable to obtain the normal by a method different from the shape. Specifically, the photometric stereo method is preferable, but considering the downsizing of the apparatus scale, the fixed light source image data and the cubic It is more preferable to use the modified photometric stereo method from the original shape information. The specific method will be described below.

  In the measurement apparatus of this embodiment, the normal information is based on a normal information calculation method (Woodha, RJ 1980. Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 1, 139-144). Required by the method. In this calculation method, the normal information is specifically obtained as follows.

  A normal of an arbitrary point (coordinate) of the fixed light source image data is obtained over the entire face as follows.

The intensity of an arbitrary point in the fixed light source image data when each of the light sources 6 (9 in this embodiment) is turned on is In (n = 1 to 9), and the normal vector of the point of interest is N. Further, the direction vector of each light source with respect to the point of interest is Dn (n = 1 to 9). Further, the distance between each light source 6 and the point of interest is Ln (n = 1 to 9). Assuming that the light intensity is proportional to the inner product of the normal of the point and the direction of the light source (Lambertian), the theoretical value Itn of the light intensity is the normal vector N of the point The direction vector Dn from the point to the light source is expressed by the following formula (2). However, c is a constant and is determined by how to take a unit system, the intensity of the light source, the exposure condition of the imaging means, and the like. Further, max (a, b) means taking a larger value of either a or b. Also, dot (A, B) represents the inner product of vector A and vector B. Ln ² excluding the right side represents the attenuation effect due to the distance of the light emitted from the light source.
Itn = max (0, c × dot (N, Dn)) / Ln ² (n is 1 to 9) (2)
When N is a fitting parameter, the value of N when the value of this equation is closest to the actual measurement is considered to be a true value, and Σ (In−max (0, c × dot (N, Dn)) / Ln ² ) ² determines the normal vector N by the method of least squares so that ² is minimized.

  It is preferable that the three-dimensional shape information and the normal line information obtained as described above are further corrected by a correction process called a hybrid method in step S16 (see FIG. 11). This correction ensures the consistency between the three-dimensional shape information and the normal information, and further improves the accuracy of the three-dimensional shape information and the normal information. The hybrid method is currently the only method to ensure the consistency of 3D shape information and normal information obtained by different methods and improve the accuracy, but when a new method is developed in the future, You may replace by such a method.

  In the measuring apparatus A of the present embodiment, as a method of correcting the three-dimensional shape information and normal information, a hybrid method (Nehab, D., Rusinkiewicz, et al., 2005, Efficiently Efficiently combining position and normals for precise 3D geometry. ACM Transactions on Graphics 24, 3, 536-543.)

  Specifically, as shown in FIG. 13, when attention is paid to the shape in the vicinity of a certain pixel in the three-dimensional shape information, four triangles formed by the pixel and the upper, lower, left, and right pixels in contact with the pixel are considered. Since each pixel includes position information in a three-dimensional space, a normal vector of each triangle can be obtained. An average of the normal vectors of each triangle is defined as a normal vector N ′ at the pixel. The normal vector N ′ obtained in this way is accurate as the high frequency component is less accurate and the low frequency component is more accurate than the normal vector N, as pointed out in the literature. In order to increase the frequency, N high frequency components and N ′ low frequency components are combined to form a new normal vector. Specifically, the normal vector N ′ is first smoothed so that information on fine changes is lost, and the result is defined as Ns ′. A general Gaussian filter is used for smoothing, but the width of the filter (that is, the cut-off frequency) varies depending on the degree of fineness of the unevenness to be measured. A value that does not cause distortion of the structure considered to be derived from pores and fine wrinkles and does not distort the overall average shape is selected in advance. For example, if the measurement target is a human face, the spatial frequency of fine irregularities does not change much, so once you select this value for one person's face, when measuring another person's face May use the same value. In the present embodiment, smoothing is performed using a Gaussian filter with a variance value of 12 on data acquired by imaging under the condition that the entire measurement target is included in 1024 × 678 pixels. The normal vector N is also smoothed under the same smoothing condition as the normal vector N ', and the result is Ns. Then, a rotation transformation that matches the obtained smoothing normal vector Ns with the normal vector N is obtained for each point, and this rotation transformation is applied to the smoothing normal vector Ns ′ to obtain the normal vector N ″ as Ask.

Next, assuming the actual shape of the face to be measured, the shape is corrected so that the error between the actual shape and the actual measurement value, that is, the three-dimensional shape information and the normal vector N ″ is minimized. When the most probable shape is the corrected shape information, the sum of squares of errors between the corrected shape information and the three-dimensional shape information, the corrected normal information obtained from the corrected shape information, and the normal vector N ″ The corrected shape information and the corrected normal information (corrected normal vector N) are obtained so that the weighted sum of the error and the square sum of the errors is minimized. Since the weighting value varies depending on the measurement system and the measurement object, the value is set in advance so that the step functions well.
In this embodiment, the face light reflection characteristic information and the face normal information to be used hereinafter refer to the corrected shape information and the corrected normal information obtained here, respectively.

Next, in step S <b> 17, the information processing means 4 obtains face light reflection characteristic information based on the movable light source image data.
The light reflection characteristic information obtained by the information processing means 4 can take a bidirectional reflection function (BRDF) or various feature amounts derived therefrom, but in this embodiment, the intensity of irradiated light is particularly constant. The gloss intensity and width in a certain case are specifically described as follows.

  First, when the light sources 2A and 2B of the movable light source are turned on in advance, when each light source is approximated as a continuum of m virtual point light sources, the light emitted from one virtual point light source is in which direction in the space. Measure how much radiation is emitted. That is, elements other than the length corresponding to one virtual point light source are shielded, and the amount of light emitted in each direction at that time is measured. In this embodiment, in order to simplify the handling, the characteristics of the light source are an axis object with respect to the axis of the line light source, and the radiation characteristics of each element in the line light source do not depend on the portion of the light source (even in the central portion, the end portion But) it was assumed to be constant. The direction and intensity for each division unit when each light source is divided into m units in the height direction are measured. The number m of division units is set according to the length of the light source, the size of the measurement target, the distance between the light source and the measurement target, and m is preferably as large as possible in terms of accuracy improvement. Since it will rise, it is preferable that it is small in the range which does not deteriorate a precision comprehensively, and in the case of this embodiment, it is preferable that m is about 100.

Next, it is assumed that the light reflection characteristic of the face follows the Torrance-Sparrow model (hereinafter referred to as TS model).
That is, it is assumed that the light reflectance f is obtained by the following equation (3), where n is the refractive index of the skin, α is the intensity of gloss, and M is the spread.
f = α × (F · G · D) / (4 cos θicos θr) (3)
However, the Fresnel term F is expressed by the following equation (4) for normal incidence, and expressed by equation (5) except for normal incidence.
F = (n−1) ² / (n + 1) ² (4)
F = (1/2) [tan ² (θi−θt) / tan ² (θi + θt) +
sin ² (θi−θt) / sin ² (θi + θt)] (5)
Moreover, the surface roughness term D is represented by the following formula (6).
D = exp (-tan ² β / M ² ) / πM ² cos ⁴ β (6)
The shape term G is expressed by the following formula (7). Here, min (a, b, c) means the smallest value among a, b, and c.
G = min (Gs, Gm, 1) (7)
Here, Gs = 2 (N · H) · (N · S) / (S · H)
Gm = 2 (N · H) · (N · V) / (V · H)
H = (S · V) / | S · V |
θi = acos (dot (N, S))
θr = acos (dot (N, V))
(Dot (A, B) is the inner product of vector A and vector B)
θt = asin (sin θi / n)
(N is the refractive index of the skin surface)
N is a modified normal vector, S is a unit vector in the direction of the light source, V is a unit vector in the direction of the imaging means, H is a unit vector that bisects S and V, β is an angle formed by N and H, θi is an incident angle, θr is a light receiving angle, and θt is a refraction angle.

Next, for each point corresponding to the face surface of the movable light source image data, a light reflection characteristic is obtained over the entire face as follows.
That is, first, paying attention to a certain point in the face, the brightness (actually measured values) I1 to In of the point when the movable light source irradiates light at each stationary position (n locations of P1 to Pn). Ask for.
And the direction V11-V1m of each said virtual point light source of the light irradiated to this point when a movable light source comes to each stationary position ... Vn1-Vnm and light quantity L1 (V11) ... ˜L1 (V1m) ˜Ln (Vn1) ˜... ˜Ln (Vnm) are determined from the measured direction and intensity data.

  Next, when each light source of the movable light source emits light at a predetermined stationary position, the brightness Ic1 to Icn of the image is obtained from V11 to Vnm and L1 (V11) to Ln (Vnm) on the assumption that it follows the TS model. . Since this calculation result includes variables (light reflection characteristic information) α and M representing the intensity and width of gloss, as shown in FIG. 14, Ic1 to Icn most closely match the actual measured values I1 to In. As described above, the light reflection characteristic information α and M are obtained by fitting using the simplex method with the variables α and M as variables. Here, the measured values I1 to In are read from images taken under two types of exposure conditions. Specifically, the coefficient C is obtained as the number of times that the pixel value of the image shot under the bright exposure condition is the same as when the same pixel value is shot under the dark exposure condition. When the pixel value of the image at the j-th still position photographed under the dark exposure condition is I1j and the pixel value of the image at the j-th still position photographed under the bright exposure condition is I2j, the bright image in the image Ij = C × I1j in the region (region where the pixel value is saturated under bright exposure conditions), and Ij = I2j in the dark region.

  In the map of the light reflection characteristic information α and M obtained as described above, if the numerical value is clearly incorrect due to the shadow of the nose, etc., the value is discarded and the surrounding value is used. Interpolate. The surface reflection characteristic information thus obtained is stored in the processing data storage area in the GPU 10C from the file device 10F by the CPU 10A.

  Prior to the real-time processing of S3 to S8 in FIG. 4 to FIG. 7, the CPU 10A has the human image extraction processing function 102, tracking processing function 103, synthetic face image information processing function 105, skin information processing function 106, shadow component information in the program 100. The processing function 107, the face image composition processing function 108, the person background composition moving image composition processing function 109, and the moving image display processing function 110 are loaded into the program area of the GPU 10C.

  In the present embodiment, the person actual moving image input device 10K is configured by a video camera connected to a computer via an IEEE 1394 interface in order to capture a person actual moving image and perform real-time processing. The video camera uses a video camera of the same model as the video camera V1 used in the background actual moving image input device 10L in order to cancel the characteristics peculiar to each model such as the aspect ratio of the pixel and the distortion in the peripheral portion. preferable. When imaging a person, as shown in FIG. 15, a polarizing plate P3 is attached to the lens of the video camera V3. A polarizing plate P3 is also attached to the illumination L3 that illuminates the person. Behind the person, a screen (green or blue) SC conventionally used for performing chroma key composition processing is arranged. Consider the direction of the illumination L3 and how the screen SC is stretched so that it appears as a single color on the image when it is picked up. The person covers the parts other than the face (below the neck and the hair) with black clothes that are not very glossy. As will be described later, since only the skin region of the face is processed, the portion other than the face in the human image in the synthesized image obtained as a result of the processing does not look unnatural in comparison with the face. It is to do. At the beginning of the actual human moving image, an image (front image) in which the face is kept stationary with the face facing the video camera V3 is captured for a certain period (several seconds) for use in tracking processing described later. It is preferable to do.

  As shown in FIG. 4, first, in the data input / storage processing step S1 ′, the CPU 10A uses the data input / storage processing function 101 in the program 100 to transfer the captured person moving image from the person actual moving image input device 10K to the computer. 10 The captured human moving image is used for real-time image processing by the GPU 10C described later.

  In the image composition processing device 1, in the human image extraction processing step S3 (see FIG. 4), the GPU 10C executes the human image extraction processing function 102 in the program 100 loaded in advance, and is taken in from the human actual moving image input device 10K. Only the person image is extracted from each frame image of the person actual moving image. The person image only extraction process is performed by a normal method conventionally used in the chroma key composition process, and the person area and the background screen area are extracted by separating specific pixel values as threshold values, and the extracted person is extracted. The image is stored in the area of the processing data storage unit in the GPU 10C. Specifically, the extraction stores a frame image of the person actual moving image, and creates a mask image in which the corresponding pixel value is 0 in the background and 1 in the person area with respect to the frame image. This is realized by referring to the mask image together when the frame image is taken out and stored.

  In the image composition processing apparatus 1, in the tracking process (tracking process) step S4 (see FIG. 4), the GPU 10C executes the tracking process function 103 in the program 100, and the person image stored in the storage area on the GPU 10C and the person image Based on the three-dimensional shape information of the face, the person's face image and the tracking information of the person's face are obtained.

  In the present embodiment, the tracking information of the person's face includes particle filtering (Oka, K., and Sato, Y. 2005. Real-time modeling of face deformation for 3-d head pose estimation. In Proceedings of the Based on IEEE International Workshop on Analysis and Modeling of Faces and Gestures, 308-320.) However, in this embodiment, in order to simplify the process, a change in shape due to facial expressions is not taken into consideration. In order to obtain a more natural result, it is preferable to consider changes in shape due to facial expressions.

  First, in order to acquire the initial coordinates of the feature points in the face image to be used for the tracking process, the GPU 10C displays the first frame of the person image stored in the storage area on the GPU 10C on the screen of the display device 10G. And a feature point (group) when the face of a preset person is viewed from the front is displayed on the display device 10G based on the three-dimensional shape information of the face stored in the storage area on the GPU 10C. Display on the screen. Examples of facial feature points include the eyes of the left and right eyes, the corners of the eyes, the nostrils, the boundary positions of the lips, the eyebrows, and the buttocks. In this embodiment, considering the simplification of processing and the accuracy of tracking information, the feature points include the top and bottom of the left and right eyes, the corners of the eyes, the nostrils, and the positions of the upper and lower boundaries of the lips. Is set. In the program 100 of this embodiment, the group of feature points can be translated and enlarged / reduced by the input device 10I on the screen of the display device 10G, and the alignment with the person image is accurate. Well done.

  The operator translates or enlarges / reduces the group of feature points by the input device 10I, and aligns the coordinates corresponding to the feature points in the person image. After the alignment, the operator sends a signal to that effect to the CPU 10A by an input operation from the input device 10I. Upon receiving this signal, the CPU 10A sends the position information of the feature points on the image to the GPU 10C. Receiving it, the GPU 10C acquires an image in the vicinity of the position corresponding to each feature point, and stores it in the storage area together with the position information of the feature point as a template. The GPU 10C starts the tracking process shown in the flowchart of FIG. 16 using the signal as a trigger.

As described above, in the present embodiment, changes due to facial expressions are not taken into consideration, and therefore the movement of the face position in the person image is based on six dimensions of translation (three dimensions) and rotation (three dimensions). Notation becomes possible.
The GPU 10C obtains the position information of the face of the current frame image from the position information of the feature point in the face of the previous and previous frame images in the person image stored in the storage area on the GPU 10C. Prepare multiple possible candidates. Candidates are chosen around the position expected when the face does not accelerate.

Specifically, the center is represented by the following formula (8).
(Position / rotation value in current frame image) = (position / rotation value in previous frame image) + ((position / rotation value in previous frame image) − (two previous frame image) Position and rotation value)) (8)
However, the second term on the right side of Equation (8) is a term relating to the speed of the face movement. Since each value of the previous frame image cannot be acquired at the start of the tracking process, the term relating to speed is set to zero.

  The GPU 10C generates Gaussian noise (random number) such that the fluctuation width is proportional to the value of the term related to the speed, and adds it to the center, so that a plurality of pieces of positional information about the center are obtained as the candidates. prepare. When the fluctuation width becomes a certain value or less, the fluctuation width is set to that value so that Gaussian noise is generated even if the term relating to the speed is zero.

The GPU 10C obtains a degree of coincidence π between the image near the feature point and the template with respect to the candidate. The degree of coincidence π is obtained by subtracting the image near the feature point and the template for each candidate point for each feature point, for each pixel, and for each RGB channel, and obtaining a square sum of the differences between the pixels. It refers to the reciprocal of it. The degree of coincidence π is specifically represented by the following formula (9).
π = 1 / Σ (Ir (x, y, λ) −It (x, y, λ)) ² (9)
Here, x and y are the coordinates of a pixel in the image, λ is any of R / G / B, Ir (x, y, λ) is the pixel of the human actual moving image, the pixel value of the channel, and It is When the template is moved according to the movement of the candidate, the pixel value of the channel of the pixel on the template at the position corresponding to the pixel of the human actual moving image, Σ represents the total pixel existing in the template and the sum over all channels .

The presence or absence of a template corresponding to each pixel in the human actual moving image can also be represented by a mask image. A mask image is created by setting the value of a pixel with a corresponding template to 1 and the value of a non-existing pixel to 0. The calculation formula (9) for the degree of coincidence π is expressed by the following formula (10).
π = 1 / Σ (mask image (x, y) × (Ir (x, y, λ) −It (x, y, λ)) ²
(10)
Here, Σ represents the sum total over all pixels in the image.
The larger this value is, the more the image near the feature point matches the template.

Focusing on a certain dimension x out of the six dimensions, when the value of the i-th candidate dimension x is b (i, x) and the matching degree is π (i), the value of the dimension x in the current frame image Is obtained by the following equation (11).
b (x) = Σb (i, x) π (i) / Σπ (i) (11)
Here, Σ represents the sum total over all selected candidates.

  In the image composition processing device 1, in the skin information processing step S5 (see FIG. 4), the GPU 10C executes the skin information processing function 106 of the program 100 loaded in advance, and the person actual stored in the storage area on the GPU 10C. Skin information including color component information and shadow component information is obtained from each frame image of the moving image. In this embodiment, the skin information is Norimichi, T., Ojima, N., Sato, K., Shiraishi, M., Shimizu, H., Nabeshima, H., Akazaki, S., Hori, K., and Miyake, Y. 2003. Image-based skin color and texture analysis / synthesis by extracting hemoglobin and melanin information in the skin.ACM Transactions on Graphics 22, 3, 770-779. It is determined based on the component analysis method.

  Consider a three-dimensional space consisting of R ′ = − log (R), G ′ = − log (G), and B ′ = − log (B) with the logarithm of each value of the RGB channel and inverted sign. It is known that when the values of each pixel of the captured skin image are projected onto the three-dimensional space, the skin color is almost collected on a certain color component plane in the three-dimensional space. It is also known that deviation from the component plane is caused by uneven illuminance, that is, shadow, in each part of the skin. In the present embodiment, this is used to separate color component information and shadow component information from each frame image as follows.

  First, the color component plane in the three-dimensional space is obtained by a statistical method. Specifically, the principal component analysis method and / or the independent component analysis method can be adopted as this statistical method. In this embodiment, since the calculation result is obtained stably, the principal component is obtained by principal component analysis. Specifically, a principal component analysis is performed on the value of each pixel of the skin projected onto the three-dimensional space, and a plane spanned by the first principal component vector and the second principal component vector is defined as the color component plane. . Since the shadow effect can be expressed by multiplying each RGB value by a value that does not depend on RGB, a shadow unit vector of (1, 1, 1) is selected as the third vector, and the three-dimensional space is represented by these three dimensions. Expressed in an oblique coordinate system with the vector as the axis.

  In order to realize an image effect such as setting the skin color of the target person to be black or white, further to the projection of the value of each pixel on the color component plane in the oblique coordinate system It is preferable to perform independent component analysis to obtain two independent component vectors and use them instead of the first and second principal component vectors. These two independent component vectors correspond to two types of pigments that bring about skin color, that is, melanin pigments and hemoglobin pigments. For example, by performing scale transformation of the axis corresponding to melanin pigments, and then performing inverse transformation, While keeping the skin tone in a natural color, it can be turned black or white. This is described in detail in the literature on the independent component analysis.

Since the values of the pixels are sufficient if they can be compared relatively, there is an arbitrary way to set the oblique coordinate system and the origin, but it is easier to handle the image data if it is non-negative. In this embodiment, after performing the principal component analysis, after obtaining the melanin pigment axis and the hemoglobin pigment axis by independent component analysis in this embodiment, as shown in FIG. It is preferable to provide an offset so that the minimum value of each of the three components does not become smaller than 0 for each pixel value of the skin color. The independent component vector and the offset are obtained in advance by performing several test shootings and analyzing the images.
Using the coordinate system thus determined, color component information and shadow component information are separated from each frame image of a person's face image, skin information consisting of these information is obtained, and the information is used for processing in the GPU 10C. Store in the data storage area.

  In the image composition processing apparatus 1, in the synthesized face image information processing step S6 (see FIG. 5), the GPU 10C executes the synthesized face image information processing function 105 of the program 100 loaded in advance and stores it in the storage area on the GPU 10C. Based on the normal information of the face in the surface reflection characteristic information of the face, the tracking information of the face obtained in each step, the position of the point light source in the light environment information, and intensity information (light quantity). As shown in FIG. 18, the shadow information of the face is obtained for each frame.

For an arbitrary pixel on each frame image, N is the normal vector of the face surface at that position, S (i) is the direction vector of each point light source when viewed from there, and I is the intensity of each point light source. When (i) is assumed, a value proportional to the illuminance of the pixel, that is, the shadow component information L is expressed by the following equation (12).
L = Σdot (N, S (i)) I (i) (12)
Here, Σ is the total over all point light sources (n point approximate light sources), and dot (A, B) is A
Represents the inner product of B and B. By multiplying the skin information from which shadows to be calculated later are removed, the skin color in consideration of the shadows can be obtained.

  In the image synthesis processing device 1, in the synthesized face image information processing step S6, the GPU 10C executes the synthesized face image information processing function 105 of the program 100, and the surface reflection characteristic information of the face stored in the storage area on the GPU, Based on the face tracking information and the light environment information obtained in each step, the surface reflection information of the face is obtained. However, in this embodiment, the surface reflection information of the face is also obtained here assuming that the light reflection characteristic information of the face follows the TS model.

In the TS model, as described above, when the refractive index of the skin is n, the intensity of gloss is α, and the spread (width) is M, the light reflectance f is obtained by the following equation (13). is there.
f = α × (F · G · D) / (4 cos θicos θr) (13)
However, F is a Fresnel term, G is a shape term, and D is a surface roughness term.
The Fresnel term F is expressed by the following formula (14) for normal incidence, and expressed by the following formula (15) except for normal incidence.
F = (n−1) ² / (n + 1) ² (14)
F = (1/2) [tan ² (θi−θt) / tan ² (θi + θt) +
sin ² (θi−θt) / sin ² (θi + θt)] (15)
Moreover, the surface roughness term D is represented by the following formula (16).
D = exp (-tan ² β / M ² ) / πM ² cos ⁴ β (16)
The shape term G is represented by the following formula (17). Here, min (a, b, c) means the smallest value among a, b, and c.
G = min (Gs, Gm, 1) (17)
Here, Gs = 2 (N · H) · (N · S) / (S · H)
Gm = 2 (N · H) · (N · V) / (V · H)
H = (S · V) / | S · V |
N is a modified normal vector, S is a unit vector in the direction of the light source, V is a unit vector in the direction of the imaging means, H is a unit vector that bisects S and V, β is an angle formed by N and H, θi is an incident angle, θr is a light receiving angle, and θt is a refraction angle. θt is obtained from the following equation (18), where n is the refractive index of the skin. Here, as described above, the corrected normal vector corrected by the hybrid method is used as the normal vector.
n = sin θi / sin θt (18)

For an arbitrary pixel on each frame image of the actual moving image of the face, N is the normal vector of the face surface at that position, S (i) is the direction vector of each point light source when viewed from there, and each point When the intensity of the light source is I (i) and the direction vector of the camera when viewed from there is V, the light reflectance F of the pixel can be expressed by the following equation (19). However, each point light source corresponds to a frame of the actual moving image of the face among the approximate point light sources obtained for each frame by the light environment information processing means, and is stored as the light environment information.
F = Σf (i) (19)
Here, Σ is the sum of calculated values over all point light sources, and f (i) is the value of the amount of reflected light obtained according to the TS model for the i-th point light source. The amount of reflected light with respect to the i-th point light source is S (i), cos θi = dot (N, S), cos θr = dot (N, V), α, S as input parameters in the TS model definition formula. M can be obtained by substituting the intensity and spread of the position of the surface reflection characteristics of the face.

  In the image composition processing apparatus 1, in the face moving image composition processing step S7 (see FIG. 6), the GPU 10C executes the shadow component information processing function 107 and the face image composition processing function 108 of the program 100 loaded in advance, and Based on the information, the shadow information of the face, and the surface reflection information of the face, the composite image of the face is synthesized.

In the present embodiment, as shown in FIG. 19, low-frequency shadow component information is obtained by applying a low-pass filter to the shadow component information, and the result obtained by dividing the shadow component information by low-frequency shadow component information is the shadow component information. High-frequency shadow component information. The high-frequency shadow component information is a composite image due to differences in frequency characteristics at high frequencies between facial color component (hemoglobin component, melanin component) information and facial shadow component, which are caused by differences in the acquisition method of each information. Used to prevent unnaturalness.
In this embodiment, the result of multiplying the shadow information of the face and the high-frequency shadow component information of the shadow component information is used as final face shadow information, which is replaced with the shadow component information obtained by the independent component analysis method. Thus, together with the melanin component information and the hemoglobin component information, the color component information considering the shadow is obtained by performing the inverse transformation of the transformation performed by the independent component analysis method. The face reflection light component information of the face is added to this result. Before the addition, each image is multiplied by an appropriate constant so that it finally becomes a natural image when it is combined with the background. This constant is necessary because the amount of light represented by the unit pixel value of each result image of the process of obtaining the surface reflection information, the process of obtaining the color component information, and the process of obtaining the background image is different.

  In the image composition processing device 1, in the person background composition moving image composition processing step S8 (see FIG. 7), the GPU 10C executes the person background composition moving image composition processing function 109 of the program 100, and the face obtained in each of the above steps. The person background synthesized moving image (final synthesized image) is synthesized based on the synthesized image and the person image and the background actual moving image stored in the file storage device 10F. Specifically, a synthesized image of a face, a corresponding one frame image in the actual human moving image, and a corresponding one frame image of the actual background moving image are sequentially combined for each frame and then connected to a moving image. To. Then, in the synthesized moving image of the person, the person image, and in the present embodiment, as shown in FIG. 20, the area of the screen SC corresponding to the background of the synthesized image of the face is replaced with the background actual moving image. A final composite image is obtained.

  As described above, in the image synthesizing apparatus and method of the present embodiment, in synthesizing a person actual moving image to be synthesized with a background actual moving image captured separately, the person's face is previously stored. The surface reflection characteristic information of the face is obtained, and the actual light environment image of the background is imaged together with the background actual motion image, and the face surface reflection characteristic information and the light environment actual moving image are captured. The light environment information obtained by point approximation of the light source from the image and the face tracking information extracted from the human actual moving image are synthesized by real-time processing to obtain face shadow information and face surface reflection information. Since each information is combined with the skin information and high-frequency shadow component information obtained from the person's face image and then combined with the background actual moving image, the lighting conditions in the person and background moving images should be substantially the same. Can It is possible to obtain a natural moving images similar to as if captured at the same place.

The present invention is not limited to the embodiment.
For example, information obtained by other methods can be adopted as the surface reflection characteristic information used for image composition.

  The skin information processing means can use an independent component analysis method instead of the principal component analysis method as an analysis method for deriving the color component plane. The light environment information processing means may use a surface light source, a line light source or the like instead of the point light source as long as the light environment actual moving image can be appropriately converted. If the frequency characteristics can be matched between facial color component (hemoglobin component, melanin component) information and facial shadow component by devising each information acquisition and processing, the high-frequency shadow component information of the shadow component is not used. May be.

  There is no particular limitation on the measurement target of the surface reflection characteristic measuring apparatus of the present invention. In addition to the surface reflection characteristics of each part of the body other than the face, the surface reflection characteristics of the article can be measured.

  The person background synthesized moving image obtained by the present invention is used for, for example, an attraction in an amusement facility, a real-time synthesis process in a television, or the like.

It is a block diagram including the data flow which shows the outline of one Embodiment of the image synthesizing | combining apparatus of this invention. 1 is a block diagram in which an embodiment of an image composition apparatus of the present invention is applied to a general-purpose computer. It is a data flow figure showing a processing step of an image composition method of the present invention. It is a data flow figure showing a processing step of an image composition method of the present invention. It is a data flow figure showing a processing step of an image composition method of the present invention. It is a data flow figure showing a processing step of an image composition method of the present invention. It is a data flow figure showing a processing step of an image composition method of the present invention. It is a schematic diagram of the imaging means used for imaging of a background real image and a light environment actual moving image in the image composition method of the present invention. It is a figure which shows the flow in the point approximation method of the light source for calculating | requiring the light environment information used in the image synthesis method of this invention. It is a schematic diagram which shows one Embodiment of the measuring apparatus of the surface reflection characteristic used in the image composition method of this invention, (a) is a top view, (b) is a side view. It is a flowchart which shows the outline of the measurement procedure in the surface reflection characteristic measuring apparatus of the said embodiment. It is a figure for demonstrating the calculation step of the three-dimensional shape information by the spatial coding method in the information processing means of the surface reflection characteristic measuring apparatus of the said embodiment. It is a figure for demonstrating the correction step of the three-dimensional shape information and normal line information by the hybrid method in the information processing means of the surface reflection characteristic measuring apparatus of the embodiment. It is a conceptual diagram for demonstrating the calculation step of the light reflection characteristic information by the simplex method in the information processing means of the surface reflection characteristic measuring apparatus of the said embodiment, (a) is a series of frame images imaged, (b) FIG. 5 is a diagram showing a graph (point) of pixel values when focusing on a specific position in a series of frame images, and a graph (dashed line) as a result of fitting using the reflection intensity and width as parameters. It is a figure which shows the apparatus used for imaging of a person actual moving image in the image composition method of this invention, (a) is a top view, (b) is a side view. It is a flowchart explaining the tracking process step in the image composition method of the embodiment. It is the schematic explaining the step which calculates | requires the skin information which consists of color component information (a hemoglobin component and a melanin component) and shadow component information from a person's face image by a principal component analysis method and an independent component analysis method. It is a flowchart explaining the synthetic face image information processing step in the image composition method of the embodiment. It is a flowchart explaining the face image composition processing step in the image composition method of the embodiment. It is a flowchart explaining the person background synthetic | combination moving image composition process step in the image composition method of the embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image composition apparatus 11 Person actual moving image input / storage means 12 Background actual moving image input / storage means 13 Optical environment actual moving image input / storage means 14 Surface reflection characteristic information input / storage means 21 Person image extraction processing means 22 Tracking process Means 23 Light environment information processing means 24 Synthetic face information processing means 25 Skin information processing means 26 Shadow component information processing means 27 Face image composition processing means 28 Human background composition moving image composition processing means 29 Moving image display processing means 10 Computer system 100 Image Composition program A Surface reflection characteristic measuring device 1A Light reflection characteristic information measuring means 1B Three-dimensional shape information measuring means 1C Normal information measuring means 2 Movable light source 2A, 2B Light source 21A, 21B Shield plate 3 Imaging means 3A Polarizer 4 Information processing means DESCRIPTION OF SYMBOLS 5 Projection means 6 Fixed light source 6A Polarizer 7 Control means 8 Computer system 9 frames

Claims (7)

An image synthesizer for synthesizing a synthesized image of a person's face synthesized based on an actual moving image of a person with an actual moving image of a background captured separately from the actual moving image of the person,
Person actual moving image input / storage means for inputting or storing the actual moving image of the person;
Background actual moving image input / storage means for inputting or storing the actual moving image of the background;
Light environment actual moving image input / storage means for inputting or storing the light environment actual moving image captured together with the background actual moving image;
Surface reflection characteristic information input / storage means for inputting or storing the surface reflection characteristic information of the person's face composed of three-dimensional shape information, normal information and light reflection characteristic information of the person's face;
Person image extraction processing means for extracting a person image from each frame image of the person's actual moving image stored in the person actual moving image input / storage means;
Based on the extracted person image and the three-dimensional shape information of the face input or stored by the surface reflection characteristic information input / storage means, the position information of the face image of the person and the specific part of the face of the person And tracking processing means for obtaining face tracking information comprising the face direction information;
Light environment information processing for obtaining light environment information corresponding to each frame image of the background actual moving image based on each frame image of the light environment actual moving image input or stored by the light environment actual moving image input / storage means Means,
Synthetic face image information processing means for determining shadow information and surface reflection information of the face based on the face surface reflection characteristic information, the face tracking information, and the light environment information;
Skin information processing means for obtaining skin information comprising color component information and shadow component information from each frame image of the face image of the person;
Face image synthesis processing means for synthesizing the synthesized image of the face based on the skin information, the shadow information of the face, and the surface reflection information of the face;
An image synthesizing apparatus comprising: a person background synthesized moving image synthesizing unit that synthesizes a person background synthesized moving image based on the synthesized image of the face, the person image, and the background actual moving image.   The skin information processing means obtains skin information including the color component information and shadow component information from each frame image of the person's face image using a principal component analysis method and / or an independent component analysis method. The image composition apparatus described.   The light environment information processing means includes a light source according to each frame image of the background actual moving image based on each frame image of the light environment actual moving image input or stored by the light environment actual moving image input / storage means. The image synthesis apparatus according to claim 1, wherein light environment information is obtained by approximating a point light source.   The face image synthesis processing means includes shadow component information processing means for obtaining high-frequency shadow component information from the shadow component information, and in addition to the skin information, the face shadow information, and the face surface reflected light information, The image synthesizing apparatus according to claim 1, wherein the synthesized image of the face is synthesized based on high-frequency shadow component information.   An optical environment in which the optical environment actual moving image is disposed behind a video camera for background imaging that captures the actual moving image of the background, and is set so that the optical axes of the lenses are positioned in the same straight line 2. A moving image including a sphere or a spherical shell, which is imaged by an imaging video camera, is disposed in front of the optical environment imaging video camera, and has a mirror-finished surface passing through the center of the optical axis. The image synthesis device according to any one of? The actual moving image of the person, the actual moving image of the background captured separately from the actual moving image of the person, the actual light image of the background captured with the actual moving image of the background, and the tertiary of the face of the person A data input / storage processing step for inputting or storing surface reflection characteristic information of the person's face composed of original shape information, normal line information, and light reflection characteristic information;
An optical environment information processing step for obtaining optical environment information according to each frame image of the background actual moving image, based on each frame image of the optical environment actual moving image;
A person image extraction processing step of extracting a person image from each frame image of the actual moving image of the person;
A tracking processing step for obtaining tracking information of the face image of the person and the face of the person based on the three-dimensional shape information of the person image and the face;
A synthetic face image information processing step for obtaining shadow information and surface reflection information of the face based on the surface reflection characteristic information of the face, the tracking information of the face, and the light environment information;
Skin information processing step for obtaining skin information consisting of color component information and shadow component information from the face image of the person,
A face image synthesis processing step of synthesizing the synthesized image of the face based on the skin information, the shadow information of the face, and the surface reflection information of the face;
An image synthesis method comprising: a person background synthesized moving image synthesizing step of synthesizing a person background synthesized moving image based on the face synthesized image, the person image, and the background actual moving image.   An actual moving image of a person, an actual moving image of a background captured separately from the actual moving image of the person, an optical environment actual moving image of the background captured together with the actual moving image of the background, and the person A data input / storage processing function for inputting or storing the surface reflection characteristic information of the person's face composed of the three-dimensional shape information of the face, normal information and light reflection characteristic information, or each frame of the actual moving image of the person A human image extraction processing function for extracting a human image from an image; a tracking processing function for obtaining tracking information of the human face image and the human face based on the human image and the three-dimensional shape information of the face; A light environment information processing function for obtaining light environment information corresponding to each frame image of the background actual motion image stored based on each frame image of the environment actual motion image, the surface reflection characteristic information of the face, and the face Tracking information And a synthetic face image information processing function for obtaining shadow information and surface reflection information of the face based on the light environment information, and skin for obtaining skin information comprising color component information and shadow component information from the face image of the person An information processing function, a face image synthesis processing function for synthesizing the synthesized image of the face based on the skin information, the shadow information of the face, and the surface reflection information of the face, the synthesized image of the face, and the person image And an image composition program for realizing a person background composition moving image composition processing function for composing a person background composition moving image based on the actual background moving image.

Images