JP2002232782A - Image processor, method therefor and record medium for program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an apparatus for image processing capable of executing with ease an image compositing processing to change a specific target object to be replaced in a material video by a target object for pasting imaged by a camera. SOLUTION: In a method for image processing, the image processor acquires imaging direction data of the specific target object to be replaced (for example, the face of a specific person on the screen) in a material video of a movie, dram or the like, or image data generated based on the 3DCG and executes, the pasting procesing by compositing the target object (for example, the face of a user) to be pasted imaged by a camera whose position is controlled based on the imaging direction data. According to the constitution, image composition can be attained without forcing various motions to the target object to be pasted (for example, the face of the user).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, an image processing method, and a program storage medium. More particularly, the present invention relates to a camera that moves on a predetermined orbit in synchronization with a scene (material video) such as a movie or a drama. An image processing apparatus, an image processing method, and a program for capturing a user's face and synthesizing it with the face of a character in a material video to generate a video in which the user enters interactively on behalf of a character in the material video It relates to a storage medium.

[0002]

2. Description of the Related Art In a scene such as a movie or a drama, a user's own image is pasted on a person appearing in the scene to generate a video in which the user interactively enters a scene such as a movie or a drama on behalf of the character. As the system, for example, the following are proposed.

First, a three-dimensional shape of a user's face and an image to be attached to a character's face in a movie image are acquired. Next, synthesis processing is executed based on the fitting information of the face model for the face part of the character in the movie. This fitting process is called a match move, and is common as a special effect of a movie. However, this match move processing requires processing in consideration of, for example, the occurrence of discontinuity or vibration in the time direction in each frame constituting each scene of a movie, and as a result, the synthesis processing in frame units is required. .

The face model used in the fitting process can be moved in a three-dimensional space, and the scene of the movie actually progresses, and the user can interactively operate his or her own face, which has been replaced by a character. Also, if, for example, a user speaks words toward the microphone, the mouth shape of the user's face synthesized to speak the words changes. In addition, the expression can be arbitrarily changed by key operation. (For details, see Shigeo Morishima, "Face Recognition / Synthesis and Possibility of New Media," Proceedings of the 6th Image Sensing Symposium, pp.415-424, June 2000.) With regard to the movement of the face, it is conceivable to measure the movement of the user's face and reflect the result in the face model.

In the above-mentioned conventional fitting processing method, it is necessary to capture a user's three-dimensional shape as data to be pasted into an image. Further, even if the technique is advanced by this approach, there are limitations on the mouth shapes and expressions that can be expressed, and unnatural mouth movements and expressions remain in the synthesized image. In order to eliminate such unnaturalness, the user is required to perform exactly the same operation as the characters in the movie with respect to the direction and position of the face, and the face data of the direction and position is extracted and synthesized. Although it is conceivable, it is impossible to perform exactly the same operation as a character such as a movie, which is not realistic. It is also conceivable to take out a synthesized image of each scene after the synthesizing process and execute a minute adjustment process. However, if such a post-process is executed, a real-time interactive system cannot be established.

[0006]

SUMMARY OF THE INVENTION The present invention has been made in view of the problems of the conventional fitting processing as described above, and has been developed for a variety of image data based on a movie, a drama, or 3DCG. It is an object of the present invention to provide an image processing apparatus and an image processing method that can easily and interactively execute an image pasting process of a face or another object and generate a natural synthesized image.

[0007]

SUMMARY OF THE INVENTION A first aspect of the present invention is as follows.
An image processing apparatus for performing a process of replacing a specific object in material image data with a replacement target object and replacing an image of the replacement target object with image data of a paste target object captured by a camera. Image capturing means for controlling an image capturing camera position in accordance with camera position information generated based on the position information to execute image capturing of the object to be pasted, and capturing the image of the object to be replaced in the image data by the image capturing An image synthesizing unit that executes an image synthesizing process of changing the image of the paste target object photographed by the unit.

Further, in one embodiment of the image processing apparatus of the present invention, the image processing apparatus includes: a material video storage unit storing material image data including the replacement target object; Camera position information storage means for storing camera position information generated based on the camera position information, wherein the camera position information stored in the camera position information storage means corresponds to each frame of the image data stored in the material video storage means. Output to the image capturing unit of the camera position information recorded in the camera position information storage unit, and output of the material video recorded in the material video storage unit to the image synthesizing unit. Are executed as synchronized processing, and the image synthesizing unit inputs the image data from the material image storage unit. Material video and the image of the paste target object input from the image capturing means are input in parallel, and based on the parallel input data, the image of the replacement target object in the material image is It is characterized in that it is configured to execute processing for changing to an image.

Further, in one embodiment of the image processing apparatus of the present invention, the image processing apparatus further includes a captured image extracting unit for extracting only an image of an object to be pasted from the image captured by the image capturing unit. Has,
The captured image extracting means outputs the extracted image of the paste target object to the image synthesizing means.

Further, in one embodiment of the image processing apparatus of the present invention, the image processing apparatus further stores hidden mask information storing hidden mask information as occluded area information of the object to be replaced in the material image data. Means for inputting hidden mask information from the hidden mask information accumulating means, and regarding the mask area, not the image data of the object to be pasted from the image capturing means but the material A synthesizing process for selecting data of image data as output data is performed.

Further, in one embodiment of the image processing apparatus of the present invention, the image capturing means includes a camera for capturing an image of an object to be pasted, and a camera moving means for moving the camera according to the camera position information. It is characterized by having.

Further, in one embodiment of the image processing apparatus according to the present invention, the image processing apparatus further includes a lighting unit storing lighting position information generated based on luminance information of the replacement target object in the material image data. The image capturing device includes a position information accumulating unit, and the image capturing unit includes a lighting unit that irradiates the paste target object and a lighting moving unit that moves the lighting unit according to the lighting position information.

Further, in one embodiment of the image processing apparatus of the present invention, the image processing apparatus further includes a replacement object position information storage unit that stores position information of the replacement object in the material image data. The image synthesizing unit adjusts the size and position of the image of the paste target object input from the image capturing unit based on the position information of the replacement target object input from the replacement target object position information storage unit. It is characterized in that it is configured to execute processing.

Further, in one embodiment of the image processing apparatus of the present invention, the image processing apparatus further includes a three-dimensional model storage unit storing three-dimensional model data, and a three-dimensional model storage unit stored in the three-dimensional model storage unit. Virtual camera information storage means for storing virtual shooting viewpoint information of the replacement target object included in the model data; material video generating means for generating material video data based on the three-dimensional model data and the virtual shooting viewpoint information; The image capturing means controls and pastes a photographing camera position according to camera position information generated based on virtual photographing viewpoint information of a replacement target object included in the material video generated by the material video generating means. It is characterized in that it is configured to capture an image of a target object.

Further, according to a second aspect of the present invention, a process for replacing a specific object in material image data as an object to be replaced and replacing the image of the object to be replaced with image data of an object to be pasted taken by a camera. An image processing method to be executed, wherein an image capturing step of controlling an image capturing camera position according to camera position information generated based on the image capturing position information of the replacement target object and capturing an image of the paste target object, An image synthesizing step of executing an image synthesizing process of changing an image of the replacement target object in the image data into an image of the paste target object photographed in the image capturing step. is there.

Further, in one embodiment of the image processing method of the present invention, the image processing method further includes a camera position information storing step of generating and storing camera position information based on shooting direction information of the object to be replaced. The output of the camera position information generated in the camera position information storage step to the image capturing means, and the output of the material video recorded in the material video storage means to the image synthesizing means are executed as synchronized processing, In the image synthesizing step, the material image input from the material image storing means and the image of the paste target object input from the image capturing means are input in parallel, and the material image is input based on the parallel input data. Change the image of the object to be replaced in the image of the object to be pasted And executes the management.

Further, in one embodiment of the image processing method of the present invention, the image processing method further includes a captured image extracting step of extracting only the image of the object to be pasted from the image captured in the image capturing step. It is characterized by having.

Further, in one embodiment of the image processing method of the present invention, the image processing method further comprises generating and storing hidden mask information as occluded area information of the object to be replaced in the material image data. The image synthesizing step includes inputting the hidden mask information, and regarding the mask area, not the image data of the object to be pasted captured in the image capturing step, but the material image data. A synthesis process of selecting the data as the output data.

Further, in one embodiment of the image processing method of the present invention, the image processing method further generates and stores illumination position information generated based on luminance information of the object to be replaced in the material image data. The image capturing step moves the illumination for irradiating the paste target object according to the illumination position information generated in the illumination position information accumulation step.

Further, in one embodiment of the image processing method according to the present invention, the image processing method further includes a step of accumulating position information of the object to be replaced, which generates and stores position information of the object to be replaced in the material image data. Then, the image synthesizing step, based on the position information of the replacement target object generated in the replacement target object position information accumulation step,
A size adjustment and a position adjustment process of the image of the object to be pasted captured in the image capturing step are performed.

Further, in one embodiment of the image processing method of the present invention, the image processing method further includes generating virtual photographing viewpoint information of the replacement target object included in the three-dimensional model data stored in the three-dimensional model storage means. Storing a virtual camera information to be stored, a material video generating step of generating material video data based on the three-dimensional model data and the virtual shooting viewpoint information, and the image capturing step includes: Controlling the photographing camera position according to the camera position information generated based on the virtual photographing viewpoint information of the replacement target object included in the material video generated in the step, and executing the image capturing of the paste target object. .

Further, a third aspect of the present invention is a process for replacing a specific object in material image data with a replacement target object and replacing the image of the replacement target object with image data of a paste target object photographed by a camera. A program storage medium for providing a computer program to be executed on a computer system, wherein the computer program controls a shooting camera position according to camera position information generated based on shooting position information of the replacement target object. Image capturing step of executing image capturing of the object to be pasted and image combining processing for changing the image of the object to be replaced in the image data to the image of the object to be pasted captured in the image capturing step In the program storage medium comprising: an image synthesizing step of executing, a.

The program storage medium of the present invention is a medium that provides a computer program in a computer-readable format to a general-purpose computer system that can execute various program codes. The form of the medium is not particularly limited, such as a recording medium such as a CD, an FD, and an MO, and a transmission medium such as a network.

Such a program storage medium defines a structural or functional cooperative relationship between the computer program and the storage medium for realizing the functions of a predetermined computer program on a computer system. Things. In other words, by installing the computer program into the computer system via the storage medium, a cooperative operation is exerted on the computer system, and the same operation and effect as the other aspects of the present invention can be obtained. You can do it.

Still other objects, features and advantages of the present invention are:
It will become apparent from the following more detailed description based on the embodiments of the present invention and the accompanying drawings.

[0026]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 FIG. 1 is a block diagram of an interactive image processing apparatus according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a specific configuration example of the image capturing unit 100 in the image processing apparatus of FIG. The outline of the image processing apparatus of the present invention will be described with reference to FIGS. In the embodiment described here, it is assumed that a process of pasting a face image of a user who captures the face of a character in moving image data such as a movie or a drama using the image capturing unit 100 illustrated in FIG. explain. In this embodiment, an example in which the captured image is a user's face will be described. However, other objects, for example, various subjects such as an entire person, a car, a building, and a scenery can be configured as the captured object.

In the following description, an example of processing for replacing the face of a person in an original image such as a movie or drama with the face of a user captured by a camera will be described. In the following description, an object to be replaced, such as a person's face in a material image such as a movie or a drama, is referred to as a replacement target object, such as a user's face captured by a camera.
An object to be pasted on the original image is called a pasting target object.

The camera position information storage unit 1 stores material data stored in the material storage unit 8 in advance, for example, the face 5 of the user in accordance with the direction (direction) of the face of a specific character in the movie and the position information. Camera position information for capturing images is stored as time-sequential sequential data. That is, the camera position information storage unit 1 captures a face image, which is an object to be pasted, from a direction coinciding with an image object (here, the face of a specific character) to be replaced by the material video storage unit 8. Is stored as time-sequential sequential data.

The camera position information stored in the camera position information storage unit 1 is sent to a camera movement control unit 2 constituting the image capturing unit 100 (ex. See FIG. 2). The camera movement control unit 2 controls the camera movement unit 3 based on the camera position information input from the camera position information storage unit 1. As shown in FIG. 2, the camera moving unit 3 is a device that can move to a desired position and direction in a space, for example, like an arm robot. The camera 4 attached to the tip of the camera moving unit 3 captures the face 5 of the user. The image captured by the camera 4 is a captured image extraction unit 6
Is sent as time-series data.

The captured image extracting unit 6 extracts only partial data (in this example, face area image data) to be synthesized from the image captured by the camera 4 of the image capturing unit 100. The extracted image is sent to the image synthesizing unit 7, where the synthesizing process is executed.

The material video storage unit 8 stores video of a movie or drama scene (hereinafter, referred to as material video), and the stored images are sent to the image synthesizing unit 7 in time series. Hidden mask information is stored in the hidden mask information storage unit 9 and is sent to the image synthesizing unit 7 in time series in synchronization with the output of the stored image from the material video storage unit 8. Hidden mask information storage unit 9
The hidden mask information stored in is that the part of the replacement target object in the material image
For example, it is hidden area information in the case of being hidden behind a pillar, a building, or the like.

The image synthesizing unit 7 outputs material video data input from the material video accumulating unit 8 instead of the image extracted by the captured image extracting unit 6 for the pixel area in accordance with the hidden mask information. Pasting of the captured image to the material video can be executed faithfully. For example, when the image of a hand or a pillar covers the face of a character in the material video and part or all of the face is hidden, synthesis based on the hidden mask information stored in the hidden mask information storage unit 9 By executing the processing, the hidden portion is faithfully reproduced even in the composite image.

The image synthesizing unit 7 generates an image obtained by synthesizing the material video and the image of the user's face extracted from the captured image using hidden mask information. However, when there is no state where the face is hidden by another object, the composition can be performed without the hidden mask information. The generated image is sent to the output unit 10 and output. The output unit is, for example, C
It is a monitor device such as an RT or LCD, a recording device such as a CD, DVD, or video, or a communication means for data communication.

The camera position information recorded in the camera position information storage unit 1, the material image recorded in the material image storage unit 8, and the hidden mask information recorded in the hidden mask information storage unit 9 are read out at a certain time. Synchronized in series.

FIG. 2 shows the configuration of the image capturing unit 100.
This will be described with reference to FIG. The configuration in FIG. 2 shows a configuration in which a video of a user's face is captured as an object to be pasted. The user 20 fixes the face to be the object to be pasted on the fixed base 21. The positional relationship between the fixed base 21 and the camera moving unit 3 (here, an arm robot) is known. The camera 4 attached to the camera moving unit 3
An image of the user's face is captured while moving under the control of the camera movement control unit 2 according to the camera position information stored in the camera position information storage unit 1 described above.

Here, if the camera moving unit 3 can move the camera 4 to a desired position and direction,
The configuration is not limited to the configuration of the arm robot shown in FIG. For example, as shown in FIG. 2, the fixed base 21 is attached with a plate painted in a single color such as blue or green with a hole for projecting an object to be pasted (face), and the face image extraction unit can easily extract the face. Configuration. Alternatively, a configuration may be adopted in which a portion other than the paste target object (ex. User's face) is covered with a cloth of blue, green, or the like.

Since the image captured by the camera 4 is an image including the surrounding image of the face, the captured image extracting unit 6 shown in FIG. 1 executes a process of extracting only the face image to be pasted. For example, an object to be pasted (ex. User's face) is photographed by the camera 4 together with the fixed base 21 having monochromatic color information such as blue or green, and monochromatic color information data is deleted from the photographed image to obtain a desired area. It is possible to extract the paste target object image, that is, to use the chroma key technique.

A display 25 for presenting the progress of the material video and the line information at that time to the user is arranged at a position visible to the user, and the user looks at the material video displayed on the display 25 to be pasted. A configuration in which the mouth is moved based on the expression of the characters or the lines may be used. With such a processing configuration,
The user's face according to the expression closer to the character can be pasted on the material video.

The camera position information storage unit 1 shown in FIG.
Camera position information based on material video such as a movie stored in the material video storage unit is stored in advance. The direction data is obtained based on a specific object (for example, the face of a specific character) that is a replacement target object in the material video, and the position of the camera that captures the user's face 5 from the same direction is defined as camera position information. I do.

The region of the face of the character in the material video may be specified manually or may be automatically extracted. As a specific image extraction processing method from an image, for example,
H. Rowley, S. Baluja, and T. Kanade, "Neural Network-
Based Face Detection, "IEEETransactions on Pattern
Analysis and Machine Intelligence, Vol. 20, No. 1,
January, 1998, pp. 23-38.

A method for obtaining camera position information will be described with reference to FIG. FIG. 3 shows camera position information for setting a camera position for performing image capturing processing of an object to be pasted (user's face) with respect to images (a) and (d) of two frames p and q of a material video. It is a figure explaining the acquisition processing of.

First, each frame image (a),
An orthogonal coordinate axis is set based on the position and orientation of the character 30 as the replacement target object in the image (d). Here, the origin is the base of the neck, Z is the direction of the face, Y is the top of the head, and X is the direction perpendicular to the YZ plane. In addition, the detection method of the direction of the face in the image is, for example,
T. Horrprasert, Y. Yacoob, and L. Davis, "Computing 3-
D head orientation froma monocular image sequenc
e, "Proc. of the second International Conference o
n Automatic Face and Gesture Recognition, pp.242-2
47, 1996.

In the environments (b) and (e) in which the user's face 40 is captured by the camera 50, the coordinate system is fixed to the user's face, and the way of taking the X, Y, and Z axes corresponds to each. The position of the camera 50 is set so as to correspond to the characters in the material video of each frame shown in the frame images (a) and (d). That is, the camera position of the frame of the material video with respect to the image (a) is the user's face
(A), in an environment in which
The two coordinate axes in (b) are set as camera positions that almost coincide.

As shown in the result (c), the camera 50 can capture the user's face 40 having the same position, orientation, and size as the face of the character 30 in (a). (C) is an image obtained by deleting the surrounding image and extracting only the user's face 40 as a captured image using the above-described chroma key technique.

Similarly, the image (d) of the frame of the material video
The camera position with respect to (d) and (e) in the environment where the user's face 40 is captured by the camera 50 in (e).
Are set as camera positions that match the two coordinate axes. As shown in the result (f), the camera 50 can capture the user's face 40 having the same position, orientation, and size as the face of the character 30 in (d).

Incidentally, FIG. 3 shows (a), (b),
Each of (c) and (d), (e), and (f) is a camera position setting process for one frame image of the material image, and such a camera position setting process is executed for each frame of the material image. These are stored in the camera position information storage unit 1 as time series information.

The stored camera position information is output to the camera movement control unit 2, and the camera movement unit moves the camera 4 in accordance with the camera position information to capture an object to be pasted (ex. User's face). For example, the user's face from the same direction as the face of a specific character in the material video can be captured in chronological order. In order to further reduce discomfort in the synthesized image, it is preferable to perform appropriate parameter setting and correction processing in consideration of the focal length of the camera and the distortion caused by the lens. Note that these parameter setting and correction processing are executed by the image capturing unit 100 or the image synthesizing unit 7.

As described above, in the material video, not only the position and orientation of the face due to camera movement such as camera movement and zooming, but also the position and orientation of the face due to the movement or turning of the character Can be realized only by moving the position of the camera, and the user can keep the position of the face fixed. As described above, the material video is displayed on the display 25 shown in FIG. 2, the facial expression of the person and the line information are presented to the user, and the mouth is moved based on the expression or the line of the character to be pasted. By performing the above, the actor himself can enjoy the performance of the face and voice, and it is possible to synthesize more realistic image data.

Next, a process of synthesizing with hidden mask information in the image synthesizing section 7 will be described with reference to FIG. As described above, the hidden mask information is shielding area information in the case where a part of the replacement target object is hidden by a near object, for example, a pillar or a building.

As shown in FIG. 4A, if a frame such as a pillar 60 exists in a frame of the material image so as to hide the face of the character 30 to be replaced, the image synthesizing unit 7 In the combining process, it is necessary to draw the pillar 60, that is, output the pillar 60. Therefore, a mask as shown in FIG. 4B is set. Pixels in the mask area 70 are not output data of the captured image extraction unit 6 but always execute processing of selecting a material image output from the material image storage unit 8, so that the user's face can be selected in the same manner as the material image. Is output as an image hidden by the pillar 60, and the problem of occlusion is solved. In addition,
The image synthesizing unit 7 performs a process of adding an effect of alpha blending (blur process) around the mask area 70, so that a more natural synthesis can be performed.

Next, the processing flow in the image processing apparatus of the present invention will be described with reference to camera position information and mask information generation processing.
The description will be made by using the flow separately for the synthesis process.

First, the procedure of the camera position information and mask information generation processing will be described with reference to FIG. In step S101, i (i indicating the frame number as an initial value)
= 1 to n, where n is the number of frames of the material video), i =
1 is performed.

Next, in step S102, an i-frame image of the material video is obtained, and in step S103, an object image to be replaced (ex. A face of a specific character) is selected from the i-frame image of the material video.
Extract face position and orientation information. Next, based on these information, in step S104, camera position information as the photographing direction and position information of the paste target object (ex. Face) is generated. This process is the process described above with reference to FIG. This camera position information is stored in the camera position information storage unit 1.

Further, in step S105, hidden mask information generation processing is performed. This is the processing described above with reference to FIG. 4. In the frame i of the material video, areas that exist in a form that hides the face of the character to be replaced are extracted, and these areas are set as mask areas. (Refer to FIG. 4B.) In this process, mask information is generated corresponding to each frame of the material video and stored in the hidden mask information storage unit 9.

Next, in step S106, it is determined whether or not the processing has been completed for all the frames. If there is an unprocessed frame, the increment processing of the frame number i is executed in step S107. The process is executed for frame i + 1.

When the generation of the camera position information and the hidden mask information for all the frames is completed, step S1 is performed.
At 08, the parameters of the image are adjusted so that each frame has a smooth connection. This is a process of adjusting parameters so that discontinuity and vibration do not occur when an image is viewed as a moving image. Note that this processing may be performed at the end in a lump, or may be a frame unit processing in which each time information about each frame is generated, adjustment is made in consideration of continuity with past generated frame information.

Next, the procedure of the synthesizing process will be described with reference to FIG. First, in step S201, FIG.
2. An image of the paste target object (ex. User's face) captured by the camera 4 in Step 2 is acquired.

Next, in step S202, the captured image extracting unit 6 causes the object to be pasted (ex.
A face) region extraction process is executed. For this processing, as described above, it is possible to use the chroma key technique of deleting the single color information from the captured image and extracting the paste target object image as a desired area.

Next, in step S203, the image synthesizing section 7 executes a synthesizing process of the captured image and the material video. In the synthesizing process, the position, size, direction, and the like of the replacement target object (ex. The face of the character) of the material video and the paste target object, for example, the user's face extracted from the captured image, match. Therefore, it is only necessary to overwrite the extracted user's face area on the material image. As described above, by adding an alpha blending effect around the face area on which the pasting process has been performed, a more natural composition can be achieved. Also,
When there is hidden mask information output from the hidden mask information storage unit 7, the image synthesizing unit 7 selects the output data of the material image for the mask area.

For example, when the color, brightness, etc. of the object to be pasted are changed depending on the photographing direction, these parameters are matched between frames, or a process of matching the parameters with the image of each frame. The parameter synthesizing process may be executed by the image synthesizing unit 7 in step S203 ′ as an optional step. The process of step S203 'is an optional process, and is not always necessary. This parameter adjustment processing can be automatically executed under a certain condition setting. For example, if the luminance value of the paste target object differs from the previous frame by a certain threshold or more due to the difference in the shooting direction, the luminance value of the paste target object is changed, or the average luminance value of the material image and the paste When the luminance value of the paste target object is different from the luminance value of the paste target object by a certain threshold value or more, processing such as changing the luminance value of the paste target object is automatically executed. In addition, the outline of the character's face region in the material video is different from the contour of the user's face, and if it does not fit in the user's region, the character's face in the material video will remain when overwritten. It is desirable to execute a process of generating and replacing the background (or skin) in the face region (particularly the peripheral portion) of the character in the video.

The image thus synthesized is output to an output unit (ex. Display monitor, data storage means, communication means) in step S204. In step S205, it is determined whether or not the material video has been completed, and if not completed, the processing of step S201 and subsequent steps is repeatedly executed.

Further, in order to make the composition more visually correct, when the user first fixes the face on the fixed base, correction of the displacement of the fixed base and the user's solid state such as face size and skin color are performed. The process of calibrating the difference may be performed before the process of step S201.

As described above, according to the image processing apparatus of the present invention, a specific replacement target object (ex. A specific character's face) of an image stored in the material video storage unit is pasted by a camera. It is possible to quickly and easily execute the image synthesizing process of changing and outputting the target object (ex. User's face) as a process synchronized with the reproduction of the stored image.

The user obtains an image corresponding to the object to be replaced with the material image by moving the camera without fixing the face to the fixed base shown in FIG. 2 and performing an operation such as moving or changing the direction. Can be attached.

[Embodiment 2] FIG. 7 is a block diagram of an interactive image processing apparatus according to a second embodiment of the present invention. FIG. 8 is a diagram showing a specific configuration example of the image capturing unit 80 in the image processing apparatus of FIG. FIG.
The outline of a second embodiment of the image processing apparatus of the present invention will be described with reference to FIG.

The lighting position information storage section 11 stores lighting position information. The lighting position information is used to capture a replacement target object in the material data stored in advance in the material storage unit 8, for example, a user's face 5 having a luminance distribution corresponding to the luminance information of the face of a specific character in a movie. Lighting 1
4 is stored as time-sequential sequential data. That is, the illumination 1 for capturing the face image of the paste target object having the luminance distribution corresponding to the luminance distribution of the image object (here, the face of a specific character) to be replaced by the material video storage unit 8
4 is stored as sequential data.

The illumination position information stored in the illumination position information storage unit 11 is sent to the illumination movement control unit 12 constituting the image capturing unit 80 (ex. FIG. 8). The illumination position information includes, for example, time-series information on the position and direction of the tip (illumination 14) of the illumination moving unit 13. Lighting movement control unit 12
Then, the illumination moving unit 13 is controlled according to the illumination position information.
The illumination moving unit 13 is a device that can move to a desired position and direction in a space, for example, like an arm robot. Illumination 14 attached to the tip of illumination moving unit 13
Illuminates the face 5 of the user.

The camera position information storage unit 1 stores material data stored in the material storage unit 8 in advance, for example, the face 5 of the user in accordance with the direction (direction) of the face of a specific character in the movie and the position information. Camera position information for capturing images is stored as time-sequential sequential data. That is, the position information of the camera for capturing the face image, which is the object to be pasted, from the direction corresponding to the image object (here, the face of a specific character) to be replaced in the material video storage unit 8 is stored as sequential data. Has accumulated.

The camera position information stored in the camera position information storage unit 1 is stored in an image capturing unit 80 (ex. FIG. 8).
(See FIG. 2). The camera movement control unit 2 controls the camera movement unit 3 based on the camera position information input from the camera position information storage unit 1. As shown in FIG. 8, the camera moving unit 3 is a device that can move to a desired position and direction in a space, for example, like an arm robot. The camera 4 attached to the tip of the camera moving unit 3 captures the face 5 of the user. The image captured by the camera 4 is a captured image extraction unit 6
Sent in chronological order.

The captured image extracting section 6 extracts only partial data (in this example, face area image data) to be synthesized from the image captured by the camera 4 of the image capturing section 100. The extracted image is sent to the image synthesizing unit 7, where the synthesizing process is executed.

The material video storage unit 8 stores video of a movie or drama scene (hereinafter, referred to as material video), and the stored images are sent to the image synthesizing unit 7 in a time series. Hidden mask information is stored in the hidden mask information storage unit 9 and is sent to the image synthesizing unit 7 in time series in synchronization with the output of the stored image from the material video storage unit 8. The synthesizing process in the image synthesizing unit 7 and the process for the hidden mask information are the same as those in the first embodiment, and thus the description is omitted. The image generated by the image synthesizing unit 7 is sent to the output unit 10 and output. The output unit is, for example, a monitor device such as a CRT or LCD, a recording device such as a CD, DVD, or video, or a communication unit for data communication.

The camera position information stored in the camera position information storage unit 1, the illumination position information stored in the illumination position information storage unit 11, the material image stored in the material image storage unit 8, and the hidden mask information storage Reading of the hidden mask information recorded in the unit 9 is performed in a time-series manner.

The configuration of the image capturing section 80 will be described with reference to FIG. The configuration in FIG. 8 illustrates a configuration in which a video of a user's face is captured as an object to be pasted. The user 20 fixes the face to be the object to be pasted on the fixed base 21. The positional relationship between the fixed base 21 and the camera moving unit 3 (here, an arm robot) is known. Camera 4 attached to camera moving unit 3
Captures an image of the user's face while moving under the control of the camera movement control unit 2 according to the camera position information stored in the camera position information storage unit 1 described above.

At this time, the illumination 14 attached to the illumination moving unit 13 moves to the user's face while moving under the control of the illumination movement control unit 12 according to the illumination position information stored in the illumination position information storage unit. Turn on the lights. here,
The camera moving unit 3 and the lighting moving unit 13 do not need to be arm robots as long as they can move the camera 4 and the lighting 14 to desired positions and directions. For example, as another configuration example of the illumination moving unit 13 and the illumination 14, a configuration in which a large number of light sources are arranged around the user and the light sources blink may be used to provide a lighting effect from various directions. It is.

For example, as shown in FIG. 8, a fixed base 21 is provided with a plate painted in a single color such as blue or green with a hole for projecting an object to be pasted (face), and a face image extraction unit extracts the face. The configuration is easy to remove. Alternatively, a configuration may be adopted in which a portion other than the paste target object (ex. User's face) is covered with a cloth of blue, green, or the like.

Since the image captured by the camera 4 is an image including the surrounding image of the face, the captured image extracting unit 6 shown in FIG. 7 executes a process of extracting only the face image to be pasted. For example, an object to be pasted (ex. User's face) is photographed by the camera 4 together with the fixed base 21 having monochromatic color information such as blue or green, and monochromatic color information data is deleted from the photographed image to obtain a desired area. It is possible to extract the paste target object image, that is, to use the chroma key technique.

The display 25 for presenting the progress of the material video and the line information at that time to the user is arranged at a position where the user can see the material video, and the user looks at the material video displayed on the display 25 to be pasted. A configuration in which the mouth is moved based on the expression of the characters or the lines may be used. With such a processing configuration,
The user's face according to the expression closer to the character can be pasted on the material video.

In the present embodiment, by controlling the illumination, it is possible to make the synthesized image more natural and high quality than the first embodiment. The illumination position information is created by analyzing the image and estimating the position of the light source in each frame of the material video. Therefore, if the number of light sources is increased, higher quality synthesis can be performed. Also, in the case of a material video where a scene where a character is turning around with a camera fixed is shot, the positional relationship between the camera and the lighting does not change, so the lighting is attached to the camera moving part as Is also good.

[Embodiment 3] FIG. 9 is a block diagram of an interactive image processing apparatus according to a third embodiment of the present invention. The camera position information recorded in the camera position information storage unit 1 is sent to the camera movement control unit 2. The camera position information is information that is generated in advance based on the replacement target object of each frame of the material video as in the first embodiment, and the camera 4 that captures the paste target object (ex. It consists of time-series information on the position and direction of the tip of the camera moving unit 3 to be driven.

The camera movement control unit 2 controls the camera movement unit 3 according to the camera position information recorded in the camera position information storage unit 1. The camera moving unit 3 is a device that can move to a desired position and direction in a space, for example, like an arm robot. The camera 4 attached to the tip of the camera moving unit 3 captures the face 5 of the user. Images captured by the camera 4 are sent to the captured image extraction unit 6 in a time series.

The captured image extracting section 6 extracts only the face portion as an object to be pasted from the image captured by the camera 4 using, for example, the chroma key technique, and sends the extracted image to the image synthesizing section 7. In the material video storage unit 8,
Video of a movie or drama scene (hereinafter referred to as material video) is stored, and the images are sent to the image synthesizing unit 7 in time series. The hidden mask information is stored in the hidden mask information storage unit 9 and is sent to the image synthesizing unit 7 in time series.
Hidden mask information is applied to the face of a character that is the replacement target object in the material video, for example, when a part or all of the face is hidden by an image of a hand or a pillar covering the front, This is information (see FIG. 4) for realizing the hiding.

Face position information is recorded in the replacement target object position information storage section 15 and is sent to the image synthesizing section 7 in time series. The face position information is information for superimposing the user's face region that has been photographed and extracted on the character's face region in the material video, the positional relationship, the scale of the region, and the rotation in the image if necessary. It consists of information.

The image synthesizing unit 7 includes a material image input from the material image storing unit 8, an image of the user's face as an object to be pasted input from the captured image extracting unit 6, and a hidden mask information storing unit. A composite image is generated using the hidden mask information input from step 9 and the face position information input from the replacement target object position information storage unit 15. The image synthesizing unit 7 performs size adjustment and position adjustment processing of the image of the paste target object using the face position information input from the replacement target object position information storage unit 15. However, when there is no state where the face is hidden by another object, the composition can be performed without the hidden mask information. The generated image is sent to the output unit 10 and output. The output unit is, for example, a monitor device such as a CRT or LCD, a recording device such as a CD, DVD, or video, or a communication unit for data communication. Note that the camera position information recorded in the camera position information accumulation unit 1, the material image recorded in the material image accumulation unit 8, the hidden mask information recorded in the hidden mask information accumulation unit 9, and the replacement target object position information accumulation unit The reading of the face position information recorded in No. 15 is performed in a time-series manner.

The method of obtaining the camera position information and the face position information in the present embodiment will be described with reference to a specific example with reference to FIG. The meaning of the camera position information here is different from that of the first embodiment. For example, as shown in FIG.
For the image (a) of a frame of a material video including the face 90 of a character as a replacement target object, orthogonal coordinate axes are set based on the position and orientation of the character. Here, the origin is the base of the neck, Z is the direction of the face, Y is the direction above the head, and X is the direction perpendicular to the YZ plane.

In the environment (b) in which the user's face 92 as an object to be pasted is photographed by the camera 94, the coordinate system is fixed to the user's face, and the axes are determined by the characters in the material video. The camera position is set to be the same.

When the user's face 92 as an object to be pasted is photographed by the camera 94 with the camera position set in this way, as shown in FIG.
The user's face 96 in the same direction as the character's face 90 in FIG.
Can be captured. (C) is an image obtained by extracting only the user's face from the image captured by the camera 94 in (b). In other words, the camera position information is a camera position that has a different position and size in the face image but has the same face direction as the face in the material video image.

Further, the position information of the face 90 of the character in the image of the material video (a), the face 90 of the character in the image of the material video (a), and the extracted face image 96 of (c) And the enlargement / reduction information of the image are used as face position information.

At the time of the synthesizing process in the image synthesizing unit 7, the position of the user's face as the object to be pasted, which is extracted by using the scale-down information for each frame input from the object position information storing unit 15 for replacement. And the scale is changed to generate a face image 98 of the user of the paste target object as shown in FIG.

As described above, the position information of the replacement target object (ex. The face of the character) in the image of the material video (a), the size of the replacement target object, and the size of the captured image of the paste target object The enlargement / reduction information based on the comparison is stored in the replacement target object position information storage unit 15, and the composition processing of the pasting target object with respect to the replacement target object of the material video can be executed using the position information and the enlargement / reduction information.

By executing such a synthesizing process, as in the first embodiment, the camera position is changed according to the size of the image of the object to be replaced, that is, the process of moving the camera away from or closer to the subject is omitted. It is possible to take an image taking only the shooting direction of the camera into consideration. The effect of image deformation due to the camera's focal length and lens distortion remains, so either correct it or use a policy of "Allow some deformation if the face is small and distant." Image capture is possible.

The procedure of the camera position information, face position information, and mask information generation processing will be described with reference to FIG. In step S301, a setting process of i = 1 is executed for i (i = 1 to n, where n is the number of frames of a material video) indicating a frame number as an initial value.

Next, in step S302, an i-frame image of the material video is obtained. In step S303, an object image to be replaced (ex. A face of a specific character) is selected from the i-frame image of the material video.
Extract face position and orientation information. The face position information also includes the size. Basically, assuming that the size of an object to be pasted (ex. User's face) captured by the camera 4 in FIG. 9 is constant, an object image to be replaced with a material video (ex. Face of a specific character) Can be set based on the size of the image.

Next, based on these information, step S
At 304, the paste target object (ex.
Camera position information as a shooting direction of a face is generated, and further, position information of an object to be pasted (ex. Face) and face position information as scaling information are generated. This process is the process described above with reference to FIG. The generated camera position information is stored in the camera position information storage unit 1, and the face position information is stored in the replacement target object position information storage unit 15.

Further, in step S305, a process of generating hidden mask information is performed. This is the processing described above with reference to FIG. 4. In the frame i of the material video, areas that exist in a form that hides the face of the character to be replaced are extracted, and these areas are set as mask areas. (Refer to FIG. 4B.) In this process, mask information is generated corresponding to each frame of the material video and stored in the hidden mask information storage unit 9.

Next, in step S306, it is determined whether or not the processing for all the frames has been completed. If there is an unprocessed frame, the process of incrementing the frame number i is executed in step S307. The process is executed for frame i + 1.

When the generation of camera position information, face position information, and hidden mask information for all frames is completed,
In step S308, adjustment of image parameters is performed so that each frame has a smooth connection. This is a process of adjusting parameters so that discontinuity and vibration do not occur when an image is viewed as a moving image.
In addition, this processing is performed at the end collectively, or
The frame unit processing may be adjusted in consideration of continuity with past generated frame information every time information about each frame is generated.

The image synthesizing processing procedure of the third embodiment is as follows.
This is almost the same as FIG. 6 described in the first embodiment. However, the processing performed by the image synthesis unit that generates an image obtained by synthesizing the material video and the image of the user's face extracted from the captured image is different. Since the faces of the characters in the material video and the user's face extracted from the captured image are oriented only in direction, an image with the same position and size is generated using face position information, and Will be overwritten.

In this embodiment, by adding face position information to the first embodiment, it is possible to reduce the moving range of the camera and make the system compact. In the first embodiment, for example, when the characters in the material video are far away and the face size is small, the photographing camera must be moved far when capturing the user's face. In this embodiment, only the direction of the camera needs to be controlled, and it is not necessary to consider the distance.

[Embodiment 4] Next, as Embodiment 4 of the image processing apparatus of the present invention, a 3DCG (computer graphics) character image is used as a material image instead of a material image of a movie or drama scene. A system having a configuration in which this object is set as a replacement target object will be described. For example, there is a system that generates a video in which the user takes the place of the character and enters the interactive system using the face of a 3DCG (computer graphics) character as a replacement target object and the user's face as a paste target object.

FIG. 12 is a block diagram of an image processing apparatus that generates a video in which a user takes the face of a 3DCG (computer graphics) character as a material video and enters the face interactively.

The 3DCG model storage section 31 is a storage section in which a three-dimensional model of a CG character and its motion information are recorded in a time series. The virtual camera information storage unit 32
Information about a virtual camera assumed to have been set in correspondence with the viewpoint of the DCG object is stored. That is, focal length, image size, 3
Position information of the virtual camera with respect to the DCG model is stored as time-series data.

The material video generation unit 33 performs rendering from the information of the 3DCG model stored in the 3DCG model storage unit 31 and the virtual camera information stored in the virtual camera information storage unit 32, and generates a source video.

In the camera position information generating unit 34, the object orientation information obtained from the 3D CG model information in the material image generating unit 33 and the object to be replaced in the material image generated as a result of the rendering process based on the virtual camera information The orientation and position information of the (ex. Character's face) is obtained, and the position information of the camera for capturing the paste target object (ex. The face of the user) 5 is time-series sequential data corresponding to the information. Generate as In other words, the position information of the camera for capturing the face image, which is the object to be pasted, from the direction corresponding to the image object (here, the face of the specific 3DCG character) to be replaced by the material video generation unit 33 is described as sequential data. Generate and accumulate as At this time, camera position information is generated in consideration of actual parameters of the camera 4 (focal length, distortion due to a lens, and the like).

That is, the image capturing means 100 according to the present embodiment determines the photographing camera position in accordance with the camera position information generated based on the virtual photographing viewpoint information of the replacement target object included in the material image generated by the material image generating unit 33. To execute image capture of the paste target object.

In the hidden mask information generating section 35, 3DCG
Hidden mask information is generated from the model information and the virtual camera information. The meaning of the hidden mask information is the same as in the system described in the first embodiment. The subsequent processing configuration is the same as that of the first embodiment.

In the present system, at the time of rendering, it is possible to combine a background composed of 3DCG data and other objects. In addition, light source setting processing from an arbitrary direction is also possible when generating a material video. Based on the light source information generated in such a rendering process,
As in the second embodiment described above, it is also possible to adopt a configuration in which the illumination direction for irradiating the paste target object (ex. User's face) is set.

The motion information of the 3DCG model and the position information of the virtual camera do not use the accumulated information. For example, the motion information of the user is acquired and the same operation as the user is performed. As a processing configuration to be reflected as a configuration, an interactive 3DCG composite image may be generated.

In the above-described embodiment, the configuration in which the object to be pasted (ex. User's face) is fixed in an environment in which the image of the object to be pasted (ex. User's face) is captured has been described. However, the position and orientation of the user's face are measured in real time, for example, by attaching a sensor such as a gyro to the object to be pasted (ex. User's face), and the information is used as illumination position information and camera position information. , And the face position information, the paste target object (ex.
Instead of fixing the user's face, it is also possible to acquire and combine captured images of the paste target object (ex. User's face).

Note that the series of processes described in each of the above embodiments can be performed not only by hardware but also by software. That is, a configuration can be adopted in which the program is executed by a general-purpose computer or a microcomputer. When a series of processes is performed by software, a program constituting the software is installed in, for example, a general-purpose computer or a one-chip microcomputer. FIG. 13 illustrates a configuration example of an embodiment of a computer on which a program for executing the above-described series of processes is installed.

The program is stored in a hard disk 205 or a ROM 2 as a recording medium built in the computer.
03 can be recorded in advance. Alternatively, the program may be a floppy disk, a CD-RO
M (Compact Disc Read Only Memory), MO (Magneto op
tical) disk, DVD (Digital Versatile Disc), magnetic disk, semiconductor memory, or other removable recording medium 210 can be temporarily or permanently stored (recorded). Such a removable recording medium 21
0 can be provided as so-called package software.

The program can be installed in the computer from the removable recording medium 210 as described above, can be wirelessly transferred from a download site to the computer via a digital satellite broadcasting artificial satellite, or can be connected to a LAN (Local Area). Network) or the Internet, and the program can be transferred to the computer by wire, and the computer can receive the transferred program by the communication unit 208 and install the program on the built-in hard disk 205.

The computer has a CPU (Central Process).
ing Unit) 202. In the CPU 202,
An input / output interface 211 is connected via the bus 201, and the CPU 202 receives a command via the input / output interface 210 by operating the input unit 207 via a keyboard, a mouse, or the like. In accordance with this, the program stored in the ROM (Read Only Memory) 203 is executed. The image of the paste target object in the above-described embodiment is
3D, under the control of the CPU 202, for example, a material video stored in the removable recording medium 210.
Synthesis processing with data such as CG data is executed. Although only one configuration is shown as the removable recording medium 210 in the figure, various data such as camera position information,
The illumination position information, the material image, the hidden mask information, and the like may be stored in separate storage media and connected.
Some of these data are stored on the hard disk 20
5 may be stored.

The CPU 202 is not limited to the ROM storage program, but may be a program stored in the hard disk 205, a program transferred from a satellite or a network, received by the communication unit 208 and installed in the hard disk 205, or mounted on the drive 209. The program read from the removable recording medium 210 and installed on the hard disk 205 is
It can also be loaded into an AM (Random Access Memory) 204 and executed.

Thus, the CPU 202 performs the processing according to each of the above-described embodiments or the processing performed according to the above-described block diagrams and flowcharts. Then, the CPU 202 transmits the processing result as necessary,
For example, via the input / output interface 211, the LCD
(Liquid CryStal Display), output from an output unit 206 including a speaker, or the like, or transmission from the communication unit 208, and further, recording on the hard disk 205.

Here, in this specification, processing steps for writing a program for causing a computer to perform various processes do not necessarily have to be processed in chronological order in the order described in the flowchart, and may be performed in parallel. Alternatively, it also includes processing executed individually (for example, parallel processing or processing by an object).

The program may be processed by one computer, or may be processed in a distributed manner by a plurality of computers. Further, the program may be transferred to a remote computer and executed.

The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the spirit of the present invention. That is, the present invention has been disclosed by way of example, and should not be construed as limiting. In order to determine the gist of the present invention, the claims described at the beginning should be considered. For example, in the embodiment, a system for replacing a person's face with a character's face in a material image is provided.However, not only a person's face, any object in a material image can be replaced with another object. Can be interactive.

[0118]

As described above, according to the image processing apparatus, the image processing method, and the program storage medium of the present invention, a material video such as a movie or a drama, or 3
A process of synthesizing and pasting a paste target object (ex. User's face) photographed by a camera to a specific replacement target object (ex. A specific character's face) in image data generated based on DCG This is realized without imposing various movements on the paste target object (ex. User's face).

Further, according to the image processing apparatus, the image processing method, and the program storage medium of the present invention, there is no need to acquire the three-dimensional shape of the object to be pasted (ex. User's face), and it Can be used as it is, the replacement target object (e
x. There is no need to analyze the motion or voice of a specific character's face) or synthesize facial expressions, and a natural synthetic image can be easily generated. Further, camera position information and the like as information necessary for the image synthesizing process may be obtained in advance, and the amount of calculation becomes extremely small during operation, which is effective as a real-time interactive system.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a first embodiment of an image processing apparatus according to the present invention.

FIG. 2 is a diagram illustrating a configuration example of an image capturing unit of the first embodiment of the image processing apparatus according to the present invention.

FIG. 3 is a diagram illustrating a process of acquiring camera position information in the image processing apparatus of the present invention.

FIG. 4 is a diagram illustrating a process of acquiring hidden mask information in the image processing apparatus of the present invention.

FIG. 5 is a flowchart illustrating a process of acquiring camera position information and hidden mask information in the image processing apparatus of the present invention.

FIG. 6 is a flowchart illustrating an image synthesizing process in the image processing apparatus according to the present invention.

FIG. 7 is a block diagram showing a configuration of a second embodiment of the image processing apparatus of the present invention.

FIG. 8 is a diagram illustrating a configuration example of an image capturing unit according to a second embodiment of the image processing apparatus of the present invention.

FIG. 9 is a block diagram showing a configuration of a third embodiment of the image processing apparatus of the present invention.

FIG. 10 is a diagram illustrating a process of acquiring camera position information and face position information in a third embodiment of the image processing apparatus according to the present invention.

FIG. 11 is a flowchart illustrating a process of acquiring camera position information and face position information in a third embodiment of the image processing apparatus of the present invention.

FIG. 12 is a block diagram illustrating a configuration of an image processing apparatus according to a fourth embodiment of the present invention.

FIG. 13 is a block diagram illustrating a configuration of a processing unit when processing in the image processing apparatus of the present invention is executed by software.

[Explanation of symbols]

 1 Camera position information storage unit 2 Camera movement control unit 3 Camera movement unit 4 Camera 6 Captured image extraction unit 7 Image synthesis unit 8 Material video storage unit 9 Hidden mask information storage unit 10 Output unit 100 Image capture unit 20 User 21 Fixed Table 25 Display 30 Character 40 User's face 50 Camera 60 Column 70 Mask area 11 Lighting position information storage unit 12 Lighting movement control unit 13 Lighting movement unit 14 Lighting 80 Image capturing unit 15 Replacement object position information storage unit 90 Character 92, 96, 98 User's face 94 Camera 31 3D CG model storage unit 32 Virtual camera information storage unit 33 Material video generation unit 34 Camera position information generation unit 35 Hidden mask information generation unit 201 Bus 202 CPU 203 ROM 204 RAM 205 Hard disk 206 Output Department 207 Input unit 208 Communication unit 209 Drive 210 Removable recording medium 211 Input / output interface 212 Camera

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification FI FI Theme coat ゛ (Reference) G06T 15/70 G06T 15/70 A H04N 5/225 H04N 5/225 Z 5/262 5/262 13/00 13/00 13/02 13/02 F term (reference) 5B050 AA09 BA08 BA09 BA10 BA11 BA12 EA07 EA12 EA19 EA24 EA27 FA02 5B057 AA20 BA02 CA13 CB13 CC03 CD05 CE08 CH20 DB03 DC08 5C022 AB68 AC00 CA01 CA02 5C023 AA03 A17A17 DA02 DA03 DA08 5C061 AA20 AB03 AB08 AB12

Claims (16)

[Claims] An image processing apparatus for executing a process of setting a specific object in material image data as a replacement target object and replacing an image of the replacement target object with image data of a paste target object photographed by a camera. Image capturing means for controlling the capturing camera position in accordance with camera position information generated based on the capturing position information of the replacement target object to capture an image of the paste target object, and an image of the replacement target object in the image data And an image synthesizing unit that executes an image synthesizing process of changing the image into an image of the paste target object captured by the image capturing unit. 2. An image processing apparatus comprising: a material video storage unit storing material image data including the replacement target object; and a camera storing camera position information generated based on shooting direction information of the replacement target object. A position information storage unit, wherein the camera position information stored in the camera position information storage unit is configured as time-series data corresponding to each frame of the image data stored in the material video storage unit; Outputting the camera position information recorded in the storage means to the image capturing means, and outputting the material video recorded in the material video storage means to the image synthesizing means as synchronized processing; The means is a material video input from the material video storage means, and is input from the image capturing means. An image of a paste target object is input in parallel, and processing for changing an image of a replacement target object in the material image to an image of the paste target object based on the parallel input data is executed. The image processing apparatus according to claim 1, wherein: 3. The image processing apparatus further includes a captured image extracting unit that extracts only an image of an object to be pasted from an image captured by the image capturing unit, wherein the captured image extracting unit includes: The image processing apparatus according to claim 1, wherein the image processing unit outputs the extracted image of the paste target object to the image combining unit. 4. The image processing apparatus further comprises: hidden mask information storage means for storing hidden mask information as occluded area information of the object to be replaced in the material image data; A combining process of inputting hidden mask information from hidden mask information storage means and selecting data of the material image data as output data instead of image data of an object to be pasted from the image capturing means for a mask area. 3. The image processing apparatus according to claim 1, wherein 5. The image capturing device according to claim 1, wherein: a camera for capturing an image of the object to be pasted; and a camera moving device for moving the camera in accordance with the camera position information. 3. The image processing device according to 2. 6. The image processing apparatus further includes lighting position information storage means for storing lighting position information generated based on luminance information of the replacement target object in the material image data. 2. The apparatus according to claim 1, wherein the means comprises: illumination for irradiating the paste target object; and illumination moving means for moving the illumination in accordance with the illumination position information.
Or the image processing device according to 2. 7. The image processing apparatus further includes a replacement target object position information storage unit that stores position information of the replacement target object in the material image data, and wherein the image combining unit stores the replacement target object position. It is characterized in that, based on the position information of the replacement target object input from the information storage means, a size adjustment and position adjustment processing of the image of the paste target object input from the image capturing means is performed. The image processing device according to claim 1. 8. The image processing apparatus further comprises: three-dimensional model storage means for storing three-dimensional model data; and a virtual shooting viewpoint of a replacement target object included in the three-dimensional model data stored in the three-dimensional model storage means. Virtual camera information storing means for storing information; material video generating means for generating material video data based on the three-dimensional model data and the virtual shooting viewpoint information; the image capturing means comprising: The configuration is such that the photographing camera position is controlled according to the camera position information generated based on the virtual photographing viewpoint information of the replacement target object included in the material video generated by the video generation means, and the image of the paste target object is captured. The image processing apparatus according to claim 1, wherein: 9. An image processing method for executing a process of setting a specific object in material image data as a replacement target object and replacing an image of the replacement target object with image data of a pasting target object photographed by a camera. An image capturing step of controlling an image capturing camera position in accordance with camera position information generated based on image capturing position information of the replacement target object and capturing an image of the paste target object; and an image of the replacement target object in the image data An image synthesizing step of executing an image synthesizing process of changing the image to the image of the paste target object captured in the image capturing step. 10. The image processing method further includes a camera position information accumulating step of generating and accumulating camera position information based on photographing direction information of the replacement target object, wherein the camera position information is generated in the camera position information accumulating step. Output of the camera position information to the image capturing means, and
The output of the material video recorded in the material video storage means to the image synthesizing means is executed as synchronized processing, and the image synthesizing step comprises: Inputting the image of the paste target object to be performed in parallel, and performing a process of changing the image of the replacement target object in the material image to the image of the paste target object based on the parallel input data. The image processing method according to claim 9. 11. The image processing method according to claim 9, further comprising a photographed image extracting step of extracting only an image of the object to be pasted from the image photographed in the image photographing step. The image processing method according to 1. 12. The image processing method further comprises a hidden mask information accumulating step of generating and storing hidden mask information as occluded area information of the object to be replaced in the material image data. The hidden mask information is input, and for the mask area, a synthesis process is performed to select not the image data of the paste target object captured in the image capturing step but the data of the material image data as output data. The image processing method according to claim 9, wherein the image processing is performed. 13. The image processing method according to claim 1, further comprising the step of: generating and storing illumination position information generated based on luminance information of the replacement target object in the material image data; The photographing step moves the illumination for irradiating the paste target object according to the illumination position information generated in the illumination position information accumulation step.
0. The image processing method according to 0. 14. The image processing method further includes a replacement object position information accumulating step of generating and storing position information of the replacement object in the material image data, wherein the image combining step includes: The image processing device according to claim 1, further comprising: performing size adjustment and position adjustment processing of the image of the paste target object captured in the image capturing step based on the position information of the replacement target object generated in the position information accumulation step. Item 10. The image processing method according to item 9 or 10. 15. The image processing method further comprising: storing virtual camera information for generating and storing virtual photographing viewpoint information of a replacement target object included in the three-dimensional model data stored in the three-dimensional model storage means; A source video generating step of generating source video data based on the model data and the virtual shooting viewpoint information; and the image capturing step includes a replacement target object included in the source video generated by the source video generating step. 11. The image processing method according to claim 9, wherein the photographing camera position is controlled in accordance with the camera position information generated based on the virtual photographing viewpoint information to capture an image of the paste target object. 16. A computer / computer which performs a process of setting a specific object in material image data as an object to be replaced and replacing the image of the object to be replaced with image data of an object to be pasted taken by a camera.
A program storage medium for providing a computer program to be executed on a system, wherein the computer program controls a photographing camera position in accordance with camera position information generated based on photographing position information of the replacement target object and pastes the image. Performing an image capturing step of capturing an image of the object to be attached; and performing an image combining process of changing an image of the replacement target object in the image data to an image of the paste target object captured in the image capturing step. A program storage medium, comprising: an image synthesis step.