JPH03278095A - Device and method for synthesizing image - Google Patents

Abstract

PURPOSE:To enrich video expressing means by utilizing natural image whose visual points or lighting conditions are not completely matched with each other from a previously stored image data base and forming a new synthetic image corresponding to a purpose by image synthesis. CONSTITUTION:This image synthesizer is provided with the image data base 22 for storing two-dimensional image data photographed under applied visual point and lighting conditions, selecting means 25, 26 for selecting an image to be synthesized out of the two-dimensional image data stored in the data base 22, a means 21 for separating the selected image to be synthesized into a solid image to be mapped and a mapping pattern, a means 21 for inputting the output of the separating means 21 and reconstituting an image under newly set lighting and visual point conditions, and a means 21 for synthesizing the reconstituted image with another image. Consequently an image corresponding to the changes of lighting and a visual point can be optionally formed and image synthesis matched lighting and visual point conditions between optional plural image can be attained.

Description

[Detailed Description of the Invention] 1. Field of Industrial Application 1. The present invention is a method for processing and correcting pre-stored images using a computer to create advertising images, broadcasting images, simulation images, etc. , relates to an image synthesis device and an image synthesis method.

[Summary of the invention] Conventionally, as a method of synthesizing natural images, a method of cutting out an object using chroma key or key frame and displaying it by superimposing it on a background image has been put into practical use, and has been used in many commercial films, FSX movies, etc. It's being used.

However, in order to use these methods, it is necessary to prepare a blue back studio, miniature models, etc.
Furthermore, advanced techniques are required to match the viewpoints of the two surfaces to be synthesized and to match the illumination conditions.

Therefore, when it is desired to actually use this method, it is often difficult due to cost and material constraints, and it is hoped that a simpler method will be realized.

Therefore, the present invention separates a natural image to be synthesized into a three-dimensional object to be mapped and a surface mapping pattern, obtains three-dimensional image data that is independent of lighting and viewpoint conditions, and then applies computer graphics techniques. Using this technology, images can be synthesized under any newly desired viewpoint and illumination conditions.

According to the present invention, existing images stored in an image database etc. are used and the above operations are applied to them, without requiring large-scale facilities/equipment or advanced photography techniques unlike the conventional technology. By doing so, it is possible to easily obtain a composite image that suits the purpose.

[Prior art 1] Conventionally, when performing image synthesis, as shown in FIG. 3, a background image 33 obtained by photographing a background scene 36 with a video camera 31, and a blue background
A model 39 to be synthesized is placed in front of a lighting device 38 installed under the same conditions as the background scene 36, and the model 39 to be synthesized is photographed by a video camera 32. A compositing method has been used in which the image 34 is simply cut out two-dimensionally and superimposed to obtain a composite image 35.

In this case, in order to match the viewpoint and lighting conditions of the two images to be combined, a model 39 of the object is created, and a camera technician uses large-scale studio equipment to separate the background by hue. It was necessary to use advanced technology to match viewpoints and lighting conditions.

] Problem 1 to be Solved by the Invention According to the conventional composition method, for example, as shown in FIG. If the two images are to be combined with an image taken from a different location, two images whose illumination conditions and viewpoint conditions do not match will be combined, resulting in an unnatural image.

Therefore, it is necessary to obtain an image whose illumination conditions and viewpoint conditions completely match. It is impossible to prepare in advance images corresponding to all viewpoints and lighting conditions so as to always satisfy these conditions.

Therefore, as mentioned in the section on conventional technology, it was necessary to prepare a model and studio equipment to be used, and to make full use of the advanced skills of a photographer.

Therefore, in order to solve the above-mentioned problems, it is an object of the present invention to convert the illumination conditions and viewpoint conditions of images to be combined in accordance with the purpose of combining, so that when two images are combined, completely unnatural images may occur. An object of the present invention is to provide an image compositing device and an image compositing method that prevent image compositing from occurring.

[Means for Solving the Problems 1] An image synthesis device according to the present invention includes an image database that stores two-dimensional image data photographed under given viewpoints and lighting conditions, and two-dimensional image data stored in the database. A selection means for selecting an image to be synthesized from dimensional image data, a separation means for separating the selected synthesis target into a solid to be mapped and a mapping pattern, and the output of the separation means are introduced and newly set. The image forming apparatus includes a reconstructing means for reconstructing an image under the specified illumination conditions and viewpoint conditions, and a synthesizing means for superimposing and synthesizing the reconstructed image and another image.

Further, the image synthesis method according to the present invention selects images to be synthesized, separates the images to be synthesized into a mapping pattern and a solid to be mapped, and creates an image under illumination conditions and viewpoint conditions after synthesis. is generated using a computer graphics method, the boundary pattern is extended for the missing portion of the mapping pattern by changing the viewpoint, the unnatural-looking portion is corrected, and the image is combined with the background image. be.

[Function] According to the present invention, first, the particularity of the illumination conditions of a given image is removed, and a mapping pattern image that is independent (irrelevant) of the illumination conditions can be obtained. Furthermore, the viewpoint of the three-dimensional object to be mapped, which is the object of pasting the mapping pattern image, can be freely changed to obtain a changed image.

Therefore, it is possible to generate images that freely respond to changes in illumination and viewpoint. By performing such operations on two images, it is possible to realize image synthesis in which illumination and viewpoint conditions match between arbitrary images.

Specifically, as shown in FIG. 5, first, image 2, which has been stored in the image database 22 (see FIG. 2) in advance, is
By using 3, there is no need to create a model or go to the site to take photographs. In order to solve the problems of illumination and viewpoint conditions that arise at that time, the object 51 in the image is divided into a mapping pattern 53 consisting of a pattern on the surface and a mapping target solid 52 to which the mapping pattern is pasted. The mapped solid 52 is modeled according to the instructions of the operator 27 (see FIG. 2), and a mapping pattern is created in consideration of the lighting conditions. 53
is estimated by the electronic computer 21 (see FIG. 2).

Furthermore, for the combination of the mapping pattern 62 and the mapped solid 61 shown in FIG. 6, images under newly set illumination conditions and viewpoint conditions are calculated, and each image is A new reconstructed image 63 is obtained by superimposing the images according to the following.

In the reconstructed image 63, as the viewpoint changes, parts that cannot be covered by the mapping pattern are covered by extending the surrounding patterns based on the operator's instructions.
If there are any parts that look unnatural, the operator can perform corrections to obtain a corrected image 64 under new lighting conditions.

[Embodiment 1] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

As shown in FIG. 1, a background image 12 showing a desk and an image to be combined 11 showing an apple are combined to create an image that looks like an apple placed naturally on the desk. 19
An example of obtaining the following will be described.

In the background image 12, the lighting is shining from the right, and in the image 13 of an apple to be synthesized, the lighting is shining from the left.

Furthermore, the desk in the background image 12 is photographed from a viewpoint far above, and the image to be combined 11 is photographed from a viewpoint directly beside the apple.

Therefore, if the images are simply superimposed as is, the result will be an unnatural image.

Therefore, let us consider a case where an image of an apple is synthesized by matching the illumination conditions and viewpoint conditions of the background image. To do this, first, the solid to be mapped 16 and the mapping pattern 15 are separated.

As an example of a specific separation method, as shown in FIG. 7, for the image 13 of an apple to be synthesized (see FIG. 1), a pseudo three-dimensional apple whose outline matches that of the apple is created. Assuming that it is a collection of cylinders, a wire frame model 71 of the solid pine tree is set.

Next, an image 72 when a --like white background pattern is placed on top of it is calculated using the Computer Crafigos method. At this time, a point (x, y) on the wire frame model 71 of the solid to be mapped is a corresponding point (x' y°) and the brightness Px 'y' of that point (.

Next, the brightness Qx'y' of the corresponding point on the pine pink pattern is calculated using the following formula with respect to the brightness Qxy of the corresponding point on the apple image 13 (see FIG. 1) C image to be synthesized.

Qx ' y ' = Qxy * Po / Pxy For this mapping pattern image and the object to be mapped, the illumination conditions of the background image 12 (see Figure 1) and the natural position on the desk in the background image 12 are determined. Using computer graphics techniques, we calculate a new way of looking at the apple, given the viewpoint conditions for the apple. Then, by superimposing the apple image resulting from this calculation on the background image 12, a natural composite image is obtained.

In this case, the illumination and viewpoint conditions of the apple image 13 to be combined are matched to the background image 12, but the above operations are applied to both the background image 12 and the apple image 13 to be combined. , a composite image under a new third viewpoint and lighting conditions can also be created.

FIG. 2 is an example of an apparatus configuration for implementing the present invention. In implementing the present invention, images inputted by an image input device such as a video camera 28 or an image scanner 29 are stored in the image database 22 in advance. The operator 27 selects an image to be synthesized from among the images 23 stored in the image database 22 by operating the operating keyboard 25 and mouse 26 while looking at the image display device 24, and selects an image to be synthesized from among the images 23 stored in the image database 22. 21, image synthesis is performed by the method described above.

FIG. 8 is a flowchart outlining an image compositing process according to one embodiment of the present invention. As shown in this figure, first of all, an image to be synthesized is selected from an existing image database (step S1).

Next, this image to be combined is separated into a mapping pattern and a solid to be mapped, which are independent of illumination conditions and viewpoint conditions (step S2).

Next, the illumination conditions and viewpoint conditions for the desired combined image are given, and an image under those conditions is generated using a computer graphics method (step S3).
.

Next, by changing the viewpoint, the boundary pattern is extended for the missing part of the mapping pattern to complete the entire visible part under the new viewpoint condition (step S4
).

Next, if there are any parts that look unnatural, they are corrected by manual operation by the operator (step S5).

Finally, the image obtained by these operations is combined with the background image. (Step S6).

Effects of the Invention 1 By implementing the present invention, image synthesis can be easily performed without requiring large-scale facilities and equipment and advanced photographing techniques as in the prior art.

Furthermore, by utilizing natural images in a pre-stored image database whose viewpoints and illumination conditions are not completely compatible, it is possible to create a new composite image according to the purpose through image synthesis.

As a result of the above, image synthesis has come to be used in many fields where image synthesis has not been applied before, making it possible to enrich the means of image expression.

[Brief explanation of the drawing]

FIG. 1 shows a case where a background image 12 showing a desk and a composite image 11 showing an apple are combined to obtain a composite image 19 that looks like an apple is placed on the desk naturally. Figure 2 is a block diagram showing an example of the configuration of an apparatus for performing image synthesis according to the present invention; Figure 3 is a plan view of a background image taken on-site and a composite image taken in a studio using a model. Figure 4 shows a background image of a desk illuminated from the left viewed from above, and an apple illuminated from the right viewed from the front. Figure 5 shows an example of an unnatural composite image when the target image is combined with the target image. The diagram shown in Figure 6 shows that a new mapping image is generated by giving a new viewpoint and illumination conditions to the separated object to be mapped and the mapping pattern from which the influence of lighting has been removed, and the insufficient parts are further removed. A diagram showing the process of modifying and generating an image for compositing. Figure 7 shows the lighting correction to give a mapped solid to the image to be synthesized and obtain a pine pink pattern that removes the influence of lighting. FIG. 8, which is a diagram explaining the processing, is a flowchart showing the image synthesis process in one embodiment of the present invention. 11... Image to be synthesized, 12... Background image, 13... Apple image, 14... Desk, 15... Mapping pattern, 16... Three-dimensional object to be mapped, 19... Composition 21... Computer, 22... Image database, 23... Accumulated images, 27... Operator, 28... Video camera, 29... Image scanner 31... Video camera, 32... Video camera, 33... Background image, 34... Image to be combined, 35... Combined image. 36... Background scene, 37... Blue pack background, 38... Lighting device, 39... Model, 51... Target object, 52... Three-dimensional object to be mapped, 53... Pine Ping pattern, 61... Solid to be mapped, 62... Mapping pattern, 63... Reconstructed image, 64... Corrected image, 71... Wire frame model. Figure 2 Figure 5 Figure 7

Claims (1)

[Claims] 1) An image database that stores two-dimensional image data taken under given viewpoints and lighting conditions; and an image to be synthesized from the two-dimensional image data stored in the database. a selection means for selecting the selected compositing target, a separation means for separating the selected synthesis target into a mapped solid and a mapping pattern, and an output of the separation means is introduced, and the output of the separation means is introduced and An image synthesizing device comprising: a reconstruction means for reconstructing an image into an image; and a synthesis means for superimposing and synthesizing the reconstructed image and another image. 2) Select images to be synthesized, separate the images to be synthesized into a mapping pattern and a three-dimensional object to be mapped, and use computer graphics techniques to generate images under the lighting and viewpoint conditions after synthesis. An image synthesis method characterized by: generating a pattern, extending a boundary pattern for a missing portion of the mapping pattern by changing the viewpoint, correcting a portion that looks unnatural, and compositing with a background image.