KR101843018B1 - System and Method for Video Composition - Google Patents

Abstract

The present invention relates to a system and a method for composing an image, which reduce costs for producing a composite image. The image composing system according to the present invention comprises: a background image input part receiving a 2D background image; a foreground object image input part receiving a 2D foreground object image; a modeling part forming a first background projection surface which has a height and has a center part of a concave curved surface in a virtual space, projecting at least a part of the 2D background image on the first background projection surface at a virtual projection point on the virtual space, and locating the 2D foreground object image between the virtual projection point on the virtual space and the first background projection surface; and a composite image generation part generating the composite image including at least a part of the 2D background image and at least a part of the 2D foreground object image at an angle of view of a virtual camera located in the virtual space.

Description

Technical Field [0001] The present invention relates to a video synthesis system,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesis system and an image synthesis method, and more particularly, to an image synthesis system and an image synthesis method capable of naturally synthesizing a background image and a foreground image according to a change in camera viewpoint.

BACKGROUND ART [0002] A background synthesis method for synthesizing a foreground image such as a person photographed by a real-world camera and a background image such as an outdoor landscape in accordance with the development of image capturing technology and image editing technology is widely used. In particular, chroma key synthesis method using green screen, background separation technique by depth measurement of image, spread of image editing software, and the like, the cost, manpower and time required for background synthesis are remarkably reduced, In addition to Professionally Generated Content (PGC) produced by experts, background synthesis techniques are also used in UGC (User Generated Content) produced by general users.

On the other hand, in the background synthesis, it is general to generate a composite image of a fixed angle of view by synthesizing a 2D foreground image of a fixed angle of view and a 2D background image of a fixed angle of view in a planar manner. There is a limitation in that a synthesized image of a monotonous composition can be produced only by a flat image matching technique because there is an uncomfortable feeling in a synthesized image when synthesizing a foreground image using a camera work such as a camera panning or a camera moving to a fixed 2D background image.

Japanese Patent Registration No. 4140006, entitled " Photographing System for Composite Image Generation " of FUJINON CORP, discloses a photographing system in which a camera control signal such as a pan, tilt, zoom and focus applied to a camera for foreground image photography is equally applied to a background image photographing camera do. In this method, since the foreground image and the background image share the same angle of view, there is an effect that a composite image without discomfort can be generated despite the panning motion or the moving motion of the camera. However, since only the background image is synchronously synchronized with the foreground image, the 2D still image or video image taken independently of the foreground image can not be used as the background image, .

Milos Jovanovic, US Publication No. 2013-0335414, entitled " Visualization of Three-Dimensional Models of Objects in Two-Dimension Environment ", discloses a method of analyzing a 2D background image and converting it into a 3D stereoscopic structure. According to this method, since the 2D background image is converted into the 3D stereoscopic structure, the background image is adaptively converted so as to match the camera work of the 2D foreground image even when synthesized with the 2D foreground image. However, it is very difficult to realize a 3D stereoscopic structure with only a single 2D still image from which atypical background objects are photographed. In addition, since a considerable effort, time, and cost are put into modeling the background objects in the 3D stereoscopic structure, it is difficult to acquire various background images, and the cost for synthesized images is increased.

Korean Patent No. 0795552, "VIDEO-TOUCHING VIRTUAL VIRTUAL STUDIO DEVICE" is a technology that combines a 2D background image into a plane of a virtual space and a background synthesis technique using 3D rendering, which places a 2D foreground image on the foreground of the background image . According to this method, since the 2D foreground image and the 2D background image are spaced apart from each other in the virtual space, the distance between the foreground image and the background image is varied depending on the panning motion of the virtual camera or the moving motion of the virtual camera, There is an effect that can be done. However, there is a problem that the 2D background image is textured on the plane, and thus the perspective of the background image can not be expressed. Secondly, when the 2D foreground image is a person, for example, when a whole-body shot is carried by a virtual camera, there is a problem that the foot, which is the lower end of the 2D foreground image, is moved to another position without being fixed to the background image according to the panning motion or the moving motion of the virtual camera . Due to these problems, the conventional art has a problem in that it can not meet a demand of a user who desires a more dynamic camera work and a high-quality composite image.

 US Patent No. 6,124,859 to Hitachi, US Pat. No. 6,124,859, entitled " Picture Conversion Method and Medium Used Therefor ", and Korean Patent No. 0533328, " Method of Generating 3D Images from 2D Images " A method of implementing a virtual space of a form is disclosed. According to this method, there is an advantage that the image in the 2D background image can be solidified through analysis of the vanishing point. However, in the case of a 2D background image in which the vanishing point is not clear, there is a problem that the 2D background image that can be used can not be used because the method of the related art can not be applied, and the background image due to the movement of the virtual camera due to the linear 3D background There is a problem that distortion occurs.

Korean Unexamined Patent Publication No. 2007-0098361 entitled " Image synthesis apparatus and method for synthesizing a two-dimensional background image in a three-dimensional space ", is a method of setting a 2D background image in a 3D virtual space as an image plane and generating a floor plane A background synthesis method is disclosed. According to this method, since the bottom surface of the foreground image is fixed to the background because there is a separate floor plane, the foreground image is fixed to the floor according to the panning motion or the moving motion of the virtual camera, thereby generating a natural composite image. However, there is a problem that a floor plane corresponding to a 2D background image must be directly generated through a separate graphic operation in order to configure the floor plane so as not to be discomfort with the 2D background image.

Therefore, the 2D foreground image and the 2D background image are given a 3D sensation in the 2D background image independently photographed with the 2D foreground image, while realizing a perspective of the natural image between the foreground image and the background image according to the panning motion or the moving motion of the camera, High-quality, low-cost background synthesis method capable of implementing a work.

JP 4140006 B2 US 2013-0335414 A1 KR 0795552 B1 US 6124859 A KR 0533328 B1 KR 2007-0098361 A

The image synthesis system and the image synthesis method according to the present invention can be applied to an image processing system in which a 2D background image is projected onto a first background projection surface formed in a concave curved surface on a virtual space and a 2D foreground image is disposed on a foreground surface of a first background, The 2D image of the 2D background image is generated by the panning motion or the moving motion of the camera, thereby generating a composite image that provides a 2D foreground image and a natural perspective, and the 2D background image photographed independently from the 2D foreground image is general- It is an object of the present invention to reduce the cost of generating a composite image.

The image synthesizing system and the image synthesizing method according to the embodiment of the present invention may further include a second background projection surface forming a bottom surface and arranging the lower end of the foreground image so as to contact the second background projection surface, The foreground image is fixed to the background image in order to provide a composite image of high quality without discomfort.

The image synthesis system and the image synthesis method according to the embodiment of the present invention are characterized in that a region in which the first background projection surface and the second background projection surface are in contact form a curved surface so that the panning operation of the virtual camera or the synthesis And aims at minimizing distortion that may occur in an image.

The image synthesis system and the image synthesis method according to the embodiment of the present invention generate a natural and dynamic synthetic image through virtual camera control synchronized with the camera work such as a panning operation or a moving operation of a real- .

The image synthesis system according to the present invention comprises: a background image input unit receiving a 2D background image; A foreground object image input unit receiving a 2D foreground object image; Forming a first background projection surface having a height and a concave curved surface at a central portion in a virtual space, projecting at least a part of the 2D background image on the first background projection surface at a virtual projection point on the virtual space, A modeling unit for arranging the 2D foreground object image between the virtual projection point and the first background projection plane; And a synthetic image generating unit generating at least a part of the 2D background image and at least a part of the 2D foreground object image at an angle of view of the virtual camera disposed in the virtual space, .

In the image synthesis system according to the embodiment of the present invention, the 2D background image is a still image, the 2D foreground object image, and the composite image are video images.

In the image synthesis system according to the embodiment of the present invention, the 2D background image, the 2D foreground object image, and the composite image are video images.

In the image synthesizing system according to the embodiment of the present invention, the modeling unit may further generate a second background projection plane extending from a lower end of the first background projection plane to form a bottom face, and the first background projection plane and the second background And at least a part of the 2D background image is projected through the projection plane.

In the image synthesizing system according to the embodiment of the present invention, the modeling unit may arrange the 2D foreground object image so that the lower end of the 2D foreground object image contacts the second background projection plane.

In the image synthesizing system according to the embodiment of the present invention, the modeling unit may further generate a third background projection plane extending from the upper end of the first background projection plane to form an upper face, and the first background projection plane, And projecting at least a part of the 2D background image through the third background projection surface.

In the image synthesizing system according to the embodiment of the present invention, the modeling unit may be configured to form the first background projection plane and the second background projection plane so that the first background projection plane and the second background projection plane form an interior angle of 90 to 120 degrees with respect to each other. .

 In the image synthesizing system according to the embodiment of the present invention, the modeling unit may process the area touched by the first background projection surface and the second background projection surface as a curved surface to generate the first background projection surface and the second background projection surface .

In the image synthesis system according to the embodiment of the present invention, the foreground object image input unit receives the video image generated by the panning operation of the real-world camera as the 2D foreground object image, The virtual camera is panned and controlled based on first camera control data defining a panning operation of the virtual camera to generate the composite image.

In the image synthesis system according to the embodiment of the present invention, the foreground object image input unit receives the video image generated by the moving operation of the real-world camera as the 2D foreground object image, Wherein the virtual camera is moved and controlled based on second camera control data defining a moving operation of the real-world camera to generate the composite image.

According to an aspect of the present invention, there is provided a method of synthesizing an image, comprising: receiving a 2D background image by a video synthesis system; Receiving the 2D foreground object image by the image synthesis system; Forming a first background projection surface in a virtual space, the first background projection surface having a predetermined height and a central concave surface; Projecting at least a portion of the 2D background image to the first background projection plane at a virtual projection point on the virtual space; Arranging the 2D foreground object image between the virtual projection point on the virtual space and the first background projection plane; And generating a composite image by the image synthesis system including at least a part of the 2D background image and at least a part of the 2D foreground object image at an angle of view of a virtual camera disposed in the virtual space. .

In the image synthesis method according to an embodiment of the present invention, the 2D background image is a still image, the 2D foreground object image, and the composite image are video images.

 In the image synthesis method according to an embodiment of the present invention, the 2D background image, the 2D foreground object image, and the composite image are video images.

In the image synthesis method according to the embodiment of the present invention, the step of generating the background projection plane may further include generating a second background projection plane extending from a lower end of the first background projection plane to form a bottom plane, And the step of projecting the 2D background image is characterized in that the image synthesizing system projects at least a part of the 2D background image via the first background projection plane and the second background projection plane.

In the image synthesis method according to the embodiment of the present invention, the step of arranging the 2D foreground object image may include the step of arranging the 2D foreground object image so that the lower end of the 2D foreground object image comes into contact with the second background projection plane, Is disposed.

In the image synthesizing method according to the embodiment of the present invention, the step of generating the background projection plane may further include generating a third background projection plane extending from the upper end of the first background projection plane to form an upper plane, And projecting the 2D background image is characterized in that the image synthesizing system projects at least a part of the 2D background image via the first background projection plane, the second background projection plane, and the third background projection plane.

In the image synthesizing method according to an embodiment of the present invention, the step of generating the background projection plane may include a step of generating the background projection plane so that the first background projection plane and the second background projection plane form an interior angle of 90 degrees to 120 degrees The first background projection plane and the second background projection plane.

 In the image synthesizing method according to the embodiment of the present invention, the step of generating the background projection plane may include a step of processing the area, which the image synthesizing system contacts with the first background projection plane and the second background projection plane, Thereby generating a projection plane and the second background projection plane.

In the image synthesis method according to an embodiment of the present invention, the step of receiving the 2D foreground object image may include receiving the video image generated by the panning operation of the real-world camera as the 2D foreground object image, The generating of the composite image is characterized in that the image synthesizing system generates the composite image by controlling the panning of the virtual camera based on first camera control data defining a panning operation of the real camera.

In the image synthesizing method according to the embodiment of the present invention, the step of receiving the 2D foreground object image may include: receiving the video image generated by the moving operation of the real camera in the image synthesizing system as the 2D foreground object image, Wherein the generating of the composite image is performed by moving the virtual camera based on second camera control data defining a moving operation of the real camera to generate the composite image.

The image synthesis system and the image synthesis method according to the present invention can be applied to an image processing system in which a 2D background image is projected onto a first background projection surface formed in a concave curved surface on a virtual space and a 2D foreground image is disposed on a foreground surface of a first background, The 2D image of the 2D background image is generated by the panning motion or the moving motion of the camera, thereby generating a composite image that provides a 2D foreground image and a natural perspective, and the 2D background image photographed independently from the 2D foreground image is general- It is possible to reduce the cost of generating a composite image.

The image synthesizing system and the image synthesizing method according to the embodiment of the present invention may further include a second background projection surface forming a bottom surface and arranging the lower end of the foreground image so as to contact the second background projection surface, The foreground image is fixed to the background image, thereby providing a high-quality composite image without discomfort.

The image synthesis system and the image synthesis method according to the embodiment of the present invention are characterized in that a region in which the first background projection surface and the second background projection surface are in contact form a curved surface so that the panning operation of the virtual camera or the synthesis Thereby providing an effect of minimizing distortion that may occur in the image.

The image synthesis system and the image synthesis method according to the embodiment of the present invention generate a natural and dynamic synthetic image through virtual camera control synchronized with the camera work such as a panning operation or a moving operation of a real- Effect.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing a video synthesis system according to the present invention; FIG.
FIG. 2 is a layout diagram of a background projection plane, a 2D foreground object image, and a virtual camera in a virtual space according to an embodiment of the present invention; FIG.
3 is a cross-sectional view of a first background projection surface according to an embodiment of the present invention.
4 and 5 are a layout diagram of a background projection plane, a 2D foreground object image, and a virtual camera in a virtual space according to another embodiment of the present invention.
6 is an illustration of modeling of a 2D background image and a 2D foreground object image projected on a background projection plane in a virtual space according to an embodiment of the present invention;
FIGS. 7 and 8 are exemplary views of a composite image to which modeling of FIG. 6 is applied. FIG.
9 is a flowchart of a process of the image synthesizing method according to the present invention.

The terms and words used in the present specification and claims should not be construed in a conventional or dictionary sense and should be construed in accordance with the principles by which the inventor can define terms and concepts to best describe his invention And should be construed in accordance with the meaning and concept consistent with the technical idea of the present invention.

Therefore, the embodiments described in the present specification and the configurations shown in the drawings correspond to the preferred embodiments of the present invention and do not represent all the technical ideas of the present invention. Therefore, There may be equivalents and variations.

The specification that " comprises " any element in any specification throughout the specification does not exclude other elements, but may also include other elements, unless the context clearly indicates otherwise. Also, the terms " part, " " system, " and the like, which are described in the specification, refer to a unit for processing at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software, Or may be included in each of the other devices.

Figure 1 illustrates an image synthesis system 100 in accordance with the present invention. The image synthesis system 100 according to the present invention includes a background image input unit 110, a foreground object image input unit 120, a modeling unit 130, and a composite image generation unit 140. The image synthesizing system 100 can be implemented by a program and an arithmetic processing unit for driving the same. In this case, each of the components can be implemented as a functional module constituting a program.

The background image input unit 110 receives a 2D background image. The background image input unit 110 may be implemented by a method of providing an interface for the user to select a 2D background image or an interface for uploading a 2D background image held by the user. The 2D background image is not particularly limited and may be a still image in the form of a normal image file or a video image in the form of a video file. However, in order to improve the quality of the composite image, a 2D background image is preferable to a high resolution image.

The foreground object image input unit 120 receives the 2D foreground object image 310. The foreground object image input unit 120 may be implemented by a method of providing an interface for the user to select the 2D foreground object image 310 or an interface for uploading the 2D background image that the user owns. The 2D foreground object image 310 is an image in which the background is removed from the video image photographed by the video camera. In the case of using a TOF-type camera or a multi-lens camera capable of measuring the depth of an object in order to give a three-dimensional sensation of a foreground object surface according to an embodiment, the 2D foreground object image 310 may include a 2D foreground object image 310, And the depth information of the image.

The modeling unit 130 spatially combines the 2D background image and the 2D foreground object image 310 in the 3D virtual space. Specifically, the modeling unit 130 forms a first background projection plane 210 having a height and a concave curved surface at the center thereof in a virtual space. The first background projection plane 210 may be formed in a semi-cylindrical shape as shown in FIG. Depending on the embodiment, the shape of the concave curved surface can be varied. For example, the cross-sectional surface of the first background projection surface 210 may be a part of a circular shape as shown in Fig. 3 (a), a part of an elliptical shape as shown in Fig. 3 (b) May be implemented as part of the shape. Or the first background projection surface 210 may be embodied in a three-dimensional form such as a quarter-spherical shape. According to the shape of the first background projection plane 210, the center area of the 2D background image is formed to be distant from the virtual camera 320 and the 2D foreground object image 310 in the virtual space, and the left and right edge areas are virtual Since the distance between the camera 320 and the 2D foreground object image 310 is relatively close to the virtual space, it is possible to provide an appropriate perspective according to the panning motion or the moving motion of the virtual camera 320, Since it can provide an appropriate perspective without image analysis, it provides the effect of utilizing various images or video as a 2D background image.

According to an embodiment, it is preferable to adaptively change the first background projection plane 210 according to the image of the 2D background image. The distance between the center of the first background projection plane 210 and the virtual camera 320 is shortened by reducing the curvature of the concave portion when the object distance at the center portion of the 2D background image is close. The distance between the center of the first background projection plane 210 and the virtual camera 320 can be increased by increasing the curvature of the concave portion. In addition, when the vanishing point of the 2D background image can be analyzed, the shape of the first background projection plane 210 can be formed as a left-right asymmetric shape so that the concave portion is positioned at the vanishing point.

Next, the modeling unit 130 projects at least a part of the 2D background image on the first background projection plane 210 at the virtual projection point in the virtual space. It is preferable that the virtual projection point is set as a point in the virtual space corresponding to the relative positional relationship between the real-world camera photographed with the 2D foreground object image 310 and the photographed foreground object. The 2D background image projected on the first background projection plane 210 at the virtual projection point is seen as a 2D background image without distortion in spite of the three-dimensional shape of the first background projection plane 210. [ The projection of the background image onto the first background projection plane 210 is performed by converting the background image in a form in which the background image is not distorted when viewed from the virtual projection point and texturing the converted background image on the first background projection plane 210 ≪ / RTI >

According to an embodiment, the 2D background image may be entirely projected onto the first background projection plane 210, or a part of the 2D background image may be projected onto the first background projection plane 210.

Next, the modeling unit 130 functions to arrange the 2D foreground object image 310 between the virtual projection point on the virtual space and the first background projection plane 210. The 2D foreground object image 310 may be configured to be disposed in an area enclosed by the first background projection plane 210 as shown in FIG. FIG. 2 illustrates that the 2D foreground object image 310 includes a background of a square shape. However, as shown in FIG. 7, the 2D foreground object image 310 includes only a background image of a person .

The composite image generating unit 140 generates at least a part of the 2D background image and at least a part of the 2D foreground object image 310 at the angle of view of the virtual camera 320 disposed in the virtual space to generate a composite image . The virtual camera 320 may preferably be located at the same position as the virtual projection point or at a position adjacent to the virtual projection point. The virtual camera 320 can perform functions such as pan, tilt, zoom, and the like, and can perform a camera movement using the 2D foreground object image 310 and the 2D background image having a fixed angle of view by the operation of the virtual camera 320 The overlapping area between the foreground and the background can be smoothly generated in accordance with the movement of the virtual camera 320 through the 2D background image projected on the first background projection plane 210 concave at the center. There is an effect that perspective can be given while changing.

According to an embodiment, it is preferable to project a 2D background image on a three-dimensional structure composed of a plurality of background projection surfaces. For example, in the case of a 2D background image depicting a floor, it is necessary to synthesize the bottom side and the bottom of a subject such as a human foot to be naturally connected to each other. 4, the modeling unit 130 further generates a second background projection plane 220 extending from a lower end of the first background projection plane 210 to form a bottom face, And a second background projection plane (220).

Since the second background projection plane 220 forms the bottom surface of the projecting body, the image of the bottom part positioned at the lower end of the 2D background image is naturally projected onto the second background projection plane 220. At this time, the modeling unit 130 preferably arranges the 2D foreground object image 310 such that the lower end of the 2D foreground object image 310 comes into contact with the second background projection plane 220. The modeling unit 130 may further represent a shadow effect according to the 2D foreground object image 310 on the second background projection plane 220 in contact with the 2D foreground object image 310, It is preferable to further express the reflection effect according to the 2D foreground object image 310 when the floor depicted in FIG.

The 2D foreground object image 310 is projected on the second background projection plane 220 even if the manipulation of the 2D foreground object image 310 and the virtual camera 320 capturing the 2D background image occurs in the 3D virtual space It is fixed at the lower part of the 2D background image, and it is possible to generate a natural dynamic synthetic image according to the camera work. For example, when the panning operation for centering the 2D foreground object image 310 is performed simultaneously with the moving operation of moving the x-axis in parallel with the virtual camera 320, the 2D foreground object 310, which is relatively close to the virtual camera 320, A part of the 2D background image projected on the object image 310 and the second background projection plane 220 has a small displacement on the screen but is projected on the center of the first background projection plane 210 located relatively far from the virtual camera 320 A part of the 2D background image has a large displacement on the screen, so that a dynamic synthetic image having a natural perspective can be generated.

At this time, the modeling unit 130 generates the first background projection plane 210 and the second background projection plane 220 so that the first background projection plane 210 and the second background projection plane 220 form an interior angle of 90 to 120 degrees with respect to each other . In the virtual space, the virtual camera 320 is positioned at the same position as or adjacent to the virtual projection point. At this time, the position of the virtual projection point is determined based on the relative position of the actual camera and the 2D foreground object image 310, which captured the 2D foreground object image 310 so that the 2D foreground object image 310, . ≪ / RTI > Since the actual camera for photographing a person is usually photographed at a position about 1 m to 1.5 m above the ground, the virtual projection point is located somewhat above the floor of the virtual space. Therefore, even if the first background projection plane 210 and the second background projection plane 220 have an interior angle of 90 degrees, the virtual projection point can project a part of the 2D background image on the second background projection plane 220. The first background projection plane 210 and the second background projection plane 220 may be configured to have an angle of 90 degrees or more to project a greater portion of the 2D background image to the second background projection plane 220 according to an embodiment. In this case, however, it is preferable to have an internal angle of 90 degrees to 120 degrees in order to maintain the stereoscopic effect of the 2D background image. In the case of forming a larger internal angle, too much of the 2D background image may be projected onto the first background projection plane 210, which is the bottom surface, and the stereoscopic effect of the background image may be degraded.

 On the other hand, since the angle of the portion where the first background projection plane 210 and the second background projection plane 220 meet changes rapidly, image distortion occurs at corner portions due to the fan motion, tilting motion, . In order to solve this problem, it is preferable that the first background projection plane 210 and the second background projection plane 220 are smoothly connected to each other as shown in FIG. To do this, the modeling unit 130 processes a region where the first background projection plane 210 and the second background projection plane 220 are touched to form a first background projection plane 210 and a second background projection plane 220 desirable. With this configuration, there is an effect that the image distortion of the corner portion, which may occur according to the operation of the virtual camera 320, can be minimized.

The upper part of the 2D background image generally includes a sky or a ceiling of a building. In this case, the uppermost region is the region closest to the camera that captured the background image, and the farther away from the camera, the closer the region is. 5 and 6, according to an embodiment for realizing the perspective of the upper region, the modeling unit 130 may include a third background projection plane 210 extending from the upper end of the first background projection plane 210, It is preferable to further generate the second background image 230 and project at least a part of the 2D background image through the first background projection plane 210, the second background projection plane 220, and the third background projection plane 230. [ According to this embodiment, the background projection plane includes a third background projection plane 230 as a top face, a first background projection plane 210 as a side face, and a second background projection plane 220 as a bottom face, . According to this configuration, the upper end of the 2D background image projected from the virtual projection point is projected onto the third background projection plane 230, and the uppermost partial image is formed closest to the virtual camera 320. The center of the 2D background image projected at the virtual projection point is projected onto the first background projection plane 210 and a partial image at both ends is formed close to the virtual camera 320. [ The lower end of the 2D background image projected at the virtual projection point is projected onto the second background projection plane 220 and the lowermost partial image is formed closest to the virtual camera 320. [

FIG. 6 shows a 2D background image disposed over the first background projection plane 210, the second background projection plane 220, and the third background projection plane 230, and a 2D foreground object image 310 placed in the three- FIG. 5 shows an embodiment of a virtual space. FIGS. 7 and 8 show composite images synthesized in the virtual space applied in FIG. 6, FIG. 7 shows synthesized images when the virtual camera 320 is located on the left side on the basis of a virtual projection point, FIG. And a composite image when the camera 320 is positioned on the right side on the basis of a virtual projection point. Due to the three-dimensional shape of the background projection plane, a 2D background corresponding to the projected distance to the center of the virtual camera 320 and the first background projection plane 210 located at a distance from the left side to the right of the virtual camera 320 The 2D foreground object image 310 located close to the virtual camera 320 and the portion of the 2D background image corresponding to the bottom projected on the second background projection plane 220 change with small displacements It is possible to generate a synthetic image which realizes a natural three-dimensional feeling at a low cost without a high-cost operation such as 3D modeling.

On the other hand, when photographing a 2D background image, the real world camera may dynamically photograph a subject, which is a 2D foreground object image 310, through a panning operation or a moving operation. At this time, it is preferable that the composite image is generated through the camera work of the virtual camera 320 interlocked with the camera work applied to the 2D foreground object image 310. To this end, the foreground object image input unit 120 receives the video image generated by the panning operation of the real-world camera as the 2D foreground object image 310, and the synthesized image generation unit 140 generates the synthesized image, It is preferable that the virtual camera 320 is panned and controlled based on the camera control data to generate a composite image. The first camera control data defining the panning operation of the real-world camera can acquire panning angle information, angular velocity information, and the like through a panning detection unit that detects panning of the camera when manually photographing a real-world camera, Can be acquired through a panning control command for controlling the real-world camera. Alternatively, the first camera control data may be generated by estimating a panning motion of a real-world camera through image analysis performed by a real-world camera.

Since the 2D foreground object image 310 is an image whose angle has already been changed by the panning operation of the real world camera, the modeling unit 130 determines that the 2D foreground object image 310 is a moving image of the virtual camera 320, It is more preferable to rotate the 2D foreground object image 310 such that the 2D foreground object image 310 is continuously opposed to the 2D foreground object image 310. The 2D foreground object image 310 generated by reflecting the panning motion of the real world camera can be synthesized with the 3D background background projection plane and the 2D background image transformed with a natural background by the panning operation of the virtual camera 320 have.

In an embodiment according to the moving operation, the foreground object image input unit 120 receives the video image generated by the moving operation of the real-world camera as the 2D foreground object image 310, and the synthesized image generation unit 140 generates the moving foreground It is preferable to move the virtual camera 320 based on the second camera control data defining the operation to generate a composite image. Moving the camera includes both lateral movement, longitudinal movement, and vertical movement, including curvilinear motion in free space. The second camera control data defining the moving motion of the real world camera can acquire the moving direction, the moving speed, the moving speed, and the like through the moving sensing unit that detects the moving of the camera when manually photographing the real world camera, When a real-world camera is automatically photographed, it can be acquired through a moving control command for controlling the real-world camera. Alternatively, the second camera control data can be generated by estimating the moving motion of the real-world camera through image analysis taken by a real-world camera.

In the above-described embodiment, the generation of the composite image according to the panning operation and the moving operation of the camera has been described. However, the present invention is not limited to this, and the camera operation including the panning operation, the moving operation, the zooming operation, the tilting operation, A composite image can be generated.

Hereinafter, an image synthesis method according to the present invention will be described with reference to FIG. First, the image synthesizing system 100 receives the 2D background image (s10). The 2D background image may be a 2D still image or a 2D video image, which is used as a background image of a composite image, and does not require spatial information or depth information of an object photographed on a 2D background image.

Next, the image synthesizing system 100 receives the 2D foreground object image 310 (s20). The 2D foreground object image 310 is a video image obtained by removing a background from a video image photographed by a real-world camera and extracting only one or a plurality of photographed objects, and is used as a foreground image in a composite image. The 2D foreground object image 310 does not require spatial information of the photographed object, but may be configured to further include depth information of the surface to enhance stereoscopic effect depending on the embodiment.

Meanwhile, the 2D background image input step s10 and the 2D foreground object image input step s20 may be performed simultaneously or the 2D foreground object image input step s20 may precede the 2D background image input step s10.

Next, the image synthesizing system 100 performs a step (s30) of forming a first background projection plane 210 having a predetermined height and a concave curved surface in the virtual space as shown in FIG. Due to the shape of the first background projection plane 210, the center part of the 2D background image is located far away from the virtual camera 320, and the left and right sides of the 2D background image are located close to the virtual camera 320. The first background projection plane 210 may be formed in various curved surfaces as shown in FIG. When the first background projection plane 210 is formed in a polygonal shape, a discontinuous background may be depicted according to the movement of the virtual camera 320 due to a sudden change in angle at the corner of the polygon, More preferable.

Next, the image synthesizing system 100 performs step (s40) of projecting at least a part of the 2D background image on the first background projection plane 210 at the virtual projection point in the virtual space. All of the 2D background image may be projected on the first background projection plane 210 and all of the 2D background image may be projected. At this time, the 2D background image to be projected is a 2D background image projected on the first background projection plane 210, which is a stereoscopic image when viewed from a virtual projection point corresponding to the position where the virtual camera 320 is located or adjacent thereto, The 2D background image projecting step s40 is performed in such a manner that the 2D background image transformed into the shape is textured on the first background projection plane 210. [

Next, the image synthesizing system 100 performs a step (s50) of arranging the 2D foreground object image 310 between the virtual projection point on the virtual space and the first background projection plane 210. The 2D foreground object image 310 is spaced a certain distance from the first background projection plane 210 in the virtual space and is elevated to a position spaced apart from the virtual projection point by a certain distance. The 2D foreground object image 310 can be naturally synthesized with the 2D background image as shown in FIG.

According to the embodiment, it is preferable to generate a background projection plane in a three-dimensional shape composed of a plurality of background projection planes as shown in Figs. 4, 5, and 6. According to the embodiment of FIG. 4, step s30 of generating a background projection plane may be performed by the image synthesizing system 100 further comprising a second background projection plane 220 extending from the lower end of the first background projection plane 210, (S40) of projecting the 2D background image is configured so that the image synthesizing system 100 projects at least a part of the 2D background image through the first background projection plane 210 and the second background projection plane 220 do. The bottom portion of the 2D background image is projected onto the second background projection plane 220 forming the bottom of the background projection plane and the second background projection plane 220 is closer to the virtual camera 320 than the first background projection plane 210 So that a natural three-dimensional effect can be formed.

At this time, the step of arranging the 2D foreground object image (s50) includes arranging the 2D foreground object image 310 so that the lower end of the 2D foreground object image 310 comes into contact with the second background projection plane 220 So that the 2D foreground object image 310 is fixed to the 2D background image corresponding to the ground despite the movement of the virtual camera 320, thereby providing a high quality composite image.

The step s30 of generating the background projection plane may be performed by the image synthesizing system 100 so that the first background projection plane 210 and the second background projection plane 220 form an interior angle of 90 to 120 degrees with respect to each other. 210 and a second background projection plane 220. [

 As shown in FIG. 6, the step s30 of generating a background projection plane may be performed by the image synthesizing system 100 by processing an area touched by the first background projection plane 210 and the second background projection plane 220, It is preferable to generate the background projection plane 210 and the second background projection plane 220. With this configuration, it is possible to prevent the background image from being distorted when the virtual camera 320 is moved due to a sudden change in angle in the region where the first background projection plane 210 and the second background projection plane 220 meet .

According to the embodiment of Figures 5 and 6, the step s30 of creating the background projection plane may be performed by the image synthesizing system 100, not only the second background projection plane but also the upper surface of the first background projection plane 210, Step s40 of generating a third background projection plane 230 and projecting a 2D background image may be performed when the image synthesis system 100 has the first background projection plane 210, the second background projection plane 220, And to project at least a part of the 2D background image through the background projection plane 230. [ According to this embodiment, since the background such as the ceiling or the sky corresponding to the upper part of the 2D background image is projected on the third background projection surface 230, it is possible to enhance the perspective of the background.

The background projection plane forming step s30, the 2D background image projecting step s40, and the 2D foreground object image placement step s50 described above may be simultaneously performed by the image synthesizing system 100 or may be performed in a different order.

Finally, the image synthesizing system 100 includes at least a part of the 2D background image and at least a part of the 2D foreground object image 310 at the angle of view of the virtual camera 320 disposed in the virtual space, s60. In the synthetic image generating step s60, the virtual camera 320 preferably generates a more dynamic synthetic image through a panning operation, a moving operation, a tilting operation, a zooming operation, and the like.

When the 2D foreground object image 310 is photographed using a camera work of a real-world camera, the virtual camera 320 preferably generates a synthetic image through the same camera work as the real-world camera. For this, in the panning operation, the 2D foreground object image input step s20 receives the video image generated by the panning operation of the real camera in the image synthesizing system 100 as the 2D foreground object image 310, The generating step s60 is configured so that the image synthesizing system 100 generates a synthesized image by controlling the panning of the virtual camera 320 based on the first camera control data defining the panning operation of the real camera.

In the case of the moving operation, the 2D foreground object image input step s20 receives the video image generated by the moving operation of the real world camera as the 2D foreground object image 310 by the image synthesizing system 100 and generates the synthesized image (s60) is configured such that the image synthesizing system (100) moves the virtual camera (320) based on the second camera control data defining the moving operation of the real camera to generate a synthesized image.

The detailed description of the invention in this specification is intended to illustrate the preferred embodiments, and the invention is not limited to these embodiments. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. .

100: image synthesis system 110: background image input unit
120: foreground object image input unit 130: modeling unit
140: composite image generation unit 210: first background projection plane
220: second background projection plane 230: third background projection plane
310: 2D foreground object image 320: virtual camera

Claims (20)

A background image input unit receiving a 2D background image;
A foreground object image input unit receiving a 2D foreground object image;
Forming a first background projection surface having a height and a concave curved surface at a central portion in a virtual space, projecting at least a part of the 2D background image on the first background projection surface at a virtual projection point on the virtual space, A modeling unit for arranging the 2D foreground object image between the virtual projection point and the first background projection plane; And
And a synthesized image generation unit that generates a synthesized image by including at least a part of the 2D background image and at least a part of the 2D foreground object image at an angle of view of a virtual camera disposed in the virtual space,
The modeling unit,
Further generating a second background projection plane extending from a lower end of the first background projection plane to form a bottom surface and projecting at least a part of the 2D background image via the first background projection plane and the second background projection plane,
Wherein the first background projection plane and the second background projection plane are generated by processing a region in which the first background projection plane and the second background projection plane are in contact with a curved surface.
The method according to claim 1,
Wherein the 2D background image is a still image, the 2D foreground object image, and the composite image are video images.
The method according to claim 1,
Wherein the 2D background image, the 2D foreground object image, and the composite image are video images.
delete The apparatus of claim 1,
And arranges the 2D foreground object image so that a lower end of the 2D foreground object image is in contact with the second background projection plane.
The apparatus of claim 1,
And a third background projection plane extending from an upper end of the first background projection plane to form an upper surface; and generating at least a part of the 2D background image through the first background projection plane, the second background projection plane, To the image synthesizing system.
The apparatus of claim 1,
Wherein the first background projection plane and the second background projection plane are generated so that the first background projection plane and the second background projection plane form an interior angle of 90 to 120 degrees with respect to each other.
delete The method according to claim 1,
The foreground object image input unit receives a video image generated by a panning operation of a real-world camera as the 2D foreground object image,
Wherein the composite image generating unit generates the composite image by controlling the panning of the virtual camera based on first camera control data defining a panning operation of the real camera.
The method according to claim 1,
Wherein the foreground object image input unit receives the video image generated by the moving operation of the real-world camera as the 2D foreground object image,
Wherein the composite image generating unit generates the composite image by moving the virtual camera based on second camera control data defining the moving operation of the real camera.
Receiving a 2D background image by a video synthesis system;
Receiving the 2D foreground object image by the image synthesis system;
Forming a first background projection surface in a virtual space, the first background projection surface having a predetermined height and a central concave surface;
Projecting at least a portion of the 2D background image to the first background projection plane at a virtual projection point on the virtual space;
Arranging the 2D foreground object image between the virtual projection point on the virtual space and the first background projection plane; And
And the image synthesizing system includes at least a part of the 2D background image and at least a part of the 2D foreground object image at an angle of view of a virtual camera disposed in the virtual space to generate a composite image. and,
Wherein the step of forming the first background projection plane further comprises the step of generating a second background projection plane extending from a lower end of the first background projection plane to form a bottom plane,
Wherein projecting the 2D background image comprises projecting at least a portion of the 2D background image via the first background projection plane and the second background projection plane,
The step of generating the first background projection plane may include generating the first background projection plane and the second background projection plane by processing the area touched by the first background projection plane and the second background projection plane as a curved surface A method of synthesizing images.
12. The method of claim 11,
Wherein the 2D background image is a still image, the 2D foreground object image, and the composite image are video images.
12. The method of claim 11,
Wherein the 2D background image, the 2D foreground object image, and the composite image are video images.
delete 12. The method of claim 11,
Wherein the step of arranging the 2D foreground object image places the 2D foreground object image so that the lower end of the 2D foreground object image comes into contact with the second background projection plane.
12. The method of claim 11,
Wherein the generating of the first background projection plane further comprises generating a third background projection plane extending from an upper end of the first background projection plane to form an upper plane,
Wherein projecting the 2D background image projects at least a portion of the 2D background image via the first background projection plane, the second background projection plane, and the third background projection plane. Synthesis method.
12. The method of claim 11,
Wherein the step of generating the first background projection plane comprises the step of moving the first background projection plane and the second background projection plane so that the first background projection plane and the second background projection plane form an interior angle of 90 to 120 degrees with respect to each other And generating the image data.
delete 12. The method of claim 11,
Wherein the receiving of the 2D foreground object image comprises receiving the video image generated by the panning operation of the real-world camera as the 2D foreground object image,
Wherein the generating of the composite image comprises generating the composite image by controlling the panning of the virtual camera based on first camera control data defining a panning operation of the real camera, Way.
12. The method of claim 11,
Wherein the receiving of the 2D foreground object image comprises receiving the video image generated by the moving operation of the real-world camera as the 2D foreground object image,
Wherein the generating of the composite image comprises moving the virtual camera on the basis of the second camera control data defining the moving operation of the real camera to generate the composite image, Way.

Images