JP4108171B2 - Image synthesizer - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image synthesizing apparatus in digital image processing, and in particular, when a still image / moving image obtained by actual shooting is used as a background image and another still image / computer graphic (hereinafter referred to as CG) image is synthesized thereon. The present invention relates to an image composition apparatus that performs composition without a sense of incongruity in consideration of a positional relationship with a background image.
[0002]
[Prior art]
FIG. 27 shows a conventional image composition display device similar to the image composition display device disclosed in, for example, Japanese Patent Laid-Open No. 5-11727, and synthesizes an image captured by a video camera at a position intended for computer graphics. To display.
In FIG. 27, reference numeral 73 denotes a video camera, 74 denotes a direction position recognition sensor such as a geomagnetic sensor or a gyro attached to the video camera 73, and 75 denotes a computer RGB which is a computer image signal based on the direction and size of the computer image. CG device for outputting an image signal, 76 is a computer image frame memory for inputting a computer RGB image signal output from the CG device 75, and 78 is a binary image obtained by binarizing an NTSC signal which is a video RGB image signal from the video camera 73. A video signal conversion device that outputs a video RGB image signal that is a valued video image signal, 79 is a video image frame memory that inputs the video RGB image signal from the video signal conversion device 78 frame by frame, and 80 is a video image frame memory 79 and computer image frames An image composition unit 81 synthesizes both RGB image signals from the memory 76 and outputs a composite RGB image signal, which is a composite binary image signal. 81 is a composite RGB image obtained by binarizing and binarizing the composite RGB image signal. An RGB signal conversion device 82 for converting an image signal into a synthesized NTSC image signal is a display as a display device.
[0003]
Next, the operation will be described.
The video camera 73 includes a direction position recognition sensor 74 that detects an imaging direction of a predetermined area and a current position of the video camera 73. When the video camera 73 captures a predetermined area, the CG device 75 uses the camera viewpoint / direction information detected from the direction position recognition sensor 74 and uses the same viewpoint as the image captured by the video camera 73. A visible CG image is generated. As a result, the size and orientation of the CG image are corrected in order to make the CG object appear to exist at the intended position in the image even if the position and orientation taken by the video camera 73 are changed randomly. The generated image and displayed on the display 82 can be displayed without a sense of incongruity.
[0004]
[Problems to be solved by the invention]
The conventional image composition display device is a device for synthesizing and displaying an image generated on the basis of the CG model created by the CG device 75 with respect to the image taken by the video camera 73, and displaying the video. In order to synthesize another image with the image photographed by the camera 73, there was a problem that a CG model had to be created in advance.
[0005]
In addition, in order to synthesize another image with the moving image shot by the video camera 73, a direction position recognition sensor 74 for detecting the direction and position of the camera is always set while shooting and the direction is set.・ There was a problem that it was necessary to continue detecting position information.
[0006]
In addition, when a CG image is synthesized with an image captured by the video camera 73, since the positional relationship between the subject in the image captured by the video camera 73 and the CG image is not considered, for example, Even when a building based on a CG image is synthesized and displayed over one tree in an image photographed by the video camera 73, the CG image is displayed on the tree, so the tree in front of the CG image is displayed. There was a problem that the image of was hidden.
[0007]
The present invention has been made to solve the above-described problems. Even if a CG model is not created in advance, a still image can be prepared to synthesize another image in a background image without a sense of incongruity. In addition, it is possible to synthesize another image in the background moving image without a sense of incongruity by a simple operation, and to consider both images in consideration of the context of the subject in the background image and the additional image added to the background image. An object is to obtain an image composition device having a composition function.
[0008]
[Means for Solving the Problems]
  An image composition apparatus according to the present invention includes a background image input unit for inputting a background image, an additional image input unit for inputting an additional image, and a background image three-dimensional information input unit for inputting three-dimensional information of the background image. The additional image relative position defining means for defining the relative position of the additional image on the background image, and the additional image on the background image based on the relative position of the additional image defined by the additional image relative position defining means The background image is divided for each image area in accordance with the three-dimensional information inputted by the background image three-dimensional information input means, and the additional image shape correcting means for correcting the shape of the additional image before synthesis. A background image region dividing unit for determining the relationship between each image region divided by the background image region dividing unit and the additional image; An image for combining the additional image whose shape has been corrected by the additional image shape correcting unit with a background image and for combining the image region determined to be in front of the additional image by the context determination unit from above the additional image By synthesizing the image, another image is synthesized in consideration of the context of the subject in the background image.At the same time, the positional relationship between the map information storing means storing the map information and the image area divided by the background image area dividing means is extracted with reference to the map information around the background image from the map information storing means. The background image region positional relationship extracting means is further reflected in the composite image, and the background image region positional relationship extracting means is divided in the background image. Find the center of gravity of each image area, associate the camera viewpoint position with the background image and the center of gravity of the image area on the map information, and display the map information from the camera viewpoint position to each center of gravity point in the camera field of view. By drawing a straight line, the distance between the subject ahead of the center of gravity and the viewpoint position of the camera is obtained, and the obtained distance is associated with each image area.
  Also,An image composition apparatus according to another invention includes a background image input means for inputting a background image, an additional image input means for inputting an additional image, and a background image 3D information input means for inputting 3D information of the background image. And an additional image relative position defining means for defining the relative position of the additional image on the background image, and the additional image on the background image based on the relative position of the additional image defined by the additional image relative position defining means And dividing the background image into image areas according to the three-dimensional information input by the background image three-dimensional information input means, and additional image shape correcting means for correcting the shape of the additional image before combining the images. A background image region dividing unit for performing the determination, and a context determining unit for determining a context between each image region divided by the background image region dividing unit and the additional image, An image for combining the additional image whose shape has been corrected by the additional image shape correcting unit with a background image and for combining the image region determined to be in front of the additional image by the context determination unit from above the additional image By combining the composition means, another image is synthesized while considering the context with the subject in the background image,When the background image is divided into several image areas by the map information storage means for storing the map information and the background image area dividing means, the boundary line of the subject in the background image is displayed as the background image from the map information storage means. By dividing the image area for each subject in the background image by further comprising area boundary association means for extracting by referring to surrounding map informationAnd the area boundary associating means associates a background image and a camera viewpoint on the map information, draws a straight line from the camera viewpoint to a boundary portion of the subject reflected in the background image, The intersection of the objects is the boundary line of the subject in the background imageIt is characterized by that.
[0018]
DETAILED DESCRIPTION OF THE INVENTION
Embodiment 1 FIG.
FIG. 1 is a system configuration diagram showing Embodiment 1 of the present invention.
In FIG. 1, 1 is a background image input means for inputting an image as a background (hereinafter referred to as a background image) when the images are synthesized on a computer, and 2 is synthesized with the background image input by the background image input means. The additional image input means 3 for inputting another image (hereinafter referred to as an additional image) to perform, is an additional image arrangement means for arranging the additional image on the background image. Background image 3D information input means 4 for inputting 3D information such as a shooting position of the background image, additional image relative position defining means 5 for defining the relative position of the additional image on the background image, and addition And an additional image shape correcting unit 6 for correcting the shape of the additional image before synthesizing the additional image on the background image based on the relative position of the additional image defined by the image relative position defining unit 5. Reference numeral 7 denotes image combining means for combining the additional image whose shape has been corrected by the additional image shape correcting means 6 with the background image.
[0019]
Next, the operation will be described.
First, the background image input means 1 inputs an image as a background for image synthesis. In order to input a background image, image data taken with a digital camera is taken into a computer, or video taken with a video camera is taken into a computer as digital data. Next, the additional image input means 2 inputs an additional image to be combined with the background image. In order to input additional images, image data taken with a digital camera is taken into a computer or created with commercially available image drawing software. In this embodiment, an example in which an image obtained by photographing a rectangular plate-like object such as a billboard advertisement is combined with a background image such as a town landscape photographed by a camera will be described. At this time, information on the distance and subject size when the additional image is taken is also input.
[0020]
FIG. 2 is a diagram for explaining an additional image and information related thereto.
An image 10 of the subject 11 is shown in the photographing field 9 when the subject 11 is photographed by the photographing camera 8. The image 10 is input as an additional image. As information about the additional image, the size (Lx, Ly) of the subject 11 and the distance L1 between the camera 8 and the subject 11 are also input simultaneously. In FIG. 2, the image 10 when the subject 11 is perpendicular to the camera line of sight is an example of the additional image, but an image when the subject 11 has an angle with respect to the camera line of sight is added. An image can also be synthesized by a method similar to the method described below.
[0021]
Next, the background image 3D information input means 4 inputs information about the shooting position and shooting direction of the camera when the background image is shot.
FIG. 3 is a diagram for explaining a situation when an image used as a background image is captured.
In FIG. 3, reference numeral 12 denotes a camera for taking a background image, and reference numeral 13 denotes a background image taken by the camera 12. Further, (X2, Y2, Z2) are absolute coordinates of the shooting position when the camera 12 captures a background image, and (θx2, θy2, θz2) is the absolute coordinate of the camera when the background image is captured. Direction. Here, θx, θy, and θz represent rotation angles of the camera 12 with respect to the X-axis direction, the Y-axis direction, and the Z-axis direction, respectively.
[0022]
Next, the additional image relative position defining means 5 inputs a place where the subject 11 is located with respect to the photographing position and photographing direction of the camera 12 when the background image is photographed.
FIG. 4 is a diagram for explaining the position and orientation of the subject 11 with respect to the photographing position and photographing direction of the camera 12 that photographed the background image.
In FIG. 4, (X1, Y1, Z1) are absolute coordinates of the position of the subject 11, and (θx1, θy1, θz1) are directions in the absolute coordinates of the subject 11. Reference numeral 14 denotes an image of the subject 11 that is finally combined with the background image 13 when the position / orientation of the subject 11 is defined by the additional image relative position defining means 5. L2 represents the distance between the camera 12 and the virtually placed subject 11, and L2 = {(X2-X1) 2+ (Y2-Y1) 2+ (Z2-Z1) 2} 1/2.
[0023]
Next, the additional image shape correcting unit 6 corrects the shape of the additional image 10 to generate an image 14 to be combined with the background image 13.
FIG. 5 is a diagram for explaining a method of correcting the shape of the additional image 10 and will be described based on the relative position / orientation of the subject 11 with respect to the position / orientation of the camera 12.
FIG. 5 shows a case where the subject 11 is rotated by θy in the Y-axis direction with respect to the camera 12. Let the vertical and horizontal lengths of the additional image 10 based on the subject 11 be Lx ′ and Ly ′, respectively. As described above, the subject 11 is perpendicular to the camera 12 when the additional image 10 is taken. In FIG. 5, since the subject 11 is rotated by θy with respect to the camera 12, the width Lx ′ of the additional image 10 becomes Lx′cos (θy). This is performed by dividing the additional image 10 into strip-shaped regions having a width 1 as shown in FIG. 5 and thinning the strip-shaped regions 15 in accordance with the scale ratio.
[0024]
Further, as the subject 11 rotates, a portion that approaches or moves away from the camera 12 appears. This is dealt with by calculating the length in the Y-axis direction for the band-like region 15 that has not been thinned out, and expanding and contracting it.
In the example of FIG. 5, the leftmost band-like area 15 of the additional image 10 is the part where the subject 11 is closest to the camera 12, and thus is expanded in the Y-axis direction like the band-shaped area 16 after shape correction. The other band-shaped regions are also expanded in the Y-axis direction by the same calculation as the band-shaped region 16, and the expanded band-shaped regions are combined to generate the additional image 14 after shape correction. In the example of FIG. 5, L2 is described as the same distance as L1 (distance when an additional image is captured) for the sake of simplicity. However, when L2 is different from L1, it is added as described above. After correcting the shape of the image 10, the additional image 14 after the shape correction can be obtained by multiplying the vertical and horizontal sizes by L1 / L2.
[0025]
Although the shape correction of the additional image when the subject 11 is rotated in the Y-axis direction has been described with reference to FIG. 5, the same method is used for the shape correction of the additional image when the subject 11 is rotated in the X-axis direction. With this, shape correction can be performed. Further, when the image is rotated in the Z-axis direction, the shape correction is performed by rotating the additional image 10 on the plane of the background image 13.
Finally, the background image 13 and the shape-corrected additional image 14 are combined by the image combining means 7.
[0026]
As described above, according to the first embodiment, when inputting an image as a background and another image to be combined therewith, an additional image is combined into the background image by simultaneously inputting shooting information and subject information. Since the shape of the additional image is corrected when the image is added, it is not necessary to create a 3D CG model in advance to synthesize another image with the background image, and the background image and the other image can be easily and realistically created. Can be synthesized.
[0027]
Embodiment 2. FIG.
In the first embodiment described above, a completely different additional image is input and synthesized with respect to the background image. Next, the image portion of the subject reflected in the background image is corrected. An embodiment in such a case will be described.
FIG. 6 is a system configuration diagram according to the second embodiment.
In FIG. 6, the background image input unit 1, the additional image arrangement unit 3, the background image three-dimensional information input unit 4, the additional image relative position defining unit 5, the additional image shape correcting unit 6, and the image synthesizing unit 7 are the same as those in the first embodiment. It is the same. The difference between the second embodiment and the second embodiment is that the additional image input unit 2 searches for the additional image from the background image and specifies the image range from the background image, and the background image. This is the point constituted by the background image cutout means 18 for cutting out the image in the range designated by the range designation means 17 as an additional image and passing it to the additional image arrangement means 3.
[0028]
Next, the operation will be described.
First, in the same manner as in the first embodiment, the background image input unit 1 inputs an image as a background. Next, the background image range designating unit 17 designates an image region to be subjected to shape correction from the background image input by the background image input unit 1.
FIG. 7 is a diagram for explaining a method in which the user designates an image area to be subjected to shape correction from the background image.
Assume that it is desired to correct the image area 19 in which the subject 20 is reflected when the camera 12 captures the background image 13. The user inputs the designated range 21 using the mouse 22 on the background image 13 displayed on the display so as to include the range of the image region 19. In the example of FIG. 7, the rectangular range is designated, but it is also possible to designate a polygonal region by designating the vertex of the polygon with the mouse. The range of the image area 19 to be corrected is specified from the background image 13 by the above operation.
[0029]
Next, the background image cutout means 18 cuts out and inputs the image area 19 designated by the background image range designation means 17. Information on the subject 20 that is the actual state of the image area 19 is also input. (X1, Y1, Z1) and (θx1, θy1, θz1) are position coordinates and orientation information of the subject 20 when the camera 12 is the origin.
[0030]
The additional image arrangement means 3 uses the image area 19 cut out from the background image 13 as an additional image, and changes the shape and shape of the additional image by changing the position and orientation of the subject 20 in the same procedure as in the first embodiment. Is obtained by correcting. Unlike the first embodiment, depending on the new position / orientation of the subject 20, a part or all of the cut-out image area 19 is not included in the newly added area of the additional image and is included in the background image. A blank area may occur. In such a case, the image synthesizing unit 7 fills an area that is not included in the additional image area with a color obtained by averaging the colors of the pixels on the boundary line of the blank area.
[0031]
FIG. 8 is a diagram for explaining the filling of the blank area as described above.
In FIG. 8, reference numeral 23 denotes an additional image whose shape has been corrected, and reference numeral 24 denotes a blank area formed by the image region 19 which is not included in the additional image 23 whose shape has been corrected. In the image composition means 7, the blank area 24 becomes a filled area 25 by the average color (white in the example of FIG. 8) of the pixels on the boundary line.
[0032]
As described above, in the second embodiment, when a subject whose position or orientation is to be changed is shown in the background image, the image area of the subject is cut out from the background image, and the subject position / orientation is determined. Since the image is corrected in accordance with the change, it is possible to easily correct a part of the background image once inputted.
[0033]
Embodiment 3 FIG.
In the above embodiment, an image taken by a camera is input to input an additional image. In the third embodiment, a three-dimensional CG model is used to input an additional image.
FIG. 9 is a system configuration diagram according to the third embodiment.
In FIG. 9, the background image input means 1, the additional image arrangement means 3, the background image three-dimensional information input means 4, the additional image relative position defining means 5, the additional image shape correcting means 6 and the image composition means 7 are the same as those in the first embodiment. It is the same. The difference in the third embodiment is that the additional image input means 2 is divided into a three-dimensional model input means 26 and a three-dimensional model input means 26 for inputting a source three-dimensional CG model for generating an additional image. The image data is generated on the basis of the input three-dimensional CG model, and is constituted by the image data conversion means 27 for passing to the additional image arrangement means 3 as an additional image.
[0034]
Next, the operation will be described.
First, in the same manner as in the first embodiment, the background image input unit 1 inputs an image as a background. Next, a three-dimensional CG model from which an additional image is generated is input by the three-dimensional model input unit 26. As the 3D model input means, commercially available 3D CAD software is used. Next, the image / data conversion means 27 generates an additional image based on the input three-dimensional CG model.
[0035]
FIG. 10 is a diagram for explaining a method of synthesizing image data based on a three-dimensional model as an additional image in a background image.
In the image / data conversion unit 27, the CG 28 input by the three-dimensional model input unit 26 is placed at a distance L1 with respect to the camera position 30, and an image 29 obtained as a result in the camera field of view is taken out as an additional image.
[0036]
In the additional image arrangement unit 3, the shape of the additional image can be corrected by the same procedure as in the first embodiment, using the image 29 obtained by the image / data conversion unit 27 as an additional image.
[0037]
As described above, in the third embodiment, an additional image is generated based on the three-dimensional CG model input by the three-dimensional model input unit 26, and is synthesized with the background image while correcting the shape of the additional image. Therefore, if one additional image based on the 3D model is generated, the 3D CG model can be applied to the background image without preparing software for handling the 3D CG thereafter. Various variations of images generated from can be easily combined.
[0038]
Embodiment 4 FIG.
In the above embodiment, a still image is used as a background image. In the fourth embodiment, a moving image is used as a background image, and an additional image is synthesized for each frame of the background moving image.
FIG. 11 is a system configuration diagram according to the fourth embodiment.
In FIG. 11, the background image input means 1, the additional image input means 2, and the additional image arrangement means 3 are the same as those in the first embodiment. Furthermore, in the fourth embodiment, a moving image input unit 31 for inputting a moving image as a background, and an area for synthesizing an additional image from each frame of the moving image input by the moving image input unit 31 The additional image arranged in one frame in the moving image by the synthetic area extracting means 32 and the additional image arranging means 3 for extracting the image is added to the preceding and following frames using the area extracted by the synthetic area extracting means 32. Additional image propagating means 33 for adding images while correcting the shape of the image is added.
[0039]
Next, the operation will be described.
First, a moving image used as a background is input by the moving image input means 31. Next, the user selects one frame from the frames of the moving image input by the moving image input unit 31, and inputs the image of the selected frame as a background image by the background image input unit 1. Next, an additional image is input by the additional image input means 2 as in the first embodiment. Next, the additional image arrangement means 3 arranges the additional image in the background image.
[0040]
In the fourth embodiment, a method of synthesizing an additional image with a moving image will be described using an example of synthesizing a mural as an additional image on a wall surface of a building that is captured in a moving image taken from a road.
FIG. 12 shows a method of extracting a composite area from frames before and after the one frame by the composite area extraction means 32 from the area of the additional image arranged in one frame in the video input by the video input means 31. It is a figure for demonstrating.
In FIG. 12, reference numeral 34 denotes a moving image input by the moving image input means 31. Reference numeral 35 denotes a frame selected as a background image from the frames included in the moving image 34 by the background image input means 1. Reference numeral 36 denotes an additional image of the mural placed in the frame 35 by the additional image placement means 3. Reference numeral 37 denotes an additional image 35. The boundary line 38 is a boundary line obtained by propagating an area corresponding to the boundary line 37 defined in the frame 35 to the frames before and after the frame 35. In the example of the present embodiment, the wall surface of the building shown in the front and rear frames is propagated.
[0041]
FIG. 13 is a diagram for explaining a method of tracking a boundary line of an object shown in a moving image frame.
When the boundary line 37 in the frame 35 is placed at the same position in the next frame 39, the boundary line 40 is obtained. A region 41 having the same color as the color inside the boundary line 37 of the frame 35 is extracted inside the boundary line 40 of the frame 39. Next, the pixels having the same color are traced toward the outside of the area 41 to obtain the area 42. Finally, a similar figure 43 of the boundary line 40 that is inscribed in the area 42 is obtained, and this is used as a synthesis area in the frame 39. By using the similar figure 43 obtained here as a new boundary line and performing the same process on the frame following the frame 39, the boundary line of the object shown in the frame of the moving image can be tracked. .
[0042]
Although a simple method using the color information inside the boundary line 37 has been described here, a dynamic contour model (Snakes) can also be used as a method for tracking the boundary line of an object in a moving image. . This method is described in “Study on Tracking of Moving Object Using Active Contour Model” (1995 Information and Systems Society Conference D-237).
[0043]
Further, FIG. 14 is a diagram for explaining a method of synthesizing an additional image with each frame by the additional image propagation means 33 in accordance with the shape of the synthesis area for each frame extracted by the synthesis area extraction means 32.
The additional image 36 is divided into strip-like regions 15 having a width of 1, and the width is adjusted by performing thinning and widening in accordance with the width of the boundary line 43 of another frame. Next, the additional image 36 having the width matched to the boundary line 43 is again divided into strip-shaped areas having a width of 1, and the vertical height of each strip-shaped area is matched to the boundary line 43, whereby the additional image 36 is changed to each frame. Can be matched to the shape of the boundary line.
[0044]
As described above, in the fourth embodiment, a moving image is used as a background, and an additional image is arranged by using one frame in the moving image as a background image, thereby extracting a corresponding composite region of the preceding and following frames. Since the additional image is arranged, another image can be easily combined with the moving image.
[0045]
Embodiment 5 FIG.
In Embodiment 4 described above, when another image is combined with a moving image, only one frame that defines the arrangement position of the additional image is used, and the combined region is tracked for the frames before and after that. However, in the fifth embodiment, frames that define the arrangement position of the additional image are applied to the two frames of the synthesis start frame and the end frame, and are automatically applied to each frame between the two frames. To synthesize additional images.
[0046]
FIG. 15 is a system configuration diagram according to the fifth embodiment.
In FIG. 15, the background image input unit 1, the additional image input unit 2, the moving image input unit 31, and the additional image propagation unit 33 are the same as those in the fourth embodiment. In the fifth embodiment, the additional image placement unit 3 is an addition for determining the relative position of the additional image with respect to the synthesis start frame selected from the frames of the moving image input by the moving image input unit 31. Image first relative position defining means 44, background image three-dimensional information first input means 45 for inputting the three-dimensional information of the start frame, and additional image for determining the relative position of the additional image with respect to the synthesis end frame Second relative position defining means 46, background image three-dimensional information second input means 47 for inputting the three-dimensional information of the end frame, and additional image shape correcting means for correcting the shape of the additional image with respect to the start frame and the end frame 6 is provided.
[0047]
Next, the operation will be described.
First, the moving image input unit 31 selects a start frame and an end frame for compositing an additional image from the input moving image frames by the background image input unit 1. Next, with respect to the additional image input by the additional image input unit 2, an additional image first relative position defining unit 44, a background image three-dimensional information first input unit 45, an additional image second relative position defining unit 46, The background image three-dimensional information second input unit 47 and the additional image shape correcting unit 6 correct the shape of the additional image with respect to the start frame and the end frame. The shape correction of the additional image with respect to the start frame and the end frame is performed by the same method as in the first embodiment. Next, based on the additional images in the start frame and the end frame, the additional image complementing unit 48 generates an additional image area to be combined with the frame between the start frame and the end frame.
[0048]
FIG. 16 is a diagram for explaining a method for generating an additional image to be synthesized by the additional image complementing means 48 into a frame between the start frame and the end frame.
In the example of FIG. 16, an additional image is taken as an example as in the first embodiment.
In FIG. 16, 49 and 50 are a start frame and an end frame of synthesis in a moving image, respectively, and 51 and 52 are additional images respectively arranged on the start frame 49 and the end frame 50 by the additional image arrangement unit 3. If there are (N-1) frames between the start frame 49 and the end frame 50, the coordinates of the vertices of the area of the additional image with respect to the k-th frame of the (N-1) frame are expressed by the equation shown in FIG. Can be calculated.
[0049]
Then, according to the area of the additional image of each frame generated by the additional image complementing unit 48, the additional image propagation unit 33 synthesizes the additional image with each frame in the same manner as described in the fourth embodiment.
[0050]
As described above, in the fifth embodiment, since the arrangement position of the additional image with respect to the start frame and the end frame in the moving image is designated, the composition of the additional image with respect to the frame between the start frame and the end frame is designated. Therefore, it is possible to easily synthesize another image in a moving image without performing complicated calculation processing such as tracking an object in a moving image. In the description of the fifth embodiment, an additional image is specified for a start frame and an end frame in a moving image. However, a start / end is performed by placing a 3D CG model in the start / end frame instead of image data. It is also possible to calculate the arrangement position of the three-dimensional CG model in the frame between the frames using the same principle and combine it with the moving image.
[0051]
Embodiment 6 FIG.
In Embodiment 4 described above, when another image is synthesized with a moving image, the position of the additional image is calculated assuming that the position of the additional image is stationary with respect to the background image. In 6, the additional image moving in the background image is synthesized by defining the moving path of the subject that is the source of the additional image with respect to the background image.
[0052]
FIG. 17 is a system configuration diagram according to the sixth embodiment.
In FIG. 17, the background image input unit 1, the additional image input unit 2, the additional image arrangement unit 3, the composite region extraction unit 32, and the additional image propagation unit 33 are the same as those in the fourth embodiment. Further, it is constituted by an additional image operation defining means 54 for defining a moving path of a subject as an additional image source with respect to a background moving image.
[0053]
Next, the operation will be described.
The process until the additional image is arranged on one frame by the moving image input unit 31, the background image input unit 1, the additional image input unit 2, and the additional image arrangement unit 3 is the same as that of the fourth embodiment. Next, in the additional image operation defining means 54, the user defines the path of movement of the additional image arranged on one frame.
[0054]
FIG. 18 is a diagram for explaining the moving path of the subject defined by the additional image operation defining means 54.
In FIG. 18, 55 represents the position / orientation of the subject 28 relative to the camera 30 in the moving image input by the moving image input means 31 with respect to time. 55, the synthesis area extraction unit 32 extracts the synthesis areas before and after the frame in which the additional image 29 is arranged.
Next, in the same manner as in the fourth embodiment, the additional image propagation means 33 synthesizes the additional image with another frame.
[0055]
As described above, in the sixth embodiment, since the movement of the subject in the moving image is defined, it is possible to easily synthesize the image of the moving subject with the moving image. In the description of the sixth embodiment, an additional image is specified as a background image in a moving image. However, by placing a three-dimensional CG model in the background image instead of image data, 3 in the frames before and after the background image is displayed. Since the motion of the three-dimensional CG model can be defined using the same principle and synthesized with a moving image, the moving three-dimensional CG model and the moving image can be easily synthesized.
[0056]
Embodiment 7 FIG.
In the above embodiment, when another image is combined with the background image, since the context of the subject in the background image and the subject in the additional image is not considered, the entire additional image is always taken into account. Is displayed on the background image, but in the seventh embodiment, the background image is divided into regions for each subject, and the additional image is synthesized using the position information.
[0057]
FIG. 19 is a system configuration diagram in the seventh embodiment.
In FIG. 19, the background image input unit 1, the additional image input unit 2, the additional image arrangement unit 3, and the image composition unit 7 are the same as those in the first embodiment. Further, the background image input means 1 for dividing the background image into areas for each subject, the image area divided by the background image area dividing means 56, and the front and rear of the additional image And a context determining means 57 for determining the relationship.
[0058]
Next, the operation will be described.
The process is the same as that of the first embodiment until the additional image is arranged on the background image by the background image input unit 1, the additional image input unit 2, and the additional image arrangement unit 3. The background image area dividing means 56 divides the background image for each subject and adds information about the distance from the camera viewpoint for each area.
FIG. 20 is a diagram for explaining a method of performing region division of a background image.
While viewing the background image 13 displayed on the display, the user designates the apex of the area while operating the mouse for each area of the subject to define a polygonal area. Each designated area has an area No. And distance information 58 from the camera viewpoint are input in association with each other.
[0059]
Next, based on the distance information from the camera viewpoint of each image region of the background image obtained by the background image region dividing unit 56 and the position information of the additional image by the additional image arranging unit 3, the context determining unit 57 The order in which the additional images are combined is determined. The image composition means 7 displays the image area farther from the camera viewpoint than the additional image by the context determining means 57, then the additional image, and finally the image area closer to the camera viewpoint than the additional image. Is synthesized.
[0060]
FIG. 21 is a diagram for explaining the background image and the additional image synthesized based on the order determined by the context determination unit 57.
When the additional image 59 with a distance of 15 from the camera is combined with the background image divided into regions, the background image with a distance from the camera smaller than 15 is combined on the additional image 59 and finally. A composite image 60 is obtained.
[0061]
As described above, in the seventh embodiment, the background image is divided into image areas for each subject, and distance information from the camera is associated with each image area. Therefore, when an additional image is combined with the background image It is possible to easily obtain a composite image that correctly reflects the context of the subject.
When a moving image is input as a background image, the region of the subject is defined for each frame in the moving image by dividing the region by the background image region dividing unit 56, and the region number and the camera viewpoint are assigned to each region. The distance information from is input in association. Further, by arranging the additional images in the same manner as in the fourth to sixth embodiments, the same effect can be obtained even when a moving image is input as the background image.
[0062]
Embodiment 8 FIG.
In the seventh embodiment described above, in order to divide the background image into regions for each subject and synthesize additional images using the positional information, the background image region dividing means allows the user to read from the camera for each image region. In the eighth embodiment, the distance from the camera for each image area is input by referring to the map information regarding the background image.
[0063]
FIG. 22 is a system configuration diagram according to the eighth embodiment.
In FIG. 22, the background image input means 1, the additional image input means 2, the additional image arrangement means 3, the image composition means 7, the background image area dividing means 56, and the context determination means 57 are the same as in the seventh embodiment. . Further, the map information storage means 62 for storing the geographical information of the scene shown in the background image input by the background image input means 1 and the map information stored in the map information storage means 62 are referred to. A background image area positional relationship extracting means 61 for associating distance information from the camera with an area in the background image is provided.
[0064]
Next, the operation will be described.
Until the background image input unit 1, the additional image input unit 2, and the additional image arrangement unit 3 arrange the additional image on the background image, and until the background image region dividing unit 56 divides the background image into image regions for each subject. This is the same as in the seventh embodiment. Next, the distance from the camera is associated with each image area divided from the background image by the background image area positional relationship extracting means 61.
[0065]
FIG. 23 is a diagram for explaining the function of the background image region positional relationship extracting means 61. In the eighth embodiment, an example in which a landscape of a town building is taken as a background image and map information around the taken background image is saved in the map information saving means 62 will be described. The barycentric point 66 of each image area divided in the background image 64 is obtained. Next, the field of view of the camera, that is, the background image 64, the viewpoint position 65 of the camera, and the barycentric point 66 of the image area are associated on the map information 63. The map information 63 includes information on the shape and position of the building shown in the background image. Accordingly, by drawing a straight line on the map information 63 from the camera viewpoint 65 to each center of gravity in the camera field of view, the distance between the actual building ahead of the center of gravity and the camera viewpoint 65 is determined. Can be sought. The distance thus obtained is associated with each barycentric point 66, that is, each image area.
[0066]
The background image and the additional image are synthesized in the same manner as in the seventh embodiment, using the distance between the image region in the background image associated with the background image region positional relationship extracting means 61 and the camera viewpoint.
[0067]
As described above, in the eighth embodiment, after the background image is divided into image areas for each subject, each image area and distance information from the camera are associated using map information. Thus, it is possible to easily determine the context of the subject when the additional image is combined.
When a moving image is input as the background image, distance information from the camera is associated with the image region divided for each frame in the moving image using the background image region positional relationship extracting means 61. Further, by arranging the additional images in the same manner as in the fourth to sixth embodiments, the same effect can be obtained even when a moving image is input as the background image.
[0068]
Embodiment 9 FIG.
In Embodiment 7 described above, in order to divide the background image into regions for each subject, the user designates the image region by operating the mouse while looking at the display. In Embodiment 9, however, the map relating to the background image is specified. By referring to the information, the image area for each subject is divided from the background image.
[0069]
FIG. 24 is a system configuration diagram according to the ninth embodiment.
In FIG. 24, the background image input unit 1, the additional image input unit 2, the additional image arrangement unit 3, the image composition unit 7, the background image region division unit 56, the context determination unit 57, and the map information storage unit 62 are implemented. This is the same as the seventh embodiment. Further, an area boundary associating unit 67 for dividing the image area for each subject from the background image with reference to the map information stored in the map information storing unit 62 is provided.
[0070]
Next, the operation will be described.
The process until the additional image is arranged on the background image by the background image input unit 1, the additional image input unit 2, and the additional image arrangement unit 3 is the same as that of the seventh embodiment. Next, the area boundary associating means 67 divides the background image for each subject.
FIG. 25 is a diagram for explaining the function of the region boundary association means.
First, a background image 64 and a camera viewpoint 65 are associated on the map information 63. Next, a straight line is drawn from the camera viewpoint 65 to the boundary portion of the building of the subject reflected in the background image 64. An intersection 68 between the drawn straight line and the background image 64 is a horizontal boundary line of the subject in the background image. Further, if the height information of each building is stored in the map information 63, the image area 69 for each building of the subject can be extracted on the background image using the information. The background image area dividing means 56 displays the image area 69 extracted by the area boundary associating means 67 on the display, and if necessary, the user can correct the shape of the image area by operating the mouse. Next, the distance from the camera viewpoint is input for each image area in the same manner as in the seventh embodiment, and the background image and the additional image are combined.
[0071]
As described above, in the ninth embodiment, when a background image is divided into image areas for each subject, an approximate image area is automatically extracted using map information. When combining additional images, position information from the camera viewpoint for each subject can be easily associated. In the description of the ninth embodiment, the image area extracted by the area boundary associating unit 67 is corrected by the background image area dividing unit 56. However, the image area is used as it is without being corrected. The background image and the additional image may be combined while being divided into regions.
[0072]
Embodiment 10 FIG.
In Embodiment 7 described above, a still image is added as an image to be combined with the background image. However, in Embodiment 10, an image obtained by combining the three-dimensional CG model with the background image is obtained instead of the still image.
When a moving image is input as the background image, the image area of the subject is divided using the region boundary association unit 67 and the background image region dividing unit 56 for each frame in the moving image. Further, by arranging the additional images in the same manner as in the fourth to sixth embodiments, the same effect can be obtained even when a moving image is input as the background image.
[0073]
FIG. 26 is a system configuration diagram according to the ninth embodiment.
In FIG. 26, the background image input unit 1, the background image three-dimensional input unit 4, the image composition unit 7, the background image region division unit 56, and the context determination unit 59 are the same as those in the seventh embodiment. Also, a three-dimensional CG model input means 70 for inputting a three-dimensional CG model to be combined with the background image, and a three-dimensional CG model relative position defining means 71, 3 for defining the relative position of the three-dimensional CG model with respect to the background image. A three-dimensional CG model shape correcting unit 72 is provided for correcting the shape of the additional image to be combined with the background image according to the relative position of the three-dimensional CG model.
[0074]
Next, the operation will be described.
First, in the same manner as in the first embodiment, the background image input unit 1 inputs an image as a background. Next, the 3D CG model input means 70 inputs a 3D CG model from which an additional image is generated. As the 3D model input means, commercially available 3D CAD software is used. Next, the three-dimensional CG relative position defining means 71 defines the relative position of the input three-dimensional CG model with respect to the background image camera position. The background image camera position is obtained by the background image three-dimensional information input means 4 as in the first embodiment. Next, based on the position defined by the 3D CG model relative position defining means 71, an image in which the 3D CG appears in the background image is determined by the 3D CG shape correcting means. Thereafter, the background image and the image based on the three-dimensional CG model are synthesized by the background image region dividing unit 56, the context determining unit 57, and the image synthesizing unit 7 in the same manner as in the seventh embodiment.
[0075]
As described above, in the tenth embodiment, the additional image to be combined with the background image is generated using the three-dimensional CG model. It can be reflected and combined with the background image.
Further, as described in the seventh to ninth embodiments, the same effect can be obtained when a moving image is input as a background image.
[0076]
According to each embodiment described above, there is an effect that another image can be easily combined with the background image from various viewpoints. Further, even when a moving image is used as a background image, the image is automatically corrected and synthesized with respect to the remaining frames simply by arranging an additional image in one or two frames in the moving image. An image can be easily combined with each frame. Further, since the position information of the subject in the background image is taken into consideration, there is an effect that the context of each subject can be reflected when the background image and another image are combined. Note that the composition considering the positional relationship of the subject in the background image can be realized by using the same method even when a moving image is used as the background image.
[0077]
【The invention's effect】
As described above, according to the present invention, when inputting the background image and another image to be combined therewith, the additional image is combined into the background image by simultaneously inputting the shooting information and the subject information. Since the shape of the additional image is sometimes corrected, there is no need to create a 3D CG model in advance to synthesize another image with the background image, and the background image and the other image can be easily and realistically synthesized. can do.
[0078]
In addition, if a subject whose position or orientation is to be changed is shown in the background image, the image area of the subject is cut out from the background image and the image is corrected according to the change in the subject's position and orientation. Therefore, it is possible to easily correct a part of the background image once inputted.
[0079]
In addition, since an additional image is generated based on the 3D CG model input by the 3D model input means and the shape of the additional image is corrected, the image is combined with the background image. Once one additional image has been generated, various variations of the image generated from the 3D CG model can be easily performed on the background image without preparing software for handling 3D CG. Can be synthesized.
[0080]
In addition, by using a moving image as a background and arranging an additional image using one frame in the moving image as a background image, the corresponding combined area of the preceding and succeeding frames is extracted and the additional image is arranged. Another image can be easily synthesized in the moving image.
[0081]
In addition, since the arrangement position of the additional image with respect to the start frame and the end frame in the moving image is designated, the composition of the additional image with respect to the frame between the start frame and the end frame is performed. Another image can be easily synthesized in a moving image without performing complicated calculation processing such as tracking.
[0082]
Further, since the movement of the subject in the moving image is defined, it is possible to easily synthesize the image of the moving subject with respect to the moving image.
[0083]
Also, since the background image is divided into image areas for each subject and distance information from the camera is associated with each image area, the context of the subject is correctly reflected when the additional image is combined with the background image. A composite image can be easily obtained.
[0084]
In addition, after dividing the background image into image areas for each subject, the distance information from each camera area and the camera is associated using the map information, so when adding an additional image to the background image, The context can be easily determined.
[0085]
Also, when the background image is divided into image areas for each subject, an approximate image area is automatically extracted using the map information. Position information from the camera viewpoint can be easily associated.
[0086]
In addition, since the additional image to be combined with the background image is generated using the three-dimensional CG model, it is possible to combine the freely created three-dimensional image with the background image while correctly reflecting the context of the subject. it can.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of an image composition device according to Embodiment 1 of the present invention.
FIG. 2 is a diagram for explaining an additional image input by the additional image input unit 2 of FIG. 1 and information related thereto.
3 is a diagram for explaining information related to a background image input by a background image three-dimensional information input unit 4 of FIG. 1; FIG.
4 is a diagram for explaining the position and orientation of a subject with respect to the photographing position and photographing direction of a camera that has photographed a background image defined by the additional image relative position defining means 5 of FIG. 1;
FIG. 5 is a diagram for explaining a method of correcting the shape of an additional image by the additional image shape correcting unit 6 in FIG. 1;
FIG. 6 is a block diagram showing a configuration of an image composition device according to Embodiment 2 of the present invention.
7 is a diagram for explaining a method in which a user designates an image area to be subjected to shape correction from a background image by the background image range designating unit 17 in FIG. 6; FIG.
8 is a diagram for explaining a method of filling a blank area generated in a background image in the additional image shape correcting unit 6 in FIG. 6;
FIG. 9 is a block diagram showing a configuration of an image composition device according to Embodiment 3 of the present invention.
10 is a diagram for explaining a method of synthesizing image data based on a three-dimensional model in a background image by the image data conversion means 27 of FIG. 9; FIG.
FIG. 11 is a block diagram showing a configuration of an image composition device according to Embodiment 4 of the present invention;
12 illustrates a method of extracting a composite area in the preceding and succeeding frames by a composite area extraction unit 32 from an area of an additional image arranged in one frame in the video input by the video input unit 31 of FIG. It is a figure for doing.
13 is a diagram for explaining a method of tracking a boundary line of an object shown in a moving image frame by the synthetic area extracting unit 32 of FIG.
14 is a diagram for explaining a method of combining an additional image with each frame by the additional image propagation means 33 of FIG.
FIG. 15 is a block diagram showing a configuration of an image composition device according to Embodiment 5 of the present invention;
16 is a diagram for explaining a method of generating an additional image to be combined with a frame between a start frame and an end frame by the additional image complementing unit 48 of FIG.
FIG. 17 is a block diagram showing a configuration of an image composition device according to Embodiment 6 of the present invention;
18 is a diagram for explaining a moving path of a subject defined by the additional image motion definition unit 54 of FIG.
FIG. 19 is a block diagram showing a configuration of an image composition device according to Embodiment 7 of the present invention;
20 is a diagram for explaining a method of performing background image region division by the background image region dividing unit 56 of FIG. 19; FIG.
FIG. 21 is a diagram for explaining a background image and an additional image synthesized based on the order determined by the context determination unit 57 of FIG. 19;
FIG. 22 is a block diagram showing a configuration of an image composition device according to Embodiment 8 of the present invention.
FIG. 23 is a diagram for explaining the function of the background image region positional relationship extracting means 61 in FIG. 22;
FIG. 24 is a block diagram showing a configuration of an image composition device according to Embodiment 9 of the present invention.
25 is a diagram for explaining the function of a region boundary association unit 67 of FIG. 24. FIG.
FIG. 26 is a block diagram showing a configuration of an image composition device according to Embodiment 10 of the present invention.
FIG. 27 is a block diagram for explaining an image composition display device disclosed in Japanese Patent Laid-Open No. 5-11727;
[Explanation of symbols]
1 background image input means, 2 additional image input means, 3 additional image arrangement means,
4 background image three-dimensional information input means, 5 additional image relative position defining means, 6 additional image shape correcting means, 7 image composition means, 17 background image range specifying means, 18 background image cutting means, 26 3D model input means, 27 Image data conversion means, 31 moving image input means, 32 composite area extraction means, 33 additional image propagation means, 44 additional image first relative position defining means, 45 background image three-dimensional information first input means, 46 additional image second Relative position defining means, 47 background image three-dimensional information second input means, 48 additional image complementing means, 54 additional image motion defining means, 56 background image area dividing means, 57 context determining means, 61 background image area positional relation extracting means 62 Map information storage means 67 Area boundary association means 70 Three-dimensional CG model input means 71 Three-dimensional CG model The relative position defining means, 72 3-dimensional CG model shape correction means.

Claims (2)

Background image input means for inputting a background image;
An additional image input means for inputting an additional image;
Background image three-dimensional information input means for inputting three-dimensional information of the background image, additional image relative position defining means for defining the relative position of the additional image on the background image, and this additional image relative position defining means Additional image arrangement means having additional image shape correction means for correcting the shape of the additional image before synthesizing the additional image on the background image based on the relative position of the additional image defined by
Background image area dividing means for dividing the background image into image areas according to the three-dimensional information input by the background image three-dimensional information input means;
A context determining means for determining a context between each image area divided by the background image area dividing means and the additional image;
An image for combining the additional image whose shape has been corrected by the additional image shape correcting unit with a background image and for combining the image region determined to be in front of the additional image by the context determination unit from above the additional image With synthesis means
By synthesizing another image while considering the context with the subject in the background image,
A background for taking out the positional relationship between the map information storage means storing the map information and the image area divided by the background image area dividing means with reference to the map information around the background image from the map information storage means By further comprising image region positional relationship extraction means, the anteroposterior relationship of the subject in the background image is reflected in the composite image ,
The background image area positional relationship extracting means obtains the center of gravity of each image area divided in the background image, and associates the viewpoint position of the camera, the background image, and the center of gravity of the image area on the map information, By drawing a straight line on the map information from each viewpoint position to each centroid point in the field of view of the camera, the distance between the subject ahead of the centroid point and the viewpoint position of the camera is obtained. An image composition apparatus characterized by being associated with the Background image input means for inputting a background image;
An additional image input means for inputting an additional image;
Background image three-dimensional information input means for inputting three-dimensional information of the background image, additional image relative position defining means for defining the relative position of the additional image on the background image, and this additional image relative position defining means Additional image arrangement means having additional image shape correction means for correcting the shape of the additional image before synthesizing the additional image on the background image based on the relative position of the additional image defined by
Background image area dividing means for dividing the background image into image areas according to the three-dimensional information input by the background image three-dimensional information input means;
A context determining means for determining a context between each image area divided by the background image area dividing means and the additional image;
An image for combining the additional image whose shape has been corrected by the additional image shape correcting unit with a background image and for combining the image region determined to be in front of the additional image by the context determination unit from above the additional image With synthesis means
By synthesizing another image while considering the context with the subject in the background image,
When the background image is divided into several image areas by the map information storage means storing the map information and the background image area dividing means, the boundary line of the subject in the background image is displayed as the background image from the map information storage means. By further comprising an area boundary association means for extracting by referring to the surrounding map information, the image area for each subject in the background image is divided ,
The region boundary associating means associates a background image and a camera viewpoint on the map information, draws a straight line from the camera viewpoint to a boundary portion of the subject reflected in the background image, and intersects the straight line and the background image. Is a boundary line of the subject in the background image .

Images