JPH10336703A - Image compositing device, its method and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image compositing device by which a computer graphics CG image is simply composited on a stereoscopic video image so as to obtain the composite stereoscopic video image without a sense of incongruity in the stereoscopic sense. SOLUTION: In the case that a left and right composites image resulting from overwriting a CG image onto an object image is composited with a horizontal stripe image and the resulting image is displayed on a 3-dimension display device 130 stereoscopically, since the composite CG image is generated by using a camera parameter at object image photographing through perspective projection, a stereoscopic sensor of the composite image displayed stereoscopically does not give any sense of incongruity with respect to the stereoscopic sensor of the object. When a photographer instructs rotation, movement, magnification to the CG image displayed on the 3-dimension display device 130, the CG image based on an operation parameter changed by the instructed data is generated by a CG image generating section 105 compositing with the object image that is photographed and then the result is displayed stereoscopically on the 3-dimension display device 130 again.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image synthesizing apparatus, method and storage medium for synthesizing a computer graphics (CG) image with a photographed subject image when photographing and displaying a three-dimensional subject.

[0002]

2. Description of the Related Art Conventionally, there has been known a system which provides a stereoscopic image by obtaining a set of images having parallax from a plurality of cameras and observing each image with separate left and right eyes of an observer. .

On the other hand, there is known an image synthesizing apparatus for synthesizing an image (CG image) created by a computer with an image photographed by a camera.

[0004]

However, in a conventional system for photographing and displaying a stereoscopic image, in order to combine a CG image with a stereoscopic image, a CG image is synthesized with each of the photographed images and converted into a stereoscopic image again. Must be displayed, the size and orientation of the CG image in the image to be synthesized,
The position has to be adjusted for each of the plurality of images, and the operation of synthesizing the images is complicated.

In addition, in order to match the stereoscopic effect of the photographed image with the stereoscopic effect of the CG image, if the stereoscopic image after the synthesis is observed and the stereoscopic effect is unnatural, it is necessary to start again from the synthesis of the CG image. However, the work is difficult, and if they do not match, when the synthesized image is observed as a stereoscopic image, the synthesized image has a strange feeling.

[0006] Accordingly, the present invention provides a simple CG image for stereoscopic images.
It is a first object of the present invention to provide an image synthesizing apparatus, method, and storage medium capable of synthesizing images and obtaining a synthesized stereoscopic video without a sense of strangeness in stereoscopic effect.

In addition, the present invention provides, in addition to the first object, an image synthesizing apparatus, a method and a storage medium capable of easily realizing shooting of a stereoscopic video and synthesizing with a CG image. The purpose of.

Further, according to the present invention, in addition to the first and second objects, even when the subject of the photographed image and the CG model overlap with each other, it is possible to obtain a three-dimensional video image that does not make the concealment relationship of the synthesized three-dimensional image strange. A third object is to provide an image synthesizing apparatus, a method, and a storage medium.

[0009]

In order to achieve the above object, an image synthesizing apparatus according to the first aspect of the present invention displays a plurality of subject images, which are photographed from different viewpoints and have overlapping visual fields, in a stereoscopic view. An image synthesizing apparatus provided with a display unit capable of performing the following operations: a model information storage unit for storing model information representing a three-dimensional structure; operation parameters including a direction and a position of the stored model information; Calculating means for calculating the rotation and movement of the model information using at least the focal length of the camera at the time of shooting and camera parameters including the distance between viewpoints, and an image of the model information for which the rotation and movement have been calculated. Image generating means for generating a plurality of images by perspective projection, and image synthesizing means for synthesizing the generated plurality of model information images and the plurality of subject images; And displaying stereoscopic by said display means an image.

According to a second aspect of the present invention, there is provided an image synthesizing apparatus comprising: a plurality of imaging systems; and a display unit capable of stereoscopically displaying a plurality of subject images photographed by the plurality of imaging systems. In the apparatus, model information storage means for storing model information representing a three-dimensional structure, operation parameters including directions and positions of the stored model information, focal lengths of the plurality of imaging systems, and distances between viewpoints. Calculating means for calculating rotation and movement of the model information using camera parameters; image generation means for generating a plurality of images of the model information for which rotation and movement are calculated by perspective projection; Image combining means for combining the image of the model information and the plurality of subject images, and the combined image is stereoscopically displayed by the display means.

According to a third aspect of the present invention, in the image synthesizing apparatus according to the first or second aspect, the image synthesizing apparatus further comprises depth extracting means for extracting a depth of the plurality of subject images based on a parallax of the plurality of subject images. The image combining means combines the image of the generated model information and the subject image based on the depth of the extracted subject image.

According to a fourth aspect of the present invention, there is provided an image combining method for combining a plurality of subject images having overlapping fields of view taken from different viewpoints with a model image representing three-dimensional information created by a computer. A step of storing model information representing a three-dimensional structure, operation parameters including the orientation and position of the stored model information, and focal lengths of the plurality of imaging systems, and camera parameters including a distance between viewpoints. Using, the step of calculating the rotation and movement of the model information, the step of generating a plurality of images of the rotation and movement of the calculated model information by perspective projection, and the images of the generated plurality of model information and It is characterized by comprising a step of combining a plurality of subject images, and a step of displaying the combined image in a stereoscopic view.

According to a fifth aspect of the present invention, there is provided an image synthesizing method according to the fourth aspect, further comprising a step of extracting a depth of the subject image based on a parallax of the plurality of subject images. Combining the generated model information image and the subject image based on the depth of the extracted subject image.

In the storage medium according to the present invention, when combining a plurality of subject images taken from different viewpoints and having overlapping visual fields with a model image representing three-dimensional information created by a computer, A parameter calculation module that calculates rotation and movement of the model information using operation parameters including orientation and position, and camera parameters including the focal length of the plurality of imaging systems and the distance between viewpoints; A computer that includes an image generation module that generates a plurality of images of the calculated model information by perspective projection, and an image synthesis module that synthesizes the generated plurality of model information images and the plurality of subject images. A program for storing the program.

[0015] The storage medium according to claim 7 includes:
A depth extracting module for extracting a depth of the subject image based on a parallax of the plurality of subject images in the storage medium according to the first aspect of the present invention. The program for synthesizing the image of the model information and the subject image is stored.

[0016]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the image synthesizing apparatus, method and storage medium of the present invention will be described.

[First Embodiment] FIG. 1 is a block diagram showing a configuration of an image synthesizing apparatus according to a first embodiment. In the figure, reference numeral 200 denotes a stereo camera, which outputs left and right image data of two systems. Reference numeral 101 denotes a camera parameter storage unit that stores camera parameters of left and right image data captured by the stereo camera 200.

Reference numeral 110 denotes a depth extraction unit which extracts and outputs the depth distribution of each of the left and right image data output from the stereo camera 200. Reference numeral 102 denotes a CG operation unit for instructing a direction and a position of a CG image to be synthesized via a pointing device.

Reference numeral 103 denotes a CG parameter calculation unit.
A parameter for generating a CG image is calculated according to the data of the G operation unit 102. Reference numeral 104 denotes a model storage unit including a removable memory card or the like, which stores CG model data created in advance by an external computer.

Reference numeral 105 denotes a CG image generation unit which generates left and right image data of the CG model stored in the model storage unit 104 in accordance with the output of the CG parameter calculation unit 103.

Reference numerals 120 and 121 denote image synthesizing units, which synthesize subject image data and CG image data on the left and right, respectively, and output them. Reference numeral 106 denotes a stereo image synthesizing unit which generates stereo image data using the synthesized image data output from the two image synthesizing units 120 and 121,
07.

Reference numeral 108 denotes a display control unit, which is a display memory 1
The stereoscopic image data stored in 07 is controlled by the three-dimensional display device 130 to be displayed as a three-dimensional image. 1
An image recording unit 40 records stereo image data.

FIG. 2 is a block diagram showing the configuration of the stereo camera 200. Reference numerals 201 and 202 denote photographing lenses, which photograph stereo images from two viewpoints. 203,
Reference numeral 204 denotes an image sensor such as a CCD that captures an image as an electric signal. In these two imaging systems, the optical axes of the imaging lenses 201 and 202 are the same as those of the image sensor 203,
It is arranged so as to be parallel to the horizontal direction of 204.

Reference numerals 205 and 206 denote image sensors 203,
An image capturing control circuit 204 controls the image capturing. Reference numerals 207 and 208 denote image signal processing units which generate an image signal by holding the electric signals from the image sensors 203 and 204, automatically control the gain of the image signal, perform gradation correction, and output the image signal. . 209 and 210 are A /
A D converter converts the image signals output from the image signal processing units 207 and 208 from analog to digital, and outputs digital image data.

Reference numerals 211 and 212 denote color signal processing circuits.
Digital image data (hereinafter simply referred to as image data) output from the A / D converters 209 and 210 is
One screen is output with B24 bits as one pixel. Note that the photographing lenses 201 and 202 and the image sensor 203,
Components such as 204 have the same characteristics on the left and right.

FIG. 3 is an exploded perspective view showing the configuration of the three-dimensional display device 130. The three-dimensional display device 130 has been previously proposed by the present applicant (Japanese Patent Application No. 8-148611).
No.) and the following configuration. That is, in the drawing, reference numeral 131 denotes a liquid crystal display element that displays an image stored in the display memory 107. 132 is a backlight, 13
3 is a slit, and 134 and 135 are lenticular lenses.

The light from the backlight 132 is transmitted through the slit 1
33, the liquid crystal display element 131 is illuminated through the lenticular lenses 134 and 135. In the liquid crystal display element 131, the observer can observe the left image and the right image indicated by L and R in the drawing separately from the left eye and the right eye, respectively, in an appropriate observation range. Thus, the image displayed on the liquid crystal display element 131 can be recognized as a stereoscopic image.

Next, the operation of the image synthesizing apparatus having the above configuration will be described. The image synthesizing apparatus according to the present embodiment includes three modes, that is, a normal mode in which only a subject image that is being shot without being combined with a CG image is displayed on the three-dimensional display device 130 and recorded;
It has a combination mode in which the combined image obtained by combining the images is displayed on the three-dimensional display device 130 and the photographer can operate the CG image, and a recording mode in which the combined image in which the subject image and the CG image are combined is recorded.

First, the normal mode will be described. When a photographer turns on a power switch of a camera (not shown), the image capturing control circuit 205
The image sensor 203 controls so as to capture an image of the subject formed from the right viewpoint as an electric signal, and outputs an image signal processing unit 207, an A / D converter 209, and a color signal processing circuit 2.
Via 11, right image data is obtained. Similarly, the image capture control circuit 206 passes through the photographing lens 202,
An image from the left viewpoint of the subject formed on the image sensor 204 is controlled to be taken in as an electric signal, and the image signal processing unit 208, the A / D converter 210, and the color signal processing circuit 2 are controlled.
Via 12, left image data is obtained.

The two image data obtained at this time are output from the stereo camera 200 as image data captured at the same timing by a synchronization signal output from a synchronization signal generation circuit (not shown).

The image data output from the stereo camera 200 is sent to the image synthesizing units 120 and 121. In the normal mode, the image combining units 120 and 121 output the input image data to the stereo image combining unit 106 as it is.
The stereo image combining unit 106 combines left and right image data to generate a stereo image to be displayed on the three-dimensional display device 130, and outputs the generated stereo image.

FIG. 4 is a diagram showing the processing of the stereo image synthesizing unit 106. FIGS. 7A and 7B show left image data and right image data output from the stereo camera 200, respectively. FIG. 3C shows a stereo image synthesized by the stereo image synthesis unit 106. Stereo image synthesis unit 10
In 6, the left and right image data are combined for each horizontal line as shown in FIG. 9C, thereby generating a stereo image (also referred to as a horizontal stripe image) to be displayed on the three-dimensional display device 130. When the horizontal stripe image is written into the display memory 107, the display control unit 108 stereoscopically displays the subject image being captured by the stereo camera 200 on the three-dimensional display device 130.

The photographer performs framing while viewing the stereoscopic image displayed on the three-dimensional display device 130,
Press the shutter button (not shown). When the shutter button is pressed, the horizontal stripe image output from the stereo image synthesizing unit 106 is written not in the display memory 107 but in the image recording unit 140. Then, the image recording unit 14
0 is written to the display memory 107, and the display control unit 108 stores the data in the image recording unit 1 in the three-dimensional display device 130.
The image data recorded in 40 is stereoscopically displayed.

Next, the combining mode will be described. When the photographer presses a synthesis button (not shown), the CG image generation unit 105 confirms that the model storage unit 104 is in the mounted state, reads the CG model data stored in the model storage unit 104, and sets the initial CG parameters. To generate left and right image data of the CG model. This algorithm will be described.

First, the CG operation unit 102 outputs operation parameters to the CG parameter calculation unit 103. Here, the operation parameter indicates a rotation parameter indicating the direction of the CG model (a value indicating the amount of rotation around each of the three coordinate axes of the rectangular coordinate system) and the position of the center of the CG model in the left and right images. It consists of a position parameter (a value representing a position in the horizontal and vertical directions) and a scaling parameter for controlling the size of the CG model in the image.

The two image pickup systems of the stereo camera 200 are arranged so that the optical axes of the respective image pickup lenses 201 and 202 are parallel to the horizontal direction of the image sensor. Among them, the position parameter in the vertical direction has the same value on the left and right.

At the time of the initial operation, the CG parameter calculation unit 103 generates the initial operation parameters instead of the operation parameters output from the CG operation unit 102. In the initial operation parameters, the rotation parameter indicating the direction of the CG model is generated in the direction indicating the front of the CG model. Further, the position parameter indicating the position of the center of the CG model in the left and right images is such that the size of the CG image in the left and right images is a predetermined area (for example, １ ／ of the entire image area),
The position of the CG image in the left and right images has a predetermined parallax and is generated at a position symmetric with respect to the image center.

Then, instead of using the scaling parameters of the operation parameters, the magnification parameter indicating the size of the CG model is replaced by the camera parameters stored in the camera parameter storage unit 101 and the left and right images in the size and position parameters of the CG model. And is set based on the parallax. At this time, the size of the CG model is stored in the model storage unit 1.
04 is calculated from the three-dimensional structure of the CG model stored in the CG model.

The CG parameter calculation section 103 calculates CG parameters from the operation parameters output from the CG operation section 102 and the camera parameters stored in the camera parameter storage section 101.

Here, the camera parameters include a distance between the lens centers (viewpoints) of the two image pickup systems (hereinafter, referred to as a base line length) and a focal length of the image pickup system (image pickup from the viewpoint along the optical axis of the image pickup system). (A distance to a plane) and pixel interval data of image data.

The CG parameter is the CG model 3
It is composed of a dimensional rotation matrix, a position parameter indicating the spatial position of the CG model, and a magnification parameter indicating a ratio between the size of the CG model in the subject image space and the size of the CG model stored in the model storage unit 104.

Here, the position parameter indicating the spatial position of the CG model represents the relative positional relationship between the lens centers of the left and right imaging systems of the stereo camera 200 and the CG model, and is composed of two left and right position parameters. Also,
The rotation matrix is calculated from the rotation parameters of the operation parameters. Further, the left and right position parameters are calculated from the position parameters of the operation parameters and the difference between the positions in the horizontal direction (that is, the parameter indicating parallax) and the camera parameters. The magnification parameter is set in advance by the CG parameter calculation unit 103.
Is obtained by multiplying the magnification parameter of the variable by the magnification parameter.

The CG image generation unit 105 stores the data in the model storage unit 1
The CG model data read from the G.04 is converted into the size in the subject space according to a magnification parameter indicating the size of the CG model, and the rotation and movement of the CG model are converted according to the rotation matrix and the position parameter of the CG parameter, respectively. After that, left and right CG image data are generated by performing perspective projection using the data of the focal length and the pixel interval of the camera parameters.

FIG. 5 is a diagram showing a state of perspective projection.
L and R are viewpoints of the left and right imaging systems separated by the base line length, respectively, and IL and IR are left and right CG images with respect to the CG model. Then, the two CG image data are output to the image synthesizing units 120 and 121, respectively. Image synthesis unit 1
Reference numerals 20 and 121 respectively combine CG image data with subject image data being captured by the stereo camera 200.

FIG. 6 is a diagram showing a state in which CG image data is combined with the subject images of FIGS. 4 (A) and 4 (B). In the combining mode, the CG image is overwritten on the subject image and combined (see FIGS. 6A and 6B). Then, the combined images output from the image combining units 120 and 121 are combined with a horizontal stripe image by the same processing as in the normal mode, and are three-dimensionally displayed on the three-dimensional display device 130. At this time,
Since the synthesized CG image data is generated by perspective projection using the camera parameters at the time of capturing the subject image as described above, the three-dimensional effect of the three-dimensionally displayed composite image is uncomfortable with the three-dimensional effect of the subject. There is no.

The photographer performs a 3D operation via the interface of the CG operation unit 102 while viewing the stereoscopic image displayed on the three-dimensional display device 130. That is, the photographer can operate the three-dimensional display device 130 using the pointing device.
, An instruction to rotate, move, or change the magnification of the CG image displayed on the screen is output from the CG operation unit 102 by an amount corresponding to the instructed data.

Then, a CG image based on the changed operation parameters is generated by the CG image generation unit 105 and combined with the subject image data being photographed by the stereo camera 200, and then displayed again on the three-dimensional display device 130. Is done.

By such an operation, a stereo image obtained by synthesizing the subject image being captured and the CG image in the synthesizing mode is stereoscopically displayed on the three-dimensional display device 130, and the photographer can interactively display the CG model using the CG operation unit 102. Can be operated.

Next, the recording mode will be described. The recording mode operates when the shutter button is pressed during the combination mode. When the shutter button is pressed, image data output from the stereo camera 200 is stored in an image memory in the depth extracting unit 110.

FIG. 7 is a block diagram showing the configuration of the depth extracting unit 110. The depth extracting unit 110 is used for the image memory 11
1, 112, correspondence extraction unit 113, region division unit 114, 1
15 and a depth calculation unit 116.

In the depth extracting unit 110, the image memory 11
1 and the right image data and the left image data stored in the image memory 112, respectively.
Divides the reference right image into rectangular small areas, and searches the corresponding area of the left image for each area.

FIG. 8 is a diagram showing a state of searching for a region of the left image corresponding to each region of the right image. FIG. 7B shows the right image, which is divided into small areas indicated by broken lines. At this time, the search range for each area is estimated in advance from the camera parameters stored in the camera parameter storage unit 101. For example, for a region indicated by a thick solid line in the right image shown in FIG. 7A, a rectangular region in the left image shown in FIG. The region is extracted as a corresponding region. Then, the extraction result corresponding to each area is output as parallax data.

On the other hand, the area dividing sections 114 and 115 extract an edge portion from the image data, and use the extracted edge portion as a boundary to divide an image for each subject area based on the similarity of the image data. FIG. 9 is a diagram showing a result of region division performed on the left image shown in FIG. The left image is A,
It is divided into four areas B, C and D.

The depth calculation unit 116 includes a region division unit 114,
An average parallax is calculated based on the parallax data output from the correspondence extraction unit 113 for each of the divided subject areas output from 115, and the depth is calculated using camera parameters (from the camera center of the imaging system to the optical axis). ), And outputs a left and right depth map. These depth maps are obtained by assigning calculated depths to image data divided into subject areas. For example, a long distance is assigned to the subject areas A, C, and D shown in FIG. 9, and a short distance is assigned to the subject area B as the depth.
Thus, the depth extracting unit 110 includes the image synthesizing unit 120,
The image data and the depth map are respectively output to 121.

The CG image generation unit 105 generates two left and right CG image data based on the CG model, CG parameters, and camera parameters used in generating the CG image data, and the two CG images are equivalent to the depth map output from the depth extraction unit 110. A depth map is created for each of the CG image data, and output to the image synthesis units 120 and 121. The depth map for these CG images takes an infinite value in an area other than the CG model.

The image synthesizing units 120 and 121 temporarily store the image memories 111 and 112 of the depth extracting unit 110, respectively.
The CG image data output from the CG image generation unit 105 and the image data stored in the CG image generation unit 105 are combined with the depth map for the subject image output from the depth extraction unit 110 and the depth map for the CG image output from the CG image generation unit 105. Then, an image having a close depth is selected in a region in the CG model, synthesized, and output.

FIG. 10 shows the left image shown in FIG.
It is a figure showing the picture which combined the picture. In this embodiment,
The CG model is located at the intermediate distance, and the concealment relationship has been corrected for the synthesized image in the synthesis mode shown in FIG.

Then, by the same processing as in the normal mode,
The horizontal stripe image obtained by synthesizing the subject image captured by the stereo camera 200 and the CG image is written to the image recording unit 140, and is displayed three-dimensionally on the three-dimensional display device 130.

By such an operation, the photographer can easily synthesize a CG image with a stereoscopic video. And C
Since the camera parameters at the time of photographing with the camera are used as the CG parameters for generating the G image, it is possible to synthesize a three-dimensional image without a sense of incongruity. Also, by using the depth in the captured stereo image, a synthetic image having a natural concealment relationship can be obtained.

In this embodiment, the CG image is superimposed on the subject image in the synthesizing mode and synthesized and displayed, and in the recording mode, the depth is used and the concealing relationship is taken into account, so that the photographer is always synthesized. The user can view and operate the entire CG image. In addition, in the synthesis mode, using the depth,
The composition may be performed in consideration of the concealment relationship. At this time, when the photographer performs a 3D operation on the CG model, a synthetic image having a natural concealment relationship can be interactively viewed on the three-dimensional display device.

Further, as in the present embodiment, the depth extraction may be performed only in the CG model area without performing the depth extraction on the entire screen. Since the corresponding extraction region between the left and right subject images can be limited, this processing can be reduced.

Further, in the present embodiment, when the stereo camera has an imaging system in which the focus position is variable, the CG parameter is calculated using a camera parameter whose focal length corresponds to the focus position. , CG
Each processing of image generation and depth extraction is performed. At this time, the camera parameters are output from the stereo camera instead of being stored in the camera parameter storage unit.

In this embodiment, the case where the optical axes of the respective imaging lenses of the two imaging systems of the stereo camera are arranged in the left and right directions has been described. However, the optical axes intersect at a predetermined distance. A configuration with convergence may be used, or a configuration that can change the convergence angle may be used.

When the convergence angle can be changed, a region where the left and right images overlap can be secured in a wider distance range, and stereoscopic viewing becomes possible. When a configuration is adopted in which convergence is added to the optical axis, the value of the convergence angle is stored in the camera parameter storage unit, and the parameters of the convergence angle are used in each of the CG parameter calculation, CG image generation, and depth extraction processing.

When the convergence angle can be changed, the value of the convergence angle at the time of photographing is output as a camera parameter from the stereo camera. As these camera parameters, an input unit such as a keyboard may be provided externally and set by inputting values measured in advance. Alternatively, a known correction chart of a three-dimensional structure is photographed in advance with a stereo camera, and the focal length is calculated from the position of the feature point in the image, and the left and right of the base line length, the convergence angle, etc. Calculating parameters related to the positional relationship between the imaging systems, calculating CG parameters,
It may be used for each processing of CG image generation and depth extraction.

Further, in this embodiment, when the focal length and the aperture value of the imaging lens of the stereo camera can be changed, the amount of blur of the subject image changes according to the depth. In particular, when the focal length is long and the aperture diameter is large. When the size is large, the change in the amount of blur becomes remarkable. Therefore, when generating a CG model, a CG image that is blurred according to the focal length and the aperture value of the imaging lens is generated and combined with the subject image, so that A natural composite image can be obtained. In this case, the focal length,
The aperture value is output from the stereo camera as a camera parameter, and a CG image having a blur amount corresponding to the camera parameter is generated in the CG image generation processing.

The brightness and color of the subject image captured by the stereo camera vary depending on the aperture value of the imaging system and the processing parameters of the image signal processing unit and the color signal processing circuit.
When generating a model, the brightness and color of the generated CG image are adjusted in accordance with the aperture value of the imaging system and the processing parameters of the image signal processing unit and the color signal processing circuit, and are synthesized with the subject image. A natural composite image can be obtained. In this case, the aperture value of the imaging system, the processing parameters of the image signal processing unit and the color signal processing circuit are output as camera parameters from the stereo camera, and the brightness and the brightness of the CG image generated according to the camera parameters in the CG image generation processing are determined. Try to adjust the color.

[Second Embodiment] FIG. 11 is a block diagram showing a configuration of an image synthesizing apparatus according to a second embodiment. In the drawings, those having the same symbols as those in FIGS. 1 and 7 have substantially the same functions.

Reference numeral 300 denotes a recording medium, which is photographed by a camera,
The recorded right image data and left image data at the left and right viewpoints, parallax data obtained as a result of corresponding extraction of the left and right images, and camera parameters at the time of shooting are recorded.

The operation of the image synthesizing apparatus will be described. First, C
The G operation unit 102 outputs operation parameters to the CG parameter calculation unit 103. As in the image synthesizing apparatus of the first embodiment, the CG parameter calculation unit 103 stores the operation parameters output from the CG operation unit 102 and the recording medium 30.
The CG parameter is calculated from the camera parameter stored in 0. The CG parameter calculation unit 103 internally generates an initial operation parameter at the time of an initial operation to calculate a CG parameter.

The CG image generation unit 105 stores the data in the model storage unit 1
The data of the CG model read from the C.G.04 is converted into the size in the subject space according to the magnification parameter according to the magnification parameter, and the rotation of the CG model and the rotation matrix and position parameter of the CG parameter are respectively performed.
After the conversion of the movement, the perspective projection is performed using the data of the focal length and the pixel interval of the camera parameter to obtain left and right CG image data. Then, the two left and right CG image data are respectively combined with the image combining units 120 and 121.
Output to The image synthesizing units 120 and 121 synthesize CG image data with the left and right image data stored in the recording medium 300, respectively.

The stereo image synthesizing unit 106 combines the left and right image data for each horizontal line to generate a stereo image to be displayed on the three-dimensional display device 130. Then, the horizontal stripe image is written into the display memory 107, and the display control unit 108 controls the three-dimensional display device 13.
At 0, the subject image being photographed by the stereo camera 200 is stereoscopically displayed.

By such an operation, a stereo image obtained by synthesizing the left and right stereo images and the CG image recorded on the recording medium 300 is stereoscopically displayed on the three-dimensional display device 130, and the user interactively performs the CG operation by the CG operation unit 102.
Can manipulate the model.

When the orientation, position, and size of the CG model are determined, the user presses a combining recording button (not shown), and the subject and the CG in the image are displayed in the same manner as in the image combining apparatus of the first embodiment. The concealment relationship of the model is modified using the depth. Next, this operation will be described.

First, the region dividing units 114 and 115 extract edge portions from the left and right image data recorded on the recording medium 300, and use the extracted edge portions as boundaries, based on the similarity of the image data, for each subject region. Divide the image into

The depth calculator 116 calculates the average parallax for each of the divided subject areas output from the area dividers 114 and 115 based on the parallax data recorded on the recording medium 300. The depth is converted into a depth using the camera parameters, and a left and right depth map is output.

Further, the CG image generating unit 105
Along with the G image data, a depth map for each CG image is created and output to the image synthesis units 120 and 121. The image synthesizing units 120 and 121 are respectively provided on the recording medium 3
00 and the CG image data output from the CG image generation unit 105 with the depth map for the subject image output from the depth calculation unit 116 and the CG image output from the CG image generation unit 105 The comparison is performed using the depth map, and in a region in the CG model, an image having a close depth is selected, synthesized, and output.

Then, a horizontal stripe image is synthesized from the synthesized images output from the image synthesizing units 120 and 121 by the stereo image synthesizing unit 106 and written into the image recording unit 140, and is then stereoscopically displayed on the three-dimensional display device 130. .

In this embodiment, the horizontal stripe image obtained by synthesizing the CG model with the pre-recorded left and right stereo images is recorded as a still image, and is displayed three-dimensionally on the three-dimensional display device. It is also possible to generate an animation image of the left and right stereo images and the CG model by generating and combining images from a plurality of directions and positions of the CG model of the size determined by the user.

In this case, the image synthesizing apparatus operates the CG operation unit 1
Among the CG parameters of the plurality of CG models designated by 02, a CG parameter storage unit that stores a plurality of CG parameters according to directions and positions, and interpolates a trajectory of the CG model from the plurality of stored CG parameters. This is realized by providing a trajectory calculation unit for outputting, and causing the CG image generation unit to generate a CG image according to each CG parameter of the trajectory calculation unit.

In the above embodiment, the case where one CG model is stored in the model storage unit has been described, but a plurality of CG models may be stored. In this case, for example, a plurality of models may be displayed in a list in the peripheral area of the display unit of the three-dimensional display device, and the photographer may select the model using a pointing device or the like. Further, the number of CG models that can be combined with the subject image is not limited to one, but may be plural.

Further, in the above embodiment, the horizontal stripe image is recorded in the image recording section. However, output images of a plurality of image synthesizing sections before stereo synthesis may be recorded.

Further, in the above embodiment, the stereoscopic image is displayed by the three-dimensional display device shown in FIG. 3. However, the left and right images are alternately displayed temporally, and the liquid crystal shutter glasses synchronized with the display are displayed. The stereoscopic image may be displayed on a three-dimensional display device that allows the left and right images to be separated and observed by the left eye and the right eye, respectively, by observing the image. In this case, the output images of the two image synthesizing units are written into the two display memories, and the data in the display memories are temporally switched and displayed by the display control unit.

Although the above embodiment has been described on the assumption that the processing unit is configured by hardware, the computer may execute a program by executing a program in which each processing unit is replaced by a processing module of software. Next, a case where an image synthesizing apparatus similar to that of the first embodiment is realized by a processing module of software will be described as a third embodiment.

[Third Embodiment] FIG. 12 is a block diagram showing a configuration of an image synthesizing apparatus according to a third embodiment. In the image synthesizing apparatus according to the third embodiment, the functions of the image synthesizing units 120 and 121, the CG image generating unit 105, the CG parameter calculating unit 103, the stereo image synthesizing unit 106, and the depth extracting unit 110 according to the first embodiment are described. Instead of these hardware, CPU 301, ROM 302,
A RAM 303 and an I / O 304 are added. The description of the same components as those in the first embodiment will be omitted by retaining the same reference numerals.

FIG. 13 is a diagram showing a memory map of the ROM 302 in which the image composition processing program is stored. RO
M302 stores a plurality of processing modules constituting an image synthesis processing program. The plurality of processing modules include a CG image generation module, a CG parameter calculation module, an image synthesis module, a stereo image synthesis module, and a depth extraction module. The depth extraction module is a correspondence extraction module, a region division module, and a depth calculation module. It is composed of

In the image synthesizing apparatus of the third embodiment, each operation of the normal mode, the synthesizing mode and the recording mode is performed similarly to the image synthesizing apparatus of the first embodiment. Only the mode operation will be described.

FIG. 14 is a flowchart showing the processing procedure of the combining mode in the image combining processing program executed by the CPU 301. This process is started when the photographer presses a combining button (not shown). First, the CPU 301 confirms that the modem storage unit 104 is in the mounted state, reads the CG model data stored in the model storage unit 104, and generates left and right image data of the CG model according to the initial CG parameters ( Step S1, CG image generation module).

Operation parameters are input from the CC operation unit 102 (step S2). CG parameters are calculated from the operation parameters input from the CG operation unit 102 and the camera parameters stored in the camera parameter storage unit 101 (step S3, CG parameter calculation module).

For the data of the CG model read from the model storage unit 104, the size of the CG model is converted into the size in the subject space according to the magnification parameter.
After performing rotation and movement conversion of the CG model according to the rotation matrix and the position parameters of the CG parameters, respectively, the perspective projection is performed using the focal length and pixel interval data of the camera parameters to generate left and right CG image data. (Step S4, CG image generation module).

The generated CG image data is combined with the subject image data being captured by the stereo camera 200 (step S5, image combining module).

Using left and right combined image data obtained by combining CG image data with subject image data, generate stereo image data (horizontal stripe image data), and output the generated horizontal stripe image data to the display memory 107. Thus, a three-dimensional display is performed on the three-dimensional display device 130 (step S6, stereo image synthesis module).

Waiting for the photographer to operate the CG operation unit 102 (step S7), the operation returns to step S2.

Next, the operation in the recording mode will be described. FIG. 15 is a flowchart illustrating a processing procedure of a recording mode in the image synthesis processing program executed by the CPU 301. This process is started when a shutter button (not shown) is pressed during the combination mode.

First, the CPU 301 controls the stereo camera 2
00 is stored in the RAM 303.
From the right image data and the left image data stored in 303, the reference right image is divided into rectangular small regions, a corresponding region of the left image is searched for each region, and a corresponding extraction result in each region is obtained. Obtained as parallax data (step S1
1, corresponding extraction module).

An edge part is extracted from the image data, and the image is divided for each subject area based on the similarity of the image data with the extracted edge part as a boundary (step S12, area dividing module).

An average parallax is calculated for each of the divided subject areas based on the parallax data in step S11.
Using the camera parameters, the depth is converted into the depth (the distance from the camera center of the imaging system to the subject along the optical axis) to obtain a left and right depth map (step S13, depth calculation module).

From the left and right CG image data, a depth map for each of two CG images equivalent to the depth map obtained in step S13 is created from the CG model, CG parameters, and camera parameters used in generating the CG image data ( Step S14, CG image generation module).

The image data stored in the RAM 303 and the CG image data created in step S14 are compared using a depth map for the subject image and a depth map for the CG image. Are selected and synthesized (step S15, image synthesis module).

The horizontal stripe image data is synthesized using the left and right synthesized images obtained by synthesizing the subject image captured by the stereo camera 200 and the CG image, and is recorded in the image recording unit 140. Stereoscopic display is performed (step S16, stereo image synthesis module).

As described above, in the image synthesizing apparatus for synthesizing images using the software processing module, the same effects as those of the first embodiment can be obtained, and the function of each processing module can be easily changed or added. Can do
The versatility of the image synthesizing device can be improved.

The present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of one device. Needless to say, the present invention can be applied to a case where the present invention is achieved by supplying a program to a system or an apparatus. In this case, by reading out a storage medium storing a program represented by software for achieving the present invention into a system or an apparatus, the system or the apparatus can enjoy the effects of the present invention.

In this embodiment, a ROM is used as a storage medium.
Although the storage medium 302 is used, the storage medium is not limited to this, and for example, a floppy disk, a hard disk,
Optical disk, magneto-optical disk, CD-ROM, CD-
R, DVD, magnetic tape, nonvolatile memory card, and the like can be used.

[0104]

According to the image synthesizing apparatus according to the first aspect of the present invention, when a plurality of subject images having overlapping fields of view taken at different viewpoints by the display means are displayed in a stereoscopic manner, the tertiary image is stored by the model information storage means. A camera that stores model information representing an original structure, and includes operation parameters including the orientation and position of the stored model information and at least the focal length of the camera and the distance between viewpoints at the time of shooting of the plurality of subject images. Using the parameters, rotation and movement of the model information are calculated by calculation means, a plurality of images of the model information whose rotation and movement are calculated are generated by the image generation means by perspective projection, and the generation is performed by the image synthesis means. The plurality of model information images thus synthesized and the plurality of subject images are synthesized, and the synthesized image is stereoscopically displayed by the display means. It is possible to synthesize CG image to obtain a synthesized stereoscopic video without uncomfortable stereoscopic effect. Also,
The same effect can be obtained in the image synthesizing method according to the fourth aspect.

According to the image synthesizing device of the second aspect,
When stereoscopically displaying a plurality of subject images photographed by a plurality of imaging systems by the display means, model information representing a three-dimensional structure is stored by the model information storage means, and the orientation and position of the stored model information are stored. Using operation parameters including the following, and the camera parameters including the focal lengths of the plurality of imaging systems and the distance between viewpoints, the rotation and movement of the model information are calculated by calculation means, and the rotation and movement are calculated by the image generation means. A plurality of images of the calculated model information are generated by perspective projection, the generated plurality of model information images and the plurality of subject images are synthesized by an image synthesizing unit, and the synthesized image is displayed on the display unit. Therefore, the same effect as the image synthesizing apparatus according to the first aspect can be obtained. In addition, since a plurality of imaging systems are provided, it is possible to easily shoot a stereoscopic video, and since the camera parameters at the time of shooting can be used as they are, it is possible to easily realize synthesis with a CG image. .

According to the image synthesizing device of the third aspect,
The depth extracting means extracts the depth of the subject image from the parallax of the plurality of subject images, and the image synthesizing means determines the image of the generated model information and the subject image based on the extracted depth of the subject image. By comparing the depth of the subject image obtained from the parallax with the depth of the generated CG model, even when the subject of the captured image and the CG model overlap, the concealment relationship of the synthesized stereoscopic image is discomfort. It is possible to obtain a three-dimensional image without images. A similar effect can be obtained in the image synthesizing method according to the fifth aspect.

According to the storage medium of the present invention, when synthesizing a plurality of subject images having overlapping visual fields taken from different viewpoints and a model image representing three-dimensional information created by a computer, the model information is used. The orientation, the operation parameters including the position, the focal length of the plurality of imaging systems, using camera parameters including the distance between the viewpoint, the rotation of the model information, a parameter calculation module that calculates the movement, the rotation, An image generation module that generates a plurality of images of model information for which movement has been calculated by perspective projection, and an image synthesis module that synthesizes the generated plurality of model information images and the plurality of subject images. Since a program for causing the CG image to be stored is stored, the CG image can be easily synthesized with the three-dimensional image, and the synthesized three-dimensional image has no strange feeling. Image can be obtained by treatment of the computer. Further, the function of each processing module can be easily changed or added, and the versatility of the image synthesizing apparatus can be enhanced.

According to the storage medium of the present invention, the image processing apparatus further includes a depth extracting module for extracting a depth of the subject image based on a parallax of the plurality of subject images. Based on the depth of the subject image, the image of the generated model information and the subject image are combined, so the image captured by comparing the depth of the subject image obtained from parallax with the depth of the generated CG model Even when the subject and the CG model overlap, it is possible to obtain a stereoscopic image by the computer processing that does not make the concealment relationship of the synthesized stereoscopic image strange.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image composition device according to a first embodiment.

FIG. 2 is a block diagram illustrating a configuration of a stereo camera 200.

FIG. 3 is an exploded perspective view showing a configuration of the three-dimensional display device 130.

FIG. 4 is a diagram illustrating processing of a stereo image combining unit 106.

FIG. 5 is a diagram showing a state of perspective projection.

FIG. 6 is a diagram showing a state in which CG image data is combined with the subject images of FIGS. 4 (A) and (B).

FIG. 7 is a block diagram showing a configuration of a depth extracting unit 110.

FIG. 8 is a diagram showing a state of searching for a region of the left image corresponding to each region of the right image.

FIG. 9 is a diagram showing a result of region division performed on the left image shown in FIG.

FIG. 10 is a diagram showing an image obtained by combining a left CG image with the left image shown in FIG.

FIG. 11 is a block diagram illustrating a configuration of an image composition device according to a second embodiment.

FIG. 12 is a block diagram illustrating a configuration of an image composition device according to a third embodiment.

FIG. 13 is a ROM storing an image synthesis processing program;
FIG. 3 is a diagram illustrating a memory map of a memory 302;

FIG. 14 is a flowchart illustrating a processing procedure in a synthesis mode in an image synthesis processing program executed by the CPU 301;

FIG. 15 is a flowchart illustrating a processing procedure of a recording mode in an image synthesis processing program executed by the CPU 301;

[Explanation of symbols]

 Reference Signs List 101 camera parameter storage unit 102 CG operation unit 103 CG parameter calculation unit 105 CG image generation unit 106 stereo image synthesis unit 110 depth extraction unit 120, 121 image synthesis unit 200 stereo camera 300 recording medium 301 CPU 302 ROM

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Nao Kurahashi 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc. (72) Inventor Katsuhiko Mori 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inside the corporation

Claims (7)

[Claims] 1. An image synthesizing apparatus comprising a display unit capable of stereoscopically displaying a plurality of subject images having overlapping fields of view taken from different viewpoints, wherein the model information storage stores model information representing a three-dimensional structure. Means, an operation parameter including an orientation and a position of the stored model information, and a camera parameter including a distance between a viewpoint and a focal length of the camera at least when the plurality of subject images are captured. Calculation means for calculating rotation and movement of the image, image generation means for generating a plurality of images of the model information for which the rotation and movement have been calculated by perspective projection, and a plurality of images of the generated model information and the plurality of subjects An image synthesizing device, comprising: image synthesizing means for synthesizing an image with the image, wherein the synthesized image is stereoscopically displayed by the display means. 2. An image synthesizing apparatus comprising: a plurality of imaging systems; and a display unit capable of displaying a plurality of subject images photographed by the plurality of imaging systems in a stereoscopic manner. Model information storage means for storing information; operating parameters including the orientation and position of the stored model information; and camera parameters including a focal length of the plurality of imaging systems and a distance between viewpoints. Calculation means for calculating rotation and movement of information; image generation means for generating a plurality of images of the model information for which rotation and movement have been calculated by perspective projection; and images for the generated plurality of model information and the plurality of images. An image synthesizing apparatus, comprising: image synthesizing means for synthesizing a subject image; and displaying the synthesized image stereoscopically on the display means. 3. The apparatus according to claim 1, further comprising a depth extracting unit configured to extract a depth of the subject image based on a parallax of the plurality of subject images, wherein the image combining unit is configured to generate the model information based on the extracted depth of the subject image. The image synthesizing apparatus according to claim 1, wherein the image is synthesized with the subject image. 4. An image synthesizing method for synthesizing a plurality of subject images having overlapping visual fields photographed from different viewpoints and a model image representing three-dimensional information created by a computer, wherein the model information representing the three-dimensional structure is stored. Using the operation parameters including the orientation and the position of the stored model information, the focal lengths of the plurality of imaging systems, and the camera parameters including the distance between viewpoints. Calculating the movement; generating a plurality of images of the model information for which the rotation and movement have been calculated by perspective projection; and combining the generated plurality of model information images with the plurality of subject images. And a step of stereoscopically displaying the synthesized image. 5. The method according to claim 5, further comprising the step of extracting a depth of the subject image based on a parallax of the plurality of subject images, wherein the step of synthesizing the images includes generating the model information based on the depth of the extracted subject image. 5. The method according to claim 4, wherein the image of the subject and the subject image are combined.
The image synthesizing method as described above. 6. When compositing a plurality of subject images having overlapping visual fields taken from different viewpoints and a model image representing three-dimensional information created by a computer, an operation parameter including a direction and a position of the model information;
Using a focal length of the plurality of imaging systems, camera parameters including the distance between viewpoints, a parameter calculation module for calculating the rotation and movement of the model information, and an image of the rotation and movement calculated model information. A program to be executed by a computer including an image generation module that generates a plurality of images by perspective projection, and an image synthesis module that synthesizes the generated plurality of model information images and the plurality of subject images is stored. Characteristic storage medium. 7. A depth synthesizing module for extracting a depth of the subject image based on a parallax of the plurality of subject images, wherein the image synthesizing module for synthesizing the images includes generating the depth based on the depth of the extracted subject image. 7. The storage medium according to claim 6, wherein the program for synthesizing the image of the model information and the subject image is stored.