JPH10116359A - Camera parameter arithmetic unit and image composing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To compose three-dimensional CG image generated based on an estimated camera parameter with an animation image by automatically and precisely estimate the camera parameter from the animation image data photographed by a video camera. SOLUTION: A camera parameter setting part 11 sets the camera parameter when the image of the frame is photographed to the first frame of the animation image. The camera parameter predicting part 12 predicts the camera parameter of a next frame from the world coordinate of the feature point of an already known shape object photographed in image data of the next frame and a position within image data of the feature point of the already known shape object. A camera parameter correcting part 13 corrects the camera parameter predicted by the part 12 by using the surface color of the already known shape object copied into image data.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for synthesizing an image photographed by a video camera or the like and an image generated by three-dimensional CG (computer graphics). A camera parameter calculation device for obtaining a camera parameter at the time of shooting from image data, and a three-dimensional CG image is generated using a camera parameter obtained from data of the shot image, and the shot image and the three-dimensional CG image are synthesized. The present invention relates to an image combining device that performs

[0002]

2. Description of the Related Art The synthesis of a still image or moving image (hereinafter referred to as a photographed image) photographed by a video camera or the like and an image generated by three-dimensional CG (hereinafter referred to as a three-dimensional CG image) is performed by combining a movie or a commercial. It is widely used in video production sites, including special effects on films and the like. In particular, moving a three-dimensional CG image in accordance with the motion of a moving image is often used to give a non-existent one a reality and power.

FIG. 12 is an explanatory diagram showing an example of synthesizing a captured image and a three-dimensional CG image. FIG. 12 shows an example in which a robot that does not actually exist (for example, a robot larger than a building) is generated by three-dimensional CG and combined with an actually captured building image. In such image synthesis, it is necessary to match a camera parameter used when creating a three-dimensional CG image with a camera parameter used when a captured image is captured, so that a sense of depth and movement are harmonized. here,
The camera parameters are the position, orientation, focal length, aspect ratio of the camera, and the projection coordinates of a point drawn perpendicular to the projection screen from the camera position.

FIG. 13 is an explanatory diagram showing a sense of discomfort of a synthesized image due to mismatch of camera parameters. FIG. 13 shows an example in which buildings generated by three-dimensional CG are combined with real buildings to be combined. If the camera parameters at the time of photographing the real building and the camera parameters at the time of generating the three-dimensional CG image do not match, a sense of incongruity occurs in the composite image.

As described above, in order to combine a moving image (captured image) and a three-dimensional CG image without giving a sense of perspective and unnatural feeling to the movement, a camera parameter and a three-dimensional CG for capturing the moving image are created. It is necessary to match with the camera parameters when performing.

Accordingly, a moving image is photographed using a camera capable of measuring and recording camera parameters, and a three-dimensional CG is created based on the camera parameters recorded for each frame of the photographed moving image, thereby obtaining a moving image. (Captured image) and a three-dimensional CG image. Alternatively, a moving image is shot according to camera parameters used when generating a three-dimensional CG image.

The following technique is known as a technique for synthesizing such a photographed image and a three-dimensional CG image.

Japanese Unexamined Patent Publication (Kokai) No. 61-267182 discloses a three-dimensional image created by computer graphics based on an image such as a landscape imaged by a video camera while changing the angle and magnification, and a state captured by the video camera. An image synthesis method for synthesizing a computer graphics image obtained by changing a model is described.

Japanese Patent Laid-Open Publication No. Hei 5-181952 discloses a position detecting device for mounting a video camera on a base capable of freely moving in three dimensions and detecting the three-dimensional position, direction and angle of view of the video camera. Is mounted on a base, a three-dimensional computer graphics image is generated in the computer based on the position data of the video camera by the position detection device, and the image of the subject photographed by the video camera is synthesized with the computer graphics image. A real-time synthesizing device is described.

Japanese Patent Application Laid-Open No. 5-183808 discloses that an object to be photographed is modeled and arranged in a virtual space, and a modeled virtual object is obtained based on detection results of parameters such as the position and direction of a camera at the time of photographing. An image synthesizing apparatus is described in which the depth of the image is calculated for each pixel of an image captured by the camera, and the images are synthesized while maintaining the correct context.

Japanese Patent Laid-Open Publication No. Hei 7-146501 has a scale function that can freely expand and reduce the range that can be physically input by an operating person in order to handle a wide range of camera movement. It has a function to set the origin of three-dimensional coordinates to a target at an arbitrary position in the target, and by making the scale function with respect to this origin, the movement path in a wide space around the target with one continuous operation Can be input all at once, and at the same time, by collecting and accumulating all data related to camera work every time the video is shot, it can be used as a camera work input locality device in CG etc. that draws every video sync Is described.

A method for estimating camera parameters when a still image is photographed from a still image is described in Rod.
G. FIG. Bogart's VIEW CORRELATIO
N method (GRAPHIICS GEMS 2, ACADEM
IC PRESS pp 181-190) is known. In the VIEW CORRERATION method, a human associates feature points in image data with three-dimensional coordinates in world coordinates, and gives a position in the image at which the three-dimensional coordinates are projected for five or more feature points. , For estimating camera parameters. As feature points in the image data, as shown in FIG. 14, a shape change point such as a corner of an object or a color change point at which a color on the object surface changes is used.

It is conceivable to estimate the camera parameters of a moving image by applying the VIEW CORRELATION method. VIEW CORRELAT is applied to each image data of each frame constituting the moving image data.
If the camera parameters are estimated using the ION method and a three-dimensional CG image to be synthesized with each frame is generated based on the camera parameters estimated for each frame, a moving image and a three-dimensional CG image can be obtained.
It can be synthesized with the dimensional CG image without giving a sense of incongruity.
However, the work of associating the world coordinates of the feature points with moving image data composed of a large number of image data becomes extremely enormous, which is a problem in terms of efficiency.

Therefore, as a method of reducing the amount of work, a human associates the position of the feature point in the image data with the world coordinates with respect to the first frame of the moving image data,
It is conceivable to automatically estimate the camera parameters for the subsequent frames. This method tracks how feature points in the first frame move in subsequent frames. As a result of this tracking processing, if the positions of the feature points are known in all the frames, VIEW CORR
The camera parameters of all frames can be estimated by applying the ELATION method.

FIG. 15 is a flowchart of a method for estimating camera parameters at the time of capturing a moving image by applying the VIEW CORRELATION method. Step S1
Then, a human (user) associates the position of the feature point in the image data with the world coordinates with respect to the image data of the first frame of the moving image data. In step S2, VIEW CORRELAT is performed using this correspondence.
The camera parameters for the image data of the first frame are calculated by applying the ION method.

In step S3, when the image data of the first frame is changed to the image data of the second frame,
Track where feature points have moved. This feature point tracking method includes a residual sequential test method (Image Processing Handbook,
The University of Tokyo Press, pp 708) is used. The residual sequential test method is a general pattern matching method. The feature point in the image data of the current frame and its neighboring pixels are used as a reference pattern to search for the reference pattern in the image data of the next frame. Tracking can be done.

In step S4, the position in the image of the feature point in the second frame obtained by tracking the feature point in step S3 and the world coordinates associated with this feature point in step S1 are used. VIEW CORREL
The camera parameter of the second frame is calculated by applying the ATION method. Thereafter, steps S3 and S4 are repeated to calculate camera parameters for image data of all frames of the moving image data.

In principle, it is possible to generate a three-dimensional CG image based on the camera parameters calculated by the method shown in FIG. 15 and to combine the generated three-dimensional CG image with a moving image (captured image). It is. However, even if the tracking of the feature points is performed in step S3, a deviation occurs between the positions of the feature points in the tracking result and the actual positions of the feature points. Therefore, an error occurs in the calculation result of the camera parameters in step S4, and it is not possible to estimate a camera parameter suitable for a moving image (captured image). Therefore, using the camera parameter estimation method shown in FIG.
Synthesizing with a dimensional CG image has not been performed practically.

As described above, according to the conventional technique, three-dimensional moving image data obtained by a general camera having no special mechanism capable of measuring and recording camera parameters is used. A CG image could not be generated and combined.

[0020]

Conventionally, camera parameters (camera position, orientation, focal length, aspect ratio, projection coordinates of a point perpendicular to the projection screen from the camera position) at the time of photographing and a three-dimensional CG image The moving image (captured image) and 3
When synthesizing a three-dimensional CG image, camera parameters are measured and recorded at the time of capturing moving image data, and a three-dimensional CG image is generated based on the recorded camera parameters.
Two methods have been used, in which camera parameters at the time of generating a dimensional CG image are recorded and a moving image is shot based on the recorded camera parameters.

For this reason, a photographing device for photographing moving image data needs a mechanism for measuring and recording camera parameters or a mechanism for controlling the movement (posture) of the camera based on the camera parameters. . Since camera parameters include parameters inside the camera, such as focal length, aspect ratio, and screen coordinates of the optical axis,
The camera itself needs a mechanism to understand the camera parameters.

Therefore, the above two methods require the installation of special photographing equipment. It is impossible to combine a three-dimensional CG image with moving image data shot by a commercially available video camera or moving image data shot in the past.

On the other hand, apart from the above two methods, a method of estimating camera parameters from still image data (VIE)
A method of estimating the camera parameters in each frame of the moving image data using the W CORRELATION method, and generating and synthesizing a three-dimensional CG image using the estimated camera parameters is conceivable.

In the VIEW CORRELATION method, it is necessary to associate the positions of feature points (points at which the corners of an object or a color on the object surface change) in the image data with three-dimensional coordinates in the world coordinate system. The position of the feature point in the image data of each frame of the data is required. As a method of determining the position of the feature point, a method in which the user specifies the position of the feature point for each frame over all frames, and a method in which the user specifies the position of the feature point in the first frame, and in the subsequent frames, A method of automatically tracking feature points can be considered.

According to these two methods, in principle,
It is possible to combine a three-dimensional CG image with moving image data captured by a commercially available video camera. However, the first method is not efficient because manual intervention is required for each frame of moving image data.

In the second method, the deviation of the tracking result of the feature point causes the deviation of the camera parameter, and it is impossible to estimate the camera parameter with practical accuracy.

The present invention has been made to solve such a problem, and a technique for accurately estimating camera parameters from moving image data captured by a camera having no special mechanism. It is an object of the present invention to provide an image synthesis technique for generating and synthesizing a three-dimensional CG image based on the image.

[0028]

In order to solve the above-mentioned problems, a camera parameter calculation device according to the present invention converts an input camera parameter (temporary camera parameter) into a surface color of a known shape object photographed in image data. A camera parameter estimating unit that evaluates using a camera parameter, a camera parameter updating amount calculating unit that calculates an amount of updating a camera parameter using a nonlinear least squares method in order to improve the evaluation in the camera parameter estimating unit, and a camera updating amount calculating A camera parameter updating unit for updating the camera parameter based on the camera parameter updating amount calculated by the camera unit, and a camera parameter for terminating the iterative processing by the three processing units of the camera parameter evaluating unit, the camera parameter updating amount calculating unit, and the camera parameter updating unit. A correction end determining unit.

The camera parameter calculation device according to the present invention reduces an error between the initial value of the input camera parameter and the camera parameter at the time of capturing the image data by processing in the camera parameter correction unit.

With respect to the camera parameters input to the camera parameter correction unit, the camera parameter evaluation unit calculates an evaluation value from a difference between a color on an object having a known shape and a color at the time of being captured in the current frame.

Next, an amount of updating the camera parameters so as to improve the evaluation value is calculated by a camera parameter update amount calculating section.

The camera parameter correction end determining unit determines whether to end the camera parameter correction by using the evaluation value, the update amount, and the number of times the repetition processing is performed in the camera parameter correction unit for camera parameter correction. If not terminated, the camera parameter updating unit updates the camera parameter using the previously obtained camera parameter update amount, and inputs the camera parameter to the camera parameter evaluation unit again.

On the other hand, when the processing is completed, it is determined whether the correction of the camera parameters has succeeded or failed using the finally obtained evaluation value of the camera parameters and a threshold value set in advance by the user. I do.

[0034]

Embodiments of the present invention will be described below with reference to the accompanying drawings. FIG. 1 is an overall configuration diagram of an image synthesizing apparatus according to the present invention. The image synthesizing device 1 includes a reproducing device 2, a recording device 3, a drawing / synthesizing device 4, a keyboard 5, a pointing device 6, and an image display device 7.

The reproducing apparatus 2 supplies moving image data photographed in advance by a video camera or the like to the drawing / synthesizing apparatus 4 and supplies CG image data for generating a CG image to the drawing / synthesizing apparatus 4. And a VTR device, a memory device, a hard disk device, a CD-ROM playback device, and the like. The moving image data is continuous image data sampled at discrete times. Each piece of continuous image data is called a frame. The playback device 2
It is configured to be able to reproduce one frame at a time.

The recording device 3 is for recording a composite image (composite image of a moving image and a CG image) output from the drawing / compositing device 4, and uses a VTR device, a memory device, a hard disk device, or the like. It is composed.

The keyboard 5 constitutes a provisional camera parameter input section and a camera parameter setting section. The keyboard 5 is used to input provisional camera parameters when obtaining a camera parameter when a still image is captured from a still image, and when obtaining a camera parameter when the video is captured from a moving image. This is for inputting camera parameters when an image of a specific frame is taken. The pointing device 6 is for specifying a feature point of the captured image displayed on the screen of the image display device 7.

FIG. 2 is a block diagram of the drawing / synthesizing apparatus. The drawing / synthesizing device 4 includes a camera parameter calculating device 10, a three-dimensional CG generating unit 20, and an image synthesizing unit 30. The camera parameter calculation device 10 obtains, for each frame of the moving image, a camera parameter when an image of the frame is captured. The obtained camera parameters are supplied to the three-dimensional CG generation unit 20. The three-dimensional CG generation unit 20 generates a three-dimensional CG image based on the camera parameters obtained for each frame. 3 generated
The dimensional CG image is supplied to the image combining unit 30. The image combining unit 30 combines the moving image and the three-dimensional CG image and outputs a combined image.

FIG. 3 is a flowchart showing the operation of the drawing / synthesizing device. In step S11, the camera parameter calculation device 10 reads image data of a moving image captured by a video camera or the like one frame at a time.
Obtains the camera parameters when the frame was shot. In step S13, the three-dimensional CG generating unit 20 generates a three-dimensional CG image to be combined with the frame based on the obtained camera parameters. In step S14, the image combining unit 30 combines the moving image and the three-dimensional CG image. The drawing / synthesizing device 4 obtains camera parameters for each frame of the moving image, and performs three-dimensional CG based on the obtained parameters.
Since the image is generated and the moving image and the three-dimensional CG image are combined, a combined image having no uncomfortable feeling in the combined position or the perspective can be obtained.

FIG. 4 is a block diagram of the camera parameter calculation device. The camera parameter calculation device 10 includes a camera parameter setting unit 11, a camera parameter prediction unit 12, and a camera parameter correction unit 13. The camera parameter correction unit 13 includes a camera parameter evaluation unit 1
4, a camera parameter update amount calculation unit 15, a camera parameter update unit 16, and a camera parameter correction end determination unit 17.

The camera parameter setting section 11 supplies the frame data of the moving image to an image display control device (not shown) to display the image of the frame to be evaluated on the screen of the image display device shown in FIG. It has.

In the camera parameter setting section 11, the user sets the world coordinates of the feature point and sets the position of the feature point in the image of the specific frame or the camera parameter for the specific frame. When the position of the feature point is designated, the camera parameters of the frame are estimated from the world coordinates of the feature point and the position in the image. Then, the camera parameter setting unit 11 outputs the world coordinates of the feature point and the camera parameters of the specific frame.

The world coordinates of the feature points and the camera parameters are input by operating the keyboard 5 shown in FIG. The positions of the feature points in the image of the specific frame are input by the pointing device 6 shown in FIG.

When the camera parameter calculation device 10 sets camera parameters for a specific frame in a moving image composed of a plurality of frames, the camera parameters of another frame can be obtained.

The camera parameter prediction unit 12
Calculates the position of the feature point in the image from the camera parameters and the world coordinates of the feature point, using the moving image data, the world coordinates of the feature point, and the camera parameters of the adjacent frame as inputs, and calculates the feature from the adjacent frame to the current frame. The tracking of the point position is performed by the residual sequential test method, etc., and the camera parameters are calculated from the world coordinates of the feature points of the current frame and the feature point positions.
The camera parameters are predicted by the View Corelation method or the like, and the predicted camera parameters are supplied to the camera parameter correction unit 13.

The camera parameter correcting section 13 corrects the predicted camera parameters by using the surface color of the object of known shape in the image data. The camera parameters corrected by the camera parameter correction unit 13 are supplied to the camera parameter setting unit 11. The camera parameter setting unit 11 temporarily stores the corrected camera parameters in the frame, and supplies the corrected camera parameters to the camera parameter prediction unit 12 as initial values of the camera parameters when predicting the camera parameters of the next frame. In this way, the camera parameters of the next frame are sequentially obtained.

The camera parameter evaluation section 14 evaluates the camera parameters using the color of the known shape object photographed in the image data. This camera parameter evaluation unit 14
Calculates the position of the object having the known shape in the image data from the world coordinates and the camera parameters of the surface of the object having the known shape in the image data, and calculates the color of the calculated position in the image data. Is determined as the color of the known-shaped object. When the color of an object having a known shape in the image data is known, the camera parameter evaluation unit 14 calculates a known amount as a quantity representing a difference between the camera parameter to be evaluated and the camera parameter at the time of capturing the image. And the average of the sum of squares of the difference between the color and the color determined by the known shape object determination unit 18.

The camera parameter update amount calculator 15 calculates the amount of camera parameter update using the nonlinear square method in order to improve the evaluation in the camera parameter evaluator 14.

The camera parameter update unit 16 updates the camera parameters based on the camera parameter update amount calculated by the camera parameter update amount calculation unit 15.

The camera parameter correction end determining section 17
The evaluation value of the camera parameter calculated by the camera parameter evaluation unit 14 is compared with a first allowable error set by the user, and when the evaluation value is smaller than the first allowable error, the correction of the camera parameter is ended. Further, the camera parameter correction end determining unit 17 compares the update amount of the camera parameter calculated by the camera parameter update amount calculation unit 15 with the minimum update amount set by the user, and when the update amount is smaller than the latest update amount. It is determined that the correction of the camera parameters is completed.

Further, when the camera parameter correction end determining section 17 determines that the camera parameter correction is to be ended, the camera parameter evaluation value calculated by the camera parameter evaluation section 14 is replaced by the second allowable error set by the user. When the number of camera parameters exceeds the limit, a camera parameter correction failure determination unit 19 that determines that the correction of the camera parameter has failed is provided. When it is determined that the correction of the camera parameter has failed, the camera parameter correction end determination unit 17 generates a parameter correction request for requesting the user to correct the camera parameter whose correction has failed. doing.

FIG. 5 is a flowchart showing the overall operation of the camera parameter calculation device. In step S21,
The user inputs the first permissible error, the minimum update amount, the second permissible error, and the maximum number of repetitions used when the camera parameter correction unit 13 determines the end of the correction of the camera parameter.

In step S22, in the first image data of the moving image data, the user changes a feature point on a known shape object (a feature point due to a shape change such as a corner of the object shown in FIG. 14 or a color on the object surface). (Point), the position in the image data is associated with the three-dimensional coordinates in the world coordinate system. In step S23, VIEW COR
The camera parameters are calculated from the positions of the feature points and their world coordinates using the RELECTION method. Step S
At 24, the world coordinates of the feature point are converted by the camera parameters of the previous frame, and the position of the feature point in the image of the previous frame is corrected. In step S25, the position of a feature point in the current frame image corresponding to the corrected feature point in the previous frame image is obtained. In step S26, the camera parameters are calculated from the positions of the feature points in the image of the current frame obtained as a result of the processing in step S25 and the world coordinates associated with the feature points in step S22.

In step S27, the camera parameters calculated in step S26 are corrected. Step S2
The processes from 4 to S27 are executed for each frame of the moving image data.

The difference between the processing of the camera parameter calculation device according to the present invention shown in FIG. 5 and the method of estimating camera parameters by the combination of the prior art shown in FIG. 15 is the presence or absence of steps S21 and S26. In particular, in the present invention, the accuracy of camera parameter estimation can be improved by the processing by the camera parameter correction unit shown in step S26.

FIG. 6 is a flowchart showing the operation of the camera parameter correction unit. Camera parameter correction unit 13
Performs the correction of the camera parameters by repeating the processing of steps S31 to S34. First, in step S31, an evaluation value of a camera parameter is calculated.
In step S32, the amount of updating the camera parameter is calculated so that the evaluation value of the camera parameter obtained in step S31 is minimized. In step S33, it is determined whether the correction of the camera parameters has been completed using the first allowable error, the minimum update amount, the second allowable error, and the maximum number of repetitions already set in step S21 in FIG. I do.

The first allowable error and the minimum update amount are used to determine whether the camera parameters have been sufficiently corrected. The maximum number of repetitions is the maximum value of the number of repetitions of steps S31 to S34 in the camera parameter correction unit 13, and no further repetition is performed. The second allowable error is used to determine whether the camera parameter correction has been correctly performed when the camera parameter correction has been completed.

In step S33, the above four values (first allowable error, minimum update amount, second allowable error, and
The end of the camera parameter is determined using the maximum number of repetitions. It is determined that the correction of the camera parameter is completed. (1) When the evaluation value of the camera parameter is smaller than the first allowable error (2) When the update amount of the camera parameter is smaller than the minimum update amount (3) In the camera parameter correction unit 13, there are three cases in which the number of repetition processes in steps S31 to S34 exceeds the maximum number of repetitions.

However, when the correction of the camera parameters is terminated under conditions other than the termination condition (1), the correction of the camera parameters may have failed. Although the reason for this will be described later, it is detected in step S33 that the correction has failed using the second allowable error. When the evaluation value of the camera parameter is larger than the second allowable error, it is determined that the correction of the camera parameter has failed, and the correction of the camera parameter that has failed to be corrected is performed through the processing of steps S22 and S23 in FIG. Ask the user.

On the other hand, if none of the three end conditions is satisfied, the correction of the camera parameters does not end. In this case, in step S34, the camera parameter is updated based on the update amount of the camera parameter calculated in step S32, and the process returns to step S31 to continue the correction of the camera parameter.

Next, evaluation of camera parameters will be described. The camera parameter evaluation method according to the present invention assumes that the color of an object photographed in moving image data does not change rapidly during photographing. Based on this assumption, the color of the corresponding point on the surface of the object, that is, the color of the same point in the world coordinates substantially matches in the adjacent frame in the moving image data. This assumption is almost satisfied except when the luminous body such as neon blinks or a special object such as a mirror.

FIG. 7 is an explanatory view showing a conversion formula of coordinate conversion by central projection, and FIG. 8 is an explanatory view showing a camera model.
From the three-dimensional coordinates in the world space and the camera parameters, it is possible to calculate at which position on the image data a point on the surface of the known shape object is projected. This formula is:
Equation 1 shown in FIG. Equation 1 is derived from the camera model shown in FIG.

The camera model is composed of the center of the camera and the screen, and a point P = (x, y, z) on the object is projected on the intersection of a straight line connecting the point P and the center of the camera with the screen. The camera parameter is the position P of the center of the camera.
c, a rotation matrix R for rotating the x, y planes of the coordinate system giving the three-dimensional coordinates of the object parallel to the screen, and projected coordinates (Xc, Y) of a point perpendicular to the screen from the center of the camera.
c) and enlargement factors Xz, Yz in the coordinate axis direction of the projected coordinates on the screen. According to this camera model, a point P = (x, y, z) on the object is projected at coordinates (X, y, z).
Y). This (X, Y) is the position in the image data.

FIG. 9 is an explanatory diagram showing the operation of the known-shaped object surface color determination unit. As shown in FIG. 9, a model of a known-shaped object is created on world coordinates, and by using Equation 1, a position P ′ in image data is calculated from world coordinates and camera parameters of a point P on the object surface. it can. If the camera parameters used in Equation 1 match the camera parameters used when the image data was captured, the color of the point P ′ in the image data is the surface color of the known shape object.

When correcting the camera parameters, since one of the camera parameters of the adjacent frames is always known, the surface color of the known-shaped object obtained by using Equation 1 can be obtained. In the other frame as well, the surface color of the known shape object is obtained using the temporary camera parameters, and if there is a large difference between the obtained two surface colors, the temporary camera parameters do not match the image data. You can see that.

In the present invention, for a number of points on the surface of a known-shaped object, the root mean square (or the sum of squares) of the surface color differences obtained from adjacent frames is obtained as an evaluation value of a camera parameter. The camera parameters are evaluated depending on whether they are small.

FIG. 10 is an explanatory diagram showing the operation of the camera parameter evaluation unit. The camera parameter evaluation unit 14 includes a three-dimensional model of the known shape object and the camera parameter prediction unit 1
2, a projection image of the known-shaped object is generated based on the camera parameters supplied from the camera 2 and the color data (Ri, Gi, Bi) of the projection coordinate point (Xi, Yi) of each pixel of the generated projection image of the known-shaped object. ) And the color data R ((Xi, Yi), G (Xi, Yi), B) of the pixel at the same coordinate point in the frame image captured using the actual video camera or the like.
(Xi, Yi)), a square value of the difference is calculated for each pixel, and a mean square (or a sum of squares) of the difference of each pixel is calculated.

Note that the known shape object has its three-dimensional shape model and its existing position in the world coordinate system defined in advance.

Although the colors are represented as data in the RGB space in FIG. 10, colors in the CMY space, the HSL space, the HSV space, the XYZ space, the Luv space, the YUV space, the YIQ space, and the YCbCr space may be used.

The evaluation value of the camera parameter is a non-linear value with respect to the camera parameter, and the evaluation value is a root mean square (or a sum of squares) of the color differences at each point on the known shape object. A camera parameter that minimizes the evaluation value of the camera parameter is obtained by a multiplication method.

In this embodiment, as a method for solving the nonlinear least squares problem, Levenberg-Marquard is used.
t method (WH Press et al. Numeri)
cal Recipes in C: The Art
of ScientificComputing, 2n
d edition, Cambridge Univ.
Press). Of course, the solution of the nonlinear least squares method is not limited to this method. The parameter update and the calculation of the evaluation value are alternately performed by the non-linear least squares method, and the parameter update amount becomes small or the evaluation value becomes 0.
It ends when it approaches sufficiently. The update amount of the camera parameter is calculated based on the nonlinear least squares method.

FIG. 11 is an explanatory diagram showing a process for updating camera parameters using the nonlinear least squares method. Assuming that the evaluation value of the camera parameter is the sum of squares of the difference between the surface colors obtained from the adjacent frames, the evaluation value E is expressed by Expression 2 in FIG. (In the case of the root mean square, the evaluation value E shown in Expression 2 may be divided by the number N of points sampled on the object surface.) However, N is the number of points sampled on the object surface, and (Rp
i, Gpi, Bpi) is the color of the point Pi on the model of the known shape object obtained by using the temporary camera parameters in the image for estimating the camera parameters. Also, (R
i, Gi, Bi) is the color of the point Pi on the model of the known-shaped object obtained using the known camera parameters in the image whose camera parameters are already known. Ak is a k-th camera parameter. The order of the camera parameters in ak is determined for convenience of calculation, and the order does not affect the estimation result.

Levenberg-Marquardt
The update amount Δa of the camera parameter calculated by the processing of the method is expressed by Expression 3 in FIG. The partial differential value used in Expression 3 is obtained as the product of the density gradient of the image and the partial differential value of the projection coordinates based on the camera parameters, as shown in Expression 4. The density gradient of the image is obtained from the image data. The partial differential value of the projection coordinates based on the camera parameters can be obtained by differentiating Equation 1 with the camera parameters. λ is a stabilization parameter that changes according to the degree of convergence. When the evaluation value E becomes smaller than the previous evaluation, λ is made smaller and the camera parameters are updated. When the evaluation value becomes larger than the previous evaluation value, λ is increased, and the camera parameters remain the same as the previous camera parameters. In the present embodiment, the initial value of λ is set to 100.0, λ is increased by ３ when the evaluation value is smaller than the previous evaluation value, and tripled when the evaluation value is increased.

In the nonlinear square method, a parameter for minimizing the evaluation value is calculated. Therefore, there is a possibility that the camera parameter corrected by the camera parameter correction unit 13 is not the minimum value even if it is the minimum value. The nonlinear least squares method may not converge depending on the input image data. In these two cases, the camera parameters are not correctly corrected.

Therefore, in the camera parameter calculation device according to the present invention, the quality of the correction of the camera parameters is determined using the second allowable error set in step S21 of FIG. If the corrected camera parameter exceeds the second allowable error, it is determined that the correction of the camera parameter in the present invention has failed, and in step S22 of FIG. Request an association. By this processing, the camera parameter estimation is interrupted, or the camera parameter of the next frame is not estimated based on an incorrect camera parameter.

[0076]

As described above, according to the present invention,
If the camera parameters are set for the first frame of the moving image data via user's intervention, the camera parameters in the subsequent frames can be estimated.
According to the present invention, since the property that the color of the surface of the known-shaped object hardly changes in the adjacent frame is used, it is possible to estimate a camera parameter having no deviation from the camera parameter at the time of capturing the image data.

As described above, according to the present invention, it is possible to use a camera capable of measuring and recording camera parameters, which has been conventionally performed, without using a camera. So, moving image data and 3
It is possible to combine with the dimensional CG image without a sense of incongruity.

Further, in the present invention, the user sets thresholds (first allowable error, minimum update amount, second allowable error, and maximum number of repetitions) for judging the end of correction of camera parameters. Thus, when a camera parameter that does not reach the estimation accuracy desired by the user is detected, it is detected, and at that time, the user can correct the camera parameter.

[Brief description of the drawings]

FIG. 1 is an overall configuration diagram of an image synthesizing apparatus according to the present invention.

FIG. 2 is a block diagram of a drawing / synthesizing apparatus.

FIG. 3 is a flowchart illustrating an operation of the drawing / synthesizing apparatus.

FIG. 4 is a block diagram of a camera parameter calculation device.

FIG. 5 is a flowchart illustrating an overall operation of the camera parameter calculation device.

FIG. 6 is a flowchart illustrating an operation of a camera parameter correction unit.

FIG. 7 is an explanatory diagram showing a conversion formula of coordinate conversion by central projection.

FIG. 8 is an explanatory diagram showing a camera model.

FIG. 9 is an explanatory diagram illustrating an operation of a known-shape object surface color determination unit.

FIG. 10 is an explanatory diagram illustrating an operation of a camera parameter evaluation unit.

FIG. 11 is an explanatory diagram showing a process of updating camera parameters using the nonlinear least squares method.

FIG. 12 is an explanatory diagram illustrating an example of combining a captured image and a three-dimensional CG image.

FIG. 13 is an explanatory diagram showing a sense of discomfort of a synthesized image due to mismatch of camera parameters.

FIG. 14 is an explanatory diagram showing an example of a feature point of an image used in the VIEW CORRELATION method.

FIG. 15 shows a camera parameter V when a moving image is captured.
It is a flowchart of the method of estimating applying the IEW CORRELATION method.

[Explanation of symbols]

1 image synthesis device, 2 playback device, 3 recording device, 4
Drawing / synthesizing device, 5 keyboard, 6 pointing device, 7 image display device, 10 camera parameter calculation device, 11 camera parameter setting unit, 12 camera parameter prediction unit, 13 camera parameter correction unit, 1
4 camera parameter evaluation unit, 15 camera parameter update amount calculation unit, 16 camera parameter update unit, 17 camera parameter correction end determination unit, 18 shape known object color determination unit, 19 correction failure determination unit, 20 three-dimensional CG generation unit, 30 Image synthesis unit

Claims (19)

[Claims] 1. A camera parameter calculation device for obtaining a camera parameter when an image is captured from image data of an image captured by a video camera, a camera parameter input unit for inputting a predicted camera parameter, A camera parameter correction unit for correcting a camera parameter input from the camera parameter input unit using a surface color of a known shape object projected in the camera parameter calculation unit. 2. A camera parameter estimator for evaluating a camera parameter using a surface color of a known shape object photographed in image data, and a camera parameter estimator for improving the evaluation in the camera parameter evaluator. A camera parameter update amount calculation unit that calculates the update amount of the camera parameter using the nonlinear least squares method, and a camera parameter update unit that updates the camera parameter with the camera parameter update amount calculated by the camera parameter update amount calculation unit 2. The camera parameter correction unit according to claim 1, further comprising: a camera parameter evaluation unit, a camera parameter update amount calculation unit, and a camera parameter correction end determination unit that determines the end of the repetitive processing by the three processing units. Camera parameter calculation device. 3. The camera parameter evaluation unit calculates a position of the object having the known shape in the image data from the world coordinates of the surface of the object having the known shape in the image data and the camera parameters, 3. The camera parameter calculation device according to claim 2, further comprising a known shape object color determination unit that determines a color at a position calculated in the image data as a color of a known shape object. 4. The camera parameter evaluation section, when a color of a surface of an object having a known shape in image data is known, a camera parameter to be evaluated and a camera parameter at the time of capturing an image. 4. The camera parameter calculation device according to claim 3, wherein a square or an average of the sum of squares of a difference between a known color and a color determined by the known shape object color determination unit is used as the evaluation value representing the shift. 5. The camera parameter correction end determination unit compares the camera parameter evaluation value calculated by the camera parameter evaluation unit with a first tolerance set by a user, and determines that the evaluation value is a first tolerance. 3. The camera parameter calculation device according to claim 2, wherein it is determined that the correction of the camera parameter is ended when the value is smaller than the predetermined value. 6. The camera parameter correction end determination unit compares the camera parameter update amount calculated by the camera parameter update amount calculation unit with a minimum update amount set by a user, and the update amount is larger than the minimum update amount. 3. The camera parameter calculation device according to claim 2, wherein it is determined that the correction of the camera parameter is ended when the value is smaller. 7. The camera parameter correction end determination unit determines the number of repetitions performed by the three processing units of the camera parameter evaluation unit, the camera parameter update amount calculation unit, and the camera parameter update unit, and the maximum number of repetitions set by the user. 3. The camera parameter calculation device according to claim 2, wherein the comparison is made to determine that the correction of the camera parameter is to be terminated when the number of repetition processes exceeds the maximum number of repetitions. 8. The camera parameter correction end determining unit, when determining that the correction of the camera parameter is to be ended, sets the camera parameter evaluation value calculated by the camera parameter evaluation unit to a second allowable error set by a user. 3. The camera parameter calculation device according to claim 2, further comprising a camera parameter correction failure determination unit that determines that the correction of the camera parameter has failed when the value exceeds the predetermined value. 9. The camera parameter correction end determining unit, when determining that the correction of the camera parameter has failed, generating a parameter correction request for requesting a user to correct the camera parameter whose correction has failed. The camera parameter calculation device according to claim 2, wherein: 10. A camera parameter calculation device for obtaining each camera parameter when an image of each frame is captured from video data of a moving image composed of a plurality of frames captured by a video camera. A camera parameter setting unit for setting a camera parameter when an image of the specific frame is captured, and setting a camera parameter in predicting a camera parameter of a frame adjacent to the frame in which the camera parameter is set. The position of the feature point of the known shape object in the image data in the image data is calculated from the world coordinates of the feature point and the camera parameter, and where the calculated position of the feature point in the image data predicts the camera parameter in the image of the frame. Track whether to move and predict camera parameters A camera parameter prediction unit for predicting a camera parameter from a position and a world coordinate of a feature point obtained as a result of image tracking in a camera, and a camera parameter predicted by the camera parameter prediction unit, the known camera parameter being projected into image data. A camera parameter correction unit that corrects using the surface color of the shape object. After setting the camera parameters in one frame, the processing of the camera parameter prediction unit and the camera parameter correction unit is repeatedly performed to obtain moving image data. A camera parameter calculation device for setting camera parameters for each frame over all frames. 11. A camera parameter evaluation unit for evaluating a camera parameter using a surface color of a known shape object photographed in image data, and a camera parameter evaluation unit for improving the evaluation in the camera parameter evaluation unit. A camera parameter update amount calculation unit that calculates the update amount of the camera parameter using the nonlinear least squares method, and a camera parameter update unit that updates the camera parameter with the camera parameter update amount calculated by the camera parameter update amount calculation unit 11. The camera parameter correction unit according to claim 10, further comprising: a camera parameter evaluation unit, a camera parameter update amount calculation unit, and a camera parameter correction end determination unit that determines the end of the repetitive processing by the three processing units. Camera parameter calculation device. 12. The camera parameter evaluation unit calculates a position of an object having a known shape in the image data from world coordinates and a camera parameter of a surface of the object having a known shape in the image data, The camera parameter calculation device according to claim 11, further comprising a known shape object color determination unit that determines a color at a position calculated in the image data as a color of a known shape object. 13. The camera parameter evaluation unit, when the color of the surface of an object having a known shape shown in image data is known, compares the camera parameter to be evaluated with the camera parameter at the time of capturing the image. 12. The camera parameter calculation device according to claim 11, wherein a sum of squares or an average of the sum of squares of a difference between a known color and a color determined by the known shape object color determination unit is used as the evaluation value representing the shift. . 14. The camera parameter correction end determination unit compares the camera parameter evaluation value calculated by the camera parameter evaluation unit with a first tolerance set by a user, and determines that the evaluation value is a first tolerance. 12. The camera parameter calculation device according to claim 11, wherein it is determined that the correction of the camera parameter is to be terminated when the value is smaller than the correction value. 15. The camera parameter correction end determination unit compares the camera parameter update amount calculated by the camera parameter update amount calculation unit with a minimum update amount set by a user, and the update amount is larger than the minimum update amount. 12. The camera parameter calculation device according to claim 11, wherein it is determined that the correction of the camera parameter is ended when the value is smaller. 16. The camera parameter correction end determining unit determines the number of repetitions performed by the three processing units of the camera parameter evaluation unit, the camera parameter update amount calculation unit, and the camera parameter update unit, and the maximum number of repetitions set by the user. 12. The camera parameter calculation device according to claim 11, wherein it is determined that the correction of the camera parameter is ended when the number of repetition processes exceeds the maximum number of repetitions. 17. The camera parameter correction end determining unit, when determining that the camera parameter correction is to be ended, sets the camera parameter evaluation value calculated by the camera parameter evaluation unit to a second allowable error set by a user. The camera parameter calculation device according to claim 11, further comprising a camera parameter correction failure determination unit that determines that the correction of the camera parameter has failed when the value exceeds the predetermined value. 18. The camera parameter correction end determining unit, when determining that the correction of the camera parameter has failed, generating a parameter correction request for requesting the user to correct the camera parameter for which the correction has failed. The camera parameter calculation device according to claim 11, wherein 19. A camera parameter setting unit for setting a camera parameter when an image of a specific frame is captured for an image of a specific frame, and a camera of a frame adjacent to the frame in which the camera parameter is set. In predicting the parameters, the position in the image data of the feature point of the known shape object in the frame in which the camera parameter is set is calculated from the world coordinates of the feature point and the camera parameter, and the calculated position of the feature point in the image data A camera parameter prediction unit that tracks where the camera moves in the image of the frame for which the camera parameters are predicted, and predicts the camera parameters from the position and the world coordinates of the feature points obtained as a result of the image tracking in the frame for which the camera parameters are predicted. And camera parameters predicted by the camera parameter prediction unit And a camera parameter correction unit that corrects using a surface color of a known-shaped object projected in image data, and sets a camera parameter in one frame, and then processes the camera parameter prediction unit and the camera parameter correction unit. Is repeatedly performed to set camera parameters for each frame over all frames of the moving image data; and 3 based on the camera parameters for each frame set by the camera parameter calculation device. An image synthesizing apparatus, comprising: a three-dimensional CG generating unit that generates a three-dimensional CG image; and an image synthesizing unit that synthesizes a three-dimensional CG image generated by the three-dimensional CG generating unit with a moving image.