JP4297197B2 - Calibration processing apparatus, calibration processing method, and computer program - Google Patents

Description

  The present invention relates to a calibration processing apparatus, a calibration processing method, and a computer program. More specifically, parameters applied to adjustment between multi-view image capturing cameras that shoot a subject with a plurality of cameras arranged at different positions to generate a multi-view image, or correction processing of an acquired image of the multi-view image capturing camera. The present invention relates to a calibration processing apparatus that performs calculation processing, a calibration processing method, and a computer program.

  In recent years, use of image data such as panorama, omnidirectional images, and the like in which viewpoints can be moved in various ways is becoming popular. For example, when an image obtained by photographing a subject from a plurality of viewpoint positions and line-of-sight directions is accumulated in a storage medium such as a DVD or CD, and the accumulated image is displayed on a CRT, a liquid crystal display device, etc., the user can A system for observing an image of a subject by moving the viewpoint to a free position has been realized. Also, images taken from a plurality of viewpoint positions and line-of-sight directions are distributed via a communication system such as the Internet, and the image from the desired viewpoint position and line-of-sight direction is displayed on the display by the user operating a mouse such as a PC. A system for displaying is constructed.

  With the improvement of computer processing capability and the development of various video media playback devices, it became possible to process a huge amount of polygon data and video data (contents) that were previously difficult, and were shot with multiple cameras with different viewpoints. Real world (target) video data is processed by a computer, video equipment, or the like, and an arbitrary viewpoint video according to a user's request can be generated and presented in real time.

  In order to generate and present an arbitrary viewpoint video from images photographed from a plurality of viewpoints by processing on a computer, (1) all cameras should be able to see the same area (target object), (2) It is necessary to obtain the positional relationship between the cameras.

  Although it is possible to generate a virtual viewpoint video shot by a virtual camera at an arbitrary position between live-action cameras by image processing based on multi-view video shot by multiple cameras with different viewpoints focusing on the same target In order to generate a highly accurate virtual viewpoint video, it is necessary to accurately determine the positional relationship between the live-action cameras. There is a geometry parameter (Geometry Parameters) as a parameter indicating the positional relationship between live-action cameras. The geometry parameter is expressed by a determinant and is called a fundamental matrix or F matrix.

  Many methods for obtaining a fundamental matrix representing the positional relationship between live-action cameras have been proposed. For example, checker patterns in a three-dimensional space are simultaneously observed by several (adjacent) cameras, feature points on the checker pattern image (for example, intersection positions of black and white patterns) are extracted, and the feature points are associated with each other. A method of estimating a fundamental matrix by performing the above is known (for example, Non-Patent Document 1).

  However, (1) In an actual shooting site, the corresponding points cannot always be accurately obtained from the shot checker pattern image due to the influence of illumination light (for example, reflected light). It is not easy to estimate (expressed as F matrix). (2) Also, when observing the same checker pattern plate with a large number of cameras with different viewpoints, the observation camera often does not face the checker pattern plate. Therefore, due to pattern deformation on the observed checker pattern image, There is a problem that it is not easy to accurately detect the feature point positions in the checker pattern image.

(3) Further, when a large number of cameras are arranged so as to surround the same object (subject), not only the photographed checker pattern is deformed, but also when the flat checker pattern is used, the checker pattern is changed to 360 degrees. Even if they are arranged so as to surround the periphery, all the cameras cannot photograph the same checker pattern at the same time. At this time, it is necessary to divide the multi-viewpoint camera into several groups and shoot the checker pattern by dividing the checker pattern into a plurality of times with a plurality of cameras, which requires much labor and time. (4) Further, when performing camera calibration, it is clear about the process for determining whether or not the checker image patterns for estimating the F matrix have been sufficiently recorded and whether the spatial positions of these checker patterns are appropriate. There is a problem that the accuracy of the F matrix calculated as a result is not guaranteed.
RYTsai: An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, pp.364-374, (1986)

  The present invention has been made in view of the above-described problems, and performs highly accurate calculation of geometry parameters, which are parameters indicating the positional relationship between live-action cameras that constitute a multi-view video shooting system. It is an object of the present invention to provide a camera calibration apparatus, a camera calibration method, and a computer program that realize high-speed and highly accurate camera calibration.

  The present invention uses a light-emitting sphere that does not easily affect the illumination environment or the like as a calibration jig, estimates the F matrix while detecting and correlating the center position of the sphere in real time, and calculates the estimated F matrix and corresponding points By evaluating errors, positional relationships, etc., the number of corresponding points required and the position of the sphere to be placed in the three-dimensional space are indicated, and a more accurate F matrix is automatically estimated.

  Since the shape of the sphere does not depend on the camera observation viewpoint position, it can be observed by all cameras at various viewpoint positions. In addition, it is possible to perform camera calibration while observing and shooting a spherical image with a multi-view video shooting system having a synchronization function and recording it on a recording medium such as a memory or a hard disk on each personal computer. Then, the coordinate position (feature point) of the sphere center in each frame image is extracted by an image processing method on each personal computer, and a fundamental matrix is estimated by association between images using the extracted feature points. Furthermore, the accuracy evaluation of the estimated F matrix is executed, the number of corresponding points and the appropriate position of the corresponding points, that is, the spatial position of the sphere, are automatically presented, and high-speed and highly accurate inter-camera parameters are automatically An apparatus and method is provided for implementing dynamic estimation.

The first aspect of the present invention is:
A calibration processing device that executes parameter calculation processing applied to adjustment between multi-viewpoint image capturing cameras or correction processing of an acquired image by a multi-viewpoint image capturing camera,
An image input unit for inputting video data of a plurality of cameras obtained by photographing moving spheres from different viewpoint directions;
A feature point extraction unit that executes a process of extracting a sphere center position as a feature point from a plurality of captured image frames constituting video data of a plurality of cameras input in the image input unit;
An association processing unit that executes an association process of the sphere center position extracted by the feature point extraction unit as a feature point in the corresponding frame of each camera;
An F matrix calculation unit that calculates an F matrix [fundamental matrix] as a calibration parameter based on the feature point correspondence data associated in the association processing unit;
The feature point extraction unit obtains at least eight sphere center position data necessary for calculating a ratio of nine elements of the F matrix from a plurality of captured image frames of each camera;
The association processing unit executes an association process of eight or more spherical center positions extracted by the feature point extraction unit,
The F matrix calculation unit is configured to perform element value calculation of a fundamental matrix as a calibration parameter based on eight or more feature point correspondence data associated with each other. In the device.

  Furthermore, in one embodiment of the calibration processing apparatus of the present invention, the sphere is a light emitting sphere, and the feature point extraction unit calculates a sphere center position based on edge information of the sphere photographed in the frame. It is the structure which performs.

  Furthermore, in an embodiment of the calibration processing apparatus of the present invention, the F matrix calculation unit sets an epipolar line based on the calculated F matrix, and sets the epipolar line and a spherical center position corresponding to the feature point. The distance is calculated individually or as an average distance, and when the calculated distance is larger than a predetermined threshold, the F matrix recalculation process is executed based on feature point correspondence data including a new feature point. It is characterized by that.

  Furthermore, in an embodiment of the calibration processing apparatus of the present invention, the F matrix calculation unit performs a spherical center position in the captured image frame, that is, a feature point distribution state determination process, and feature point uneven distribution is confirmed. In this case, the F matrix recalculation process is performed based on feature point correspondence data including feature points additionally set at positions where the uneven distribution is eliminated.

  Furthermore, in an embodiment of the calibration processing apparatus of the present invention, the feature point extraction unit executes a center position calculation process based on the edge information only when an edge of the entire sphere exists in the imaging frame. It is characterized by being.

Furthermore, the second aspect of the present invention provides
It is a calibration processing method for executing parameter calculation processing applied to adjustment between multi-viewpoint image capturing cameras or correction processing of acquired images of multi-viewpoint image capturing cameras.
An image input step for inputting video data of a plurality of cameras obtained by photographing moving spheres from different viewpoint directions;
A feature point extracting step of extracting a sphere center position as a feature point from a plurality of captured image frames constituting video data of a plurality of cameras input in the image input step;
An association processing step for performing a feature point association process based on the spherical center position extracted in the feature point extraction step;
An F matrix calculation step of calculating an F matrix [fundamental matrix] as a calibration parameter based on the feature point correspondence data associated in the association processing step;
The feature point extraction step acquires at least eight sphere center position data necessary for calculating a ratio of nine elements of the F matrix from a plurality of captured image frames of each camera;
The association processing step executes an association process of eight or more spherical center positions extracted in the feature point extraction step,
The F matrix calculation step performs a calculation of element values constituting a fundamental matrix as a calibration parameter based on eight or more feature point correspondence data associated with each other. It is in.

  Furthermore, in an embodiment of the calibration processing method of the present invention, the sphere is a light emitting sphere, and the feature point extraction step performs a sphere center position calculation process based on edge information of the sphere photographed in the frame. It is characterized by performing.

  Furthermore, in an embodiment of the calibration processing method of the present invention, the F matrix calculation step sets an epipolar line based on the calculated F matrix, and a distance between the set epipolar line and the spherical center position as the feature point Are calculated individually or as an average distance, and when the calculated distance is larger than a predetermined threshold value, an F matrix recalculation process based on feature point correspondence data including a new feature point is executed.

  Furthermore, in one embodiment of the calibration processing method of the present invention, the F matrix calculation step executes a spherical center position in the captured image frame, that is, a feature point distribution state determination process, and feature point uneven distribution is confirmed. In this case, the F matrix recalculation process is performed based on the feature point correspondence data including the additional feature point additionally set at the position where the uneven distribution is eliminated.

  Furthermore, in an embodiment of the calibration processing method of the present invention, the feature point extraction step executes the center position calculation process based on the edge information only when the edge of the entire sphere exists in the shooting frame. It is characterized by.

Furthermore, the third aspect of the present invention provides
A computer program for executing parameter calculation processing applied to adjustment between multi-viewpoint image capturing cameras or correction processing of acquired images of a multi-viewpoint image capturing camera,
An image input step for inputting video data of a plurality of cameras obtained by photographing moving spheres from different viewpoint directions;
A feature point extracting step of extracting a sphere center position as a feature point from a plurality of captured image frames constituting video data of a plurality of cameras input in the image input step;
An association process step of executing an association process of the sphere center position extracted in the feature point extraction step;
An F matrix calculation step of calculating an F matrix [fundamental matrix] as a calibration parameter based on the feature point correspondence data associated in the association processing step;
The feature point extracting step is a step of acquiring spherical center position data of at least eight points necessary for calculating a ratio of nine elements of the F matrix from a plurality of captured image frames of each camera;
The association processing step is a step of executing association processing of eight or more spherical center positions extracted in the feature point extraction step;
The F matrix calculation step is a step of executing calculation of element values constituting a fundamental matrix as a calibration parameter based on eight or more feature point correspondence data associated with each other.・ It is in the program.

  The program of the present invention is, for example, a program that can be provided by a storage medium or a communication medium provided in a computer-readable format to a general-purpose system capable of executing various program codes. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the present invention, the light emitting sphere is photographed by each camera in the calibration process for executing the parameter calculation process applied to the adjustment process between the multi-viewpoint image capturing cameras or the correction process of the acquired image by the multi-viewpoint image capturing camera. Therefore, it is possible to easily detect the sphere and acquire the center position of the sphere even in a shooting site where the lighting environment is unstable, and perform the matching process using the sphere center position as the feature point for high accuracy. The F matrix can be calculated. In other words, it is possible to calculate a fundamental matrix as a geometry parameter (Geometry Parameters), which is a parameter indicating the positional relationship between live-action cameras that constitute a multi-view video shooting system, with high accuracy and high speed. Accurate calibration is possible.

  Furthermore, according to the configuration of the present invention, the light emitting sphere is used as a calibration jig, the F matrix is estimated while detecting and correlating the center position of the sphere in real time, and the error between the estimated F matrix and the corresponding point By evaluating the position relation and the like, it is possible to indicate the number of necessary corresponding points and the position of the sphere to be placed in the three-dimensional space, and to efficiently calculate a more accurate F matrix.

  Furthermore, according to the configuration of the present invention, a spherical image can be observed and photographed by a multi-viewpoint video photographing system having a synchronization function, and camera calibration can be performed while recording in a recording medium such as a memory or a hard disk on each personal computer. It is possible to estimate the fundamental matrix by extracting the coordinate position (feature point) of the sphere center in each frame image and matching the images using the extracted feature points by image processing method on each personal computer. Evaluation of matrix accuracy, determination of the number of corresponding points and appropriate positions of corresponding points, presentation, etc. can be executed on a PC, for example, and high-speed and high-precision calibration processing is executed on a simple system such as a PC. It becomes possible to do.

  Hereinafter, a camera calibration device, a camera calibration method, and a computer program according to the present invention will be described in detail with reference to the drawings.

  First, with reference to FIG. 1, a multi-viewpoint camera shooting process and a process of generating a virtual viewpoint video using a live-action video shot by a multi-viewpoint camera will be described. The subject 100 is photographed with a plurality of cameras arranged around the subject 100. Each camera captures an image of the subject 100 from a different viewpoint. The photographed image is acquired according to the number of cameras to be arranged. However, real images between the cameras cannot be acquired. Accordingly, an image between the cameras, that is, an image of the virtual viewpoint is generated by image processing based on the photographed image.

  For example, when a camera is not disposed between the camera A101 and the camera B102, an image between the camera A101 and the camera B102 is generated based on the two photographed images 111 and 112 of these cameras. This can be generated by a process for generating an image that is not actually captured, so-called view interpolation process.

  View interpolation is a technique for generating an image that can be seen from a viewpoint without an actual camera from images from a plurality of cameras. For example, [S. M. Seitz and C. R. Dyer, “View Morphing,” Proc. SIGGRAPH 96, ACM, 1996 pp. 21-30. ]It is described in. According to view interpolation, a viewpoint image without a camera can be generated based on actual acquired images based on a plurality of cameras.

  For example, in order to generate the virtual viewpoint camera image 121 as if it was captured by the virtual camera 113 between the camera A101 and the camera B102, the difference in characteristics and the relative positional relationship between the camera A101 and the camera B102 are obtained. It is necessary to perform various corrections in the generation of the virtual viewpoint image based on the parameters as the information. That is, it is necessary to perform adjustment that accurately grasps the difference in characteristics of the camera that captures the photographed image, that is, camera calibration.

  As shown in FIG. 2, the correction processing applying the calibration parameters necessary for generating the virtual viewpoint video includes a distortion aberration parameter including a distortion coefficient, a distortion center, and an aspect ratio. Camera internal / external parameters based on distortion parameter correction based on, internal parameters [A] consisting of scale factor, twist, and image center data, and external parameters [T, R] consisting of translation [T] and rotation [R] vectors There is correction. Regarding camera distortion parameters, generally, checker patterns, straight lines, and the like are photographed by a plurality of cameras that perform calibration, and adjustment parameters between the cameras are estimated based on photographed images such as checker patterns and straight lines.

  The camera internal / external parameters [A, T, R] are expressed by a determinant indicating geometry parameters as parameters indicating the positional relationship between the live-action cameras. This is called a fundamental matrix (F matrix). The F matrix is expressed by the following formula (Formula 1).

  Several methods have already been proposed for obtaining a fundamental matrix between cameras represented by the above equation. A general method for obtaining a Fundamental matrix is briefly described.

  FIG. 3 shows a general F matrix parameter estimation method. First, as shown in FIG. 3A, an image pattern (for example, a checker pattern) as a calibration jig in a three-dimensional space is photographed several times by cameras 1, 231 and cameras 2, 232 and photographed. The feature points are extracted from the images I1, 241 (camera 1, 231 actual image) and I2, 242 (camera 2, 232 actual image). As characteristic points, for example, intersections m11, m12,... And m21, m22,.

  After the feature point extraction, as shown in FIG. 3B or the following formula, the F matrix is obtained by matching the extracted feature points m1i and m2i (i = 1, 2,... N). Can be estimated.

In the above equation, by calculating f as an eigenvector corresponding to the minimum eigenvalue of W ^T W, the elements of the F matrix can be obtained.

  FIG. 4 is a diagram showing a camera calibration procedure when a checker pattern is used as a calibration jig in order to actually estimate the F matrix.

  First, in step S101, a checker pattern as a calibration jig is photographed by the camera 1 and the camera 2 from different viewpoints.

  Next, in step S102, feature points m1 (x, y) and m2 (x, y) in the checker pattern images I1 (x, y) and I2 (x, y) photographed by the cameras 1 and 2 are extracted. In step S103, association processing between each extracted feature point m1 (x, y) and m2 (x, y) is performed.

  The association process is a matching process for associating pixels in a plurality of images obtained by photographing the same object. Conventionally, “association” methods that are often used include, for example, pixel-based matching, area-based matching, and feature-based matching. Pixel-based matching is a method of searching for correspondence of points in one image as it is in the other image. Area-based matching is a method of searching for a correspondence between points in one image using a local image pattern around the point when searching for the other image. Feature-based matching is a method of extracting features such as shading edges from images and performing association using only features between images. Using these methods, pixels are associated based on feature points extracted from a plurality of images.

Finally, in step S104, the F matrix to which the above-described equation (Equation 2) is applied is calculated based on the acquired plurality of corresponding points. That is, by calculating f as an eigenvector corresponding to the minimum eigenvalue of W ^T W by associating the extracted feature points m1i and m2i (i = 1, 2,... N), F A matrix, that is, a fundamental matrix (= F parameter) is obtained.

  However, the above-described method has a problem in that the feature point extraction accuracy is lowered when the camera and the checker pattern plate are not facing each other. In addition, there may be cases where checker patterns cannot be observed simultaneously by each camera due to the arrangement of multiple cameras, and the association of feature points between images is affected by the quality of the captured image, which takes time. .

  As a technique for solving these problems, in the present invention, a moving light emitting sphere is used as a calibration tool instead of a planar checker pattern, and a plurality of cameras for photographing the light emitting sphere from different directions are installed. Is recorded, a fundamental matrix (= F parameter) is calculated based on the recorded image, and camera calibration is executed. According to the configuration of the present invention, efficient and high accuracy can be achieved without causing a problem that the feature point extraction accuracy is lowered by photographing the planar checker pattern from different directions and the checker pattern cannot be observed simultaneously by each camera. Camera calibration can be performed.

  FIG. 5 shows a configuration example in which the moving luminescent sphere is used as a calibration tool and the issuing sphere is photographed from different viewpoints. Further, FIG. 6 shows an example of the configuration of a synchronized recording processing apparatus for recorded video of each camera.

  As shown in FIG. 5, cameras 301-1 to 301-N that execute camera calibration are arranged around a light emitting sphere 300 as a calibration tool, and the light emitting sphere 300 is moved, for example, to display an image of 30 frames / second. Take a picture. In the figure, acquired images (one frame) 302-1, 2, n of cameras 1, 301-1, camera 2, 301-2, and cameras n, 301-n are shown.

  The advantages of using a light-emitting sphere as shown in FIG. 5 are (1) that the sphere can be easily detected from an image observed in an arbitrary background / environment, and (2) that the sphere has the same shape with a camera at an arbitrary viewpoint position. It can be observed as (circular), (3) the center position of the sphere in the image can be accurately estimated, and (4) the corresponding point processing between images using the center of the sphere extracted from the time series image can be easily performed. It is in.

In order to estimate the F matrix, it is necessary to determine the values of the nine elements (f ₁₁ , f ₁₂ ,... F ₃₃ ) that constitute the F matrix shown in Equation 2 above. This substantially corresponds to obtaining the ratio of nine elements of the F matrix, and corresponds to obtaining the solution of an equation with eight unknowns. Therefore, it is necessary to set a relational expression of a large number of corresponding points (at least 8 points or more).

  In the configuration of the present invention, a moving luminescent sphere is photographed, the center position of the luminescent sphere is extracted as a feature point for each frame along the time axis, and a plurality of feature points extracted from the frame along the time axis. Are used for matching between images as if they were extracted from a single shot image. For example, if photographing is performed at 30 frames / second for 5 seconds, 30 × 5 = 150 frames can be photographed by each camera, and 150 feature points (= sphere center position) can be extracted.

  In order to perform such feature point extraction and association automatically, as shown in FIG. 6, a synchronized recording apparatus for a plurality of camera images having a network synchronization mechanism is applied.

  The cameras 301 of the cameras 1 to n that shoot the light emitting sphere 300 as a calibration tool from different viewpoints receive the synchronization signals (horizontal and vertical synchronization signals) output from the synchronization signal generator 320 at the external synchronization input terminals of the cameras. And a plurality of images from multiple viewpoints observed by a plurality of cameras 1 to n are output as synchronized images.

  Each camera output video is, for example, a video signal such as a Y / C signal as a difference signal (C) between a luminance signal (Y) and an RGB color component signal, and each camera sets a video signal corresponding to each camera. Input to the AD converter 330. The AD converter 330 converts the video signal input from the camera 301 into digital data, and further converts it into compressed data (for example, motion JPEG) as necessary, and records the processed data as data. Output to a client PC (PC1 to PCn) 350 and an image monitor 360 as image recording processing apparatuses to be executed.

  A client PC (PC1 to PCn) 350 as an image recording processing apparatus includes a recording medium 356 for recording data, such as a hard disk, a DVD, and a CD.

  The video signal input from the AD converter 330 to the client PC (PC1 to PCn) 350 is input to the video signal processing unit 355 in the client PC (PC1 to PCn) 350. The video signal processing unit 355 is configured by, for example, a 1394 DV capture board.

  The plurality of PCs 1 to PCn constituting the client PC 350 record DV video from each camera on the hard disk 356 in each PC by DV capture software that can be executed on the general-purpose PC. Each client PC (PC1 to PCn) 350 and server PC 370 are connected by a network card 357 as a network interface attached to each PC and a network 380. The server PC 370 connects to each client PC (PC1 to PCn) 350. On the other hand, by outputting a recording start command, the client PCs 350 can start recording all at once.

  A configuration example of the video signal processing unit 355 configured by, for example, a 1394 DV capture board in the client PC 350 is shown in FIG.

  The IEEE 1394 port (hereinafter, 1394 port) 394 and the LINK / PHY 395 capture a digital video (DV) signal processed by the AD converter 330. The 1394 port 394 is an input port for capturing a DV signal, and the input DV signal is sent to the LINK / PHY 395. The LINK / PHY 395 primarily stores the input DV image data in the RAM 392 as a buffer means.

  The RAM 392 functions as an image data storage unit including a buffer for storing the captured DV image data. The image data primarily recorded in the RAM 392 is output and stored in the storage unit 356 such as an HDD or DVD via the storage unit I / F 396 under the control of the CPU 391. As described above, each of the client PCs (PC1 to PCn) 350 and the server PC 370 are connected by the network card 357 as a network interface mounted on each PC and the network 380, and each client PC (PC1) is connected to the server PC 370. By outputting a recording start command to .about.PCn) 350, each client PC 350 can start recording at the same time, that is, start data storage processing for the storage means 356 such as an HDD or a DVD.

  The CPU 391 functions as image data control means for controlling the entire apparatus related to capturing and storing image data and controlling the overall flow of DV image data. The CPU 391 controls the flow of image data in accordance with a program stored in the ROM 393.

  The program stored in the ROM 393 is, for example, DV capture software that can be executed on a general-purpose PC. The program is executed by a recording start command from the server PC, and the client PCs start recording all at once.

  As described above, the reference video signal (synchronization signal) from the synchronization signal generator 320 is input to the external synchronization terminals of the cameras 301-1 to 301-n, and the video output signals of the cameras 301-1 to n are converted into AD converters. 330 is converted into a digital video signal, and is stored in a recording medium such as a memory or a hard disk built in the personal computer 350 through a capture board. Since the timing until the start of image recording is not always the same depending on the execution environment of each personal computer, the recorded camera images are not normally synchronized. Therefore, when saving each camera video, the image frame number being recorded is reported to the server as measurement point data through the network, and the recorded video is recorded on the server using the measurement point data. Realized synchronization between camera images.

  Details of the synchronization adjustment process executed by the server PC will be described with reference to FIG. First, in step S301, the server PC reads the recording frame number data of each PC at each measurement point input from each client PC. This is, for example, data shown in the data example (a) of FIG.

  Here, it is assumed that the server PC has acquired the recording frame numbers at four measurement points (measurement points 1 to 4) from a plurality of client PCs (PC1 to PCn).

According to (a) a list of recording frame numbers of each PC at each measurement point input from each client PC in FIG.
At the total point 1, the recording process frame = 6,
At the total point 2, the recording processing frame = 10,
At the total point 3, the recording processing frame = 14,
At the total point 4, the recording processing frame = 19,
Indicates that the frame number is stored at each measurement point in accordance with the “save currently recorded frame number” command from the server PC.

  Next, in step S302, the server PC determines the frame No. at the PC having a large processing delay amount, here the total point 1. 3 is calculated with the PC 2 processing 3 as a reference PC.

As a result of this processing, (b) the amount of frame shift of the recorded video in each PC at each measurement point shown in FIG. 8 is obtained. In the example shown in FIG. 8, PC2 is the reference, and PC2 has a deviation amount = 0 at all measurement points. Further, for example, the shift amount of PC1 is obtained from the recording processing frame of PC1-the recording processing frame of PC2 at each measurement point. As a result, the shift amount of PC1 is
At the total point 1, the deviation amount = 6-3 = 3
At the total point 2, the deviation amount = 10−7 = 3
At the total point 3, the deviation amount = 14-11 = 3
At the total point 4, the deviation amount = 19−15 = 3
It becomes.

  For other PCs, PC3 to PCn, the recording frame No. Is obtained as the amount of deviation.

  Next, in step S303, a determination process of the deviation amount of each PC from the reference PC = PC2 is performed. This is performed, for example, by calculating an average value of the deviation amounts at each measurement point of each PC and calculating an integer value based on the average value by rounding off.

FIG. 8C shows data obtained by calculating an average value of deviation amounts at each measurement point of each PC. The amount of deviation of PC1 from PC2 is
(3 + 3 + 3 + 4) /4=3.25
It becomes.

FIG. 8D shows data obtained by rounding off the average deviation amount at each measurement point of each PC.
The displacement amount of PC1 with respect to PC2 is 3,
The shift amount of PCn with respect to PC2 is 2.
In this way, the deviation amount of each client PC from the reference PC (PC2) is determined.

  Next, in step S304, the server PC executes synchronization adjustment processing for each client PC based on the calculated deviation amount of each PC.

The processing frame No. at each time 1 to 4 of each client PC. It is executed as a process to unify. As shown in FIG. 8 (e), at times 1 to 4,
At time 1, all PC recording frames = 3
At time 2, all PC recording frames = 7
At time 3, all PC recording frames = 11
At time 4, all PC recording frames = 15
The following settings are executed.

  For example, PC1 setting frame No. Indicates the setting frame No. of the reference PC (PC2). Therefore, the value obtained by subtracting the difference 3 from the PC1 setting frame No. is used as a new frame No. in PC1. Set as. PCn setting frame No. Indicates the setting frame No. of the reference PC (PC2). Therefore, the value obtained by subtracting the difference 2 from the PCn setting frame No. is set as a new frame No. in PC1. Is set at each time.

  By executing the synchronization adjustment process in each client PC in this way, accurate inter-frame synchronization is possible.

  Through the above-described processing, each camera video can be acquired in complete synchronization. The calibration apparatus according to the present invention executes the association process of the light emitting sphere center as a feature point for each of the above-described synchronized image capturing frames, and according to the procedure described with reference to FIG. Find the F matrix.

  FIG. 9 shows a time-series image (t = 1,..., K) of a light-emitting sphere observed with one camera (camera 1, 301-1). By automatically extracting the center of the sphere in each frame image from all the camera images and performing the association, the F matrix can be estimated at high speed and with high accuracy.

  In the configuration of the present invention, the center point of the light emitting sphere is a feature point for executing the association process. The processing for obtaining the center point of the light emitting sphere is executed in each frame of each camera image. In order to determine the elements of the F matrix, it is necessary to associate at least eight feature points as described above, and feature points are extracted from the captured image of the camera that performs calibration in at least eight frames. The determination process of the center position of the light emitting sphere performed as the feature point extraction process will be described with reference to FIG.

  The procedure of the center point position determination process of the luminous sphere executed as the feature point extraction process in the configuration of the present invention will be described according to the process flow of FIG. FIG. 10 shows a processing flow for obtaining the center pixel position of a sphere from one frame image. First, in step S501, a captured image frame of a light emitting sphere is input.

  In step S502, the edge of the sphere is detected from the input image. When the coordinate value of the edge pixel is in contact with the four sides of the acquired frame image, it is determined that a part of the sphere is missing, and feature points are not extracted in that frame (time). This determination process is step S503.

  When the entire light emitting sphere is observed as an image in the input frame (step S503: Yes), in step S504, the initial center position of the sphere is determined by projection processing based on the coordinate value of the edge pixel. Specifically, the difference between the xy coordinate values of the pixels on the left and right edges of the sphere is projected on the X axis and the Y axis, respectively, and distributions for X and Y are obtained. ) And size (radius).

  In step S505, the estimated initial center and size are applied to a circular model, and pixel points on edges that deviate significantly from the model are deleted as exceptional points (Outliers). Finally, in step S506, the edge pixel point after the exception point (Outlier) is deleted is applied to the circular model, and the center position (and size) of the sphere is determined.

  FIG. 11 is a diagram illustrating a processing example in which the center position of each sphere is detected from the time series images observed by the adjacent cameras, and set as if they were feature points extracted from one image.

  Assume that an F matrix for camera calibration for adjustment processing between the cameras 1 and 501 and the cameras 2 and 502 for photographing the light emitting sphere 500 from different directions is obtained. In this case, as described above, it is necessary to extract a plurality of feature points (at least 8) from a plurality of frames.

  In the configuration of the present invention, the feature point is the center position of the light emitting sphere in the light emitting sphere photographed image, and the center position of the light emitting sphere in each frame is extracted as the feature point. Each frame image of the shooting frames t = 1 to k of the cameras 1 and 501 and the cameras 2 and 502 is an image that has been subjected to the synchronization adjustment processing described with reference to FIG. 6, and each corresponding frame is a synchronized image. It is. By executing the spherical center position coordinate determination process described with reference to FIG. 10 for k frames of t = 1 to k, k feature points are obtained for each camera. However, as described above, when the spherical image protrudes from the frame image, the center cannot be obtained, so that there are a maximum of k feature points.

  As described above, by obtaining at least 8 feature points and executing the association process, it is possible to determine the elements of the F matrix consisting of 9 elements. Eight frames capable of obtaining at least the center position of the light emitting sphere are selected from the synchronized frame image, and the feature point (sphere center position) is obtained. When they are combined into one, feature point distribution data 511 and 512 shown in FIG. 11 are obtained.

  The feature point distribution data 511 includes t = 1, 2,. . The distribution of feature points (sphere center position) of each frame of k is shown. p1 (1) is the sphere center position in the frame at t = 1, p1 (2) is the sphere center position in the frame at t = 2, and p1 (k) is the sphere center position in the frame at t = k. is there.

  The feature point distribution data 512 includes t = 1, 2,. . The distribution of feature points (sphere center position) of each frame of k is shown. p2 (1) is the sphere center position in the frame at t = 1, p2 (2) is the sphere center position in the frame at t = 2, and p2 (k) is the sphere center position in the frame at t = k. is there.

  Corresponding feature points of these feature point distribution data 511 and 512, that is, p1 (1) and p2 (1), p1 (2) and p2 (2),... P1 (k) and p2 (k) The association process for the feature points is executed according to the process procedure of the flow described above with reference to FIG. 4, and the F matrix can be obtained by applying the above formula (Formula 2).

  FIG. 12 is a flowchart illustrating a processing procedure for obtaining a spherical center position as a feature point from each image frame based on a time-series image including a plurality of frames. Each processing step in the flowchart shown in FIG. 12 will be described. Here, the camera No. of the feature point extraction process target. M, and camera no. Let m be the number of image frames taken by m.

  In step S701, an observation (captured) image Im (t) (t = 1, 2,..., K) of the m-th (m = 1, 2,..., N) camera is input to the calibration processing apparatus. .

  In step S702, t = 1 is sequentially set as an initial value of a processing frame for executing processing for each frame. In step S703, the camera image Im (t) is read, and in accordance with the procedure described above with reference to FIG. 10, the detection of the sphere in the read frame and the estimation process of the center position thereof are executed.

  When the image of the sphere is not detected in the processing target image or the entire sphere is not included (step S704: No), the sphere center position estimation process is not executed as described above. In step S712, Pm (t) = Null is stored in the memory as the frame correspondence data indicating that the result, that is, the center position has not been obtained.

  When the entire sphere is detected in the image, estimation of the center position of the sphere is executed according to the method of FIG. 10 described above, and the result is stored as the center position data Pm (t) corresponding to the frame in step S705. Save to.

  Here, Pm (t) means the spherical center position of the t (t = 1,..., N) frame in the m-th camera image. In step S706, it is determined whether or not all frames have been processed, that is, [t <k? ] Is executed, and if there is an unprocessed frame, update processing of [t = t + 1] is executed in step S711, and spherical center position estimation processing and storage processing are executed in step S703 and thereafter. .

  The process shown in FIG. 12 is executed for all N cameras, and the center position Pm (t) of the ball (light emitting sphere) in a plurality of frames is obtained from all camera images.

  The set of feature point distribution data shown in FIG. 11 is configured and configured using a set of frames excluding a frame for which the center position is not obtained from a frame taken by an adjacent camera that performs calibration processing or a remote camera. The feature point association process of the corresponding frame in the feature point distribution data is executed to calculate the F matrix. The calculation process of the F matrix by the feature point association is executed by the same method as described above with reference to FIGS. When the F matrix is determined, it is possible to correct a captured image of each camera using the F matrix, and it is possible to generate a highly accurate virtual viewpoint camera image from actual captured images of a plurality of cameras.

  FIG. 13 is a block diagram illustrating the functional configuration of the calibration processing apparatus of the present invention. Various processes executed in the calibration processing apparatus of the present invention can be executed as processes according to a computer program. Specifically, various processes are executed under the control of the CPU as the control means. FIG. 13 is a block diagram illustrating functions executed by the calibration processing apparatus of the present invention, individually explaining the processes executed by the CPU as the control means. A specific hardware configuration will be described later.

  The block diagram of FIG. 13 will be described. The image input unit 601 inputs image data obtained by photographing a light emitting sphere. In many cases, the calibration process is intended to adjust an image between two cameras, and frame images of two cameras obtained by photographing a light emitting sphere from different viewpoints are input. This input frame image is an image acquired in the configuration described above with reference to FIG. 6, and each frame image of each camera is synchronized image data.

  The feature point extraction unit 602 obtains the center position of the light emitting sphere of each frame as a feature point according to the procedure described with reference to FIGS. The association processing unit 603 uses the feature point distribution data (for example, feature point distribution data 511 and 512 shown in FIG. 11) consisting of a plurality of feature points obtained based on the images captured by the cameras. The association process is executed. This association process is executed by, for example, Pixel-based matching, Area-based matching, Feature-based matching, and the like.

  Once the association process is executed, the F matrix calculation unit 604 then executes an F matrix calculation process based on the association data. The F matrix calculation processing based on the association data is executed according to the processing described above with reference to FIGS. 3 and 4, and the F matrix is obtained according to the above-described equation (Equation 2).

  The obtained F matrix is output to the virtual viewpoint image generation unit 605, and calibration of acquired images of the two cameras based on the F matrix, that is, correction processing is executed, and the actual between the two cameras based on the corrected image. An image of a virtual viewpoint camera that has not been shot is generated.

  Note that calibration as adjustment processing of the camera itself may be executed based on the calculated F matrix. Whether to correct the acquired image or to adjust the camera itself is a matter of choice.

  As explained earlier, from a multi-view video shot by multiple cameras with different viewpoints focusing on the same target, an arbitrary virtual viewpoint video as if shot with a virtual camera is generated between those live-action cameras In order to express it, it is necessary to accurately determine the positional relationship between live-action cameras. The fundamental matrix (Fundamental matrix) as a parameter indicating the positional relationship between the live-action cameras enables the accurate positional relationship between the two cameras to be acquired, and correction and composition of the captured image based on the determined positional relationship can be executed with high accuracy. It is possible to generate a highly accurate image of a virtual viewpoint camera that is not actually captured between the two cameras.

  Next, an optimal F matrix calculation processing procedure executed by the calibration processing apparatus of the present invention will be described with reference to a flowchart shown in FIG.

  In step S801, first, the luminous sphere is placed at a position where each camera to be calibrated can be observed. For example, the light emitting sphere is placed and moved where the optical axes of the cameras intersect. In step S802, the light-emitting sphere moving by each camera is photographed. In this photographing process, as described with reference to FIGS. 5 to 8, the photographing frames of the cameras are acquired as synchronized frames.

  In step S803, the spherical center position as a feature point is detected from the captured image of each camera. The detection of the sphere center position is executed according to the procedure described with reference to FIGS.

  In step S804, the detected spherical center position is used as a feature point of each camera image, and a process for associating the calibration target camera images is performed. The feature point association process is executed according to the procedure described with reference to FIGS.

  In step S805, it is determined whether the number of corresponding points has reached 8. As described above, in order to calculate the ratio of the nine elements of the F matrix, it is necessary to associate eight feature points. If the number of corresponding points is less than 8, the process returns to step S801, and images taken by a plurality of cameras to be calibrated are acquired while moving the light emitting sphere to change the position of the sphere.

  When the number of corresponding points reaches 8 or more, the process proceeds to step S806, and the F matrix is calculated based on the corresponding points of the feature points. The calculation of the F matrix is performed according to the equation (Equation 2) as described with reference to FIGS. In addition, a photographed image by the camera is photographed for several seconds at, for example, 30 frames / second in one photographing process, 100 or more frames are acquired, and feature point extraction processing is executed from these many frames. The

  Therefore, feature points to be associated with each other are randomly selected from 100 or more feature points, and feature point association processing is performed based on the selected points. Further, calculation of the F matrix is executed based on the associated feature points. However, since the association of feature points is only performed based on the correlation between images, the association process is often executed by mistake. If there is such an error, an accurate F matrix cannot be obtained. Accordingly, it is necessary to perform processing for obtaining an accurate F matrix by excluding combinations of feature points that are incorrectly associated.

  The processing after step S807 is processing for eliminating feature points that are determined to be inappropriate for calculation of the F matrix, such as feature points that are incorrectly associated.

  In step S807, an epipolar line is calculated using the calculated F matrix. In step S808, the distance (error) between the corresponding point used for F matrix estimation and the epipolar line is calculated. If the error is larger than a predetermined threshold, the sphere position is changed, the center is detected, and a new corresponding point is obtained.

  With reference to FIG. 15 and FIG. 16, the processing of steps S807 and 808, that is, generation of epipolar lines and calculation processing of distances (errors) between corresponding points used for F matrix estimation and epipolar lines will be described. First, generation of epipolar lines will be described. An epipolar line is a line set based on the positional relationship between two cameras, and is generally generated by parallax detection by a stereo method.

  The principle of the stereo method will be briefly explained. The stereo method associates pixels in a plurality of images obtained by photographing the same object from two or more viewpoints (different gaze directions) using a plurality of cameras, thereby determining the position of the measurement object in the three-dimensional space. It is what you want. For example, the same object is photographed from different viewpoints by the reference camera and the detection camera, and the distance of the measurement object in each image is measured by the principle of triangulation.

  FIG. 15 is a diagram for explaining the principle of the stereo method. The reference camera (Camera 1) and the detection camera (Camera 2) capture the same object from different viewpoints. Consider obtaining the depth of a point “mb” in an image taken by a reference camera.

  The objects that appear at the point “mb” in the image captured by the reference camera are “m1”, “m2”, and “m3” in the images captured by the detection camera capturing the same object from different viewpoints. It will be developed on a straight line. This straight line is referred to as an epipolar line Lp.

  The position of the point “mb” in the reference camera appears on a straight line called “epipolar line” in the image by the detection camera. As long as the point P to be imaged (a point existing on a straight line including P1, P2, and P3) exists on the line of sight of the reference camera, regardless of the depth, that is, the distance from the reference camera, on the reference image It appears at the same observation point “mb”. On the other hand, the point P on the image captured by the detection camera appears on the epipolar line at a position corresponding to the distance between the reference camera and the observation point P.

  In the configuration of the present invention, feature points are extracted based on the captured images of the two cameras to be calibrated, and the F matrix is calculated in step S806 based on the extracted feature points. This F matrix is a parameter indicating a positional relationship between two cameras, and an epipolar line can be set based on the positional relationship parameter.

  The setting of epipolar lines will be described with reference to FIG. FIGS. 16A and 16B are diagrams collectively showing feature points (sphere centers) obtained in a plurality of frames taken by the camera to be calibrated. This corresponds to the feature point distribution data 511 and 512 shown in FIG. Further, (b) shows epipolar lines L1, L2, Li set based on the F matrix obtained in step S806 of the flow shown in FIG. These epipolar lines are epipolar lines set on the basis of, for example, eight feature points that have been subjected to the association process for calculating the F matrix.

  For example, from the F matrix estimated using corresponding points mli (i = 1,..., P) and m2i (i = 1,..., P) in the images of the cameras 1 and 2, the following expression (Expression 3) (Equation 4), for each feature point (corresponding point) m1i (i = 1,..., P) in the camera 1 image, an epipolar line Li on the camera 2 image is obtained. Similarly, an epipolar line in the camera 1 image is obtained for each feature point in the camera 2 image.

  Based on the above formula (formula 3), one straight line (epipolar line) shown in the following formula (formula 4) is set.

  However, the feature points used for F matrix calculation are feature points randomly selected from a large number of feature points existing in a large number of frames taken by two cameras. If the F matrix is accurately calculated, the feature point will be on the epipolar line set based on the F matrix, but the feature point is not accurately extracted or correctly associated. Then, the feature point is at a position shifted from the epipolar line. When the deviation is large, it is determined that the feature point association processing is inaccurate and the F matrix is also inaccurate.

  As an index for determining the accuracy of the F matrix, the distance Di between the feature point and the epipolar line is calculated. As shown in FIG. 16, the epipolar lines L1, L2,..., Li set based on the F matrix and the characteristic points m21, m22,. . distances D1, D2,. . Find Di.

  In step S809, each calculated distance D1, D2,. . Di is compared with a predetermined threshold value. If there is a feature point m2i having a distance greater than the threshold, in step S810, the feature point set, that is, the feature point set m1i, m2i of the camera 1 and camera 2 is excluded from the correspondence target, and the step In step S806, the F matrix is calculated again based on the association processing data of other feature points that can be associated.

  In step S809, each calculated distance D1, D2,. . If all of Di are equal to or less than a predetermined threshold value, in step S811, each calculated distance D1, D2,. . An average value of Di is calculated, and the average value is compared with a predetermined second threshold value. If the distance average value is equal to or greater than the second threshold value, in step S806 again, the F matrix is calculated again based on association processing data of other feature points that can be associated.

  In step S811, each calculated distance D1, D2,. . If the average value of Di is less than the second threshold value, the process proceeds to step S812, and the feature point distribution is determined. The feature point distribution determination process will be described with reference to FIG. When the feature point for executing the association process is biased to a part of the captured image frame, it is difficult to obtain an accurate positional relationship between the cameras. That is, it is difficult to accurately calculate the F matrix. Therefore, it is assumed that the feature point distribution for executing the association processing is scattered throughout the entire captured image frame, and in order to perform more accurate F matrix calculation, the feature point distribution is confirmed. A new feature point is set at the position, and the F matrix is calculated again.

  As shown in FIG. 17, the coordinate values of the corresponding point m1i (i = 1,..., P) on the image of the camera 1 and the corresponding point m2i (i = 1,. As shown in the following formula (Formula 5), each coordinate distribution (average value of x and y) is obtained.

  When the coordinate distribution of each camera obtained by the above equations is away from the image centers (xc1, yc1) and (xc2, yc2), it is determined that the feature points are unevenly distributed, and new feature points are added. I do. Specifically, the distance between the coordinate distribution value of each camera obtained by the above equations and the image center (xc1, yc1), (xc2, yc2) is compared with a predetermined third threshold value, and the threshold value is calculated. If it is large, it is determined that the feature point distribution is biased, and the process advances from step S813 to step S814 to set a necessary feature point position. The additional feature point to be instructed is set at a position to reduce the deviation of the coordinate distribution obtained in the above formula (Formula 5). For example, the points m1k and m2k shown in FIG.

  In step S815, the sphere is moved to a position where the instructed feature point can be acquired, and then the process returns to step S802 to execute camera shooting of the sphere again, add the additional feature point, F matrix calculation processing is executed.

  If it is determined in step S813 that the feature points are not biased, it is determined that there is no feature point bias, and the processing is ended with the obtained F matrix as the final F matrix.

  Thus, the processing procedure of image observation-> sphere detection-> sphere center estimation-> correspondence-> F matrix estimation-> epipolar line calculation-> error calculation between corresponding points and epipolar lines-> error value evaluation is as follows: Repeat until the error evaluation is below a certain threshold. Furthermore, when the above error value becomes smaller than a certain threshold value, it is determined whether or not the position of the corresponding point (that is, the spatial position of the sphere) is biased using disparity information between images. If the corresponding points are biased, the spatial position where the sphere should be placed is presented and the above-described repetitive operation is performed. As a result, highly accurate camera calibration can be performed even in field shooting where the lighting environment is unstable.

  As described above, according to the processing procedure described with reference to FIG. 14, in the process of calculating the F matrix by associating the spherical center position obtained from the spherical image acquired by the calibration target camera as the feature point, Even if an error occurs in selecting a feature point for which association setting is performed or an error occurs in the association processing, the error can be corrected. In addition, even when feature points are unevenly distributed, it is possible to check the uneven distribution state and calculate a new F matrix by adding feature points to positions where uneven distribution is not present, so that a more accurate F matrix can be calculated. Is possible. Accordingly, it is possible to perform camera calibration based on a high-precision F matrix, or correction processing of an acquired image, and generation of a high-precision virtual viewpoint image based on the correction processing.

  Next, a specific hardware configuration example of the calibration processing apparatus according to the present invention will be described with reference to FIG. A CPU (Central processing Unit) 901 is a processor that executes the processing program described with reference to the above-described flowcharts and an OS (Operating System). A ROM (Read-Only-Memory) 902 stores programs executed by the CPU 901 or fixed data as calculation parameters. A RAM (Random Access Memory) 903 is used as a storage area and work area for programs executed in the processing of the CPU 901 and parameters that change as appropriate in the program processing. The HDD 904 executes control of the hard disk, and executes storage processing and reading processing of various data and programs with respect to the hard disk.

  The bus 910 is configured by a PCI (Peripheral Component Internet / Interface) bus or the like, and enables data transfer with each module and each available power device via the input / output interface 911.

  The input unit 905 includes an image data input unit, a keyboard, a pointing device, and the like. The input unit 905 inputs image data acquired by a camera that performs calibration, and inputs various commands and data to the CPU 901. The output unit 906 is, for example, a CRT or a liquid crystal display that displays a captured image, a feature point extraction processed image, or a virtual viewpoint image generated based on a camera acquired image after F matrix calculation.

  The communication unit 907 executes communication processing with other devices. For example, the images of a plurality of cameras acquired in the system that executes the image synchronous acquisition process shown in FIG. 6 are input. Based on the input image, the above-described F matrix calculation process, virtual viewpoint image generation process, and the like are executed under the control of the CPU 901 as the control unit. Note that the image data to be processed may be input not only via the communication unit but also via an A / V input unit configured in the input unit 905, or an HDD connected to the drive 908. Alternatively, image data stored in a removable recording medium 909 such as a CD or DVD may be input as a processing target image.

  A drive 908 executes recording and reproduction of a removable recording medium 909 such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disc, a DVD (Digital Versatile Disc), a magnetic disc, and a semiconductor memory. The program or data is read from each removable recording medium 909, and the program or data storage process for the removable recording medium 909 is executed.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims section described at the beginning should be considered.

  The series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a storage medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  In addition to installing the program from a removable recording medium as described above, the program is wirelessly transferred from a download site to the computer, or transferred to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this way and install it in a storage medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary.

  As described above, according to the configuration of the present invention, in the calibration process for executing the parameter calculation process applied to the adjustment between the multi-viewpoint image capturing cameras or the correction process of the acquired image by the multi-viewpoint image capturing camera, Since the light-emitting sphere is photographed by each camera, it is possible to easily detect the sphere and acquire the center position of the sphere even in shooting scenes where the lighting environment is unstable, and the association processing is performed using the sphere center position as a feature point. By executing this, it is possible to calculate the F matrix with high accuracy. In other words, it is possible to calculate a fundamental matrix as a geometry parameter (Geometry Parameters), which is a parameter indicating the positional relationship between live-action cameras that constitute a multi-view video shooting system, with high accuracy and high speed. Accurate calibration is possible.

  Furthermore, according to the configuration of the present invention, a light emitting sphere that does not easily affect the lighting environment or the like is used as a calibration jig, the F matrix is estimated while the center position of the sphere is detected and associated in real time, and the estimated F By evaluating the error and positional relationship between the matrix and the corresponding points, the number of corresponding points required and the position of the sphere to be placed in the three-dimensional space are indicated, and a more accurate F matrix is efficiently calculated. It becomes possible.

  Furthermore, according to the configuration of the present invention, a spherical image can be observed and photographed by a multi-viewpoint video photographing system having a synchronization function, and camera calibration can be performed while recording in a recording medium such as a memory or a hard disk on each personal computer. It is possible to estimate the fundamental matrix by extracting the coordinate position (feature point) of the sphere center in each frame image and matching the images using the extracted feature points by image processing method on each personal computer. Evaluation of matrix accuracy, determination of the number of corresponding points and appropriate positions of corresponding points, presentation, etc. can be executed on a PC, for example, and high-speed and high-precision calibration processing is executed on a simple system such as a PC. It becomes possible to do.

It is a figure explaining the process which produces | generates a virtual viewpoint image | video using the multi-view camera imaging | photography process and the real image image | photographed with the multi-view camera. It is a figure explaining the various parameters applied to a calibration process. It is a figure which shows the general F parameter estimation method. It is the figure which showed the camera calibration procedure at the time of using a checker pattern as a calibration jig | tool in order to estimate F matrix. It is a figure which shows the structural example which image | photographs a light emission sphere from a different viewpoint, using a light emission sphere as a calibration tool. It is a figure explaining the synchronous recording structure of the several camera image | video provided with the network synchronization mechanism. It is a figure explaining the structural example of the video signal processing part comprised by a DV capture board. It is a figure explaining the detail of the synchronous adjustment process performed with server PC. It is a figure explaining the time-sequential image (t = 1, ..., k) of the light-emitting sphere observed with one camera. It is a figure explaining the procedure of the center point position determination process of the light emission sphere performed as a feature point extraction process. It is a figure explaining the process which detects the center position of each sphere from the time series image observed with the adjacent camera, and sets like the feature point extracted from one image. It is a figure which shows the flowchart which shows the process sequence which calculates | requires the spherical center position as a feature point from each image frame based on the time series image which consists of several frames. It is a block diagram explaining the functional structure of the calibration processing apparatus of this invention. It is a figure explaining the calculation process procedure of the optimal F matrix which the calibration processing apparatus of this invention performs. It is a figure explaining an epipolar line. It is a figure explaining the calculation process of the distance (error) of the production | generation of an epipolar line, and the corresponding point used for F matrix estimation, and an epipolar line. It is a figure explaining a feature point distribution determination process. It is a figure explaining the specific hardware structural example of the calibration processing apparatus which concerns on this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Subject 101,102 Camera 111,112 Actual image 113 Virtual viewpoint camera 121 Virtual viewpoint camera image 301 Camera 302 Actual image 301 Camera 320 Synchronization signal generating part 330 A / D converter 350 Client PC
355 Video signal processing unit 356 Storage unit 357 Network card 360 Image monitor 370 Server PC
380 Network 391 CPU
392 RAM
393 ROM
394 1394 port 395 LINK / PHY
396 Storage unit 500 Luminescent sphere 501, 502 Camera 511, 512 Feature point distribution data 601 Image input unit 602 Feature point extraction unit 603 Association processing unit 604 F matrix calculation unit 605 Virtual viewpoint image generation unit 901 CPU
902 ROM
903 RAM
904 HDD
905 input unit 906 output unit 907 communication unit 908 drive 909 removable recording medium 910 bus 911 input / output interface

Claims (11)

A calibration processing device that executes parameter calculation processing applied to adjustment between multi-viewpoint image capturing cameras or correction processing of an acquired image by a multi-viewpoint image capturing camera,
An image input unit for inputting video data of a plurality of cameras obtained by photographing moving spheres from different viewpoint directions;
A feature point extraction unit that executes a process of extracting a sphere center position as a feature point from a plurality of captured image frames constituting video data of a plurality of cameras input in the image input unit;
An association processing unit that executes an association process of the sphere center position extracted by the feature point extraction unit as a feature point in the corresponding frame of each camera;
An F matrix calculation unit that calculates an F matrix [fundamental matrix] as a calibration parameter based on the feature point correspondence data associated in the association processing unit;
The feature point extraction unit obtains at least eight sphere center position data necessary for calculating a ratio of nine elements of the F matrix from a plurality of captured image frames of each camera;
The association processing unit executes an association process of eight or more spherical center positions extracted by the feature point extraction unit,
The F matrix calculation unit is configured to perform element value calculation of a fundamental matrix as a calibration parameter based on eight or more feature point correspondence data associated with each other. apparatus. The sphere is a light emitting sphere,
The calibration processing apparatus according to claim 1, wherein the feature point extraction unit is configured to execute a process of calculating a sphere center position based on edge information of a sphere photographed in a frame.   The F matrix calculation unit sets an epipolar line based on the calculated F matrix, calculates the distance between the set epipolar line and the spherical center position corresponding to the feature point individually or as an average distance, and the calculated distance is The calibration processing apparatus according to claim 1, wherein when the threshold value is larger than a predetermined threshold, the F matrix recalculation process is executed based on feature point correspondence data including a new feature point. .   The F matrix calculation unit executes a spherical center position in the captured image frame, that is, a feature point distribution state determination process, and when feature point uneven distribution is confirmed, a feature point additionally set at a position where the uneven distribution is eliminated. The calibration processing apparatus according to claim 1, wherein the F matrix recalculation process is executed based on the feature point correspondence data included.   2. The calibration according to claim 1, wherein the feature point extraction unit is configured to execute a center position calculation process based on edge information only when an edge of the entire sphere exists in the imaging frame. Processing equipment. It is a calibration processing method for executing parameter calculation processing applied to adjustment between multi-viewpoint image capturing cameras or correction processing of acquired images of multi-viewpoint image capturing cameras.
An image input step for inputting video data of a plurality of cameras obtained by photographing moving spheres from different viewpoint directions;
A feature point extracting step of extracting a sphere center position as a feature point from a plurality of captured image frames constituting video data of a plurality of cameras input in the image input step;
An association processing step for performing a feature point association process based on the spherical center position extracted in the feature point extraction step;
An F matrix calculation step of calculating an F matrix [fundamental matrix] as a calibration parameter based on the feature point correspondence data associated in the association processing step;
The feature point extraction step acquires at least eight sphere center position data necessary for calculating a ratio of nine elements of the F matrix from a plurality of captured image frames of each camera;
The association processing step executes an association process of eight or more spherical center positions extracted in the feature point extraction step,
The F matrix calculation step performs calculation of element values constituting a fundamental matrix as a calibration parameter based on eight or more feature point correspondence data associated with each other. . The sphere is a light emitting sphere,
The calibration processing method according to claim 6, wherein the feature point extraction step executes a calculation process of a sphere center position based on edge information of the sphere photographed in the frame.   The F matrix calculation step sets an epipolar line based on the calculated F matrix, calculates the distance between the set epipolar line and the spherical center position as the feature point individually or as an average distance, and the calculated distance is calculated in advance. The calibration processing method according to claim 6, wherein when the threshold value is larger than a predetermined threshold value, an F matrix recalculation process based on feature point correspondence data including a new feature point is executed.   In the F matrix calculation step, a spherical center position in the photographed image frame, that is, a feature point distribution state determination process is executed, and when feature point uneven distribution is confirmed, an additional feature point additionally set at a position where the uneven distribution is eliminated The calibration processing method according to claim 6, wherein an F-matrix recalculation process based on feature point correspondence data including is executed.   The calibration processing method according to claim 6, wherein the feature point extracting step executes a center position calculation process based on the edge information only when an edge of the entire sphere exists in the imaging frame. A computer program for executing parameter calculation processing applied to adjustment between multi-viewpoint image capturing cameras or correction processing of acquired images of a multi-viewpoint image capturing camera,
An image input step for inputting video data of a plurality of cameras obtained by photographing moving spheres from different viewpoint directions;
A feature point extracting step of extracting a sphere center position as a feature point from a plurality of captured image frames constituting video data of a plurality of cameras input in the image input step;
An association process step of executing an association process of the sphere center position extracted in the feature point extraction step;
An F matrix calculation step of calculating an F matrix [fundamental matrix] as a calibration parameter based on the feature point correspondence data associated in the association processing step;
The feature point extracting step is a step of acquiring spherical center position data of at least eight points necessary for calculating a ratio of nine elements of the F matrix from a plurality of captured image frames of each camera;
The association processing step is a step of executing association processing of eight or more spherical center positions extracted in the feature point extraction step;
The F matrix calculation step is a step of executing calculation of element values constituting a fundamental matrix as a calibration parameter based on eight or more feature point correspondence data associated with each other. ·program.