JP2002354506A - Stereoscopic chart for correcting camera, acquisition method of correction parameter for camera, correcting information processing device for camera, and program thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a stereoscopic chart, a parameter acquisition method, and an information processing device capable of widely correcting a camera with high accuracy. SOLUTION: Unit figures whose sizes are encoded at different ratios are formed on the sides of a stereoscopic chart 2. The ratios of the unit figures are calculated on the basis of the picked-up image of the stereoscopic chart 2, and the calculation result is collated with an actual value by which the position and attitude of an image pickup point are obtained. The moving average of the height of the unit figures is set nearly proportional to a distance from an apex, and a limitation imposed on an image pickup distance can be reduced. A movable camera 11 is mounted on an image pickup camera 13, and the stereoscopic chart 2 is imaged at the same time when the image of an object 30 is picked up. The position and attitude of the camera 11 are obtained on the basis of the image of the chart 2 picked up by the camera 11 through the above procedure, and the position and attitude of the object imaging camera 13 are found on the basis of a positional and attitude relation between the cameras 11 and 13, where the above relation is previously obtained.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for constructing a three-dimensional image model of a subject by photographing the subject with a camera from a plurality of directions. In particular, the present invention specifies a relative relationship between a plurality of photographing positions. The present invention relates to a camera calibration three-dimensional chart used for performing the calibration and a technique for determining a camera calibration parameter using the chart.

[0002]

2. Description of the Related Art A three-dimensional image model of a subject can be obtained by photographing a three-dimensional subject from a plurality of directions and combining a plurality of image data obtained thereby. In other words, if data of the external parameters (such as the position and orientation of the camera) and the internal parameters (such as the focal length) of the camera can be obtained for each image of the subject photographed from a plurality of directions, the shape-from-silhouette method Thus, a three-dimensional model can be reconstructed from the silhouette image of the subject. For more information on this shape from silhouette method, see W.Ni
em, "Robust and Fast Modeling of 3D Natural Objec
ts from Multiple Views "SPIE ProceedingsImage and
Video Proceeding II vol. 2182, 1994, pp. 388-397. Hereinafter, the external parameters and the internal parameters of the camera are collectively referred to as “calibration parameters (of the camera)”. In the case where the internal parameters among the calibration parameters are known and the camera calibration by the internal parameters has been completed, If the external parameters of the camera are obtained, a three-dimensional image model of the subject can be constructed.

One method of photographing a subject from a plurality of directions is a fixed arrangement method in which a plurality of cameras are fixedly arranged at different positions to photograph a subject. However, in this fixed arrangement method, since a plurality of cameras must be fixedly arranged in a photographing studio or the like, the photographing equipment becomes large-scale.

Accordingly, a moving photographing method has been proposed in which a user moves around a subject with one hand-held camera while sequentially photographing the subject from a plurality of directions to obtain an image of the entire periphery of the subject. I have.

However, in order to determine the external parameters of the camera using this moving photographing method, it is necessary to specify the position and orientation of the camera for each photographing.

As a method for measuring external parameters of a camera for such a purpose, a magnetic method, an ultrasonic method, an optical method, and the like have been conventionally proposed. Among them, the magnetic method is a method of detecting the geomagnetism or the like at the camera position, and the ultrasonic method is a method of detecting an ultrasonic wave from a predetermined ultrasonic wave to specify the position, posture, and the like of the camera. The optical method includes a method using a stereo camera, a method of installing a calibration chart larger than the field of view, and the like.

[0007] Of these methods, in the magnetic method, it is difficult to measure with high accuracy when the object is made of metal, and the ultrasonic method makes the apparatus expensive.

On the other hand, as a conventional optical system,
Japanese Patent Laid-Open No. 2000-270343 discloses a method in which a single planar chart on which a non-uniform matrix pattern is drawn is arranged at a predetermined position, and the position and posture relationship between the planar chart and the camera are identified by observing the same with a camera. Is disclosed. According to this, since the relative relationship between the position and orientation of the camera with respect to the coordinate system fixed to the planar chart can be known, if the positional and orientation relationship between the planar chart and the subject is fixed, the subject can be viewed from a plurality of directions. By observing the pattern on the planar chart with the camera every time an image is taken, the position and orientation of the camera at that time can be specified in the absolute coordinate system.

[0009]

However, the flat chart has a narrow angle range in which it can be observed, and cannot be observed from a direction exceeding 90 degrees from the normal direction of the flat chart, so that the movable range of the camera is large. Limited.
In addition, even if the camera is within the range in which the planar chart can be observed, when the direction of the camera deviates greatly from the normal direction of the planar chart, the observation accuracy of the pattern on the planar chart decreases, and as a result, the external There is a disadvantage that the parameter determination accuracy is not good.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems in the prior art, and is an external device of a camera which can obtain a high measurement accuracy while securing a wide movable range while using an optical system. The purpose is to realize a parameter acquisition technology.

[0011]

In order to solve the above-mentioned problems, a camera calibration chart according to the first aspect of the present invention has a pyramid having a bottom surface and a plurality of side surfaces, and is displayed on each of the plurality of side surfaces. And a plurality of charts, each of the plurality of charts includes a set of a plurality of unit figures each having a known size, and the geometric shape of each of the plurality of unit figures is within the set. Regarding both between the plurality of charts, irrespective of the relative position and relative attitude between each unit figure and the observation position, the coding is performed so as to be mutually identifiable by observation from the observation position. .

According to a second aspect of the present invention, in the camera calibration three-dimensional chart according to the first aspect, the plurality of unit figures increase in size closer to the bottom surface of the cone.

According to a third aspect of the present invention, in the three-dimensional camera calibration chart of the second aspect, the pyramid is a pyramid, and the plurality of charts include a relative position and a relative attitude between each chart and an observation position. Regardless, it is characterized by being coded so as to be mutually identifiable by observation from the observation position.

According to a fourth aspect of the present invention, in the camera three-dimensional chart of the third aspect, the plurality of unit figures are a plurality of first straight lines parallel to a bottom surface of the cone and a vertex of the cone. A plurality of trapezoids formed by intersections with a plurality of second straight lines extending radially from corresponding positions, wherein a cross ratio of the sizes of the plurality of trapezoids is coded.

According to a fifth aspect of the present invention, in the camera calibration three-dimensional chart of the fourth aspect, a marker is provided at a vertex of the pyramid.

According to a sixth aspect of the present invention, in the three-dimensional chart for camera calibration according to the fifth aspect, the marker includes a luminous body.

According to a seventh aspect of the present invention, in the three-dimensional camera calibration chart of the third aspect, the pyramid is a truncated pyramid.

According to an eighth aspect of the present invention, in the camera calibration three-dimensional chart according to any one of the first to seventh aspects, the unit figures adjacent to each other among the plurality of unit figures have different lightness or hue. Characterized by the following color:

According to a ninth aspect of the present invention, there is provided a camera calibration chart including a cone having a bottom surface and a side surface, and a chart displayed over one round of the side surface, wherein each of the charts has a known size. Including a set of a plurality of unit figures having a geometric shape of each of the plurality of unit figures, irrespective of a relative position and a relative posture between each unit figure and an observation position inside the set. It is characterized by being coded so as to be mutually identifiable by observation from an observation position.

According to a tenth aspect of the present invention, there is provided a method for acquiring a calibration parameter of a camera, wherein the three-dimensional chart for camera calibration according to any one of the first to eighth aspects is arranged in a space in which a subject is to be accommodated. And at least one of the plurality of charts of the camera calibration three-dimensional chart.
Observing two charts from an observation position attached to the camera, extracting at least one unit graphic included in the observed chart as a target unit graphic, and extracting the target unit graphic from the plurality of charts and the plurality of charts. The step of identifying in the unit figure of the, the actual size specified in advance for the identified target unit figure, from the relationship between the observation size of the target unit figure viewed from the observation position, the observation position and Specifying a calibration parameter of the camera that depends on a relative position and a relative attitude with respect to the camera calibration three-dimensional chart.

[0021] The invention of claim 11 is an apparatus for obtaining a calibration parameter of a camera, wherein the apparatus is for determining a calibration parameter of a camera.
Extracting at least one unit graphic as a target unit graphic from a chart image obtained by observing any one of the camera calibration three-dimensional charts from the observation position, and extracting the target unit graphic between the plurality of charts and the plurality of charts. Identification means for identifying among the unit figures of the above, the relationship between the actual size specified in advance for the identified target unit figure and the observation size of the target unit figure viewed from the observation position, Calculating means for calculating a calibration parameter of the camera depending on a relative position and a relative attitude between the camera and the three-dimensional chart for camera calibration.

According to a twelfth aspect of the present invention, a program is executed by a computer to cause the computer to function as a camera information processing apparatus according to the eleventh aspect.

[0023]

DESCRIPTION OF THE PREFERRED EMBODIMENTS <Overview of System Configuration> FIG.
1 is a diagram illustrating a configuration of an imaging operation system 1 to which an embodiment of the present invention is applied, and FIG. 2 is a block diagram of the imaging operation system 1. In FIG. 1, an imaging operation system 1
Is provided with a portable camera system 10 capable of capturing a three-dimensional image of a subject 30 and a three-dimensional chart 2 for camera calibration arranged near the subject 30 in a space accommodating the subject 30. . The three-dimensional chart 2 is a three-dimensional object in which a chart pattern is formed on each side surface of a substantially pyramid-shaped main body, as will be described in detail later.
Hanged from. The chart support 250 includes an inverted L-shaped arm 252 extending from the pedestal 251, and the three-dimensional chart 2 is fixed near the tip of the arm 252. Preferably, the three-dimensional chart 2 is suspended substantially above the subject 30.

The camera system 10 includes a subject photographing camera (hereinafter, abbreviated as "subject camera") 13 having a function as a digital camera. A movable camera 11 is mounted on the upper part of the subject camera 13 via a mounting mechanism 12 so as to be freely changeable in posture.
The movable camera 11 specifies a relative position and orientation relationship between the three-dimensional chart 2 and the movable camera 11 by photographing a plurality of unit figures UP included in a pattern (see FIG. 3) on the three-dimensional chart 2. And 3D chart 2
Is used to detect the position and orientation of the subject camera 13 in an absolute coordinate system fixed relative to.

Although not shown in FIG. 1, as shown in FIG. 2, the photographing operation system 1 may include a notebook-type portable computer 15, for example. The computer 15 can transmit and receive commands and data to and from the camera system 10 by wireless communication via the communication interface 15a.

FIG. 3 is a side view of the three-dimensional chart 2. The three-dimensional chart 2 has a three-dimensional chart main body 203 and a chart pattern CP formed on the surface of the three-dimensional chart main body 203.

The three-dimensional chart main body 203 has a polygonal pyramid-shaped display section 204 and a truncated pyramid-shaped support section 205 integrated with each other, and has a hollow interior. The chart pattern CP is a set of patterns P1 to Pn provided on each side surface T1 to Tn (n is an integer of 3 or more) of the display unit 204. Preferably, the number n of the sides of the pyramid is n = 3 to 3
6, and more preferably n = 6 to 12. Each of the patterns P1 to Pn formed on each of the side surfaces T1 to Tn is a planar pattern. However, by arranging the patterns P1 to Pn three-dimensionally, the chart pattern CP as a set of the patterns P1 to Pn becomes It has a three-dimensional pattern. Each of the patterns P1 to Pn is a set of a plurality of trapezoids, each of which functions as a unit graphic, the details of which will be described later.

A marker 201 serving as a reference point when the movable camera 11 tracks the chart pattern CP is provided at the apex of the polygonal pyramid constituting the display unit 204.
, A light emitting diode (LED) is attached so that the movable camera 11 can easily and accurately recognize the position of the three-dimensional chart 2. Although not shown in FIG. 3, a marker power supply 202 (FIG. 2) for supplying light-emitting power to the light-emitting diodes is built in the three-dimensional chart 2.

<Outline of Movable Camera 11> FIG. 4 is a front view of the movable camera 11, and FIG. 5 is a block diagram of the movable camera 11. As shown in FIG. 5, the movable camera 1
In 1, a spherical unit 116 is formed by integrating a lens unit 110 and a two-dimensional light receiving element 111 for photoelectrically converting a two-dimensional image formed by the lens unit 110.
It is stored in. The two-dimensional light receiving element 111 is a CCD array. The lens unit 110 includes a fixed lens 110
a and the zoom lens 110b, and an aperture / shutter mechanism 110e exists between them.

As shown in FIG. 4, the spherical unit 116 is connected to a fixed part 114 via a posture device 113, and together with the elements built in the spherical unit 116, pivots about ± 70 ° in the pan direction (θ (Rotation) and a tilt (φ rotation) of ± 70 ° in the tilt direction are possible.
In order to perform the rotation drive in the pan direction and the rotation drive in the tilt direction, a posture device 113 having a plurality of piezo elements is arranged at the base of the spherical unit 116. Further, a zoom operation corresponding to driving of the zoom lens 110b is also performed by another piezo element. By applying a sawtooth wave signal to these piezo elements, the element to be driven by the piezo element jogs, and a required movement is given to the element by repetition.
The turning angle in the pan direction and the elevation angle in the tilt direction are detected by angle sensors 126p and 126t such as encoders, respectively, and the driving amount of the zoom lens 110b is detected by a sensor 126z also formed by an encoder. These drive mechanisms are described in, for example,
9-18000 and JP-A-1999-41504.

The control operation unit 120 is provided with the two-dimensional light receiving element 11
An image processing unit 121 that performs a process such as image recognition by inputting a signal from the image processing unit 1 and an image memory 122 that stores an image signal obtained by the image processing unit 121. Further, a camera control unit 123 that generates drive signals for the zoom lens 110b, the attitude device 113, and the aperture / shutter mechanism unit 110e and outputs the drive signals to them is provided.
The image processing unit 121 and the camera control unit 123 can wirelessly communicate with the subject camera 13 via the communication unit 124 and the communication device 112. By this communication, image data is transmitted to the subject camera 13, and various information is transmitted and received between the movable camera 11 and the subject camera 13. In the movable camera 11 of this embodiment, an IR for performing infrared communication is used as the communication device 112.
An infrared element corresponding to a DA (Infrared Data Association) interface is used.

As shown in FIG. 4, the first mounting groove 115a and the second mounting groove 115b provided on the fixing portion 114 are
It is used for attaching the fixing part 114 to the subject camera 11. Further, a tracking button 117 is provided for a mode in which the movable camera 11 automatically tracks the three-dimensional chart 2 (hereinafter, abbreviated as an “automatic tracking mode”) and a mode in which the tracking is performed according to a user instruction from the subject camera 13 (hereinafter, referred to as “automatic tracking mode”). Button for switching between "manual mode").

FIG. 6 is a diagram showing a main part of the information processing function of the movable camera 11 as viewed from the hardware configuration, and FIG. 7 is a diagram showing a data flow in the movable camera 11. In FIG. 6, the control operation unit 12 of the movable camera 11
0 is a program 1 that includes a CPU 130, a ROM 131, and a RAM 132, and implements various operations described below.
31a is stored in the ROM 131.

The two-dimensional light receiving element 111 has an RG for each pixel.
B, and the light focused on the two-dimensional light receiving element 111
1 for each of the three RGB primary color components. The signal obtained in this way is converted into a digital image signal by the A / D converter 141, and undergoes white balance correction, γ correction, and the like in the image corrector 142. The corrected image signal is stored in the image memory 122. The first image data D1 in FIG. 7 corresponds to the corrected image signal.

The recognition unit 145 and the attitude control unit 146 shown in FIG.
Are realized as a part of the functions of the CPU 130, the ROM 131, the RAM 132, and the like.

The recognition unit 145 is activated in response to a user instruction from the tracking button 117, and
Generates the tracking data DF for recognizing the image of the three-dimensional chart 2 from the acquired first image data D1 and tracking the image of the three-dimensional chart 2 in the first image data D1.

The posture control unit 146 controls the posture device 113 based on a user's instruction received from the subject camera 13 in the manual mode. After the tracking button 117 is pressed and the mode is changed to the automatic tracking mode, the image of the three-dimensional chart 2 is changed to the CCD 111 by the processing described later.
The attitude device 113 is controlled so that an image is always formed thereon.

The CPU 130 further has a function of creating the rotation angle data DR of FIG. 7 by the processing described later.

When the shutter button of the subject camera 13 is pressed, the first image data D1 and the rotation angle data DR are transmitted to the subject camera 13 via the communication unit 124.

<Outline of Camera 13 for Object> FIG. 8 is a diagram showing a main part of the information processing function of the camera 13 for object, viewed from the hardware configuration, and FIG. It is a figure showing a flow. The subject camera 13 includes a CPU 150, a RAM 151, and a ROM 1
A program 152 a for realizing various operations of the subject camera 13, which will be described later, includes a ROM 152.
Is stored in Elements such as a shutter button 161, a flash 162, a monitor color display 163 installed on the back surface, and an operation button group 164 are also electrically connected to the CPU 150.

As shown in FIG. 8 and FIG. 9, light incident from the subject 30 via the lens unit 155 is converted to a CC signal, which is provided with one of RGB filters for each pixel.
An image is formed on a two-dimensional light receiving element 156 such as a D array, and the two-dimensional light receiving element 156 performs photoelectric conversion for each of the three RGB primary color components. The signal thus obtained is converted into a digital image signal by the A / D converter 157, and undergoes white balance correction, γ correction, and the like in the image corrector 158. The corrected image signal is stored in the image memory 159. When the shutter button 161 is pressed, shooting is performed, and the image signal stored in the image memory 159 is stored in the RAM 151 as second image data D2.

The communication unit 167 transmits and receives control signals for various parts of the movable camera 11 and various information such as acquired image data to and from the movable camera 11 via the communication device 168. For example, in the manual mode, a signal obtained by the user operating a part of the operation buttons 164 is transmitted to the movable camera 11 so that the posture device 113 of the movable camera 11 is manually operated by the user. It becomes possible to operate. In addition, in response to the pressing of the shutter button 161 of the camera 13 for the subject, it is also possible to simultaneously shoot the camera 13 for the subject and the movable camera 11.

The communication device 168 includes the movable camera 11
(Infrared Data As) for infrared communication with
An infrared element as an interface, and is driven by the communication unit 167.

The card slot 165 is for the camera 1 for the subject.
3 is used to mount a memory card 166, and the memory card 166 can store photographed image data and the like.

The extraction unit 171, the operation unit 173, and the display control unit 174 of FIG.
1. Functions implemented by the ROM 152 and the like.

The extracting unit 171 converts the first image data D1 received from the movable camera 11 through the communication unit 167 from the first image data D1.
Four points on the three-dimensional chart 2 are extracted to create first extracted point data DP1. Further, the extraction unit 171 similarly calculates
Second image data D acquired by subject camera 13
Then, four points on the three-dimensional chart 2 are extracted from No. 2 to create second extraction point data DP2.

The operation unit 173 calculates the first extraction point data DP
1. Rotation angle data DR and second extraction point data DP2
Then, the relative position and orientation of the movable camera 11 and the subject camera 13 are obtained, and the relative position data DPS is created. Further, the first extraction point data D1 and the rotation angle data D
From R and the relative position data DPS, the relative position and orientation between the subject photographing camera 13 and the three-dimensional chart 2 are obtained, and photographing data DM is created. The shooting data DM is R
Stored in the AM 151.

The display control unit 174 includes an operation button group 164
The second image data D2 based on the user's instruction from
And the photographing data DM from the RAM 151,
The data is stored in the memory card 166. The display control unit 17
4 is a display 1 that performs necessary processing for various data.
It also has a function of displaying on the RAM 63 and reading various data stored in the memory card 166 onto the RAM 151.

<Principle of Camera Calibration> When an image is obtained by photographing the subject 30 with the subject camera 13 from an arbitrary direction, the image is taken with respect to the three-dimensional chart 2 or the absolute coordinate system fixed thereto when the photographing is performed. , Subject camera 1
It is necessary to specify the relative positions and postures of the three as external parameters. This is because, when constructing a three-dimensional image model of the subject 30 by combining the images obtained by photographing from a plurality of directions, a spatial correlation between the images is required.

However, the subject 30 is actually photographed.
The stereoscopic camera within the angle of view of the subject camera 13
In some cases it is difficult to insert Then, X₀: Coordinate system fixed to three-dimensional chart 2 (absolute coordinates
System); X₁: A coordinate system fixed to the movable camera 11 (first row
Cull coordinate system); X_Two: A coordinate system fixed to the subject camera 13 (second coordinate
Coordinate system); τ₀₁: First local coordinate system X₁From absolute coordinate system X₀Change to
Commutation relation; τ₀₂: Second local coordinate system X_TwoFrom absolute coordinate system X₀Change to
Commutation relation; τ₁₂: First local coordinate system X_TwoTo the second local coordinate system
X₁Use the following relation that holds when₀₁, Τ
₀₂, Τ₁₂And Q described below ₀, Q_TwoAre not shown).

Τ ₀₂ = τ ₀₁ · τ _{12 If} τ ₀₁ and τ ₁₂ are known, τ ₀₂ is obtained. If τ ₀₂ is obtained, the subject camera 1 that has captured the two-dimensional image
Position and orientation of the second local coordinate system X ₂ of 3, by acting on the transformation tau _02, is determined as the position and orientation in the absolute coordinate system X _0. A matrix representing the position and orientation of the subject camera 13 in the absolute coordinate system is Q ₀ ,
When the matrix representing the position and orientation of the object camera 13 in the second local coordinate system X ₂ and Q _2,

Equation 2 Q ₀ = ｛τ ₀₁ · τ ₁₂ ｝ Q ₂ = τ ₀₂ · Q ₂

Therefore, each time the subject 30 is photographed by the subject camera 13 while moving with respect to the subject 30, a conversion relationship τ ₀₂ corresponding to the photographing is obtained and attached to the photographed image to obtain a plurality of images. By combining images taken in the directions with X ₀ , a three-dimensional image model of the subject 30 can be obtained.

A specific process for realizing this principle (details will be described later) is roughly classified into a first sub-process and a second sub-process.

* First sub-process: This is a sub-process for specifying the conversion relationship τ ₁₂ between the two camera coordinate systems.

First, the three-dimensional chart 2 is moved to the movable camera 11.
And at the same time taken by the subject camera 13, using the result thereof shooting, external parameters of each camera, that obtains the position and orientation of each camera in the absolute coordinate system X _0.

This is because the conversion relation τ ₀₂ , τ in that state
_This corresponds to specifying ₀₁ . And from Equation 1,

From the relationship of τ ₁₂ = (τ ₀₁ ) ⁻¹ τ ₀₂ , a conversion relationship τ ₁₂ between the first local coordinate system X ₁ and the second local coordinate system X ₂ is obtained.

[0059] Further, the rotation angle of the movable camera 11 theta, the value of φ, respectively angle sensor 126p, because it is known values detected by 126t, separating the rotation angle-dependent parts from the conversion relationship tau _12, Reference conversion relation τ when the movable camera 11 is in the reference posture (θ = 0, φ = 0)
₁₂ (0,0) can be obtained. This reference conversion relation τ
₁₂ (0,0) is an operator who does not change even if the camera system 10 is moved or the movable camera 11 is rotated.
When the conversion relation τ ₁₂ (0,0) is determined, the conversion relation τ ₁₂ has the rotation angles θ and φ as variables.

The conversion relation τ ₁₂ thus obtained is
Does not depend on the position and orientation of the entire camera system 10 in the absolute coordinate system X _0, available camera system 10 to convert operations therein be moved to another location.

* Second sub-process: This uses the result of the first sub-process to obtain image data by photographing the subject 30 from a plurality of directions, and to obtain a second local image for each of the image data. This is a subprocess for adding information corresponding to the conversion relationship τ ₀₂ from the coordinate system X ₂ to the absolute coordinate system X ₀ .

In the second sub-process, the subject camera 1
3 and the movable camera 11
To photograph the three-dimensional chart 2. The conversion relationship τ ₀₁ from the first local coordinate system X ₁ to the absolute coordinate system X ₀ is specified from the image data of the three-dimensional chart 2 captured by the movable camera 11.

On the other hand, since the rotation angle dependency of the transformation relation τ ₁₂ from the second local coordinate system X ₂ to the first local coordinate system X ₁ is specified by the first sub-process, the object 3
From the values of the rotation angles θ and φ when shooting 0, the conversion relation τ
_Twelve specific contents are specified. Therefore, the conversion relation τ
A conversion relationship τ ₀₂ obtained by combining ₁₂ and τ ₀₁ can be obtained from Expression 4.

Τ ₀₂ = τ ₀₁ · τ _{12 The} information expressing the conversion relationship τ ₀₂ is stored in association with the image obtained by the subject camera 13.

The second sub-process is executed each time the subject 30 is photographed from a plurality of directions, whereby a group of information for obtaining a three-dimensional image model is obtained.

<Photographing and Calibration Process> FIGS. 10 and 11 show a photographing and calibration process according to the above principle. The movable camera 11 and the subject camera 13 are included in steps S1 to S5.
This corresponds to the above-described first sub-process in which the three-dimensional chart 2 is photographed at the same time to obtain the relative position and orientation of the two. Also, after step S6, the subject 3 is actually
This corresponds to the above-described second sub-process for performing shooting of 0.

(1) Determination of relative position between cameras (first
Sub-process): First, each of the movable camera 11 and the subject camera 13 reads information of internal parameters held by itself and calibrates the internal parameters based on the information (step S1). Such internal parameters include the focal length of the lens system. For example, when the substantial imaging magnification M is small due to the relatively short focal length, the image of the three-dimensional chart 2 is formed on the two-dimensional light receiving element with a relatively small size. When the substantial imaging magnification M is large due to the relatively long focal length, the image of the three-dimensional chart 2 is
An image is formed on the two-dimensional light receiving element in a relatively large size. Therefore, by dividing the size of the image on the two-dimensional light receiving element by the substantial imaging magnification M, the image is converted into a reference state independent of the focal length. Such processing is camera calibration based on internal parameters, which can be automatically performed in the subject camera 13.

After the calibration based on the internal parameters is completed, the user holds the camera system 10 and turns the subject camera 13 at the three-dimensional chart 2. Next, while maintaining this posture, the three-dimensional chart 2 is moved by the movable camera 1.
The rotation angle of the lens unit 110 is manually designated so as to fall within the field angle of 3 (step S2). In this operation, the output image of the movable camera 13 is displayed on the display 163.
, The user can visually check whether the three-dimensional chart 2 has entered the angle of view of the movable camera 13.

When the user presses the automatic tracking button 117 after the three-dimensional chart 2 enters the angle of view of the movable camera 11, the automatic tracking program is activated. Attitude control unit 1
A drive output is provided from 46 to the attitude device 113, and the movable camera 11 is automatically controlled so that the three-dimensional chart 2 is always at the center of the angle of view while tracking the marker 201. Here, when the user presses the shutter button 161, the first image data D1 in the movable camera 11 becomes
The second image data D2 is obtained by the subject camera 13 (step S3). FIG. 12 shows an example of image data obtained simultaneously by the movable camera 11 and the subject camera 13. 12A shows an example of a captured image of the movable camera 11, and FIG. 12B shows an example of an image of the subject camera 13. 12A and 12B, the image plane is a plane defined by an xy rectangular coordinate system,
The direction perpendicular to the xy plane and toward the front of the image is the z-axis. The i-th layer is based on the definition in FIG. 18 described later.

Then, in this step S3, for example, the four grid points C1 to C4 common in FIGS.
Specify the two-dimensional coordinate values of C4 on each image plane,
The movable camera 11 and the subject camera 1 are processed by processing the two-dimensional coordinate values using an algorithm described later.
3 and the respective external parameters are calculated.

When the photographing is completed, the first image data D1 and the rotation angle data DR of the attitude device 113 are transmitted from the movable camera 11 to the subject camera 13 by communication. The subject camera 13 calculates the external parameters of the movable camera 11, that is, the relative position and the relative attitude of the movable camera 11 with respect to the three-dimensional chart 2 based on these. Further, from the second image data D2, the external parameters of the camera 13 for the subject, that is, the relative position and the relative attitude of the camera 13 for the subject with respect to the three-dimensional chart 2 are calculated (step S4). If the information necessary for this calculation cannot be obtained, the process returns to step S3, and the photographing of the three-dimensional chart 2 is repeated.

The calculation of the external parameters includes: 1) internal parameters of the camera; and 2) three-dimensional coordinate values of four or more points on the same plane fixed in the absolute coordinate system, and 3) It can be performed under the condition that two-dimensional coordinate values of points on the captured image corresponding to these points can be calculated. An algorithm that can be used for this calculation is disclosed in, for example, the following document, and is hereinafter referred to as a “multipoint analysis algorithm”.

L. Quan, Z. Lan, "Linear N-Point Camer
a Pose Determination ", IEEE Trans.PAMI 21 (8) 199
9: ・ Takahashi, Ishii, Makino, Nakashizu, "Measurement of marker position and orientation from monocular image for virtual reality interface", Electronic Information Transactions AJ79 1996.

Next, the external parameters of the cameras 11 and 13 obtained in step S4 and the movable camera 1
The relative position and orientation of the movable camera 11 and the subject camera 13 (hereinafter, these are referred to as “inter-camera parameters PMC”) are obtained from the one rotation angle data DR (step S5).

FIG. 13 shows the state of the coordinate conversion used in step S5. The definition of each coordinate system in FIG. 13 is as follows.

X ₀ : three-dimensional rectangular coordinate system (absolute coordinate system) fixed relative to the three-dimensional chart 2; θ: turning angle of the movable camera 11; φ: elevation angle of the movable camera 11; X ₁ (θ, φ): a three-dimensional rectangular coordinate system (first local coordinate system) corresponding to the observation space from the movable camera 11; X _1h : first local coordinate system when both angles θ, φ are zero X ₂ … 3 corresponding to the observation space from the camera for subject 13
Dimensional rectangular coordinate system (second local coordinate system).

In step S4, each camera 1
Since 1,13 external parameters are obtained, the absolute coordinate system
The position and orientation of the first local coordinate system X ₁ (θ, φ) in X ₀ is determined, and thus the first local coordinate system X ₁ (θ, φ,
φ) to the absolute coordinate system X ₀ is the rotation matrix R _C1 ,
And the translation vector T _C1 ,

Equation 5 X ₀ = R _C1 X ₁ (θ, φ) + T _C1 Similarly, the second coordinate transformation from the local coordinate system X ₂ to the absolute coordinate system X _0, using the rotation matrix R _C2, and translation vector T _C2,

Equation 6 X ₀ = R _C2 X ₂ + T _C2 Equations 5 and 6 correspond to the above-described conversion relationships τ ₀₁ and τ ₀₂ , respectively.

The rotation conversion based on the fact that the rotation angles θ and φ are not zero is based on the rotation matrix R _X (θ, φ) and the translation vector T _X (from the design data of the rotation mechanism of the attitude device 113 of the movable camera 11. θ, φ),

Equation 7 X ₁ = R _X (θ, φ) X _1h + T _X (θ, φ) A second local coordinate system X _2, the rotation angle theta, phi coordinate transformation to both the first local coordinate system X _1h when the zero, using a rotation matrix R _h and translation vector T _h,

[0082] When expressed as the number _{8 X 1h = R h X 2} + T h, number 8 corresponds to the conversion relationship τ ₁₂ (0,0). Substituting equation 8 into equation 7 gives

Equation 9 X ₁ = R _m (θ, φ) X ₂ + T _m (θ, φ) is obtained. However,

[0084] the number _{10 R m (θ, φ)} = R X (θ, φ) R h T m (θ, φ) = R X (θ, φ) T h + T X (θ, φ). Equations 9 and 10 correspond to the conversion relationship τ ₁₂ .

Therefore, if the external parameters of the respective cameras are obtained from the respective images of the three-dimensional chart 2 taken simultaneously by the movable camera 11 and the subject camera 13, the conversion relation τ _{12 described} above is obtained. Equations 9 and 10 are specified. This is the relative position data DPS in FIG. The specific values of the angles θ and φ are detected as angle data DR by the angle sensors 126p and 126t.

(2) Subject Photographing and Camera Calibration (Second Sub-Process): When step S5 is completed, the camera system 10 is moved as appropriate to photograph the subject 30. At this time, the movable camera 11 always captures the three-dimensional chart 2 at an angle of view by automatic tracking (step S).
6).

The subject 30 by the subject camera 13
When the three-dimensional chart 2 can be photographed by the movable camera 11, the subject camera 1
When the shutter button 161 of No. 3 is pressed down, the respective cameras shoot images simultaneously (step S7). FIG.
FIG. 1 shows examples of images obtained by these cameras 11 and 13. FIG. 14A shows an example of an image obtained by imaging with the movable camera 11, and FIG. 14B shows an example of an image captured by the subject camera 13 (an image of the subject 30 is simplified by an ellipse). The way of setting the coordinate axes is the same as in FIG. In the state of FIG. 14A, the distance and direction from the three-dimensional chart 2 to the movable camera 11 are different from those in FIG.
Of these, a part different from that of FIG. However, since the movable camera 11 automatically tracks the marker 201, the marker 201 is always shown at the same position within the angle of view of the movable camera 11.

[0088] In this case, from an image obtained by the movable camera 11, as in step S4, the position and posture (external parameters P in the absolute coordinate system X ₀ of movable camera 11
M1) is obtained (step S8).

The external parameter P of the movable camera 11
M1 and the inter-camera parameter PM obtained in step S5
C, and the relative position and orientation of the subject camera 13 with respect to the three-dimensional chart 2 (that is, the external parameter PM2 of the subject camera 13) from the respective values θ ′ and φ ′ of the rotation angles θ and φ of the movable camera 11 in step S7. ) Is obtained (step S9).

After the photographing, the external parameter P of the camera 13 for photographing at the time of photographing together with the image data D2 of the photographing object 30
The value of M2 is stored in the RAM 151 or the memory card 17
6 (step S10).

The photographing processing from step S6 is performed on the subject 3
While changing the direction of the camera system 10 with respect to
Repeat several times. Then, when the number of pieces of second image data D2 necessary for constructing the three-dimensional image model of the subject 30 and the value of the external parameter PM2 for each are obtained, the second sub-process is completed. The external parameter PM2 corresponds to the photographing data DM in FIG.

FIG. 15 shows the state of the coordinate conversion used in step S9. In FIG. 15, the definitions of symbols such as the coordinate systems X ₀ , X ₁ (θ, φ), and X ₂ are common to FIG. The camera rotation angles of the movable camera 11 at the time of shooting an object are θ ′ and φ ′.

[0093] The image of the three-dimensional chart 2 obtained by the movable camera 11, by analyzing at multiple points analysis algorithms, the position and orientation of the movable camera 11 in the absolute coordinate system X ₀ (i.e. external parameters of movable camera 11 PM1) is specified, and thereby the conversion relation with the first local coordinate system X ₁ (θ ′, φ ′) of the movable camera 11:

Equation 11 X ₀ = R _CP1 X ₁ (θ ′, φ ′) + T _CP1 is determined.

The first local coordinate system X ₁ (θ ′, φ ′)
When the conversion relationship between the second local coordinate system X _2, the number 9 Number 10
By

Equation 12 X ₁ (θ ′, φ ′) = R _m (θ ′, φ ′) X ₂ + T _m (θ ′, φ ′) R _m (θ ′, φ ′) = R _X (θ ', φ') R _h T _m (θ ', φ') = R _X (θ ', φ') T _h + T _X (θ ', φ').

[0097] Therefore, Equation 11, from Equation 12, the position and orientation expressed by the second local coordinate system X _2, conversion relationship for converting the position and orientation of the absolute coordinate system X _0,

Equation 13 X ₀ = R _CP1 R _m (θ ′, φ ′) X ₂ + R _CP1 T _m (θ ′, φ ′) + T _CP1

Of the quantities appearing in Equation 13, the rotation matrix
R _CP1 and translation vector T _CP1 are the movable camera 11
And is determined from the external parameter PM1 of the movable camera 11. Also, the rotation matrix R _m (θ ′,
φ ′) and the translation vector T _m (θ ′, φ ′) are converted to the function forms R _m (θ, φ) and T _m (θ, φ) specified in advance.
It is determined by substituting the angle values θ ′ and φ ′ detected by the angle sensors 126p and 126t.

Therefore, Expression 13 is given by

Equation 14 X ₀ = R _CP2 X ₂ + T _CP2 R _CP2 = R _CP1 R _m (θ ′, φ ′) T _CP2 = R _CP1 T _m (θ ′, φ ′) + T _CP1 Then, the rotation matrix R _CP2 and the translation vector T _CP2 are converted to the external parameters PM2 of the camera 13 for a subject.
It is the content which expressed. They are photography data DM
(FIG. 9) is stored together with the second image data D2, and is used to combine images of the subject 30 obtained from a plurality of directions when constructing a three-dimensional image model based on the second image data D2. You. The construction of the three-dimensional image model may be performed by the computer 15 or may be performed by another arithmetic system.

<Identification of Chart by Cross Ratio Coding> A method of coding the side surface of the three-dimensional chart 2 will be described below. As shown in FIG. 3, the display section 204 of the three-dimensional chart main body 203 is a regular polygonal pyramid, and each side surface T1 to Tn has the same isosceles triangle shape. The surface has a bottom surface direction DR of a triangle forming the side surface.
1 (see FIG. 16) and a plurality of radial straight lines L2 passing through a vertex x0 corresponding to a vertex of the three-dimensional chart 2 are drawn. A trapezoidal unit figure UP formed by the intersection of these straight lines (hereinafter referred to as a “unit trapezoid”) is alternately painted with different brightness colors so that straight lines can be easily extracted during image processing. High contrast pattern. Typically, the first set of unit trapezoids UP1 is black, and the second set of unit trapezoids U1 is black.
P2 is white.

The sizes of these unit trapezoids UP are coded by the cross ratio. More specifically, 1) the interval between the straight line groups L1 forming these unit trapezoids UP, and 2) the interval between the intersections (grid points) of the straight line groups L1 and L2 in the bottom direction DR1. Each is coded by cross ratio. FIG. 17 shows the concept of the cross ratio. The cross ratio is a value which does not change by spatial projection through an arbitrary viewpoint, and is a cross ratio DR obtained from four points P1 to P4 on a straight line existing in a three-dimensional space. :

DR = Va / Vb Va = dis (P0P1) · dis (P2P3) Vb = dis (P0P2) · dis (P1P3) where the symbol dis (P0P1) indicates the distance between the point P0 and the point P1. : When the straight line is projected onto an arbitrary plane through the viewpoint O, the four points P′1 to P′1 to P4 corresponding to the four points P1 to P4
Compound ratio DR 'obtained from P'4:

Equation 16 DR ′ = Va ′ / Vb ′ Va ′ = dis (P′0P′1) · dis (P′2P′3) Vb ′ = dis (P′0P′2) · dis (P′1P It is known to be equal to '3).

By utilizing this property, FIGS.
, The distance between the straight line groups L1 forming the unit trapezoid UP is coded as a cross ratio for each layer forming each unit trapezoid UP, and the distance between the lattice points in the bottom direction DR1 is calculated as the side surface T1. ＴTn, the unit trapezoids UP included in the image of the three-dimensional chart 2 captured by the movable camera 11 or the subject camera 13 are converted to the side surfaces T 1 to Tn of the three-dimensional chart 2. It is possible to uniquely identify which unit trapezoid exists on which side of which. An example is shown below.

FIG. 16 shows an example in which the interval between the straight lines L1 arranged in the direction perpendicular to the bottom surface (vertex direction DR2) is coded by the cross ratio. Among a plurality of unit trapezoids, the compound ratio of the height of three consecutive unit trapezoids is 3
It is coded differently for each set of unit trapezoids.

That is, with the vertex x0 of the three-dimensional chart 2 as an end point, straight lines x1, x2... Parallel to the base are defined, and the "i-th layer" is defined as "a straight line xi and a straight line x (i + 1)". When it is defined as a "region between," the i-th layer to the (i + 3) -th layer (i = 1,
2,...) Of the respective positions in the vertex direction DR2,

Equation 17 DRi = Vai / (Vbi · Vb (i + 1)) Vai = dis (xix (i + 1)) · dis (x (i + 2) x (i + 3)) Vbi = dis ( xix (i + 1)) + dis (x (i + 1) x (i + 2)) Vb (i + 1) = dis (x (i + 1) x (i + 2)) + dis (x (i + 2) x (i + 3)) or rewrite this,

Expression 18 DRi = Vai / VBi Vai = dis (xix (i + 1)) · dis (x (i + 2) x (i + 3)) VBi = dis (xix (i + 2)) + dis ( x (i + 1) x (i + 3)), and the respective cross ratios DRi are values as shown in FIG. The size (width and height) of each unit trapezoid increases as it approaches the bottom of the pyramid.

Also, in this embodiment, the vertex direction DR2
Are determined so as to be substantially proportional to the distance from the apex. That is, from FIG. 18, the position of each layer is “17.000, 22.500, 31.000.
The difference between them is 5.500 (= 22.500-17.000) 8.500 (= 31.000-22.500)..., And the moving average of four consecutive differences among these series of differences is as shown in FIG. As can be seen from FIG. 19, the moving average of the difference between the positions of the layers (layer thickness) is gradually increasing, but the “moving average / (position of the layer from the top)”, that is, the variation of the value corresponding to the proportionality coefficient Is within about 20%. Therefore, the moving average is almost proportional to the distance from the vertex.

On the other hand, the distance between the intersections of the straight lines y1, y2... Extending radially from the apex with the straight lines x1, x2. Determined so that it can be identified. FIG. 20 shows an example in which the three-dimensional chart 2 is a hexagonal pyramid. In FIGS. 16 and 20, a, b, c, and d are intervals between the straight lines y1, y2,.

In this example,

Expression 19: DRα = (a · c) / {(a + b) · (b + c)} DRβ = (b · d) / {(b + c) · (c + d)} Each of the cross ratios DRβ: 1) Within each side Tj (j = 1-6), a straight line x
1, x2,..., And their values are coded such that they are common to the columns at their respective intersections with the straight lines y1, y2, and 2) are different from each other in different aspects.

<Identification of Photographing Location> FIG. 21 is a flowchart showing a process of identifying which part of the three-dimensional chart 2 is being photographed by the movable camera 11 and the subject camera 13 when photographing the three-dimensional chart 2. is there. FIG. 22 is an explanatory diagram of straight line grouping.

* Line grouping: First, light and dark edges of a captured image are extracted (step S91). Various methods are known for extracting edges, such as the Sobel operator. For example, Makoto Nagao, "Image Recognition Theory"
The algorithm disclosed in Corona, 1983 is used.
FIG. 22B shows an example in which edges are extracted from the image shown in FIG.

Next, a straight line is extracted from the extracted edges (step S92). How to extract straight lines is HOUGH
Transformation is known as a general method. For example, a plurality of straight lines can be extracted from an edge image using a method described in the above-mentioned Makoto Nagao's document, and the equation of a straight line in a two-dimensional plane on imaging can be determined. FIG. 22C shows an example in which edges are extracted from FIG. 22B.

The plurality of extracted straight lines are grouped into a plurality of groups as follows according to the properties of the straight lines (step S93).

[0119]-parallel straight lines (Hereinafter, this product is the α _i .i
Represents a straight line group having the same inclination); a straight line passing through the end point of each straight line belonging to α _i (this is β
); A straight line group intersecting α _i (hereinafter referred to as γ _i , where i corresponds to i of the intersecting straight line group α _i ).

In the example of FIG. 22C, the straight lines are grouped into two straight line groups α ₁ and α ₂ having different inclinations, corresponding straight line groups γ ₁ and γ ₂ , and furthermore, an intersection straight line β. Will be.

Further, since the intersection straight lines β and γ _i intersect at one point, this intersection corresponds to the marker 201 of the three-dimensional chart 2. Thus, it can be determined that the straight line corresponding to the side of the cone forming the three-dimensional chart 2 is the straight line β, the straight line parallel to the bottom surface is the straight line α _i , and the straight line passing through the side surface of the cone is the straight line γ _i . Thereby, the cross ratio of the intersections associated with each of the unit trapezoids on the image is calculated.

[0122] ※ Identification of photographing locations: First, before photographing, the straight lines in each side T1~Tn of 3D chart 2, intersections expressed in the absolute coordinate system X ₀ of (grid point) coordinates and, their The data of the cross ratio calculated from the interval between the intersections is stored in the RAM 151 (FIG. 8) in advance.
Then, the three-dimensional chart 2 is output by the camera 11 or 13.
Is photographed, three layers or unit trapezoids that are continuous in the vertex direction DR2 are specified from the image obtained thereby, and the cross ratio is calculated from their height.

FIG. 23 shows the same side view of the three-dimensional chart 2.
5 shows examples of images taken at different distances. FIG.
FIG. 23A shows an example taken from a long distance, and FIG. 23B shows an example taken from a short distance. The interval in the vertex direction DR2 of the unit trapezoid is substantially proportional to the distance from the vertex.

In order to calculate the cross ratio for the interval between the straight lines, it is only necessary to observe four consecutive points on the same straight line. That is, it is only necessary that four straight lines that are within the angle of view and that are continuous at intervals sufficient to accurately calculate the cross ratio can be observed in relation to another straight line that intersects them.
When a large number of straight lines exist in the image, for example, four straight lines having an interval equal to or larger than a predetermined threshold interval with the next straight line and arranged at a position immediately above the vertex (marker 201) and closest to the vertex Select Then, four intersections between the four straight lines and one straight line extending along the vertex direction DR2 are extracted, and the cross ratio of the intervals is calculated. Among the data obtained in this extraction, the data obtained by shooting with the movable camera 11 is the first extraction point data DP1 in FIG. 9, and the data obtained by shooting with the camera 13 for the subject is the second extraction point data DP2. is there.

In FIG. 23 (a), each is in the bottom direction D
Four straight lines x7 to x10 extending to R1 and defining the top and bottom of each of the seventh to ninth layers, and in FIG. 23B, four straight lines defining the top and bottom of each of the third to fifth layers x
3 to x6 can be selected as these four straight lines. By selecting the four straight lines in this manner, it is possible to sufficiently calculate the cross ratio in any image. If a plurality of sides have been photographed, for example, the side closest to the center of the image is selected.

Further, since the three-dimensional chart 2 has a pyramid shape, even if the three-dimensional chart 2 is photographed from various directions,
If the marker 201 is detected by automatic tracking, it is possible to sufficiently observe at least one side surface.

One of a plurality of unit trapezoids existing in the area sandwiched by the four straight lines selected in this way is
Select as the target unit trapezoid (target unit figure). The target unit trapezoid can be selected, for example, by a selection rule such that the unit trapezoid sandwiched between the two straight lines on the middle side among the above four straight lines and closest to the center of the screen is selected. In the example of FIG. 23, for example, unit trapezoids UPA and UPB can be selected as target unit trapezoids.

Then, for each of the four straight lines, the vertex direction DR
A cross ratio is obtained as a target cross ratio from the intervals on the image for No. 2, and the cross ratio value (FIG. 18) of the linear interval of each side surface stored in the RAM 151 in advance is compared with the target cross ratio to be collated. Thereby, the four straight lines are converted into the three-dimensional chart 2
It is possible to specify which layer of the three-dimensional chart 2 is the four straight lines defining the layers from which layer and the target unit trapezoid (step S94).

By increasing the size of the unit trapezoid in the vertex direction closer to the bottom surface of the pyramid, there are the following advantages.

First, from a relatively short distance, the three-dimensional chart 2
When only a relatively small number of unit trapezoids are present in the image (FIG. 23B), the marker 2
A relatively large unit trapezoid close to 01 is photographed.

Conversely, from a relatively long distance, the three-dimensional chart 2
When the photographing size of each unit trapezoid becomes relatively small by photographing, a unit trapezoid having a large actual size near the bottom surface of the pyramid exists in the image (FIG. 23).
(A)), it does not become much smaller as the observation size on the image.

Therefore, when photographing from a short distance,
In both the case of shooting from a long distance, the image always contains a unit figure having a size sufficient to ensure the accuracy in image processing, and as a result, it does not depend much on the shooting distance. The calculation accuracy can be increased. This is an advantage of increasing the size of the unit trapezoid in the vertex direction closer to the bottom surface of the pyramid.

By identifying the control unit trapezoid,
The relative position and orientation of the camera relative to the three-dimensional chart 2, i.e. the calculation of the external parameters in the absolute coordinate system X ₀ is enabled (step S95). This will be described below.

First, in the RAM 151, the compound ratio of the distances a, b, c, and d for each of the side surfaces T1 to Tn of the pyramid shown in FIG.
And of information each side T1~Tn of the pyramid is oriented in the direction of the absolute coordinate system X ₀ throat, associated with one another, it is previously stored as a table. Therefore, on one of the four straight lines (for example, in the example of FIG. 23A, a straight line x7 to which a side near the vertex among the sides of the target unit trapezoid belongs), an image of four intersections continuous in the bottom direction DR1 is displayed. Are identified, the cross ratio of those intervals is calculated, and the calculated ratio is compared with the above table to identify the side to which the target unit trapezoid belongs as the observation side almost directly facing the camera at that time. And depending on whether the observation side is one of sides Tl to Tn, the relative orientation of the camera relative to the three-dimensional chart 2 can be known in the absolute coordinate system X _0.

In order to know the relative attitude of the camera with respect to the three-dimensional chart 2 in more detail, for example, the coordinate value of the four vertices of the target unit trapezoid is used to define the outer periphery of the target unit figure.
Find the ratio of the lengths of the sides. This ratio changes depending on how much the imaging axis of the camera is tilted from the normal direction of the target unit trapezoid in the absolute coordinate system. Therefore, the direction of the imaging axis of the camera from the normal direction of the side surface can be specified from this ratio.

The target unit trapezoid is specified, and the three-dimensional chart 2
When the relative posture of the camera with respect to is obtained, the actual size information of the unit trapezoid corresponding to the target unit trapezoid is read out of the actual size information of each unit trapezoid stored in the RAM 151 in advance. Then, for the target unit trapezoid, the ratio r between the size on the image and its actual size is determined. The ratio r is a function of the distance L between the three-dimensional chart 2 and the camera and the relative attitude of the camera with respect to the three-dimensional chart 2, but since the relative attitude is obtained as described above,
Eventually, the distance L can be expressed as a function f (r) of the ratio r. Therefore, the distance L can be calculated from the ratio r by storing an arithmetic expression or a numerical value table corresponding to the function f (r). Distance L and 3D chart 2
The relative position of the camera with respect to the three-dimensional chart 2 is obtained from the relative posture of the camera with respect to.

As described above, by photographing the three-dimensional chart 2 according to the present embodiment, it is possible to accurately obtain the external parameters among the calibration parameters for calibrating the position and orientation of the camera.

<Modifications> ◎ Calculation of external parameters by selecting not only one target unit graphic but also a plurality of target unit graphics, calculating external parameters for each of them, and averaging the obtained external parameters. The accuracy is further improved.

In the above embodiment, the camera for subject 1
In step 3, the external parameters of the camera are calculated, and the subject camera 13 functions as a calibration information processing device. Instead, a computer 15 (FIG. 2) communicates with both cameras 11, 13 and this computer 15
Accordingly, it is possible to control the synchronization of the cameras 11 and 13, perform arithmetic processing on image data, and save data. In this case, the computer 15 functions as a camera information processing device.

A configuration may be adopted in which the movable camera 11 performs arithmetic processing of image data to calculate external parameters, and communicates with the computer 15 or the subject camera 13. Information transmission between these devices may be performed via a communication cable.

In the above embodiment, the calculation of the external parameters of the movable camera 11 is performed only when the subject camera 13 performs shooting, but the three-dimensional chart captured by the movable camera 11 in real time is used. From the two images,
By sequentially finding out what part of the three-dimensional chart 2 is being captured,
The external parameters may be calculated in real time. In this case, the position of the vertex of the three-dimensional chart 2 on the image is
The marker 201 can be omitted because it is grasped in real time.

Also, in the above embodiment, after the straight lines extracted from the image are grouped, the intersection of the straight lines,
That is, the lattice point coordinates of the unit trapezoid are obtained, but after extracting the coordinates of the lattice points by the corner extraction operator,
A straight line passing through the grid points may be determined. Alternatively, how to project a straight line on the three-dimensional chart 2 to match with the obtained edge image may be obtained by repeatedly performing an arithmetic operation process, thereby calculating an external parameter.

In extracting the four straight lines in the above embodiment, the vertices of the three-dimensional chart 2 are not always essential. Therefore, as the three-dimensional chart 2, a three-dimensional body in which a portion including a vertex of the pyramid can be cut off by a plane parallel to the bottom surface, for example, a truncated pyramid-shaped solid as shown in FIG.

The adjacent unit trapezoids may be painted in different hues.

The movable camera 11 and the subject camera 13 may be indirectly connected via a connecting mechanism.

The three-dimensional chart 2 can be placed on the floor. In this case, it is preferable that the bottom surface of the pyramid face downward.

The three-dimensional chart 2 may be a polygonal pyramid (unequal polygonal pyramid) other than a regular polygonal pyramid. However, the use of the regular polygonal pyramid has an advantage that the observation accuracy of the unit figure from each surrounding direction is substantially the same.

立体 As shown in FIG. 24B, the three-dimensional chart 2
May be used as a cone. In this case, since the chart is drawn on a curved surface as a single side surface, the cross ratio of the interval between the curves is determined when the unit trapezoid is identified. In the case of a cone, there is only one side,
It is not necessary to distinguish the side surfaces from each other, and the cross ratios of the sides of each unit trapezoid in a range making a circuit along the side surface are all different. In the case of a cone, a unit figure can be observed with the same accuracy in all directions around the cone.

The manual operation of the movable camera 11
An apparatus configuration that can be performed by operation keys of the subject camera 13 may be used.

The marker 201 may be a material which does not have a light emitting property and is coated with a fluorescent paint, or the like, and even a simple vertex can be used for tracking.

[0151]

As described above, according to the first to eighth aspects of the present invention, the unit graphic exists in each direction, and at least one unit is obtained even when the three-dimensional chart is observed from any direction. The camera calibration parameters can be observed from a direction close to the front of the camera, and the coding can identify the unit figure formed on which side of the cone and on which position. High measurement accuracy can be obtained while securing a wide movable range in measurement.

In particular, according to the second aspect of the present invention, since the size increases as the position becomes closer to the bottom surface of the cone, the image of the chart is photographed with the vertex side as the center, regardless of the distance between the three-dimensional chart and the camera. Instead, the size of the unit graphic in the image can be made relatively large.

In particular, according to the third aspect of the present invention, since each cone is used, each side surface becomes a flat surface, and image processing thereof is easy.

In particular, according to the fourth aspect of the present invention, since the coding is performed by the cross ratio of the height of the trapezoid, the unit figure can be identified by a relatively simple algorithm.

In particular, according to the fifth aspect of the present invention, since the marker is provided at the vertex of the pyramid, the tracking is easy and accurate.

In particular, according to the invention of claim 6, since the marker includes the luminous body, tracking thereof is particularly easy and accurate.

In particular, in the invention according to the eighth aspect, since the adjacent unit figures have different brightness or hue colors, the image recognition is easy.

According to the ninth aspect of the present invention, the unit figure is present in each direction with respect to one side of the cone, and at least one unit figure is close to the front of the three-dimensional chart from any direction. In addition to being observable from the direction, the coding enables the unit figure to be identified in which part of the cone the unit figure is formed. High measurement accuracy can be obtained.

According to the tenth to twelfth aspects of the present invention, the calibration parameters of the camera can be measured with high measurement accuracy while securing a wide movable range by using the three-dimensional charts of the first to eighth aspects. It is possible.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration example of an imaging operation system using a calibration chart according to an embodiment.

FIG. 2 is a block diagram of the imaging operation system of FIG. 1;

FIG. 3 is a diagram illustrating an example of a side surface of the calibration three-dimensional chart according to the embodiment;

FIG. 4 is a front view of a movable camera for photographing a chart.

FIG. 5 is a block diagram of a movable camera for photographing a chart.

FIG. 6 is a diagram showing a main part of an information processing function of the movable camera for photographing a chart.

FIG. 7 is a diagram showing a data flow of a movable camera for photographing a chart.

FIG. 8 is a diagram illustrating a main part of an information processing function of the digital camera for photographing a subject.

FIG. 9 is a diagram showing a data flow of a movable camera for photographing a chart.

FIG. 10 is a diagram showing a procedure corresponding to a first sub-process among procedures of photographing and calibration using the camera calibration chart according to the present embodiment.

FIG. 11 is a diagram showing a procedure corresponding to a second sub-process among the procedures of photographing and calibration using the camera calibration chart according to the present embodiment.

FIG. 12 is a diagram illustrating an example of imaging when a movable camera for photographing a chart and a digital camera for photographing a subject simultaneously photograph a camera calibration chart according to the present invention.

FIG. 13 is a diagram illustrating a state of coordinate conversion used when calculating a relative position and orientation of a digital camera for photographing a subject with respect to a movable camera for photographing a chart.

FIG. 14 is a diagram showing an example of imaging by a movable camera for chart imaging and a digital camera for object photography when the object photographing digital camera photographs an object.

FIG. 15 is a diagram illustrating a state of coordinate conversion used for calculating a relative position and orientation of a subject photographing digital camera with respect to a chart photographing movable camera.

FIG. 16 is a diagram illustrating an example in which the size of a unit trapezoid is coded by a cross ratio on the side surface of the three-dimensional chart according to the embodiment.

FIG. 17 is a diagram illustrating that the cross ratio does not change due to projection.

FIG. 18 is a diagram illustrating an example of a cross ratio used for coding.

FIG. 19 is a diagram showing that the moving average of the interval between straight lines obtained from FIG. 18 is almost proportional to the distance from the vertex.

FIG. 20 is a diagram illustrating an example of coding used for each surface when the three-dimensional chart according to the present embodiment is a hexagonal pyramid.

FIG. 21 is a diagram showing a procedure for identifying a photographing location on a three-dimensional chart from an image photographed of the three-dimensional chart according to the present embodiment.

FIG. 22 is a diagram illustrating an example of extraction of a straight line in imaging of a three-dimensional chart according to the present embodiment.

FIG. 23 is a diagram illustrating an example of imaging when the same side surface of the three-dimensional chart according to the present embodiment is imaged from different distances.

FIG. 24 is a diagram showing an example of a three-dimensional chart which is a modification of the present embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Imaging calculation system 2 3D chart for calibration 11 Movable camera 13 Camera for subject 30 Subject 201 Marker UP Unit trapezoid (unit figure)

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 5/232 H04N 5/232 Z 13/02 13/02 // G01B 11/00 G01B 11/00 H11 / 26 11/26 H (72) Inventor Koji Fujiwara F-term (reference) 2F065 AA04 AA31 FF05 MM25 2H054 AA01 2H059 AA04 AA21 5C022 2-13-13 Azuchicho, Chuo-ku, Osaka-shi, Osaka AB62 AC26 AC69 5C061 AA20 AB03 AB08 BB11 CC01

Claims (12)

[Claims] 1. A conical body having a bottom surface and a plurality of side surfaces; and a plurality of charts displayed on each of the plurality of side surfaces, wherein each of the plurality of charts has a known size. Including a set of a plurality of unit figures, the geometric shape of each of the plurality of unit figures is a relative position and a relative position between each unit figure and an observation position, both inside the set and between the plurality of charts. What is claimed is: 1. A three-dimensional chart for camera calibration, which is coded so as to be mutually identifiable by observation from the observation position irrespective of a target posture. 2. The three-dimensional chart for camera calibration according to claim 1, wherein the plurality of unit figures increase in size closer to the bottom surface of the cone. 3. The three-dimensional chart for camera calibration according to claim 2, wherein the pyramid is a pyramid, and the plurality of charts include the plurality of charts irrespective of a relative position and a relative posture between each chart and an observation position. A three-dimensional chart for camera calibration, which is coded so as to be mutually identifiable by observation from a position. 4. The three-dimensional chart for camera calibration according to claim 3, wherein the plurality of unit figures are radially arranged from a plurality of first straight lines parallel to a bottom surface of the cone and a position corresponding to a vertex of the cone. A plurality of trapezoids formed by intersecting with a plurality of second straight lines extending to a plurality of trapezoids, and a cross ratio of the sizes of the plurality of trapezoids is coded. 5. The three-dimensional chart for camera calibration according to claim 4, wherein a marker is provided at a vertex of the pyramid. 6. The three-dimensional chart for camera calibration according to claim 5, wherein the marker includes a light emitter. 7. The three-dimensional chart for camera calibration according to claim 3, wherein the pyramid is a truncated pyramid. 8. The camera calibration three-dimensional chart according to claim 1, wherein mutually adjacent unit figures of the plurality of unit figures have different brightness or hue colors. A three-dimensional chart for camera calibration. 9. A pyramid having a bottom surface and a side surface, and a chart displayed over one side of the side surface, wherein the chart includes a set of a plurality of unit figures each having a known size. The respective geometric shapes of the plurality of unit figures can be distinguished from each other by observation from the observation position, regardless of the relative position and relative attitude between each unit figure and the observation position inside the set. 3D chart for camera calibration characterized in that it is coded. 10. A method of acquiring a calibration parameter of a camera, comprising: arranging the three-dimensional chart for camera calibration according to claim 1 in a space in which a subject is to be accommodated; Observing at least one of the plurality of charts of the camera calibration three-dimensional chart from an observation position attached to the camera; and extracting at least one unit figure included in the observed chart as a target unit figure. , The target unit figure is
A step of identifying among the plurality of charts and the plurality of unit figures, an actual size previously specified for the identified target unit figure, and an observation size of the target unit figure viewed from the observation position. Obtaining a calibration parameter of the camera, comprising: determining a calibration parameter of the camera depending on a relative position and a relative posture between the observation position and the three-dimensional chart for camera calibration from the relationship. Method. 11. An apparatus for obtaining a camera calibration parameter, wherein at least one unit is obtained from a chart image obtained by observing the camera calibration three-dimensional chart according to claim 1 from the observation position. An identification unit for extracting a figure as a target unit figure, identifying the target unit figure between the plurality of charts and in the plurality of unit figures, and an actual unit specified in advance for the identified target unit figure From the relationship between the size and the observation size of the target unit figure viewed from the observation position, the camera calibration parameters dependent on the relative position and relative attitude between the observation position and the camera calibration stereo chart are calculated. An information processing apparatus for calibrating a camera, comprising: 12. A program executed by a computer to cause the computer to function as the camera information processing apparatus according to claim 11.