JP2003021505A - Image photographing system and mobile imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system for easy acquisition with precision, the position and attitude of a person and object in the space. SOLUTION: On the side surface of a solid chart 2, a plurality of unit graphics, of which the sizes are coded with referent compound ratios, are arranged. The compound ratio of the unit graphic is calculated from the photographed image of the solid chart 2, which is collated with the actual value to acquire the position/attitude of an imaging point. An object-photographing camera 13 is fitted with a mobile camera 11 which automatically tracks the solid chart 2, and the mobile camera 11 makes the three-dimensional chart 2 to be imaged, at the same time when photographing an object 30. The position/attitude of the object-photographing camera 13 is acquired, based on the position/attitude of the mobile camera 11 which is acquired by the procedure based on the imaged picture as well as the relative position/attitude between both cameras which has been acquired in advance.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for specifying a relative relationship between a three-dimensional chart (reference object) fixed in a space and a photographing position of a movable photographing device.

[0002]

2. Description of the Related Art A technique for obtaining a relative position and a relative posture of a person or an object in space is required in the field of virtual reality. For example, `` L.
Quan, Z.Lan, "Linear N-Point Camera Pose Determinat
ion ", IEEE Trans.PAMI 21 (8) 1999", and "Takahashi,
Ishii, Makino, Nakashizuka, "Measurement of Position and Pose of Marker from Monocular Image for Artificial Reality Interface", Electronic Information Journal AJ79 1996, discloses algorithms for obtaining them. According to these techniques, when a reference object is photographed, the relative position and relative attitude of the photographing position with respect to the space can be obtained using the photographed image. Hereinafter, the algorithm disclosed here will be referred to as a “multipoint analysis algorithm”.

As an application field of the "multipoint analysis algorithm", for example, in Japanese Unexamined Patent Publication No. 6-080403, a camera is attached to a moving body and a chart having a known three-dimensional position is photographed to detect the position and orientation of the moving body. Techniques for doing so are disclosed.

There is also known a technique in which a three-dimensional image model of a subject is obtained by photographing a three-dimensional subject from a plurality of directions and combining a plurality of image data obtained thereby. That is, if the data of the external parameters of the camera (such as the position and orientation of the camera) and the internal parameters (such as the focal length) can be obtained for each of the images obtained by shooting the subject from a plurality of directions, the shape from silhouette method can be used. Thus, a three-dimensional model can be reconstructed from the silhouette image of the subject. For more information about this shape from silhouette method, see W. Niem, "Robust and Fast Modeling of 3D Natu.
ral Objects from Multiple Views "SPIE Proceedings
Image and Video Proceeding II vol.2182,1994, pp.388
-397. Below, the external and internal parameters of the camera are referred to as “(calibration parameters of the camera)”.
If the internal parameters of these calibration parameters are known and the calibration of the camera by the internal parameters has been completed, if the external parameters of the camera are obtained, then a three-dimensional image model of the subject can be constructed. It will be possible.

By the way, one method for photographing an object from a plurality of directions in this way is a fixed arrangement method in which a plurality of cameras are fixedly arranged at different positions and the object is photographed. However, in this fixed arrangement method, since a plurality of cameras must be fixedly arranged in a photographing studio or the like, the photographing equipment becomes large.

Therefore, a moving photographing method has been proposed in which a user holds a single hand-held camera and moves around the subject while sequentially photographing the subject from a plurality of directions to obtain an image of the entire circumference of the subject. There is.

However, in order to determine the external parameters of the camera in this moving photographing method, it is necessary to specify the position and orientation of the camera at each photographing.

As a method for measuring the external parameters of the camera for such a purpose, a magnetic method, an ultrasonic method, an optical method, etc. have been conventionally proposed. Among them, the magnetic method is a method for detecting the geomagnetism or the like at the camera position, and the ultrasonic method is a method for detecting the ultrasonic wave from a predetermined ultrasonic wave to specify the position and posture of the camera. The optical method includes a method of using a stereo camera and a method of setting a calibration chart larger than the field of view.

Of these methods, the magnetic method makes it difficult to measure accurately when the object is made of metal, and the ultrasonic method makes the apparatus expensive.

On the other hand, as a conventional optical system,
A single plane chart on which a non-uniform matrix pattern is drawn is placed at a predetermined position, and by observing it with a camera, the position and orientation relationship between the plane chart and the camera can be calculated using a "multipoint analysis algorithm". The specifying method is disclosed in Japanese Patent Laid-Open No. 2000-270343. According to this, the relative relationship between the position and orientation of the camera with respect to the coordinate system fixed on the plane chart can be known. Therefore, if the position and orientation relationship between the plane chart and the object is fixed, the object can be viewed from a plurality of directions. By observing the pattern on the plane chart with the camera every time you take a picture,
The position and orientation of the camera at that time can be specified in the absolute coordinate system.

[0011]

However, in the technique disclosed in Japanese Patent Laid-Open No. 6-080403, a large plane chart must be prepared in the background in order to keep the chart within the photographing range of the moving body. There was a problem that the imaging equipment would be large-scale.

Also, in the technique disclosed in Japanese Patent Laid-Open No. 2000-270343, even if the camera is within the range where the chart can be observed, if the camera is located in the peripheral portion of the angle of view of the camera, the pattern on the chart However, there is a drawback in that the accuracy of observation decreases, and as a result, the accuracy of determining the external parameters of the camera is poor.

The present invention has been made in order to solve the above-mentioned problems in the prior art. While using an optical system, a wide movable range is ensured and a high measurement accuracy is achieved without burdening the user. The objective is to realize a technique for acquiring the external parameters of the camera so that

[0014]

According to a first aspect of the present invention, there is provided an image photographing system, which comprises a movable photographing device and a movable photographing device.
A movable body photographing device, which is provided with a reference object having a dimensionally known shape, and which rotates the driving means with respect to a fixed part; The driving means and the control means for controlling the driving means so that the photographing means can photograph the reference object when the movable photographing device moves.

The invention according to claim 2 is the image photographing system according to claim 1, further comprising a calibration information processing device, wherein the movable photographing device is connected to the calibration information processing device. Further comprising first communication means for performing data communication with the calibration information processing apparatus, the second communication means for performing data communication with the movable imaging device, and the second communication means for receiving the second communication means. Based on the first image data,
A first calculation means is provided for calculating a first parameter for calibration of the movable image capturing device, which is dependent on a relative position and a relative orientation of the image capturing position of the movable image capturing device and the reference object.

According to a third aspect of the invention, there is provided the image photographing system according to the second aspect, further comprising a subject photographing device for obtaining a subject image as second image data,
The movable photographing device has a mounting means for mounting the fixed part to another object, a means for obtaining a rotation angle of the photographing means from the reference posture rotated by the driving means, and the subject photographing device by the mounting means. And an image pickup control unit attached to the image pickup unit for causing the image pickup unit to obtain the first image data in association with the acquisition of the second image data of the subject image pickup apparatus. Apparatus includes a third communication means for transmitting the second image data to the calibration information processing apparatus, and the calibration information processing apparatus, based on the second image data obtained by photographing the reference object, A second parameter for calculating a second parameter for calibration of the subject photographing device that depends on a relative position and a relative posture of the photographing position of the photographing device and the reference object
Based on the calculation means, the first parameter, the second parameter, and the rotation angle, depending on the relative position and relative attitude between the photographing position of the photographing means in the reference posture and the photographing position of the subject photographing device. A third calculating means for calculating a third parameter for calibration of the object photographing device, and a relative position between the photographing position of the object photographing device and the reference object based on the first parameter and the third parameter. It further comprises a fourth calculation means for calculating a fourth parameter for calibration of the device for photographing an object depending on the position and the relative attitude.

The invention described in claim 4 is the image photographing system according to claim 2, wherein the movable photographing device comprises:
Attachment means for attaching the fixing part to another object, means for obtaining a rotation angle of the photographing means from the reference posture rotated by the driving means, relative to the photographing position in the reference posture of the photographing means and the fixing part A means for acquiring reference relative data depending on a position and a relative attitude; and a fifth means depending on a relative position and relative attitude between the photographing position of the photographing means and the fixed portion based on the rotation angle and the reference relative data. Further comprising means for calculating a parameter, wherein the calibration information processing device uses the fifth parameter and the first parameter received by the second communication means to determine the position of the fixed part and the relative position of the reference object, A sixth parameter, which depends on the relative pose, is calculated.

According to a fifth aspect of the present invention, in the image photographing system according to the fourth aspect, the movable photographing device comprises:
It can be attached to a human body by the attaching means.

According to a sixth aspect of the present invention, there is provided a movable photographing device, wherein the photographing means for obtaining an image of a reference object having a known three-dimensional shape as the first image data and the fixed part. The imaging device includes a driving device that rotationally drives the imaging device, and a control device that controls the driving device so that the imaging device can image the reference object when the movable imaging device moves.

According to a seventh aspect of the present invention, there is provided the movable photographing apparatus according to the sixth aspect, wherein the fixing portion is attached to a subject photographing device that acquires a subject image as second image data. And an image capturing control unit that causes the image capturing unit to acquire the first image data in association with the acquisition of the second image data of the subject image capturing apparatus.

The invention described in claim 8 is the movable photographing device according to claim 6, wherein the fixing means is a reference for the mounting means for mounting the fixed portion to another object and the photographing means rotated by the driving means. And a relative attitude between the photographing position of the photographing means and the fixed part based on the positional relationship between the photographing means and the fixed part in the reference posture and the rotation angle. Calculate a fifth parameter that depends on

According to a ninth aspect of the invention, there is provided the movable photographing apparatus according to the eighth aspect, which can be attached to a human body by the attaching means.

[0023]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<1. First Embodiment><Outline of System Configuration> FIG. 1 is a diagram showing a configuration of an image capturing system 1 to which an embodiment of the present invention is applied, and FIG. 2 is a block diagram of the image capturing system 1. is there. Figure 1
In the image capturing system 1, the three-dimensional object 30
A portable camera system 10 capable of picking up an image and a stereoscopic chart 2 for camera calibration arranged in the vicinity of the subject 30 in a space accommodating the subject 30. The three-dimensional chart 2 is a reference object having a known three-dimensional shape, and is formed of, for example, a three-dimensional object in which a chart pattern is provided on each side surface of a substantially pyramidal main body, as described later in detail. This three-dimensional chart 2 is a chart support tool 2.
Suspended from 50. The chart support 250 includes an inverted L-shaped arm 252 extending from the base 251.
The three-dimensional chart 2 is fixed near the tip of the arm 252. Preferably, the three-dimensional chart 2 is suspended substantially above the subject 30.

The camera system 10 includes a subject photographing camera (hereinafter, abbreviated as "subject camera") 13 having a function as a digital camera. The subject camera 13 also has various calculation functions, which will be described later, as a calibration information processing device in the present system. Further, a movable camera 11 is attached to the upper portion of the subject camera 13 via an attachment mechanism 12 so that the posture can be changed.
The movable camera 11 identifies a relative position-orientation relationship between the three-dimensional chart 2 and the movable camera 11 by photographing a plurality of unit figures UP included in the pattern (see FIG. 3) on the three-dimensional chart 2. And then 3D chart 2
It is used to detect the position and orientation of the subject camera 13 in an absolute coordinate system that is relatively fixed with respect to.

Although not shown in FIG. 1, as shown in FIG. 2, the image capturing system 1 includes, for example, a notebook type portable computer 1 as a calibration information processing device.
5 may be provided. The computer 15 can send and receive commands and data to and from the camera system 10 by wireless communication via the communication interface 15a.

<Outline of Solid Chart> FIG. 3 is a side view of the solid chart 2. The three-dimensional chart 2 has a three-dimensional chart body 203 and a chart pattern CP formed on the surface of the three-dimensional chart body 203.

Of the three-dimensional chart main body 203, a polygonal pyramid-shaped display portion 204 and a truncated pyramidal support portion 205 are integrated, and the inside thereof is hollow. The chart pattern CP is a set of patterns P1 to Pn provided on each side surface T1 to Tn (n is an integer of 3 or more) of the display section 204. Preferably, the number n of side surfaces of the polygonal pyramid is n = 3 to 3
6 and more preferably n = 6-12. Although the respective patterns P1 to Pn formed on the respective side surfaces T1 to Tn are planar patterns, the chart patterns CP as a set of the patterns P1 to Pn are formed by arranging the patterns P1 to Pn three-dimensionally. It has a three-dimensional pattern. Each of the patterns P1 to Pn is a set of a plurality of trapezoids each of which functions as a unit figure, the details of which will be described later.

A marker 201, which is a reference point when the movable camera 11 tracks the chart pattern CP, is provided at the apex of the polygonal pyramid forming the display section 204.
As an example, a light emitting diode (LED) is attached so that the position of the three-dimensional chart 2 can be easily and accurately recognized by the movable camera 11. Although not shown in FIG. 3, a marker power supply 202 (FIG. 2) for supplying light emission power to the light emitting diode is built in the three-dimensional chart 2.

<Outline of Movable Camera 11> FIG. 4 is a front view of the movable camera 11, and FIG. 5 is a block diagram of the movable camera 11. As shown in FIG. 5, the movable camera 1
In 1, the spherical unit 116 is formed by integrating the lens unit 110 and the two-dimensional light receiving element 111 that photoelectrically converts the two-dimensional image formed by the lens unit 110.
Is stored in. The two-dimensional light receiving element 111 is a CCD array. The lens unit 110 is a fixed lens 110.
It is a combination of a and a zoom lens 110b, and an aperture / shutter mechanism section 110e exists between them.

As shown in FIG. 4, the spherical unit 116 is connected to the fixed portion 114 via the attitude device 113, and together with the respective elements incorporated in the spherical unit 116, ± about the pan direction ± about the fixed portion 114. It is possible to turn by 70 ° (θ rotation) and ascend / descend (± rotate) about ± 70 ° in the tilt direction. Then, in order to perform the rotational driving in the pan direction and the rotational driving in the tilt direction, the attitude device 113 having a plurality of piezo elements is incorporated into the spherical unit 11.
It is located at the base of 6. In addition, the zoom lens 110
The zoom operation corresponding to the drive of b is also performed by the piezo element different from the above. By giving a sawtooth wave signal to these piezo elements, the element to be driven by the piezo element is moved slightly, and the required movement is given to the object element by repetition thereof. The turning angle in the pan direction and the depression / elevation angle in the tilt direction are respectively determined by an angle sensor 126 such as an encoder.
The zoom lens 110b detected by p.126t
The driving amount of the sensor 126 is also composed of an encoder.
detected by z. For these drive mechanisms,
For example, JP-A-11-18000 and JP-A-11-41
504.

The control calculation unit 120 includes the two-dimensional light receiving element 11
An image processing unit 121 for inputting a signal from No. 1 to perform processing such as image recognition, and an image memory 122 for storing the image signal obtained by the image processing unit 121 are provided. Further, a camera control unit 123 is provided which generates drive signals for the zoom lens 110b, the posture device 113, and the aperture / shutter mechanism unit 110e and outputs them.
The image processing unit 121 and the camera control unit 123 can wirelessly communicate with the subject camera 13 via the communication unit 124 and the communication device 112. By this communication, the image data is transmitted to the subject camera 13, and various information is transmitted and received between the movable camera 11 and the subject camera 13. In the movable camera 11 of this embodiment, an IR for performing infrared communication is used as the communication device 112.
An infrared element compatible with a DA (Infrared Data Association) interface is used.

As shown in FIG. 4, the first mounting groove 115a and the second mounting groove 115b provided in the fixed portion 114 are
It is used to attach the fixed portion 114 to the camera 13 for a subject. In addition, the tracking button 117 is a mode in which the movable camera 11 automatically tracks the stereoscopic chart 2 (hereinafter, abbreviated as “automatic tracking mode”) and a mode in which the camera for subject 13 tracks the user according to a user's instruction (hereinafter, referred to as “automatic tracking mode”). It is a button for switching between "manual mode" and abbreviated.

FIG. 6 is a diagram showing the main part of the information processing function of the movable camera 11 as viewed from the hardware configuration, and FIG. 7 is a diagram showing the data flow in the movable camera 11. In FIG. 6, the control calculation unit 12 of the movable camera 11
A program 0 includes a CPU 130, a ROM 131, and a RAM 132, and realizes various operations described below.
31a is stored in the ROM 131.

The two-dimensional light receiving element 111 has an RG for each pixel.
One of the filters B is attached, and the light imaged on the two-dimensional light receiving element 111 is
1 performs photoelectric conversion for each of the RGB three primary color components. The signal thus obtained is converted into a digital image signal by the A / D conversion unit 141, and the image correction unit 142 receives white balance correction and γ correction. The corrected image signal is stored in the image memory 122. The first image data D1 in FIG. 7 corresponds to this corrected image signal.

The recognition unit 145 and the attitude control unit 146 shown in FIG.
Are implemented as part of the functions of the CPU 130, ROM 131, RAM 132, and the like.

The recognition unit 145 is activated in response to the user's instruction from the tracking button 117, and the movable camera 11 is activated.
The tracking data DF for recognizing the image of the stereo chart 2 from the acquired first image data D1 and tracking the image of the stereo chart 2 in the first image data D1 is created.

The posture control unit 146 controls the posture device 113 based on the user's instruction received from the subject camera 13 in the manual mode. After the tracking button 117 is pressed and the mode is changed to the automatic tracking mode, the attitude device 113 is controlled by the processing described later so that the image of the stereo chart 2 is always formed on the two-dimensional light receiving element 111. It

The CPU 130 also has a function of creating the rotation angle data DR shown in FIG. 7 by the processing described later.

When the shutter button of the subject camera 13 is pressed, the first image data D1 and the rotation angle data DR are transmitted to the subject camera 13 as the calibration information processing device via the communication section 124. And is used for various calculations. That is, the movable camera 11 works in conjunction with the shutter button of the subject camera 13 to generate the first image data D1.
Is controlled so that the acquisition of

<Principle of Automatic Tracking> FIG. 8 is a flowchart showing details of the automatic tracking operation in the movable camera 11.
First, in the manual mode, the angle of view of the movable camera 11 is adjusted so that the marker 201 of the three-dimensional chart 2 is included (step S11), and the tracking button 117 is pressed (step S12) to change the mode to the automatic tracking mode.

Next, the movable camera 11 acquires the first image data D1 (step S13), and the recognition unit 145 detects the position of the photographing range of the marker 201 of the three-dimensional chart 2 by image recognition (step S14). , By comparing with the position in the immediately preceding shooting range, the variation amount and the direction are obtained, and the tracking data DF is generated (step S15).

Further, the attitude control unit 146 causes the tracking data D
Based on F, the posture device 113 is controlled so that the marker 201 of the three-dimensional chart 2 is photographed at the center of the angle of view (step S16). Then, the processing from step S13 is further repeated to acquire new first image data D1.

As a result, the movable camera 11 can acquire the image of the three-dimensional chart 2 as the first image data D1. Further, even when the movable camera 11 moves, the attitude device 113 can be controlled so that the movable camera 11 can capture the stereo chart 2, and the stereo chart 2 required for the processing described later is always displayed. The first image data D1 photographed at the center of the angle of view can be photographed without the user's awareness.

<Outline of Subject Camera 13> FIG. 9 is a diagram showing the main part of the information processing function of the subject camera 13 as viewed from the hardware configuration. FIG. 10 shows the data in the subject camera 13. It is a figure which shows a flow. The subject camera 13 includes a CPU 150, a RAM 151 and a ROM.
The program 152a, which is provided with 152 and realizes various operations of the subject camera 13 described later, is stored in the ROM 15
It is stored in 2. Elements such as the shutter button 161, the flash 162, the monitor color display 163 installed on the rear surface, and the operation buttons 164 are also electrically coupled to the CPU 150.

As shown in FIGS. 9 and 10, the light incident from the subject 30 via the lens unit 155 is provided with a filter of any of RGB for each pixel, C
An image is formed on the two-dimensional light receiving element 156 such as a CD array, and 2
The three-dimensional light receiving element 156 photoelectrically converts each of the RGB three primary color components. The signal thus obtained is converted into a digital image signal by the A / D conversion unit 157, and the image correction unit 158 receives white balance correction, γ correction and the like. The corrected image signal is stored in the image memory 159. Photographing is executed by pressing the shutter button 161, and the image signal stored in the image memory 159 is stored in the RAM 151 as the second image data D2.

The communication unit 167 transmits / receives to / from the movable camera 11 various control information such as control signals of each unit of the movable camera 11 and acquired image data via the communication device 168. For example, in the manual mode, by transmitting a signal obtained by the user operating a part of the operation buttons 164 to the movable camera 11, the attitude device 113 of the movable camera 11 is manually operated by the user. It becomes possible to operate. In addition, in response to the pressing of the shutter button 161 of the subject camera 13, it is possible to perform the simultaneous shooting of the subject camera 13 and the movable camera 11.

The communication device 168 is the movable camera 11
IRDA (Infrared Data As)
Infrared element as an interface, which is driven by the communication unit 167.

The card slot 165 is used for the subject camera 1.
3 is used to mount the memory card 166, and the memory card 166 can store photographed image data and the like.

The extraction unit 171, the calculation unit 173, and the display control unit 174 of FIG. 10 are the CPU 150 and the RAM 1 of FIG.
51, ROM 152 and the like.

The extraction unit 171 extracts from the first image data D1 received from the movable camera 11 via the communication unit 167,
The four points on the three-dimensional chart 2 are extracted to create the first extraction point data DP1. In addition, the extraction unit 171 similarly
Second image data D acquired by the subject camera 13
The second extraction point data DP2 is created by extracting the four points on the three-dimensional chart 2 from 2.

The calculation unit 173 calculates the first extraction point data DP.
1, rotation angle data DR, and second extraction point data DP2
From the above, the relative position and orientation of the movable camera 11 and the subject camera 13 are obtained, and the relative position data DPS is created. Furthermore, the first extraction point data DP1 and the rotation angle data D
From R and the relative position data DPS, the relative position and orientation between the subject camera 13 and the three-dimensional chart 2 is obtained,
Imaging data DM is created. Image data DM is RAM
It is stored in 151.

The display control unit 174 has operation buttons 164.
The second image data D2 based on the user's instruction from
And the shooting data DM are acquired from the RAM 151,
Save in memory card 166. In addition, the display control unit 17
4 displays the data by performing necessary processing on various data.
It also has a function of displaying on 63 and reading various data stored in the memory card 166 onto the RAM 151.

<Principle of Camera Calibration> When a subject 30 is photographed by the subject camera 13 from an arbitrary direction to obtain an image, the three-dimensional chart 2 or the absolute coordinate system fixed thereto is taken when the photograph is taken. , Subject camera 1
It is necessary to specify the relative position and orientation of No. 3 as external parameters. This is because the spatial mutual relation between the images is required to construct the three-dimensional image model of the subject 30 by combining the images obtained by photographing from a plurality of directions.

However, the subject 30 is actually photographed.
When shooting, the three-dimensional char is placed within the angle of view of the subject camera 13.
It may be difficult to insert G2. Therefore, X₀: Coordinate system fixed to 3D chart 2 (absolute coordinates
system); X₁: Coordinate system fixed to the movable camera 11 (first row
Cull coordinate system); X₂: The coordinate system fixed to the subject camera 13 (second
Local coordinate system); τ₀₁: First local coordinate system X₁To absolute coordinate system X₀Change to
Conversion relationship; τ₀₂: Second local coordinate system X₂To absolute coordinate system X₀Change to
Conversion relationship; τ₁₂: Second local coordinate system X₂To the first local coordinate system
X₁The following relation, which holds when
To use. (Τ₀₁, Τ₀₂, Τ₁₂And Q described later₀, Q₂Is
(Not shown) τ₀₂   = τ₀₁・ Τ₁₂  ... (Equation 1) Transformation relation τ₀₁, Τ₁₂Is known, the transformation relation τ₀₂Is wanted
Can be Transformation relation τ ₀₂Is calculated, the two-dimensional image
Second local coordinate system X of the camera 13 for the photographed subject₂
The position / orientation at₀₂To act
By this, the absolute coordinate system X₀As the position and posture in
Desired. Of the subject camera 13 in the absolute coordinate system
Q is the matrix that represents the position and orientation₀, The second local coordinate system X₂To
Q is the matrix that represents the position and orientation of the subject camera 13 in
₂Then,     Q₀   = {Τ₀₁・ Τ₁₂} Q₂ = τ₀₂・ Q₂  ... (Equation 2) It will be obtained like.

Therefore, each time the subject camera 13 photographs the subject 30 while moving with respect to the subject 30, a conversion relation τ ₀₂ corresponding to the photographing is obtained, and if this is attached to the photographed image, a plurality of images are obtained. The three-dimensional image model of the subject 30 can be obtained by combining the images captured in the directions with X ₀ .

Specific processes (details will be described later) for realizing this principle are roughly classified into a first sub-process and a second sub-process.

* First sub-process: This is a sub-process for specifying the conversion relation τ ₁₂ between two camera coordinate systems.

First, the three-dimensional chart 2 is moved to the movable camera 11
And the camera 13 for the subject are simultaneously photographed, and the photographed results are used to determine the external parameter of each camera, that is, the position and orientation of each camera in the absolute coordinate system X ₀ .

This is the conversion relation τ ₀₂ , τ in that state.
Corresponds to specifying ₀₁ . Then, from the relation of τ ₁₂ = (τ ₀₁ ) ⁻¹ τ ₀₂ (Equation 3) obtained from the equation ₁ , the conversion relation τ between the first local coordinate system X ₁ and the second local coordinate system X _{2 Get twelve} .

Since the values of the rotation angles θ and φ of the movable camera 11 are known values detected by the angle sensors 126p and 126t, respectively, the rotation angle dependent portion is separated from the conversion relation τ ₁₂ . Reference conversion relation τ when the movable camera 11 is in the reference posture (θ = 0, φ = 0)
₁₂ (0,0) can be obtained. This reference conversion relation τ
₁₂ (0,0) is an operator that does not change even when the camera system 10 is moved or the movable camera 11 is rotated.
When the standard conversion relation τ ₁₂ (0,0) is determined, the conversion relation τ ₁₂ is
It has the rotation angles θ and φ as variables.

The conversion relation τ ₁₂ obtained in this way is
Since it does not depend on the position or orientation of the entire camera system 10 in the absolute coordinate system X _{0, even if} the camera system 10 is moved to another place, it can be used for conversion calculation there.

* Second sub-process: This is to use the result of the first sub-process while photographing the subject 30 from a plurality of directions to obtain image data, and for each of those image data, the second local process. This is a sub-process of adding information corresponding to the conversion relationship τ ₀₂ from the coordinate system X ₂ to the absolute coordinate system X ₀ .

In the second sub-process, the subject camera 1
At the same time when the subject 30 is photographed with the movable camera 11
Take a picture of the stereo chart 2. The conversion relationship τ ₀₁ from the first local coordinate system X ₁ to the absolute coordinate system X ₀ is specified from the image data of the three-dimensional chart 2 taken by the movable camera 11.

On the other hand, the rotation angle dependence of the conversion relation τ ₁₂ from the second local coordinate system X ₂ to the first local coordinate system X ₁ is specified by the first sub-process, so that the object 3
From the values of rotation angles θ and φ when shooting 0, the conversion relation τ
_Twelve specific contents are identified. Therefore, the transformation relation τ
_The conversion relation τ ₀₂ obtained by combining ₁₂ and τ ₀₁ can be obtained from the equation 4.

Τ ₀₂ = τ ₀₁ · τ ₁₂ (Equation 4) Then, the information expressing this conversion relation τ ₀₂ is stored together with the image obtained by the subject camera 13.

The second sub-process is executed each time the subject 30 is photographed from a plurality of directions, whereby a group of information for obtaining the three-dimensional image model is obtained.

<Photographing and Calibration Process> FIGS. 11 and 12 are diagrams showing a photographing and calibration process according to the above principle. Of these, from step S1 to step S5, the movable camera 11 and the subject camera 13
This corresponds to the first sub-process in which the relative position / orientation of the three-dimensional chart 2 is obtained by simultaneously photographing the three-dimensional chart 2 with. Further, after step S6, the subject 3 is actually
This corresponds to the second sub-process of performing 0 photographing.

(1) Determination of relative position between cameras (first
(Sub-process): First, each of the movable camera 11 and the subject camera 13 reads out internal parameter information held by itself (step S1). The internal parameters are parameters for specifying the line-of-sight direction of each pixel of the light receiving element, and are the focal length of the lens system, the positional relationship between the lens optical axis and the light receiving element, the pixel pitch, and the like. These parameters are calibrated in advance.

After this, the user holds the camera system 10 in his hand and points the camera 13 for the subject at the three-dimensional chart 2. Next, while maintaining this posture, the lens unit 1 is adjusted so that the three-dimensional chart 2 enters the angle of view of the movable camera 11.
The rotation angle of 10 is manually specified (step S2). During this operation, the output image of the movable camera 13 is displayed live on the display 163, which allows the user to visually confirm whether or not the three-dimensional chart 2 has entered the angle of view of the movable camera 13. .

When the user presses the tracking button 117 after the three-dimensional chart 2 enters the angle of view of the movable camera 11,
The automatic tracking program is activated. Posture control unit 146
The driving output is given from the posture device 113 to the marker 2
While tracking 01, the movable camera 11 is automatically controlled so that the three-dimensional chart 2 is always in the center of the angle of view (see the flowchart of FIG. 8). When the user presses the shutter button 161, the movable camera 11 obtains the first image data D1 and the subject camera 13 obtains the second image data D2 (step S3). Figure 1
3 shows an example of image data obtained by the movable camera 11 and the subject camera 13 at the same time. Of these, Figure 13
13A is an example of a captured image of the movable camera 11, FIG.
Is an example of a captured image of the camera 13 for a subject. FIG.
In both (a) and (b), the image plane is a plane defined by the xy orthogonal coordinate system, and the direction perpendicular to the xy plane and facing the image is the z axis. The i-th
The term “layer” is based on the definition in FIG. 19 described later.

When the photographing is completed, the first image data D1 and the rotation angle data DR of the attitude device 113 are sent from the movable camera 11 to the object camera 13 by communication,
Four grid points C1 to C4 common to FIGS. 13A and 13B
The two-dimensional coordinate value of each is specified on each image plane (first
This corresponds to creating the extraction point data DP1 and the second extraction point data DP2. ), The two-dimensional coordinate values thereof are processed by the above-described multi-point algorithm to obtain the movable camera 11 and the camera for the subject based on the first image data D1 and the second image data D2 obtained by capturing the three-dimensional chart 2. The external parameters of each camera depending on the relative position and relative attitude between the photographing position 13 and the three-dimensional chart 2 can be calculated, and the position and attitude of each camera in the absolute coordinate system can be obtained (step S4). ). If the information necessary for this calculation cannot be obtained, the process returns to step S3 and the photographing of the three-dimensional chart 2 is repeated.

The external parameters are calculated by 1) the internal parameters of the camera, and 2) the three-dimensional coordinate values of four or more points on the same plane fixed in the absolute coordinate system, and 3) It can be performed under the condition that the two-dimensional coordinate values of the points on the captured image corresponding to these points can be calculated.

Next, from the external parameters of the movable camera 11 and the subject camera 13 obtained in step S4 and the rotation angle data DR of the movable camera 11, the movable camera 11 and the subject camera 13 are selected. Relative position / orientation is obtained (step S5).

FIG. 14 shows a state of coordinate conversion used in step S5. The definition of each coordinate system in FIG. 14 is as follows.

X ₀ ... Three-dimensional orthogonal coordinate system (absolute coordinate system) fixed relative to the three-dimensional chart 2; θ ... Rotation angle of the movable camera 11; φ ... Depression angle of the movable camera 11; X ₁ (θ, φ) ... 3D orthogonal coordinate system (first local coordinate system) corresponding to the observation space from the movable camera 11; X _1h ... 1st local coordinate system when both angles θ and φ are zero ; X ₂ ... 3 corresponding to the observation space from the subject camera 13
Dimensional Cartesian coordinate system (second local coordinate system).

In step S4, the movable camera 11 and
And the external parameters of the subject camera 13 are
Absolute coordinate system X₀ First Local Zodiac
Standard X ₁The position and orientation of (θ, φ) are determined and
1st local coordinate system X₁Absolute coordinate system X from (θ, φ)₀ To
The coordinate transformation is the rotation matrix R_C1, And the translation vector T_C1
Using, X₀ = R_C1X₁(Θ, φ) + T_C1  ... (Equation 5) It is decided like. Similarly, the second local coordinate system X₂Ridiculous
Coordinate system X₀ The coordinate transformation to is the rotation matrix R_C2 , And Taira
Line movement vector T_C2Using, X₀ = R_C2X₂+ T_C2  (Equation 6) It is decided like. These equations 5 and 6 have already been described.
Transformation relation τ₀₁, Τ₀₂Equivalent to.

The rotation conversion due to the fact that the rotation angles θ and φ are not zero is based on the design data of the rotation mechanism of the attitude device 113 of the movable camera 11 based on the rotation matrix R _X (θ, φ) and the translation vector T _X ( θ, φ) is used to obtain X ₁ = R _X (θ, φ) X _1h + T _X (θ, φ) (Equation 7). The coordinate transformation from the second local coordinate system X ₂ to the first local coordinate system X _1h when the rotation angles θ and φ are both zero is X _1h = using the rotation matrix R _h and the translation vector T _h. If it is expressed as R _h X ₂ + T _h (Equation 8), Equation 8 corresponds to the conversion relation τ ₁₂ (0,0). Further, by substituting the equation 8 into the equation 7, X ₁ = R _m (θ, φ) X ₂ + T _m (θ, φ) (Equation 9) is obtained. However, R _m (θ, φ) = R _X (θ, φ) R _h (Equation 10) T _m (θ, φ) = R _X (θ, φ) T _h + T _X (θ, φ) ) (Equation 11) Expressions 9 to 11 correspond to the conversion relationship τ ₁₂ .

Therefore, the first extraction point data DP1 and the second extraction point data DP2 are obtained from the respective images taken by the movable camera 11 and the subject camera 13 on the three-dimensional chart 2 at the same time. If the external parameter of is obtained, the conversion formulas of the formulas 9 to 11 corresponding to the conversion relation τ _{12 described} above are specified. This is the relative position data DPS of FIG. It should be noted that the external parameters of the movable camera 11 are external parameters PM1, and the external parameters of the subject camera 13 when the stereo chart is captured are external parameters PM2. Further, specific values of the angles θ and φ are given by the angle sensor 126.
The rotation angle data DR is detected by p and 126t.

As a result, based on the external parameter PM1, the external parameter PM2 and the rotation angle data DR,
The relative position data DPS depending on the relative position and the relative position between the photographing position of the movable camera 11 in the reference posture and the photographing position of the subject camera 13 can be calculated,
The subject camera 13 can be calibrated by performing a process (second sub-process) described later using this.

(2) Imaging of subject and calibration of camera (second sub-process): When step S5 is completed, the camera system 10 is appropriately moved to capture the subject 30. At this time, the movable camera 11 always captures the three-dimensional chart 2 at the angle of view by automatic tracking (step S).
6).

The object 30 is taken by the object camera 13.
However, when the three-dimensional chart 2 is ready to be photographed by the movable camera 11, the subject camera 1
When the shutter button 161 of No. 3 is pressed, images are simultaneously taken by the respective cameras (step S7). Figure 15
An example of an image obtained by the movable camera 11 and the subject camera 13 is shown in FIG. FIG. 15A is an example of an image obtained by capturing an image with the movable camera 11, and FIG. 15B is an example of an image captured by the camera 13 for a subject (the subject 3 is represented by an ellipse.
0 image is simplified). The method of taking coordinate axes is the same as in FIG. In the state of FIG. 15A, the distance and the direction from the three-dimensional chart 2 to the movable camera 11 are different from those in FIG. 13A, so that the movable camera 11 is shown in FIG. I am shooting a different part from. However, since the movable camera 11 automatically tracks the marker 201, the marker 201
Are always in the same position within the angle of view of the movable camera 11.

At this time, from the image obtained by the movable camera 11, the position and orientation of the movable camera 11 in the absolute coordinate system X ₀ (external parameter P, as in step S4).
M1) is obtained (step S8).

External parameter P of this movable camera 11
M1 and the relative position data DPS obtained in step S5,
Then, from the respective values θ ′ and φ ′ of the rotation angles θ and φ of the movable camera 11 in step S7, the subject camera 1 is obtained.
The relative position / orientation of 3 with respect to the three-dimensional chart 2 (that is, the external parameter of the subject camera 13) is obtained (step S9).

FIG. 16 shows a state of coordinate conversion used in the above step S9. In FIG. 16, symbols such as coordinate systems X ₀ , X ₁ (θ, φ), and X ₂ are defined in common with FIG. 14. The camera rotation angles of the movable camera 11 at the time of photographing a subject are represented by θ ′ and φ ′.

By analyzing the image of the three-dimensional chart 2 obtained by the movable camera 11 with a multipoint analysis algorithm, the position and orientation of the movable camera 11 in the absolute coordinate system X ₀ (that is, the external parameters of the movable camera 11). PM1) is specified, whereby the conversion relationship with the first local coordinate system X ₁ (θ ', φ') of the movable camera 11 is X ₀ = R _CP1 X ₁ (θ ', φ') + T _CP1 .. (Equation 12) is determined.

Also, the first local coordinate system X ₁ (θ ', φ')
The conversion relation between the second local coordinate system X ₂ and the second local coordinate system X ₂ is X ₁ (θ ′, φ ′) = R _m (θ ′, φ ′) X ₂ + T _m (θ ′, φ) ') ・ ・ ・ (Equation 13) R _m (θ', φ ') = R _X (θ', φ ') R _h・ ・ ・ (Equation 14) T _m (θ', φ ') = R _X ( θ ′, φ ′) T _h + T _X (θ ′, φ ′) ... (Equation 15)

Therefore, according to the equations 12 to 15, the second
The conversion relationship for converting the position and orientation expressed in the local coordinate system X ₂ to the position and orientation in the absolute coordinate system X ₀ is X ₀ = R _CP1 R _m (θ ', φ') X ₂ + R _CP1 T _m (θ ′, φ ′) + T _CP1 (16) is obtained.

Among the quantities appearing in equation 16, the rotation matrix
R _CP1 and the translation vector T _CP1 are for the movable camera 11 and are determined from the external parameter PM1 of the movable camera 11. Also, the rotation matrix R _m (θ ',
φ ′) and the translation vector T _m (θ ′, φ ′) are the functional forms R _m (θ, φ) and T _m (θ, φ) specified in advance,
It is determined by substituting the angle values θ ′ and φ ′ detected by the angle sensors 126p and 126t.

Therefore, the following _equation 16 is _converted into X ₀ = R _CP2 X ₂ + T _CP2 ( _Equation 17) R _CP2 = R _CP1 R _m (θ ', φ') ( _Equation 18) T _CP2 = R _CP1 T _m (θ ′, φ ′) + T _CP1 ... When transformed into the form of ( _Equation 19), the rotation matrix R _CP2 and the translation vector T _CP2 represent the external parameters of the camera 13 for the subject. Has become. These are the shooting data DM (Fig. 1
0) is stored together with the second image data D2 and is used to combine images of the subject 30 obtained from a plurality of directions when constructing a three-dimensional image model based on the second image data D2. The construction of this three-dimensional image model may be performed by the computer 15 or another computing system.

When the photographing data DM is obtained, the photographing data DM of the subject camera 13 at the time of photographing is recorded in the RAM 151 or the memory card 166 together with the second image data D2 of the subject 30 (step S10).

Thereby, based on the external parameter PM1, the relative position data DPS and the rotation angle data DR,
It is possible to calculate the photographic data DM for calibration of the photographic subject camera 13 that depends on the relative position and relative posture between the photographic position of the photographic subject camera 13 and the three-dimensional chart, and use the photographic data DM to calculate the shape from A three-dimensional image model of the subject 30 can be constructed by the silhouette method.

After that, the required number of second image data D2,
This second sub-process is completed when the imaging data DM for each of them are obtained.

<Identification of Chart by Coding of Cross Ratio> Now, a method of coding the side surface of the three-dimensional chart 2 will be described. As shown in FIG. 3, the display portion 204 of the three-dimensional chart body 203 is a regular polygonal pyramid, and each side surface T1 to Tn thereof has the same isosceles triangular shape. The bottom surface direction DR of the triangle forming the side surface is
1 (see FIG. 17), a plurality of straight lines L1 and a plurality of radial straight lines L2 passing through the vertex x0 corresponding to the vertex of the three-dimensional chart 2 are drawn. The trapezoidal unit figure UP (hereinafter referred to as “unit trapezoid”) created by the intersection of these straight lines is painted with alternating lightness colors so that straight lines can be easily extracted during image processing. And has a high contrast pattern. Typically, the first set of unit trapezoids UP1 is black and the second set of unit trapezoids U1 is black.
P2 is white.

The sizes of these unit trapezoids UP are coded by the cross ratio. More specifically, 1) the distance between the straight line groups L1 forming the unit trapezoid UP, and 2) the distance in the bottom surface direction DR1 at the intersection (lattice point) of the straight line group L1 and the straight line group L2. Each is coded by cross ratio. The concept of this cross ratio is shown in FIG. 18. The cross ratio is a value that does not change due to space projection through an arbitrary viewpoint, and the cross ratio DR is calculated from four points P1 to P4 on a straight line existing in a three-dimensional space. : DR = Va / Vb (Equation 20) Va = dis (P0P1) ・ dis (P2P3) ・ ・ ・ (Equation 21) Vb = dis (P0P2) ・ dis (P1P3) ・ ・ ・ (Equation 22) , The symbol dis (P0P1) indicates the distance between the points P0 and P1: is the four points P'1 corresponding to those four points P1 to P4 when the straight line is projected through the viewpoint O on an arbitrary plane. ~
Multiratio DR 'obtained from P'4: DR' = Va '/ Vb' ... (Equation 23) Va '= dis (P'0P'1) .dis (P'2P'3) ... (Equation) 24) It is known that Vb ′ = dis (P′0P′2) · dis (P′1P′3) (equation 25).

Utilizing this property, FIG. 3 and FIG.
The distance between the straight line groups L1 forming the unit trapezoid UP as shown in FIG. 3 is coded in a ratio for each layer forming each unit trapezoid UP, and the distance between the lattice points in the bottom surface direction DR1 is set to the side surface T1. If each stereoscopic chart 2 is coded in a different ratio for each Tn, each unit trapezoid UP included in the image captured by the movable camera 11 or the camera 13 for a subject is changed to the side surface T1 to Tn of the stereo chart 2. It becomes possible to uniquely identify which unit trapezoid exists on which side of the. An example is shown below.

FIG. 17 shows an example in which the intervals of the straight lines L1 arranged in the direction perpendicular to the bottom surface (vertex direction DR2) are coded by the cross ratio. Of the multiple unit trapezoids, the cross ratio of the heights of three consecutive unit trapezoids is 3
Each unit trapezoidal group is coded differently.

That is, straight lines x1, x2, ... Which are parallel to the base are defined with the vertex x0 of the three-dimensional chart 2 as an end point, and the "i-th layer" is defined as "straight line xi and straight line x (i + 1)". When defined as "region between", the i-th layer to the (i + 3) -th layer (i = 1,
2, ...), the cross ratio DRi at each position in the vertex direction DR2 direction is DRi = Vai / (Vbi.Vb (i + 1)) (Equation 26) Vai = dis (xix (i + 1) )) ・ Dis (x (i + 2) x (i + 3)) (Equation 27) Vbi = dis (xix (i + 1)) + dis (x (i + 1) x (i + 2) ) (Equation 28) Vb (i + 1) = dis (x (i + 1) x (i + 2)) + dis (x (i + 2) x (i + 3)) ・ ・ ・ (Equation) 29) Or by rewriting this, DRi = Vai / VBi (Equation 30) Vai = dis (xix (i + 1)). Dis (x (i + 2) x (i + 3)) .. (Equation 31) VBi = dis (xix (i + 2)) + dis (x (i + 1) x (i + 3)) (Equation 32), and each cross ratio DRi is defined as shown in FIG. The values are as shown in. The size (width and height) of each unit trapezoid increases as it approaches the base of the pyramid.

Further, in this embodiment, the vertex direction DR2
The moving average of the intervals of the straight lines x1, x2 ... Is determined to be approximately proportional to the distance from the vertex. That is, the position of each layer from FIG. 19 is “17.000, 22.500, 31.000 ...”.
Therefore, the difference between them is 5.500 (= 22.500-17.000) 8.500 (= 31.000-22.500) ... And four consecutive moving averages of these series of differences are as shown in FIG. As can be seen from FIG. 20, the moving average of the difference in the position of each layer (layer thickness) is gradually increasing, but the fluctuation of the value corresponding to “moving average / position of layer from apex”, that is, a proportional coefficient. Is within about 20%. Therefore, the moving average is approximately proportional to the distance from the vertex.

On the other hand, regarding the intervals in the bottom surface direction DR1 between the intersections of the straight lines y1, y2 ... Radially extending around the apex and the straight lines x1, x2. Make a decision so that you can identify them. FIG. 20 shows an example when the three-dimensional chart 2 is a hexagonal pyramid. 17 and 20, a, b, c, d are the intervals between the straight lines y1, y2 ... At the bottom.

In this example, DRα = (a · c) / {(a + b) · (b + c)} (Equation 33) DRβ = (b · d) / {(b + c) · (c + d)} ... Each of the two types of compound ratios DRα and DRβ defined as (Equation 34) is 1) a straight line x in each side Tj (j = 1 to 6).
The values are coded so that in any of 1, 1, x 2, ..., The columns of intersections with the straight lines y 1, y 2, ... Are common, and 2) different sides are different from each other.

<Identification of Shooting Part> FIG. 22 is a flowchart showing a process of identifying which part of the stereo chart 2 is being shot when the movable camera 11 and the camera for subject 13 shoot the stereo chart 2. is there. FIG. 23 is an explanatory diagram of linear grouping.

* Linear grouping: First, the gray edges of the captured image are extracted (step S91). Various methods such as the Sobel operator are known as methods for extracting edges. For example, Masao Nagao "Image Recognition"
The algorithm disclosed in Corona, 1983 is used.
FIG. 23B shows an example of extracting edges from the image of FIG.

Then, a straight line is extracted from the extracted edge (step S92). How to extract a straight line is HOUGH
Transform is known as a general method, and a plurality of straight lines can be extracted from an edge image by using the method described in the above document of Nagao Makoto, and the formula of the straight line in a two-dimensional plane on imaging can be determined. FIG. 23C shows an example of extracting edges in FIG. 23B.

The plurality of extracted straight lines are grouped into a plurality of groups as follows for each property of the straight lines (step S93).

[0106]-parallel straight lines (Hereinafter, this product is the α _i .i
Represents a group of straight lines having the same slope); ・ A straight line that passes through the end points of each straight line belonging to α _i (this is β
A group of straight lines intersecting with α _i (hereinafter, this is referred to as γ _i . I corresponds to _i of the group of intersecting straight lines α _i ).

In the example of FIG. 23 (c), each straight line is grouped into two straight line groups α ₁ and α ₂ having different slopes, straight line groups γ ₁ and γ ₂ corresponding to these straight line groups, and a crossing straight line β. It will be.

Further, since the intersecting straight lines β and γ _i intersect at one point, this intersecting point corresponds to the marker 201 of the three-dimensional chart 2. From this, it can be determined that the straight line corresponding to the side of the cone forming the three-dimensional chart 2 is the straight line β, the straight line parallel to the bottom surface is the straight line α _i , and the straight line passing through the side surface of the cone is the straight line γ _i . As a result, the cross ratio of the intersections associated with each unit trapezoid on the image is calculated.

* Identification of shooting location: First, before shooting, the coordinates of intersections (lattice points) represented by the absolute coordinate system X ₀ and the coordinates of the straight lines on each of the side surfaces T1 to Tn of the three-dimensional chart 2 are taken. The cross ratio data calculated from the intervals of the intersections are stored in advance in the RAM 151 (FIG. 9).
Then, when the three-dimensional chart 2 is photographed by the movable camera 11 or the subject camera 13, three layers continuous in the apex direction DR2 are obtained from an image obtained by the photographing.
Or, unit trapezoids are specified and the cross ratio is calculated from their heights.

FIG. 24 shows the same side face of the three-dimensional chart 2.
The image example when image | photographed from a different distance is shown. Figure 24
FIG. 24A is an example taken from a long distance, and FIG. 24B is an example taken from a short distance. The distance in the apex direction DR2 of the unit trapezoid is approximately proportional to the distance from the apex.

In order to calculate the cross ratio of the intervals between the straight lines, it is only necessary to observe four consecutive points on the same straight line. That is, it suffices if four straight lines that are within the angle of view and that are continuous at sufficient intervals for accurately calculating the cross ratio can be observed in relation to another straight line that intersects them.
When a large number of straight lines are present in the image, for example, four straight lines having a distance equal to or more than a predetermined threshold value with the next straight line and arranged immediately above the apex (marker 201) at a position closest to the apex. Select. Then, four intersections of the four straight lines and one straight line extending along the apex direction DR2 are extracted, and the cross ratio of these intervals is calculated. Of the data obtained in this extraction, the data obtained by shooting with the movable camera 11 is the first extraction point data DP1 in FIG. 10, and the data obtained by shooting with the camera 13 for the subject is the second.
It is the extraction point data DP2.

In FIG. 24 (a), each is in the bottom direction D.
Four straight lines x7 to x10 extending to R1 and defining the upper and lower sides of the seventh to ninth layers, respectively, and in FIG. 24 (b), four straight lines defining the upper and lower sides of the third to fifth layers, respectively. x
3 to x6 can be selected as these four straight lines. By selecting four straight lines in this manner, the cross ratio can be calculated sufficiently in any image. If a plurality of side surfaces are photographed, for example, the side surface closest to the image center is selected.

Since the three-dimensional chart 2 has a pyramidal shape, even if the three-dimensional chart 2 is photographed from various directions,
If the marker 201 is detected by automatic tracking, it is possible to sufficiently observe at least one side surface.

One of a plurality of unit trapezoids existing in the area sandwiched by the four straight lines thus selected is
Select as the target unit trapezoid (target unit figure). The target unit trapezoid can be selected by a selection rule such as selecting the unit trapezoid which is sandwiched by the two middle straight lines among the above four straight lines and which is closest to the center of the screen. In the example of FIG. 24, for example, the unit trapezoids UPA and UPB can be selected as the target unit trapezoids.

Then, with respect to the above four straight lines, the vertex direction DR
For 2, the cross ratio is calculated as the target cross ratio from the interval on the image, and the cross ratio value (FIG. 19) of the linear interval of each side face stored in the RAM 151 in advance is compared with the target cross ratio. As a result, the four straight lines become the three-dimensional chart 2.
It is possible to specify which layer of the three-dimensional chart is the four straight lines defining which layer to which layer, and which unit trapezoid of the target unit trapezoid is in the three-dimensional chart 2 (step S94).

By the way, the closer to the bottom surface of the pyramid, the larger the size in the apex direction of the unit trapezoid has the following advantages.

First, the three-dimensional chart 2 is obtained from a relatively short distance.
When only a relatively small number of unit trapezoids are present in the image by capturing the image (FIG. 24 (b)), the marker 2
A unit trapezoid close to 01 is photographed relatively large.

On the contrary, the three-dimensional chart 2 from a relatively long distance.
When the image capturing size of each unit trapezoid is relatively small by capturing, the unit trapezoid with a large actual size, which is close to the bottom surface of the pyramid, exists in the image (FIG. 24).
(A)), it does not become so small as the observation size on the image.

Therefore, when shooting from a short distance,
In both cases of shooting from a long distance, the image always contains a unit figure with a size sufficient to ensure accuracy in image processing, and as a result, it does not depend much on the shooting distance. The calculation accuracy can be increased. This is an advantage that the size of the unit trapezoid in the apex direction is increased as it is closer to the bottom surface of the pyramid.

By identifying the control unit trapezoid,
It is possible to calculate the relative position / orientation of the camera with respect to the three-dimensional chart 2, that is, the external parameter in the absolute coordinate system X ₀ (step S95). This will be explained below.

First, in the RAM 151, the cross ratio of the distances a, b, c, d for each of the side surfaces T1 to Tn of the pyramid shown in FIG.
Information regarding in which direction of the absolute coordinate system X ₀ each side surface T1 to Tn of the pyramid faces is associated with each other and stored in advance as a table. Therefore, on one of the four straight lines (for example, in the example of FIG. 24A, the straight line x7 of the sides of the target unit trapezoid to which the side closest to the vertex belongs) is displayed on the image of four intersections continuous in the bottom direction DR1. By specifying the coordinates of, the cross ratio of these intervals is calculated and collated with the above table, the side face to which the target unit trapezoid belongs is identified as the observation side face that is almost directly facing the camera at that time. The relative attitude of the camera with respect to the three-dimensional chart 2 can be known in the absolute coordinate system X ₀ depending on which of the side surfaces T1 to Tn the observation side surface is.

In order to know the relative posture of the camera with respect to the three-dimensional chart 2 in more detail, for example, from the coordinate values of the four vertices of the target unit trapezoid, the outer circumference of the target unit graphic is defined.
Find the ratio of side lengths. This ratio changes depending on how much the imaging axis of the camera is tilted from the normal direction of the target unit trapezoid in the absolute coordinate system. Therefore, from this ratio, the direction of the imaging axis of the camera from the normal direction of the side surface can be specified.

The target unit trapezoid is specified, and the three-dimensional chart 2
When the relative attitude of the camera with respect to is calculated, the actual size information of the unit trapezoid corresponding to the target unit trapezoid is read out from the actual size information of each unit trapezoid stored in the RAM 151 in advance. Then, for the target unit trapezoid, the ratio r between the size on the image and its actual size is obtained. The ratio r is a function of the distance L between the stereo chart 2 and the camera and the relative attitude of the camera with respect to the stereo chart 2, and since the relative attitude is obtained as described above,
After all, the distance L can be expressed as a function f (r) of the ratio r. Therefore, the distance L can be calculated from the ratio r by storing an arithmetic expression or a numerical value table corresponding to this function f (r). Distance L and 3D chart 2
The relative position of the camera with respect to the three-dimensional chart 2 can be obtained from the relative attitude of the camera with respect to.

As described above, the stereo chart 2 according to the present embodiment is photographed at the center of the angle of view by using a camera different from the camera for photographing the subject, so that the calibration for calibrating the position and orientation of the camera can be performed. Of the parameters, the external parameters can be acquired accurately.

<2. Second Embodiment> In the first embodiment, the movable camera 11 is attached to the subject camera 13 and the position of the subject camera 13 when the second image data is captured by the subject camera 13 We have described how to obtain optimal data for the shape from silhouette method by identifying postures, but this technique can also be applied to detail movements of humans by attaching multiple movable cameras to the human body. . Such data is used to reproduce the motion of a person in the virtual reality space.

FIG. 25 is a diagram showing the configuration of the image photographing system 3 according to the second embodiment constructed according to such a principle. FIG. 26 is an external view of the movable camera 11a according to the present embodiment. Hereinafter, description will be made by omitting the same configurations and the like as those in the first embodiment as appropriate.
As shown in FIG. 26, the movable camera 11a in the image capturing system 3 includes a cloth belt 118 and a detachable tape 1.
A variety of data such as image data attached to the human body by 19 is wirelessly communicated with a computer 15 as a calibration information processing device.

The movable camera 11a includes a cloth belt 118.
It has the same functions and configurations as those of the first embodiment except that it is attached to the human body by. However, the rotation angle data DR is not data indicating the drive angle itself of the posture device 113 with respect to the reference posture of the movable camera 11a, but the relative position and relative posture between the shooting position of the movable camera 11a and the fixed portion 114a in the reference posture. Is also taken into consideration.

The movable camera 11a performs the same operation and processing as those in the first embodiment. The first image data D1 and the rotation angle data DR at the time of shooting are acquired and transmitted to the computer 15 as the calibration information processing device. The acquisition of the first image data D1 is not performed in conjunction with the shutter but is automatically performed at a predetermined timing. In addition, which mobile camera 11a has acquired the data is transmitted to the computer 15 in a state in which it can be identified.

FIG. 27 is a diagram showing the main part of the information processing function of the computer 15 as seen from the hardware configuration.
FIG. 28 is a diagram showing a data flow in the computer 15. Hereinafter, configurations that perform the same functions as the information processing functions in the subject camera 13 of the first embodiment will be described using the same reference numerals as appropriate.

The computer 15 includes a CPU 180 and a RAM
A program 182 that includes a 181 and a ROM 182 and that realizes various operations of the computer 15 described later.
a is stored in the ROM 182. Elements such as the monitor color display 163 and operation buttons 164 installed on the front surface are also electrically coupled to the CPU 180.

The extraction unit 171 performs image recognition on the first image data D1 as in the first embodiment to generate first extraction point data DP1. The calculation unit 173 calculates each movable camera 11 from the first extraction point data DP1 and the rotation angle data DR.
Relative data DS indicating the relative relationship between the position of the fixed portion 114 of a and the three-dimensional chart 2 is generated.

<Principle of Calibration> X ₀ ... 3D Cartesian coordinate system (absolute coordinate system) fixed relatively to the solid chart 2; X ₁ (θ, φ) ... Observation space from the movable camera 11a 3D Cartesian coordinate system (first local coordinate system); X ₃ ... 3D Cartesian coordinate system (third local coordinate system) corresponding to the observation space from the fixing unit 114; θ ... Rotation of the movable camera 11a Angle; φ ... angle of elevation of movable camera 11a; and rotation matrix R _X1 and translation vector T _X1
Can be expressed as follows: X ₃ = R _X1 (θ, φ) X ₁ + T _X1 (θ, φ) (Equation 35) The rotation matrix R _X1 (θ, φ) and the translation vector T _X1 (θ, φ) are given as rotation angle data DR, and the photographing position of the movable camera 11 a in the reference posture of the posture device 113 and the fixed portion 114 are set. The amount depending on the relative position and the relative posture of is calibrated in advance. That is, Expression 35 corresponds to a conversion relationship for converting the first local coordinate system of the movable camera 11 into the third local coordinate system of the fixed portion.

By the multipoint analysis algorithm described above, the conversion of the movable camera 11a from the first local coordinate system to the absolute coordinate system is performed by using the rotation matrix R _c and the translation vector T _c as X ₀ = R _c X ₁ + T _c ... (Equation 36) From Equations 35 and 36, the conversion relationship between the absolute coordinate system and the third local coordinate system is X ₀ = R _c (R _X1 (θ, φ )) ^-1 X ₃ -R _c (R _X1 (θ, φ)) ⁻¹ T _X1 (θ, φ) + T _c (Equation 37)

<Photographing Operation> FIG. 29 is a flowchart showing the operation of the image photographing system 3 in the present embodiment.
The operation of the image capturing system 3 will be described with reference to FIG.
In addition, in order to simplify the description, it is assumed that there is one movable camera 11a.

First, the internal parameters of the movable camera 11a are calibrated (step S21), the orientation is adjusted so that the movable camera 11a captures the three-dimensional chart 2 within the angle of view,
Automatic tracking setting is performed (step S22). The three-dimensional chart 2 is photographed by the movable camera 11a (step S2
3) It is determined whether or not the external parameter PM1 of the movable camera 11a can be calculated (step S24), and if the external parameter PM1 cannot be calculated, step S24.
23. When the external parameter PM1 can be calculated repeatedly from 23, the rotation angle data DR is calculated (step S2).
5).

Next, the relative data DS is calculated and stored from the external parameter PM1 and the rotation angle data DR (step S27).

As described above, it is possible to calculate the relative data DS indicating the relative position and the relative attitude of the position of the movable camera 11a of the person to be traced, and to trace the person's movement easily and in detail. , It is possible to reproduce the movement of a person in a virtual reality space or the like.

<3. Modifications> The embodiments of the present invention have been described above, but the present invention is not limited to the above embodiments, and various modifications can be made.

For example, not only one target unit figure,
If a plurality of target unit figures are selected and the external parameters are calculated for each of them, and the obtained external parameters are averaged, the calculation accuracy of the external parameters is further improved.

The movable camera 11 may perform a calculation process of image data to calculate external parameters and communicate with the computer 15 or the subject camera 13. Information may be transmitted between these devices via a communication cable or data may be transferred via a storage medium such as a memory card.

In the above embodiment, the calculation of the external parameters of the movable camera 11 is executed only when the subject camera 13 takes a picture, but the three-dimensional chart 2 captured by the movable camera 11 in real time is used. The external parameter may be calculated in real time by sequentially obtaining where in the stereo chart 2 is captured from the image. In this case, the marker 201 can be omitted because the position of the vertex of the three-dimensional chart 2 on the image is grasped in real time.

The marker 201 may have no light emitting property and may be coated with fluorescent paint, and even a simple vertex can be used for tracking.

Movable camera 1 in the first embodiment
1 and the camera 13 for a subject may be indirectly connected via a connection mechanism.

The movable camera 11 may be manually operated by the operation keys of the subject camera 13.

Further, the program for causing the CPU to execute the image processing and the like according to the above-mentioned embodiment does not necessarily have to be written in the ROM in advance. The calibration information processing device may be executed after the program is read from the recording medium via a reading device or the like in advance and stored.

Further, in the above embodiment, the series of image processes are all executed by the software process by the CPU, but it is also possible to realize a part or all of the processes by a dedicated circuit.

The photographing by each movable camera 11a in the second embodiment may be performed by an instruction from the computer 15 or the like.

In the above embodiment, the camera attitude is determined by detecting four points on the pattern drawn on the chart. However, the invention is not limited to this, and the camera attitude is determined as follows. You may make it detect.

An object having a known shape (for example, a quadrangular pyramid) is placed instead of the chart of the above embodiment. The camera has model data of its shape in the camera coordinate system,
The camera orientation is detected by determining the orientation of the model when the captured image and the model image projected on the surface equivalent to the imaging surface match.

[0150]

According to the invention described in claims 1 to 5, the user is conscious by controlling the driving means so that the photographing means can photograph the reference object when the movable photographing device moves. In addition, the movable photographing device can always photograph the reference object at the center of the angle of view without using any large-scale photographing equipment.

According to a second aspect of the present invention, the calibrating information processing device is based on the first image data, and the movable photographic device depends on the relative position and relative posture between the photographic position of the movable photographic device and the reference object. By calculating the first parameter for calibration of, the position and orientation of the movable photographing device in the space can be easily grasped even when the movable photographing device moves.

According to the third aspect of the present invention, the movable photographing device is attached to the subject photographing device, and the subject photographing device is calibrated depending on the relative position and relative posture between the photographing position of the subject photographing device and the reference object. By calculating the fourth parameter for, it is possible to obtain optimum data for constructing a three-dimensional image by the shape-from-silhouette method.

According to the fourth aspect of the present invention, the movable photographing device fixing portion is attached to the object, the rotation angle from the reference posture of the photographing means rotated by the driving means is determined, and the relative position between the fixed portion and the reference object is determined. By calculating the fifth parameter depending on the position and the relative attitude, the position and the attitude of the object attached to the fixed part in the space can be easily grasped.

According to the fifth aspect of the invention, the movable photographing device can be attached to the human body by the attaching means, and the movement of the human body can be traced.

According to the invention described in claims 6 to 9, it is possible to provide a movable photographing apparatus most suitable for the system described in claims 1 to 4.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration example of an image capturing system using a calibration chart according to a first embodiment.

FIG. 2 is a block diagram of the image capturing system of FIG.

FIG. 3 is a diagram showing an example of a side surface of a calibration solid chart according to the first embodiment.

FIG. 4 is a front view of a movable chart-capturing camera.

FIG. 5 is a block diagram of a movable chart photographing camera.

FIG. 6 is a diagram illustrating a main part of an information processing function of a chart-capturing movable camera.

FIG. 7 is a diagram showing a data flow of a movable camera for photographing a chart.

FIG. 8 is a flowchart showing an automatic tracking operation of the movable chart-taking camera.

FIG. 9 is a diagram showing a main part of an information processing function of a digital camera for photographing an object.

FIG. 10 is a diagram showing a data flow of a camera for photographing an object.

FIG. 11 is a diagram showing a procedure corresponding to a first sub-process among the procedures of photographing and calibration using the camera calibration chart according to the first embodiment.

FIG. 12 is a diagram showing a procedure corresponding to a second sub-process among the procedures of photographing and calibration using the camera calibration chart according to the first embodiment.

FIG. 13 is a diagram showing an example of each image pickup when a movable camera for photographing a chart and a digital camera for photographing a subject simultaneously photograph a camera calibration chart according to the present invention.

FIG. 14 is a diagram showing a state of coordinate conversion used when calculating a relative position / orientation of a digital camera for photographing a subject with respect to a movable camera for chart photographing.

FIG. 15 is a diagram illustrating an example of image capturing by the movable chart capturing camera and the subject capturing digital camera when the subject capturing digital camera captures a subject.

FIG. 16 is a diagram showing a state of coordinate conversion used when calculating the relative position / orientation of the subject photographing digital camera with respect to the chart photographing movable camera.

FIG. 17 is a diagram showing an example in which the size of a unit trapezoid is coded by a cross ratio on the side surface of the three-dimensional chart according to the first embodiment.

FIG. 18 is a diagram illustrating that the cross ratio does not change due to projection.

FIG. 19 is a diagram illustrating an example of a cross ratio used for coding.

FIG. 20 is a diagram showing that the moving average of straight line intervals obtained from FIG. 19 is approximately proportional to the distance from the apex.

FIG. 21 is a diagram showing an example of coding used for each surface when the three-dimensional chart according to the first embodiment is a hexagonal pyramid.

FIG. 22 is a diagram showing a procedure for identifying a shooting location on a stereo chart from an image obtained by shooting a stereo chart according to the first embodiment.

FIG. 23 is a diagram illustrating an example of straight line extraction in capturing an image of the stereo chart according to the first embodiment.

FIG. 24 is a diagram illustrating an example of imaging when the same side surface of the three-dimensional chart according to the first embodiment is captured from different distances.

FIG. 25 is a diagram showing a configuration example of an image capturing system using the calibration chart according to the second embodiment.

FIG. 26 is an external view of the movable camera according to the present embodiment.

FIG. 27 is a diagram showing a main part of an information processing function of a digital camera for photographing an object.

FIG. 28 is a diagram showing a data flow of a camera for photographing an object.

FIG. 29 is a flowchart showing the operation of the image capturing system in this embodiment.

[Explanation of symbols]

1,3 image capturing system 10 camera system 11,11a Movable camera 110 lens unit 110a fixed lens 110b zoom lens 110e Aperture / shutter mechanism 111,156 Two-dimensional light receiving element 112 communication devices 113 Posture device 114, 114a Fixed part 115a First mounting groove 115b Second mounting groove 116 Spherical unit 117 Tracking button 118 fabric belt 119 Detachable tape 12 Mounting mechanism 126p, 126t angle sensor 126z sensor 13 Camera for shooting subjects 15 Computer 152a program 155 lens unit 15a communication interface 161 shutter button 162 flash 163 display 164 Operation buttons 165 card slots 166 memory card 167 Communication unit 168 communication device 2 three-dimensional chart 201 marker 202 Power supply for marker 203 3D chart body 204 display 205 support 250 chart support 251 pedestal 252 arm 30 subjects

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Kunimitsu Sakakibara             2-3-3 Azuchi-cho, Chuo-ku, Osaka-shi, Osaka Prefecture               Osaka International Building Minolta Co., Ltd. F-term (reference) 2F065 AA01 AA04 AA21 AA24 AA37                       AA39 BB05 CC16 DD02 FF04                       GG07 JJ03 JJ26 LL06 LL30                       MM25 QQ03 QQ24 QQ32 RR07                       SS02                 5C022 AB63 AC74 CA02                 5C054 AA02 CC02 CF05 HA31

Claims (9)

[Claims] 1. An image photographing system, comprising: a) a movable photographing device; and b) a reference object having a known three-dimensional shape, wherein the movable photographing device is a-1) A photographing means for acquiring an image of the reference object as the first image data; a-2) a driving means for rotationally driving the photographing means with respect to a fixed part; a-3) a case where the movable photographing device moves. An image capturing system, comprising: a control unit that controls the driving unit so that the image capturing unit can capture the reference object. 2. The image capturing system according to claim 1, further comprising: c) a calibration information processing device, wherein the movable imaging device is a-4) between the calibration information processing device and Further comprising a first communication means for performing data communication, wherein the calibration information processing device is c-1) second communication means for performing data communication with the movable imaging device, and c-2) the second. Based on the first image data received by the communication means, a first parameter for calibration of the movable photographing device that depends on a relative position and a relative posture between the photographing position of the movable photographing device and the reference object is set. An image capturing system comprising: a first calculating means for calculating. 3. The image photographing system according to claim 2, further comprising: d) a subject photographing device that acquires an image of the subject as second image data, wherein the movable photographing device is a-5). A-6) a mounting means for mounting the fixed part to another object; a-6) a means for obtaining a rotation angle of the photographing means from the reference posture rotated by the driving means; and a-7) a mounting means for photographing the subject. Imaging control means attached to the device, for causing the imaging means to acquire the first image data in association with the acquisition of the second image data of the object imaging device, The subject photographing device has a d-1) third communication means for transmitting the second image data to the calibration information processing device, and the calibration information processing device c-3) photographs the reference object. Based on the second image data, A second calculation means for calculating a second parameter for calibration of the subject photographing device depending on a relative position and a relative posture of the photographing position of the subject photographing device and the reference object, c-4) the first Calibration of the subject photographing device that depends on the relative position and relative posture between the photographing position of the photographing means in the reference posture and the photographing position of the subject photographing device, based on the parameter, the second parameter, and the rotation angle. Calculating means for calculating a third parameter for use, c-5) based on the first parameter and the third parameter, the relative position and the relative position between the photographing position of the subject photographing device and the reference object; An image capturing system, further comprising: a fourth calculation unit that calculates a fourth parameter for calibration of the device for capturing an image of a subject that depends on a posture. 4. The image photographing system according to claim 2, wherein the movable photographing device comprises a-8) mounting means for mounting the fixed portion to another object, and a-9) the driving means. A-10) means for obtaining a rotation angle of the rotated photographing means from the reference posture, and a-10) acquiring reference relative data depending on the relative position and relative posture between the photographing position and the fixed part in the reference posture of the photographing means. And a-11) means for calculating a fifth parameter depending on the relative position and relative attitude between the photographing position of the photographing means and the fixed part based on the rotation angle and the reference relative data. C-6) depending on the relative position and relative attitude of the position of the fixed part and the reference object from the fifth parameter and the first parameter received by the second communication means. First Image capturing system and calculates the parameters. 5. The image photographing system according to claim 4, wherein the movable photographing device is attachable to a human body by the attaching means. 6. A movable photographing device, comprising: photographing means for acquiring an image of a reference object having a known three-dimensional shape as first image data; and rotating the photographing means with respect to a fixed part. A movable photographing device, comprising: a driving device; and a control device that controls the driving device so that the photographing device can photograph the reference object when the movable photographing device moves. 7. The movable photographing apparatus according to claim 6, wherein the fixed portion is attached to a subject photographing device that obtains an image of the subject as second image data, and the photographing unit includes: The movable photographing apparatus, further comprising: photographing control means for causing the photographing means to obtain the first image data in association with the acquisition of the second image data of the subject photographing device. 8. The movable photographing apparatus according to claim 6, wherein an attachment means for attaching the fixed part to another object and a rotation angle of the photographing means rotated by the driving means from a reference posture are obtained. A fifth unit that depends on the relative position and relative posture between the photographing position of the photographing unit and the fixed unit from the positional relationship between the photographing unit and the fixed unit in the reference posture and the rotation angle.
A movable photographing device characterized by calculating parameters. 9. The movable photographing device according to claim 8, wherein the movable photographing device is attachable to a human body by the attaching means.