JP2003014421A - Measuring apparatus and measuring method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly measure a subject, even a large one, with high measurement accuracy while ensuring a wide movable range of a camera. SOLUTION: Two or more three-dimensional charts 2 (2a, 2b) are arranged around the subject 30. A chart pattern of a known structure is formed on the side surface of each three-dimensional chart 2. The three-dimensional charts arranged around the subject 30 are photographed by use of a movable camera 11, and the relative positional relation of each three-dimensional chart 2 is determined from the resulting images. The movable camera 11 takes the image of one three-dimensional chart 2 (a three-dimensional chart to be photographed) synchronously with the photographing of the subject 30 with a subject camera (subject photographing camera) 13. The three-dimensional information of the subject 30 is generated on the basis of the relation between the movable camera 11 and the three-dimensional chart to be photographed, the relation between the movable camera 11 and the subject camera 13, and the relative positional relation of each three-dimensional chart.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for photographing a subject with a camera from a plurality of directions and constructing a three-dimensional image model of the subject.

[0002]

2. Description of the Related Art A three-dimensional image model of a subject can be obtained by photographing a three-dimensional subject from a plurality of directions and combining a plurality of image data obtained thereby. That is, if the data of the external parameters of the camera (such as the position and orientation of the camera) and the internal parameters (such as the focal length) can be obtained for each of the images of the subject taken from a plurality of directions, the shape from silhouette method can be used. The three-dimensional model can be reconstructed from the silhouette image of the subject. For more information on this shape from silhouette method, see W. Ni.
em, "Robust and Fast Modeling of 3D Natural Objec
ts from Multiple Views "SPIE ProceedingsImage and
Video Proceeding II vol.2182,1994, pp.388-397. Hereinafter, the external parameters and internal parameters of the camera will be collectively referred to as “(camera) calibration parameters”. If the internal parameters of these calibration parameters are known and the calibration of the camera by the internal parameters has been completed, Once the external parameters of the camera are obtained, it is possible to construct a three-dimensional image model of the subject.

By the way, one method for photographing an object from a plurality of directions is a fixed arrangement method in which a plurality of cameras are fixedly arranged at different positions to photograph an object. However, in this fixed arrangement method, since a plurality of cameras must be fixed and dispersed in a photographing studio or the like, the photographing equipment becomes large in size.

Therefore, a moving photographing method has been proposed in which a user holds a single hand-held camera and moves around the subject while sequentially photographing the subject from a plurality of directions to obtain an image of the entire circumference of the subject. There is. In order to determine the external parameters of the camera in this moving photographing method, it is necessary to specify the position and orientation of the camera at each photographing.

As one of the methods for measuring the external parameters of the camera in this case, a plan chart and an optical method used have been conventionally proposed.

[0006]

However, since the plane chart has a narrow angle range in which it can be observed and cannot be observed from a direction exceeding 90 degrees from the normal direction of the plane chart, the movable range of the camera is large. It has the problem of being limited. Moreover, even if the camera is within the range where the plane chart can be observed, the observation accuracy of the pattern on the plane chart decreases when the direction of the camera deviates significantly from the normal direction of the plane chart, and as a result, the outside of the camera There is also a drawback that the accuracy of parameter determination is not good.

Particularly, when the subject is an object that is sufficiently larger than the plane chart, the conventional optical system cannot obtain accurate external parameters and cannot construct a three-dimensional image model of the subject. The problem becomes remarkable.

Therefore, the present invention has been made in view of the above problems, and it is possible to construct a three-dimensional image model with high measurement accuracy while securing a wide movable range while using an optical system. The purpose is to realize the device.

[0009]

In order to achieve the above object, the invention according to claim 1 is characterized in that a plurality of posture detecting reference objects having a known shape are arranged around a subject, and A measuring device for measuring a subject, based on an image taken by the reference object measuring camera in advance, and the plurality of posture detection reference objects by the reference object photographing camera,
First computing means for obtaining a relative positional relationship between the plurality of reference objects for attitude detection, and control means for controlling so that the reference object shooting camera performs shooting in synchronization with the measurement operation of the measuring device, Based on the image of the attitude detection reference object at the time of measurement taken by the reference object imaging camera,
Second calculation means for calculating the attitude of the measuring device.

According to a second aspect of the present invention, in the measuring apparatus according to the first aspect, the first computing means includes at least two posture-detecting reference objects within an angle of view of the reference-object photographing camera. It is characterized in that a relative positional relationship between the at least two posture detection reference objects is obtained based on each posture detection reference object included in the image captured so as to fit.

According to a third aspect of the present invention, in the measuring apparatus according to the first aspect, the first computing means changes the posture of the reference object photographing camera while the reference object photographing camera is used. The relative positions of the plurality of posture detection reference objects based on each image obtained by previously photographing the plurality of posture detection reference objects and the posture of the reference object photographing camera when the respective images are photographed. It is characterized by finding the physical position relationship.

The invention according to a fourth aspect is the first to the third aspects.
The measuring device described in any one of 1 to 3 is characterized in that the camera for photographing the reference object is variably fixed in a photographing direction with respect to the measuring device.

According to a fifth aspect of the present invention, in the measuring apparatus according to the fourth aspect, the control means sets the reference object photographing camera so as to photograph at least one of the posture detecting reference objects. It is characterized in that it has a photographing direction control means for controlling the photographing direction of the reference object photographing camera.

The invention according to a sixth aspect is the first to fifth aspects.
2. The measuring device according to claim 1, wherein the measuring device is a device for measuring a three-dimensional shape of a subject, and the three-dimensional shape obtained by a plurality of measurements is combined based on the posture at each measurement. It is characterized by that.

According to a seventh aspect of the present invention, there is provided a measuring device for measuring the subject in a state where a plurality of reference objects for posture detection having a known shape are arranged around the subject.
Based on an image captured by the reference object capturing camera and / or the subject capturing camera in advance, the subject capturing camera capturing the subject, the reference object capturing camera, and the plurality of posture detection reference objects in advance. First control means for obtaining the relative positional relationship between the plurality of reference objects for posture detection, and control for controlling the reference object photographing camera to perform photographing in synchronization with the photographing operation of the subject photographing camera. Means, and second calculation means for calculating the attitude of the measuring device based on an image of the attitude detecting reference object at the time of measurement taken by the reference object imaging camera,
It is characterized in that a predetermined measurement calculation is performed on the basis of the image captured by the camera for capturing an object and the posture at that time.

The invention according to claim 8 is a measuring method for measuring the subject in a state in which a plurality of reference objects for posture detection having a known shape are arranged around the subject,
Synchronizing with a step of photographing the plurality of posture detection reference objects with a reference object photographing camera in advance and obtaining a relative positional relationship of the plurality of posture detection reference objects based on the image, and a measurement operation of the measuring device. Then, the step of photographing at least one of the plurality of posture detection reference objects with the reference object photographing camera, and the step of measuring the posture detection reference object at the time of measurement photographed by the reference object photographing camera. Calculating the orientation of the measuring device based on the image.

According to a ninth aspect of the present invention, there is provided a measuring method for measuring the subject with a plurality of reference objects for posture detection having a known shape arranged around the subject.
A step of preliminarily photographing the plurality of posture detection reference objects with a reference object photographing camera and / or a subject photographing camera, and obtaining a relative positional relationship of the plurality of posture detection reference objects based on the image; In synchronization with the shooting operation of the object shooting camera, a step of shooting at least one of the plurality of posture detection reference objects with the reference object shooting camera, and a step of shooting with the reference object shooting camera Calculating a posture of the measuring device based on the image of the reference object for posture detection at the time of measurement, and performing a predetermined measurement calculation based on the image photographed by the camera for photographing the subject and the posture at that time And have.

[0018]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will now be described in detail with reference to the drawings.

<1. Configuration of 3D Information Generating System> FIG. 1 is a diagram showing an example of the configuration of the 3D information generating system 1 according to the present invention. This 3D information generation system 1
Is a portable camera system 10 capable of capturing a stereoscopic image of a subject 30, and a plurality of stereoscopic charts 2 for camera calibration arranged in the vicinity of the subject 30 in a space accommodating the subject 30. I have it. Note that FIG. 1 illustrates an example in which two stereo charts 2a and 2b are provided as the plurality of stereo charts 2.

As will be described in detail later, each three-dimensional chart 2 is a three-dimensional object in which a chart pattern is formed on each side surface of a substantially pyramidal main body, and is a posture detection reference object. Each three-dimensional chart 2 is suspended from the chart support 250. The chart support 250 includes an inverted L-shaped arm 252 extending from the pedestal 251, and each solid chart 2 is fixed near the tip of the arm 252. Preferably, each three-dimensional chart 2 is suspended substantially above the subject 30.

The camera system 10 is a three-dimensional information generating apparatus which is provided with a camera for photographing a subject (hereinafter, abbreviated as "camera for subject") 13 having a function as a digital camera, and is a measurement for measuring a subject. It also functions as a device. In addition, on the upper part of the camera 13 for the subject,
Movable camera 11 that functions as a camera for photographing a reference object
Is attached so that the posture can be changed. Movable camera 1
1 is a chart pattern of the three-dimensional chart 2 (see FIG. 2)
By photographing a plurality of unit figures UP included in
It is used to specify the relative position and orientation relationship between the three-dimensional chart 2 and the movable camera 11, and further to detect the position and orientation of the subject camera 13 with respect to the three-dimensional chart 2.

As shown in FIG. 1, the three-dimensional information generation system 1 may include a notebook computer 15 or the like. In that case, the computer 15 can exchange commands and data with the camera system 10 by wireless or wired data communication via a communication interface, or data exchange via a recording medium such as a memory card. Is composed of.

<2. Structure of Solid Chart> FIG. 2 is a side view of each solid chart 2. The three-dimensional chart 2 has a three-dimensional chart body 203 and a chart pattern CP formed on the surface of the three-dimensional chart body 203.

Of these, the three-dimensional chart body 203 has a polygonal pyramid-shaped display portion 204 and a truncated pyramid-shaped support portion 205 integrated with each other, and has a hollow interior. The chart pattern CP includes the side surfaces T1 to Tn (n (n
Is an integer of 3 or more) and is a set of patterns P1 to Pn. Preferably, the number n of side surfaces of the polygonal pyramid is n = 3.
To 36, and more preferably n = 6 to 12. Each pattern P1 formed on each side surface T1 to Tn
Pn is a two-dimensional pattern, but the patterns P1 to Pn are three-dimensionally arranged so that the patterns P1 to Pn are
The chart pattern CP as a set of n is a three-dimensional pattern. Each of the patterns P1 to Pn is a set of a plurality of trapezoids each functioning as a unit figure, and the shape of each unit figure is formed with high accuracy based on the design value. Therefore, the vertex position of each unit figure is known.

A marker 201, which serves as a reference point when the movable camera 11 tracks the chart pattern CP, is provided at the apex of the polygonal pyramid forming the display section 204.
As a light emitting diode (LED) is attached, the movable power source for the marker 1 is turned on by the marker power source provided inside the three-dimensional chart 2 lighting the marker 201.
1 is configured so that the position of the three-dimensional chart 2 can be recognized easily and accurately.

<3. Structure of Movable Camera 11> FIG. 3 is a front view of the movable camera 11, and FIG.
2 is a block diagram showing the internal functions of FIG. As shown in FIGS. 3 and 4, in the movable camera 11, the lens unit 1
10 and 2 imaged by this lens unit 110
A two-dimensional light receiving element 111 for photoelectrically converting a three-dimensional image is integrally housed in a spherical unit 116. The two-dimensional light receiving element 111 is a CC in which a plurality of pixels are arranged on the light receiving surface.
It is composed of a D array or the like. Lens unit 110
Is a combination of the fixed lens 110a and the zoom lens 110b, and the aperture / shutter mechanism unit 1 is interposed between them.
10e is present.

As shown in FIG. 3, the spherical unit 116 is connected to the fixed portion 114 through the attitude changing device 113, and together with the respective elements incorporated in the spherical unit 116, the pan direction is rotated about ± 70 ° ( θ rotation) and elevation (φ rotation) of ± about 70 ° in the tilt direction are possible. Then, in order to perform the rotational drive in the pan direction and the rotational drive in the tilt direction, an attitude changing device 113 having a plurality of piezo elements built therein is arranged at the base of the spherical unit 116. A zoom operation corresponding to driving the zoom lens 110b is also performed by a piezo element different from the above. By giving a sawtooth wave signal to these piezo elements, the element to be driven by the piezo element moves slightly, and the required movement is given to the object element by repeating the element. The turning angle in the pan direction and the depression angle in the tilt direction are
Angle sensors 126p and 126 such as encoders, respectively
The drive amount of the zoom lens 110b is detected by t and is also detected by the sensor 126z which is also composed of an encoder. Regarding these drive mechanisms, for example, JP-A-11-18000 and JP-A-11-41504.
It is disclosed in the publication.

As described above, the spherical unit 116 is driven by the posture changing device 113 so that the position and the posture with respect to the subject camera 13 are in an arbitrary state within the movable range.

Further, the movable camera 11 inputs an image signal obtained by the two-dimensional light receiving element 111 to perform data processing such as image processing, and a data processing unit 120 for controlling the posture changing device 113, A tracking button 117 is provided.

The data processing unit 120 includes the two-dimensional light receiving element 1
An image processing unit 121 that receives a signal from the image processing unit 11 to perform processing such as image recognition, and an image memory 122 that stores the image signal obtained by the image processing unit 121 are provided. Further, a camera control unit 123 is provided which generates drive signals for the zoom lens 110b, the posture changing device 113, and the aperture / shutter mechanism unit 110e and outputs them to the image processing unit 121 and the camera control unit 123. ,
Wireless communication with the subject camera 13 is possible via the communication unit 124 and the communication device 112. By this communication, the image data is transmitted to the subject camera 13,
Various information is transmitted and received between the movable camera 11 and the subject camera 13. In the movable camera 11 of this embodiment, as the communication device 112, an infrared element compatible with an IRDA (Infrared Data Association) interface for performing infrared communication is used.

As shown in FIG. 3, the first mounting groove 115a and the second mounting groove 115b provided in the fixed portion 114 are
It is used to attach the fixed portion 114 to the camera 13 for a subject. By attaching the fixing portion 114 to the subject camera 13, the movable camera 11 can be fixed to the subject camera 13, and for example, the turning angle and the depression angle of the movable camera 11 are 0 °. At this time, the relative position and posture relationship between the movable camera 11 and the subject camera 13 is set to a predetermined state.

Further, the tracking button 117 is a mode in which the movable camera 11 automatically tracks the three-dimensional chart 2 (hereinafter abbreviated as "automatic tracking mode") and a mode in which the camera for subject 13 tracks the stereo chart 2. (Hereinafter, abbreviated as “manual mode”) is a button for switching between.

Next, the operation of the movable camera 11 configured as described above will be described.

The two-dimensional light receiving element 111 has an R pixel for each pixel.
A filter of any one of (red), G (green), and B (blue) is attached, and the light imaged on the two-dimensional light receiving element 111 is converted by the two-dimensional light receiving element 111 into RGB color components. It is photoelectrically converted. The image signal obtained by the two-dimensional light receiving element 111 is A /
The image data is D-converted and converted into image data which is a digital signal, and further subjected to various image processing such as white balance correction and γ correction.

When the user presses the tracking button 117 to set the automatic tracking mode, the image processing unit 121
Analyzes the image data to detect the marker 201, and gives a signal indicating, for example, how much the image component of the marker 201 deviates from the image center to the camera control unit 123.
As a result, the camera control unit 123 causes the posture changing device 113 to move the image component of the marker 201 to the center of the image.
Is obtained, and the posture changing device 113 is drive-controlled based on the drive amount. Thereby, the attitude changing device 11 is configured so that the image of the three-dimensional chart 2 is always formed on the two-dimensional light receiving element 111.
3 is controlled.

On the other hand, in the manual mode, the camera control unit 12
Reference numeral 3 controls the posture changing device 113 based on a user instruction received from the subject camera 13.

Further, the camera control section 123 controls each sensor 1
26t, 126p, 126z are configured to be input, lens rotation angle information is generated based on information input from the angle sensors 126t, 126p, and zoom information is generated based on information input from the sensor 126z. To generate. The camera control unit 123 is configured to send these pieces of information to the subject camera 13 via the communication unit 124.

When the shutter button of the subject camera 13 is pressed, the movable camera 11 receives a photographing command from the subject camera 13 and, in response to the reception of the photographing command, the two-dimensional light receiving element 111. The image capturing operation is performed to generate image data. Then, the image data and the lens rotation angle information obtained at that time are transmitted to the subject camera 13 via the communication unit 124.

<4. Configuration of Subject Camera 13> Next,
The configuration of the subject camera 13 will be described. Figure 5
3 is a block diagram showing an internal configuration of a subject camera 13. FIG. As shown in FIG. 5, the subject camera 13 includes a lens unit 155, a two-dimensional light receiving element 156, and an image processing unit 1.
57, image memory 158, control unit 160, shutter button 161, flash 162, display 163, operation button 164, card slot 165, memory card 166, communication unit 167, communication device 168, memory 1
69 and a computing unit 170. Also,
The calculation unit 170 includes a first calculation unit 171 and a second calculation unit 172.
It is configured with.

The light from the subject 30 is emitted from the lens unit 1
The light enters the subject camera 13 via 55 and is imaged on the two-dimensional light receiving element 156. The two-dimensional light receiving element 156 is composed of a CCD array in which a plurality of pixels are arranged on the light receiving surface. The two-dimensional light receiving element 156 photoelectrically converts the light received for each pixel. One of RGB filters is provided for each pixel on the light-receiving surface side of the two-dimensional light receiving element 156, and the photoelectric conversion function of the two-dimensional light receiving element 156 generates an image signal having RGB color components for each pixel. It

The image processing section 157 is a two-dimensional light receiving element 156.
The image signal from is A / D converted to generate image data which is a digital signal. Further, the image processing unit 157 further performs various image processing such as white balance correction and γ correction on the image data. The image data generated by the image processing unit 157 is stored in the image memory 158 composed of a semiconductor memory or the like.

The shutter button 161 is a button for the user to instruct photographing, and the flash 162 is for illuminating the subject 30 when photographing the subject 30.
Further, the display 163 is a display device that displays an operation guide screen serving as a user interface and a photographed subject image, and the operation button 164 is a user-movable camera 11.
Is a button for performing an input operation or the like for manually changing the position and orientation of the. Card slot 1
A removable memory card 166 can be attached to the computer 65, and data can be exchanged with the computer 15 via the memory card 166. The communication unit 167 has a function of performing data communication with the movable camera 11 via the communication device 168. The memory 169 is a storage unit for temporarily storing the calculation target data by the calculation unit 170 such as the image data input from the movable camera 11 and the lens rotation angle information. Further, the memory 169 stores in advance information about internal parameters (focal length, etc.) of the movable camera 11.

The control unit 160 is composed of a CPU,
It is configured to control the operation of each of the above units. When the user presses the shutter button 161, the control unit 160
Controls the photographing operation by the two-dimensional light receiving element 156 and the image processing unit 157. The control unit 160 also generates a shooting command in response to the user's operation of pressing the shutter button 161, and transmits the shooting command to the movable camera 11 via the communication unit 167 in order to synchronize with the subject camera 13. .

Then, the control section 160 stores the photographed image data stored in the image memory 158 by the photographing operation of the subject camera 13 in the memory 169, and receives the image data received from the movable camera 11 after transmitting the photographing command. And lens rotation angle information in the memory 169
To store.

The arithmetic unit 170 is also composed of a CPU, and functions as the first arithmetic unit 171 and the second arithmetic unit 172 by executing a predetermined arithmetic program.
Such an arithmetic program may be stored in advance in a memory or the like at the manufacturing stage, or may be input later from an external device.

When the image data obtained by previously photographing a plurality of three-dimensional charts 2 by the movable camera 11 is input, the first calculation unit 171 inputs each three-dimensional chart 2 included in the image data.
The calculation function (first calculation means) for obtaining the relative positional relationship between the plurality of three-dimensional charts 2 is realized based on the image component of.

In addition, the second arithmetic unit 172 synchronizes with the subject camera 13 photographing the subject 30, and the movable camera 11 causes at least one of the three-dimensional charts of the plurality of three-dimensional charts. The target position and the shooting posture of the movable camera 11 with respect to the target solid chart are obtained based on the image data obtained from the movable camera 11. Then, the second calculation unit 172 further causes the movable camera 11
Of the photographing target and the photographing posture with respect to the photographing target three-dimensional chart, the relative position and the relative posture of the movable camera 11 and the subject camera 13, and the relative positional relationship between the photographing target three-dimensional chart and the other three-dimensional chart 2. Based on this, a calculation function (second calculation means) for obtaining three-dimensional information of the subject 30 from the captured image obtained from the subject camera 13 is realized.

Further, the control section 160 judges whether the movable camera 11 is in the manual mode or the automatic tracking mode,
In the manual mode, the user operates the operation buttons 164
The signal obtained by operating a part of the
It is transmitted and output to the movable camera 11 via 67. As a result, the posture changing device 113 of the movable camera 11 can be controlled by a user's manual operation.

<5. Principle of Three-Dimensional Information Generation> In the three-dimensional information generation system 1 configured as described above, the principle of data calculation when generating the three-dimensional information of the subject 30 will be described.

When the internal parameters of the camera (focal length, pixel density, etc.) are known, the position of each point of the subject image formed on the two-dimensional light receiving element in the photographing space of the camera is determined. It can be calculated.
For example, when one point of the subject image is formed on one pixel of the two-dimensional light receiving element, the two-dimensional coordinate value can be obtained from the image forming position of the two-dimensional light receiving element.

Therefore, by photographing the three-dimensional chart 2 having a known structure with the movable camera 11, external parameters of the movable camera 11, that is, the relative position of the movable camera 11 with respect to the three-dimensional chart 2 and The posture can be specified.

In general, the calculation of the external parameters of the camera is such that the internal parameters of the camera are known, the three-dimensional coordinate values of four or more points on the same plane fixed in the absolute coordinate system are known, and , Can be performed under the condition that the two-dimensional coordinate values of the points on the captured image corresponding to the above four points or more can be calculated.

Regarding such a calculation method, for example,
Reference "L.Quan, Z.Lan," Linear N-Point Camera Pose D
etermination, "IEEE Trans.PAMI 21 (8) 1999" and references "Takahashi, Ishii, Makino, Nakashizuka," Measurement of marker position and posture from monocular image for artificial reality interface ", Electronic Journal of Electronic Information The technology disclosed in “AJ79 1996” and the like can be applied. These techniques capture four or more points whose coordinates are known with a camera, and based on known three-dimensional coordinate values of the four points or more and two-dimensional coordinate values obtained from an image obtained by the camera, The relative position and relative attitude of the camera are calculated.

Also in this embodiment, when the movable camera 11 photographs the three-dimensional chart 2, the vertex coordinates of each unit figure included in the three-dimensional chart 2 can be obtained from the image data obtained from the two-dimensional light receiving element 111. it can. However, the vertex coordinates in this case are coordinate values with respect to the unique local coordinate system in the state where the movable camera 11 photographs the three-dimensional chart 2.

On the other hand, the three-dimensional chart 2 is fixed while being arranged around the subject 30. If a three-dimensional coordinate system unique to this three-dimensional chart 2 is set (this is referred to as "chart coordinate system"), the vertex position of each unit figure of the three-dimensional chart 2 is determined based on the design value of the three-dimensional chart 2. be able to.

When the coordinate values of at least four points of the chart pattern CP are obtained based on the image data taken by the movable camera 11, the four measured points of the three-dimensional chart 2 are specified from the correspondence of each point. Therefore, the coordinate values of the four designed points in the chart coordinate system and the movable camera 11
The relative position and orientation of the movable camera 11 with respect to the three-dimensional chart 2 can be specified from the coordinate values of the four points in the local coordinate system. And the movable camera 1
From the relationship between the 1 local coordinate system and the chart coordinate system,
Parameters for converting the local coordinate system of the movable camera 11 into the chart coordinate system are defined.

The chart coordinate system is Xc, and the movable camera 11 is used.
Let Xm be the local coordinate system of, and Rc be the translation matrix for converting the local coordinate system Xm to the chart coordinate system Xc, and Tc be the translation vector.

[0058]

[Equation 1]

The coordinate value expressed in the local coordinate system Xm of the movable camera 11 by the formula
It is possible to convert into coordinate values expressed by.

That is, the movable camera 11 relative to the three-dimensional chart 2 based on the coordinate values of at least four points of the chart pattern CP obtained by the movable camera 11 photographing the three-dimensional chart 2 and the design values of the chart pattern CP. Since the posture and the relative position can be specified, the rotation movement matrix Rc and the parallel movement vector T in the equation (1) are used.
c can be determined, which defines the coordinate transformation formula.

FIG. 6 is a diagram showing the concept of coordinate conversion in the three-dimensional information generation system 1. As shown in FIG. 6, the coordinate conversion formula of the formula 1 is a formula for converting the local coordinate system Xm set in the movable camera 11 to the chart coordinate system Xc set in the three-dimensional chart 2. That is, since the relative attitude and relative position of the movable camera 11 with respect to the three-dimensional chart 2 are specified by the movable camera 11 photographing the three-dimensional chart 2, the equation 1 based on the relative attitude and relative position is It can be defined as a conversion formula from the local coordinate system Xm to the chart coordinate system Xc.

Further, the movable camera 11 is used as the posture changing device 1.
When the posture is changed to the turning angle θ and the elevation angle φ by 13 and when the turning angle is 0 ° and the elevation angle is 0 °, respectively, when the movable camera 11 captures an image,
The coordinate system of coordinate values derived from each image is different. As shown in FIG. 6, the coordinate values derived from the image captured in the state (11b) in which the movable camera 11 is at the turning angle of 0 ° and the elevation angle of 0 ° are represented by the local coordinate system Xmo, and
The coordinate values derived from the image captured by the movable camera 11 in the turning angle θ and the elevation angle φ (11a) are represented by the local coordinate system Xm. Since these local coordinate systems Xmo and Xm are generated due to the change of the attitude of the movable camera 11, the local coordinate system Xmo
The coordinate conversion parameter between mo and Xm can be obtained in advance as an internal parameter, or can be obtained by calculation.

The local coordinate system Xmo is replaced with the local coordinate system X
Let R (θ, φ) be the rotation matrix for conversion into m, and T (θ, φ) be the translation vector.

[0064]

[Equation 2]

It is possible to convert the coordinate value represented by the local coordinate system Xmo into the coordinate value represented by the local coordinate system Xm by the equation (3). The rotation matrix R
(Θ, φ) and the translation vector T (θ, φ) are uniquely determined if θ and φ are determined. In the movable camera 11, since the angle sensors 126t and 126p generate lens rotation angle information, that is, information regarding θ and φ, if this lens rotation angle information is used,
Rotational transfer matrix R (θ, φ) and translational vector T
(Θ, φ) can be obtained.

Further, the movable camera 11 has a turning angle of 0 °.
When the moving camera 11 takes an image when the elevation angle is 0 °, the coordinate system of the coordinate value derived from the image is the coordinate value derived from the taken image when the subject camera 13 takes the image. Different from the coordinate system of. As shown in FIG. 6, the movable camera 11 has a turning angle of 0 ° and an elevation angle of 0 °.
The coordinate values derived from the image photographed in the state (11b) are represented by the local coordinate system Xmo, and the coordinate values derived from the photographed image photographed by the subject camera 13 are represented by the local coordinate system Xo. It That is, since the posture and position at which the movable camera 11 shoots and the posture and position at which the subject camera 13 shoots are different, coordinate values obtained from the respective images even if the shooting directions are simply the same direction. Are for different coordinate systems.

From the local coordinate system Xo to the local coordinate system X
The coordinate conversion to mo can be obtained in advance from a design value or the like when the movable camera 11 is fixed to the subject camera 13. However, in actual use, the movable camera 11
Since it is expected that a slight error will occur when the is attached to the subject camera 13, it is preferable to obtain the conversion parameters for performing the coordinate conversion when the subject is photographed.

For example, before the photographing operation of the subject 30, the turning angle and the elevation angle of the movable camera 11 are set to 0 °, and the movable camera 11 and the subject camera 13 preliminarily display the same three-dimensional chart 2. The relative posture and the relative position of the subject camera 13 with respect to the movable camera 11 can be specified by photographing and obtaining the same vertex coordinates on the three-dimensional chart 2 from each image. Then, by performing coordinate conversion based on the relative attitude and relative position,
Coordinate conversion from the local coordinate system Xo to the local coordinate system Xmo can be performed.

Specifically, the rotational movement matrix for converting the local coordinate system Xo into the local coordinate system Xmo is Rh,
If the translation vector is Th,

[0070]

[Equation 3]

It is possible to convert the coordinate value expressed in the local coordinate system Xmo into the coordinate value expressed in the local coordinate system Xm by the formula (1). The rotation matrix R
h and the translation vector Th are a matrix and a vector that are determined based on the relative attitude and relative position of the subject camera 13 with respect to the movable camera 11.

In this way, coordinate conversion between each coordinate system becomes possible. Then, the subject camera 13 is used as the subject 3
In synchronization with the shooting of 0, the movable camera 11 shoots the shooting target stereo chart 2 of the plurality of stereo charts 2, and thereby the coordinate values derived from the captured image of the camera for subject 13 are set to the shooting target. It can be converted into coordinate values expressed in the chart coordinate system of the three-dimensional chart 2.

Specifically, it is derived from the above equations 1 to 3.

[0074]

[Equation 4]

The coordinate value obtained in the local coordinate system Xo of the subject camera 13 is calculated by the equation
It can be converted into the coordinate value expressed by.

By the way, in this embodiment, the subject 3
As described above, a plurality of three-dimensional charts 2 are arranged around 0. Then, when the subject camera 13 photographs the subject 30, the movable camera 11 determines which of the plurality of three-dimensional charts 2 is the photographing target three-dimensional chart 2.
It is optional. Therefore, the subject camera 1
When the side surface (first side surface) of the subject 30 is photographed at 3, the movable camera 11 moves the stereo chart 2a (see FIG. 1).
Is a three-dimensional chart to be photographed, and the subject camera 1
When the other side surface (second side surface) of the subject 30 is photographed at 3, the movable camera 11 moves the three-dimensional chart 2b (see FIG. 1).
It is also conceivable to use as a three-dimensional chart to be photographed.

In such a case, even if the coordinate value is obtained from the photographed image when the subject 30 is photographed by the subject camera 13 and the coordinate conversion based on the equation (4) is performed, Is changed to the coordinate value of the chart coordinate system Xca for the three-dimensional chart 2a, and for the second side surface, the chart coordinate system Xc for the three-dimensional chart 2b.
It is changed to the coordinate value of b. Therefore, even if these coordinate values are combined, accurate three-dimensional information about the subject 30 cannot be generated.

Therefore, in this embodiment, the subject 30
The relative positional relationship between the plurality of three-dimensional charts 2 arranged around the three-dimensional chart 2 is obtained, and the chart coordinate system for each three-dimensional chart is determined based on the relative positional relationship. , "Reference coordinate system").

FIG. 7 is a diagram showing the concept of coordinate conversion for converting the chart coordinate system into the reference coordinate system. In the example of FIG. 7, it is assumed that the chart coordinate system Xca of the three-dimensional chart 2a is the reference coordinate system.

When obtaining the relative positional relationship between the plurality of three-dimensional charts 2, for example, the plurality of three-dimensional charts are set by the movable camera 11 in such a state that the plurality of three-dimensional charts 2 fit within the angle of view G1 of the movable camera 11. Take a picture of 2.

The three-dimensional chart 2a is displayed by the movable camera 11.
When is photographed, the coordinate value in the local coordinate system Xm is obtained from the image. On the other hand, since the three-dimensional chart 2a is formed with a known structure, the chart coordinate system (reference coordinate system)
The coordinate value in Xca is known by design.

Therefore, the three-dimensional chart 2a is obtained by the coordinate values of at least four points of the chart pattern CP obtained by the movable camera 11 photographing the three-dimensional chart 2a and the corresponding design values of the four chart patterns CP. The relative attitude and relative position of the movable camera 11 with respect to can be specified.

Therefore, similarly to the above-mentioned formula 1, the coordinate conversion formula from the local coordinate system Xm to the reference coordinate system Xca is

[0084]

[Equation 5]

It is defined as follows. Note that Rca and Tca in the equation of Formula 5 are respectively expressed by the local coordinate system Xm.
To the reference coordinate system Xca and the translation vector, which can be obtained from the relative attitude and relative position of the movable camera 11 with respect to the stereo chart 2a.

Similarly, when the movable camera 11 photographs the three-dimensional chart 2b, the local coordinate system X
The coordinate value at m is obtained. On the other hand, the three-dimensional chart 2b
Is also formed with a known structure, the chart coordinate system Xcb
The coordinate values at are known by design.

Therefore, the three-dimensional chart 2b is obtained by the coordinate values of at least four points of the chart pattern CP obtained by the movable camera 11 photographing the three-dimensional chart 2b and the corresponding design values of the four chart patterns CP. The relative attitude and relative position of the movable camera 11 with respect to can be specified.

Therefore, similarly to the above-mentioned formula 1, the coordinate conversion formula from the local coordinate system Xm to the chart coordinate system Xcb is

[0089]

[Equation 6]

It is defined as follows. Note that Rcb and Tcb in the equation of Formula 6 are respectively expressed by the local coordinate system Xm.
To the reference coordinate system Xcb and a translation vector and a translation vector, which can be obtained from the relative attitude and relative position of the movable camera 11 with respect to the stereo chart 2b.

Then, based on the coordinate conversion formulas of the above equations 5 and 6, the chart coordinate system Xcb is changed to the reference coordinate system Xca.
When the coordinate conversion formula to

[0092]

[Equation 7]

It becomes like this. The expression of the equation 7 is an expression showing the relative positional relationship between the three-dimensional charts 2a and 2b. Considering the case where a large number of three-dimensional charts 2 are provided, by generalizing the formula of Formula 7,

[0094]

[Equation 8]

It becomes Note that in the formula of Equation 8, i is i
The third stereo chart 2 (i) is shown, where Rci and Tci are the rotational movement matrix and translation vector from the local coordinate system Xm to the chart coordinate system Xci, respectively, and the movable camera for the stereo chart 2 (i) is shown. It can be obtained from the relative attitude and relative position of 11.

Then, when the coordinate value of the subject 30 is obtained from the photographed image obtained by the subject camera 13 by the coordinate conversion equation of the equation 4 and the coordinate conversion equation of the equation 8, the coordinate value is used as the reference coordinate system ( That is, the coordinate values can be converted into the coordinate values in the world coordinate system in the three-dimensional information generation system 1, and the three-dimensional image model of the subject 30 can be appropriately constructed.

In the above example, in order to make it easy to understand the calculation principle, a case has been described in which a plurality of three-dimensional charts 2 are photographed so as to fit within the angle of view G1 of the movable camera 11. However, it is not limited thereto.

For example, when a large number of three-dimensional charts 2 are provided, not all of the three-dimensional charts 2 are contained within the angle of view G1, but the two three-dimensional charts 2 are arranged within the angle of view G1. It is possible to adopt a method in which the photographing operation is repeated and the relative positional relationship between the two stereo charts included in each image is sequentially obtained. If the two stereo charts 2 are photographed so as to fit within the angle of view G1, there is an advantage that the relative positional relationship between the two stereo charts 2 can be easily obtained from the image.

On the other hand, a three-dimensional chart 2 that fits within the angle of view G1.
It is expected that the image component of each three-dimensional chart 2 in the image will become smaller as the number of H.sub.2 increases. In that case, the accuracy of obtaining the relative positional relationship between the three-dimensional charts decreases. For this reason, it is preferable to reduce the number of the three-dimensional charts 2 that can be accommodated within the angle of view G1 in one shooting.

From this point of view, it can be said that it is most preferable that the movable camera 11 individually captures the three-dimensional charts 2 as a method of obtaining the relative positional relationship between the plurality of three-dimensional charts 2. For example, if all the three-dimensional charts 2 can be individually photographed when the posture of the movable camera 11 is changed within the movable range,
The relative positional relationship between the three-dimensional charts 2 can be obtained by individually photographing each three-dimensional chart 2 and performing coordinate conversion based on the turning angle and the elevation angle when each image is photographed. At this time, since the movable camera 11 captures one stereo chart 2, the image component of the stereo chart 2 included in the captured image can be captured in a large state, and as a result, the movable camera for the stereo chart 2 can be captured. The position and orientation of 11 can be obtained with high accuracy.

<6. Operation Process of 3D Information Generating System 1> Next, an operation process of the 3D information generating system 1 will be described. 8 to 12 are flowcharts showing the operation process of the three-dimensional information generation system 1.

First, a plurality of three-dimensional charts 2 are used as the subject 30.
Are arranged around (step S1). The three-dimensional chart 2 may be arranged randomly at this time, but it is preferable that the three-dimensional chart 2 be arranged around the subject 30 at substantially equal intervals according to the size of the subject 30. Further, the plurality of three-dimensional charts 2 arranged around the subject 30 are fixed until the operation process is finished, more strictly, the subject photographing process of step S4 is finished.

Next, the process of calculating the relative positional relationship of the plurality of three-dimensional charts 2 is performed (step S2). The details of this processing are shown in the flowchart of FIG.

First, the user does not occlude the three-dimensional charts 2 in front of the plurality of three-dimensional charts 2, that is, one stereo chart 2 does not become a shadow of another three-dimensional chart 2. Is installed (step S21). At this time, the camera system 10 is fixed, but the movable camera 11 is installed in a state in which the angle of view can be changed by the action of the posture changing device 113.

Then, the user sets the movable camera 11 to the manual mode, and operates the operation button 16 of the subject camera 13.
By operating 4, the three-dimensional chart 2 can move the movable camera 1
The lens rotation angle of the movable camera 11 is designated so that the angle of view becomes 1 (step S22). As a result, the subject camera 13 gives an instruction to change the attitude to the movable camera 11, the shooting direction of the movable camera 11 faces in the instructed direction, and the three-dimensional chart 2 is within the angle of view of the movable camera 11. Fits in. At this time, one 3D chart 2
It is preferable to instruct the shooting direction so that is within the angle of view.

The user gives a photographing instruction of the three-dimensional chart 2 by the movable camera 11 (step S23). As a result, the movable camera 11 performs a photographing operation, and the image data of the three-dimensional chart 2 is transmitted from the movable camera 11 to the subject camera 13. At that time, the lens rotation angle information is also transmitted together with the image data. The subject camera 13 stores and saves the image data and the lens rotation angle information received from the movable camera 11 in the memory 169.

Then, the user selects the uncaptured stereo chart 2
Is present (step S24), and if so, the movable camera 11 is set in step S25 so that the uncaptured stereoscopic chart 2 is within the angle of view. As a result, the posture of the movable camera 11 is changed, and one uncaptured stereo chart 2 is displayed on the movable camera 11.
Fits within the angle of view of.

By repeating the processing of steps S23 to S25, all the three-dimensional charts 2 arranged around the subject 30 are photographed by the movable camera 11. When the user finishes photographing all the three-dimensional charts 2, the user inputs an input to that effect through the operation button 164 of the camera 13 for a subject.

In the subject camera 13, the first arithmetic unit 171 of the arithmetic unit 170 functions. Then, the first calculation unit 171
The image data obtained from the movable camera 11 stored in the memory 169 is acquired by the movable camera 1 for each stereo chart 2 when each stereo chart 2 is photographed.
The relative position and relative attitude of No. 1 are calculated (step S2
6). That is, by this process, each parameter (rotational movement matrix and translational vector) shown in the equations (5) and (6) is obtained.

Next, the first arithmetic unit 171 selects one basic stereo chart showing the reference coordinate system from the plural stereo charts 2, and the relative position and relative attitude of the movable camera 11 with respect to the basic stereo chart. And the relative position and relative attitude of the movable camera 11 with respect to another three-dimensional chart,
Based on the lens rotation angle information, the relative position and relative attitude of the other 3D chart with respect to the basic 3D chart are calculated, and the calculated data are stored in the memory 169 (step S).
27). By this processing, each parameter shown in the equation (7) or (8) is obtained. The selection of the basic 3D chart 2 may be configured such that the calculation unit 170 automatically selects any one of the plurality of 3D charts 2, or the user designates the operation button 164. You may.

As described above, the plurality of three-dimensional charts 2 are distinguished into one basic three-dimensional chart and the other three-dimensional charts, and the relative positional relationship of all the other three-dimensional charts with respect to the basic three-dimensional chart is determined. , The process of step S2 ends.

Next, a process for calculating the relative position and relative attitude between the subject camera 13 and the movable camera 11 is performed (see FIG. 8; step S3). Details of this process are shown in Fig. 1.
0 is shown in the flowchart.

First, the user adjusts the orientation of the camera system 10 so that the subject camera 13 and the movable camera 11 capture the same stereo chart 2 within their respective field angles (step S31).

Next, the user gives an instruction to photograph the same three-dimensional chart 2 by the subject camera 13 and the movable camera 11 (step S32). For example, the subject camera 13
When the shutter button 161 is pressed, the subject camera 13 and the movable camera 11 of the camera system 10 synchronously perform a shooting operation. As a result, the image data obtained by photographing the three-dimensional chart 2 with the movable camera 11 is transmitted from the movable camera 11 to the subject camera 13, and the image data is stored in the memory 169 of the subject camera 13. Further, image data obtained by photographing the three-dimensional chart 2 with the subject camera 13 is also stored in the memory 169 of the subject camera 13.

Then, in the subject camera 13, the second arithmetic unit 172 of the arithmetic unit 170 functions and the movable camera 1
1, the relative position and the relative attitude of the movable camera 11 with respect to the stereo chart 2,
That is, the calculation calculation of the external parameters of the movable camera 11 is performed (step S33). Similarly, the relative position and the relative attitude of the subject camera 13 with respect to the three-dimensional chart 2, that is, the calculation calculation of the external parameter of the subject camera 13 is performed from the image of the three-dimensional chart 2 captured by the subject camera 13 (step S33). ). Computing unit 1
Reference numeral 70 determines whether or not the external parameters for each of the movable camera 11 and the subject camera 13 have been appropriately calculated. When the external parameters cannot be calculated, the same stereo chart 2 by the two cameras is used.
The photographing operation (step S32) is repeated. On the other hand, when the external parameters can be calculated appropriately, the process proceeds to step S34.

Then, the second computing unit 172 determines the relative position and relative attitude of the two cameras from the external parameters of the subject camera 13 and the movable camera 11 and the lens rotation angle of the movable camera 11 when the camera is photographing. A camera parameter expressing is calculated (step S34). That is, 2 is obtained by the external parameter obtained in step S33.
Since the relative position and orientation of each of the two cameras with respect to the same three-dimensional chart 2 are clarified, the relative positional relationship between the two cameras is calculated by using the relationship. If the turning angle and the elevation angle of the movable camera 11 are 0 ° when the same three-dimensional chart 2 is photographed, the relative position of the two cameras and the relative position of the two cameras can be easily obtained without considering the lens rotation angle. It is possible to calculate a camera parameter expressing the relative posture. By this processing, each parameter shown in the conversion formula of the above-mentioned formula 3 or each parameter corresponding to the combined conversion formula of the above-mentioned formulas 2 and 3 is obtained.

As described above, the relative position and relative posture between the subject camera 13 and the movable camera 11 in a state where the movable camera 11 is fixed to the subject camera 13 are determined, and the process of step S3 is performed. To finish.

Next, a subject photographing process for photographing the subject 30 for which the three-dimensional information is generated is performed (FIG. 8).
See; step S4). The details of this processing are shown in the flowchart of FIG.

In step S41, the user installs the camera system 10 at an arbitrary position around the subject 30 and points the subject camera 13 toward the subject 30. Then, the movable camera 11 is set to the manual mode,
To any one of the plurality of three-dimensional charts 2 (imaging target three-dimensional chart). At this time, it is preferable to select the one that is closest to the camera system 10 from the plurality of three-dimensional charts 2 as the photographing target three-dimensional chart. This is because by setting the closest 3D chart as the shooting target 3D chart, the relative position and relative attitude of the movable camera 11 with respect to the shooting target 3D chart can be obtained with high accuracy. Then, the user sets the movable camera 11 to the automatic tracking mode at the time when the stereoscopic chart to be photographed is within the angle of view of the movable camera 11.

The user uses the subject camera 13 to capture the subject 30.
The framing adjustment of the subject camera 13 is performed to capture the image. During this time, the movable camera 11 automatically tracks the shooting target stereo chart 2, and operates so as to always capture the shooting target stereo chart within the angle of view of the movable camera 11.

Then, if the automatic tracking is performed well, the process proceeds to step S43, and if the automatic tracking fails, the process returns to step S41 so that the object stereoscopic chart is within the angle of view of the movable camera 11 again. Is manually operated (step S42).

When the framing adjustment by the subject camera 13 is completed, the user depresses the shutter button 161 of the subject camera 13 to instruct the photographing operation of the subject 30. In response to this, the subject camera 13 captures an image of the subject 30, and at the same time, the movable camera 11 captures an image of the subject stereoscopic chart 2 (step S).
43).

Then, the photographed image of the subject 30 obtained by the subject camera 13 is stored in the memory 169, and the image data and the lens rotation angle information obtained by photographing the subject stereo chart by the movable camera 11 are also stored in the memory 169. Is stored.

Then, in the camera 13 for the subject, the second calculation section 172 functions. The second calculation unit 172 is provided in the memory 1
The image obtained by the movable camera 11 stored in 69 is read out, and which of the three-dimensional charts 2 is photographed is specified as the three-dimensional chart to be photographed (step S44).

An example for specifying the photographing target stereo chart from the plurality of stereo charts 2 will be described.
For example, the chart patterns CP of a plurality of three-dimensional charts 2
Are formed in different colors. Then, the calculation unit 170 determines the color of the shooting target stereo chart in the image obtained from the movable camera 11, whereby the stereo chart shot as the shooting target stereo chart can be specified from among the plurality of stereo charts. Is.

In addition, as described above, a plurality of three-dimensional charts 2 are used.
By forming the respective chart patterns CP in different colors, it is possible to specify each three-dimensional chart 2 when the relative positional relationship of the plurality of three-dimensional charts 2 is obtained.

Then, the second calculation section 172 obtains the relative position and relative attitude of the movable camera 11 with respect to the photographing target stereo chart 2 based on the image of the photographing target stereo chart 2 obtained from the movable camera 11 (step S4
5). As a result, the external parameters of the movable camera 11 when the movable camera 11 captures the capture target stereo chart are calculated.

Then, the second operation section 172 operates in step S4.
External parameters of the movable camera 11 obtained in (5)
Based on the camera parameters (each parameter in the equation of Equation 3) obtained in step S34, and the lens rotation angle information (each parameter in the equation of equation 2) at the time of shooting in step S43. A relative position and a relative attitude of the subject camera 13 with respect to the image-capturing stereo chart are obtained (step S46).
That is, the external parameters of the subject camera 13 are obtained. As a result, each parameter in the equation (4) is obtained, and the coordinate value of the subject 30 obtained from the captured image of the subject 30 can be converted into the coordinate value in the chart coordinate system Xc of the capture target stereo chart. It will be possible.

The photographed image obtained by the subject camera 13 and the subject camera 1 obtained as described above.
Data is stored in the memory 169 by associating the external parameters of No. 3 with the specified stereoscopic chart of the photographing target.

By the processing of steps S41 to S47, the parameters for converting the image obtained from the photographed image when the subject 30 is photographed by the subject camera 13 into the chart coordinate system of the three-dimensional chart to be photographed are obtained. Become. By repeatedly performing such a process by dividing and photographing the entire circumference of the subject 30, parameters relating to the entire circumference of the subject 30 are generated.

In step S48, if the user determines that another side surface of the subject 30 is to be photographed, the processing from step S41 is repeated. In addition, when the user has performed a shooting operation on the entire circumference of the subject 30, a predetermined input is made to the operation button 164 to instruct the end of shooting. When the instruction to end the shooting is input, the camera system 10 ends the subject shooting process (step S4).

However, when the user moves around the subject 30 and repeatedly photographs the subject 30, the three-dimensional charts to be photographed by the movable camera 11 at each photographing are designated as different three-dimensional charts. For this reason,
The external parameters of the camera 13 for a subject, which are required at the time of each photographing, are different from each other in the standard three-dimensional chart.

Next, in the three-dimensional information generation system 1, a three-dimensional information generation process is performed (see FIG. 8; step S5). The details of this processing are shown in the flowchart of FIG.

In the camera 13 for the subject, the second operation unit 1
72 works. Then, the second calculation unit 172, the captured image obtained by capturing the subject 30 with the subject camera 13, the external parameters of the subject camera 13 obtained at the time of the capture, and the capture target stereoscopic chart at the time of the capture. And the information for identifying the are read from the memory 169 (step S51).

Further, the second arithmetic unit 172 has the memory 169.
And the relative position and relative attitude of the photographing target stereo chart with respect to the basic stereo chart are read out from the relative positional relationships of the stereo charts 2 obtained in advance by the first calculation unit 171 (step S2) (step S52). At this time, the second calculation unit 172 reads out the relative position and the relative attitude of the photographing target three-dimensional chart with respect to the basic three-dimensional chart, based on the information specifying the photographing target three-dimensional chart at the time of photographing, which is obtained in step S51.

The external parameters of the subject camera 13 obtained at the time of photographing, which are obtained in step S51, are the parameters shown in the equation (4). Further, the relative position and the relative attitude of the photographing target three-dimensional chart with respect to the basic three-dimensional chart obtained in step S52 are, in other words, the respective parameters shown in the above formulas 7 or 8. Therefore, if the calculation is performed based on these parameters, the coordinate value of the subject 30 obtained from the captured image when the subject camera 13 captures the subject 30 is represented by the coordinate value expressed in the reference coordinate system of the basic stereo chart. The coordinates can be converted to. That is, the coordinate conversion formula of Equation 4 and the coordinate conversion formula of Equation 7 or 8 can be obtained to obtain the coordinate value of the subject 30 in the reference coordinate system (world coordinate system). The formula can be obtained.

Then, the second calculation section 172 obtains the above-mentioned synthetic conversion formula based on the relative position and relative attitude of the object stereoscopic chart with respect to the basic stereo chart and the external parameters of the subject camera 13 to obtain a captured image. Three-dimensional information of the included subject image is generated (step S53).

Then, the second operation unit 172 uses the memory 169.
It is checked whether or not there is another photographed image in the table, and if there is, a process of steps S51 to S53 is repeated (step S5).
4). By repeating this, the coordinate values in the reference coordinate system (world coordinate system) can be obtained from all the images obtained by dividing the periphery of the subject 30 and generating three-dimensional information about the entire circumference of the subject image. It will be possible.

When the three-dimensional information about the entire circumference of the subject 30 is finally generated, the three-dimensional information is output or saved and all the processes are completed (step S5).
5).

In the above operation process, the order of each procedure may be slightly changed. For example, the process (step S2) of calculating the relative positional relationship between the plurality of three-dimensional charts 2 may be performed after the process of step S3 or the process of step S4.

As described above, in the three-dimensional information generation system 1 of this embodiment, the surroundings of the subject 30 are
A plurality of three-dimensional charts 2 having a known structure are arranged. Then, the first calculation unit 171 causes the plurality of three-dimensional charts 2 to be based on the respective three-dimensional charts included in the image captured by the movable camera 11 in advance.
The second computing unit 172 is configured to determine the relative positional relationship between the movable camera 11 and the photographing position and the photographing posture at which the movable camera 11 photographed the photographing target stereo chart, and the relative relationship between the movable camera 11 and the subject camera 13. The three-dimensional information of the subject is generated from the captured image obtained from the subject camera 13 based on the position and the relative attitude and the relative positional relationship between the capturing target stereo chart and the other stereo chart 2. It is configured.

For this reason, the movable range of the camera system 10 can be greatly expanded, and a three-dimensional image model can be constructed with high measurement accuracy. That is, when the size of the subject 30 is large, the number of stereo charts arranged around the subject 30 may be increased according to the size of the subject 30. Even if the number of three-dimensional charts increases, the relative positional relationship between the three-dimensional charts is required with high accuracy, so that it is possible to generate highly accurate three-dimensional information while expanding the movable range of the camera system 10. It becomes.

Further, when obtaining the relative positional relationship of the plurality of three-dimensional charts 2 arranged around the subject 30, at least two three-dimensional charts 2 should be captured within the angle of view of the movable camera 11. If so, the relative positional relationship between at least two three-dimensional charts 2 can be easily obtained from the image obtained by photographing, which is efficient.

When obtaining the relative positional relationship of the plurality of three-dimensional charts 2 arranged around the subject 30, the movable camera 11 is used while changing the photographing direction (that is, the photographing posture) of the movable camera 11. You may make it image | photograph several stereo charts 2. Even if such an image is taken, each image obtained from the movable camera 11,
It is possible to appropriately obtain the relative positional relationship between the plurality of three-dimensional charts 2 based on the shooting direction (shooting posture) of the movable camera 11 when shooting each image. Further, in the case of a shooting mode in which at least two stereo charts 2 are accommodated within the angle of view of the movable camera 11 as described above, the arrangement of the plurality of stereo charts 2 is limited by the angle of view of the movable camera 11. However, in the case of a shooting mode in which the movable camera 11 shoots a plurality of three-dimensional charts 2 while changing the shooting direction of the movable camera 11, the arrangement of the plurality of three-dimensional charts 2 is changed. The degree of freedom in the arrangement of the three-dimensional chart 2 is increased without being restricted by the corners.

<7. Modifications> Although the embodiments of the present invention have been described above, the present invention is not limited to the above description.

For example, in the above description, the movable camera 11 is used when obtaining the relative positional relationship between the plurality of three-dimensional charts 2.
In the above, the mode in which a plurality of three-dimensional charts 2 are photographed has been described, but a plurality of three-dimensional charts 2 may be photographed by the subject camera 13. In addition, the movable camera 11
It is also possible that both the camera and the subject camera 13 shoot different stereo charts 2 and obtain the relative positional relationship of the respective stereo charts from the images respectively obtained.

Further, in order to obtain the relative positional relationship of the plurality of three-dimensional charts 2, the movable camera 11 is fixed to a certain three-dimensional chart 2 and the movable camera 11 photographs another adjacent three-dimensional chart 2. You may do it. In this case, it is necessary that the position and orientation of the movable camera 11 when the movable camera 11 is fixed to the three-dimensional chart 2 be set in a predetermined state and be known.

Further, in the above description, an example in which the chart patterns CP of different colors are formed on each three-dimensional chart in order to distinguish each three-dimensional chart from other three-dimensional charts has been described. It is also possible to distinguish each three-dimensional chart by changing each chart or forming different character patterns or the like on each three-dimensional chart and discriminating them.

In the above description, an example in which the arithmetic unit 170 including the first arithmetic unit 171 and the second arithmetic unit 172 is provided in the subject camera 13 has been described. However, the present invention is not limited to this, and the movable unit is movable. The first calculation unit 171 may be arranged in the expression camera 11, or all or part of the functions of the calculation unit 170 may be provided in the computer 15. When the computer 15 is provided with the function as the calculation unit 170, all of the information obtained by the movable camera 11 and the information obtained by the subject camera 13 are given to the computer 15. And the arithmetic unit 1
The calculation program that realizes the functions of 70 is installed in the computer 15 and executed.

Furthermore, in the above description, an example in which a plurality of three-dimensional charts 2 are suspended above the subject 30 has been described, but the present invention is not limited to this. FIG. 13 is a diagram showing a modified example regarding the arrangement of a plurality of three-dimensional charts. As shown in FIG. 13, a plurality of three-dimensional charts 2
(2c to 2e) may be arranged on the floor surface around the subject 30. Even in this case, there is no change in the above-mentioned calculation contents.

[0151]

As described above, according to each of the first to ninth aspects of the present invention, since the relative positional relationship of a plurality of posture detection reference objects arranged around the subject is obtained, If one of the reference objects for posture detection is taken at the same time when taking the image, the subject can be appropriately measured. Therefore, it is possible to secure a wide movable range around the subject of the measuring device, and it is possible to measure the subject with high measurement accuracy.

In particular, according to the second aspect of the invention, the relative positional relationship of the plurality of posture detecting reference objects arranged around the subject can be easily obtained, and the calculation efficiency is also improved.

According to the third aspect of the present invention, it is possible to obtain the relative positional relationship of the plurality of posture detecting reference objects arranged around the subject by utilizing the characteristics of the reference object photographing camera. This increases the degree of freedom in arranging a plurality of 3D charts.

According to the fourth aspect of the invention, since the camera for photographing the reference object is variably fixed in the photographing direction with respect to the measuring device, any reference for posture detection arranged around the subject. The object can be measured, and the relative positional relationship between the plurality of attitude detection reference objects can be appropriately obtained.

According to the fifth aspect of the present invention, the shooting direction of the reference object shooting camera is controlled so that the reference object shooting camera shoots at least one attitude detection reference object. The degree of freedom in arranging the measuring device around the subject is increased.

Further, according to the invention of claim 6,
The measuring device is a device for measuring the three-dimensional shape of the subject,
Since the three-dimensional shapes obtained by a plurality of measurements are combined based on the postures at the time of each measurement, the three-dimensional shape relating to the periphery of the subject is accurately measured by the measuring device.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration example of a three-dimensional information generation system according to the present invention.

FIG. 2 is a side view of a three-dimensional chart.

FIG. 3 is a front view of a movable camera.

FIG. 4 is a block diagram showing internal functions of a movable camera.

FIG. 5 is a block diagram showing an internal configuration of a camera for photographing an object.

FIG. 6 is a diagram showing the concept of coordinate conversion in a three-dimensional information generation system.

FIG. 7 is a diagram showing a concept of coordinate conversion for converting a chart coordinate system into a reference coordinate system.

FIG. 8 is a flowchart showing an overall operation process of the three-dimensional information generation system.

FIG. 9 is a flowchart showing a part of processing in an operation process of the three-dimensional information generation system.

FIG. 10 is a flowchart showing a part of processing in an operation process of the three-dimensional information generation system.

FIG. 11 is a flowchart showing a part of processing in an operation process of the three-dimensional information generation system.

FIG. 12 is a flowchart showing a part of processing in an operation process of the three-dimensional information generation system.

FIG. 13 is a diagram showing a modified example regarding the arrangement of a plurality of three-dimensional charts.

[Explanation of Codes] 1 3D information generation system 2 (2a to 2e) 3D chart (reference object for attitude detection) 10 camera system (3D information generation device, measurement device) 11 movable camera (camera for reference object photography) 13 Camera for Shooting Subject (Subject Camera) 15 Computer 30 Subject 113 Posture Change Device 160 Control Unit (Control Unit) 170 Calculation Unit 171 First Calculation Unit (First Calculation Unit) 172 Second Calculation Unit (Second Calculation Unit) ) G1 angle of view Xc, Xm, Xmo, Xo, Xcb, Xxa coordinate system

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 5/225 H04N 5/225 Z 5L096 13/00 13/00 (72) Inventor Kunimitsu Sakakibara Osaka City, Osaka Prefecture Chuo-ku, Azuchi-chome No. 3 No. 13 Osaka International building Minolta Co., Ltd. in the F-term (reference) 2F065 AA04 AA20 AA37 AA53 BB05 BB29 FF04 FF28 FF61 FF65 FF67 GG08 JJ03 JJ26 LL06 LL30 NN20 PP05 QQ03 QQ24 QQ31 QQ32 RR10 5B047 AA07 BA02 BB04 BC16 BC23 CA14 CB09 CB22 DC07 DC09 5B057 AA20 BA11 CA12 CB13 CD14 CH01 CH11 DA07 DA16 DB03 DC08 5C022 AA13 AB62 AC27 5C061 AA29 AB04 AB24 5L096 AA09 BA20 CA03 FA67 FA69 HA05 LA05 LA11

Claims (9)

[Claims] 1. A measuring device for measuring a subject in the state where a plurality of reference objects for posture detection having a known shape are arranged around the subject, the camera for measuring a reference object, and the plurality of postures. First calculation means for obtaining a relative positional relationship between the plurality of reference objects for posture detection based on an image obtained by previously photographing the reference object for detection by the camera for photographing the reference object, and in synchronization with the measurement operation of the measuring device. Control means for controlling the reference object photographing camera to perform photographing, and the posture of the measuring device based on the image of the posture detection reference object at the time of measurement photographed by the reference object photographing camera. And a second calculation means for calculating. 2. The measuring device according to claim 1, wherein the first computing unit is photographed so that at least two of the posture detecting reference objects are within the angle of view of the reference object photographing camera. A measuring apparatus, wherein a relative positional relationship between the at least two attitude detection reference objects is obtained based on each attitude detection reference object included in the image. 3. The measuring device according to claim 1, wherein the first computing unit changes the posture of the reference object photographing camera, and the plurality of posture detection reference objects are changed by the reference object photographing camera. Is obtained in advance, and the relative position relationship of the plurality of reference objects for posture detection is obtained based on the posture of the camera for photographing the reference object when the respective images are photographed. Characteristic measuring device. 4. The measuring apparatus according to claim 1, wherein the reference object photographing camera has a photographing direction variably fixed with respect to the measuring apparatus. . 5. The measuring device according to claim 4, wherein the control unit controls the reference object photographing camera so that the reference object photographing camera photographs at least one of the posture detecting reference objects. A measuring apparatus having a photographing direction control means for controlling a photographing direction. 6. The measuring device according to claim 1, wherein the measuring device is a device for measuring a three-dimensional shape of a subject, and each of the three-dimensional shapes obtained by a plurality of measurements is different from each other. A measuring device characterized by being synthesized based on a posture at the time of measurement. 7. A measuring device for measuring the subject in a state in which a plurality of reference objects for posture detection having a known shape are arranged around the subject, the camera being a subject photographing camera for photographing the subject, A relative object of the plurality of attitude detection reference objects based on an image obtained by previously photographing the reference object shooting camera and the plurality of attitude detection reference objects with the reference object shooting camera and / or the subject shooting camera. First calculation means for obtaining a positional relationship, control means for controlling the photographing of the reference object photographing camera in synchronization with the photographing operation of the subject photographing camera, and photographing by the reference object photographing camera A second calculation means for calculating the attitude of the measuring device based on the image of the reference object for attitude detection at the time of measurement, A measuring apparatus characterized in that a predetermined measurement calculation is performed based on an image and a posture at that time. 8. A measuring method for measuring the subject in a state in which a plurality of reference objects for posture detection having a known shape are arranged around the subject, wherein the reference objects for posture detection are pre-referenced. The step of obtaining the relative positional relationship of the plurality of reference objects for posture detection based on the image captured by the object shooting camera, and the plurality of reference objects for posture detection in synchronization with the measurement operation of the measuring device. A step of photographing at least one of them with the reference object photographing camera; and a posture of the measuring device based on an image of the posture detecting reference object at the time of measurement photographed by the reference object photographing camera. And a step of calculating. 9. A measuring method for measuring the subject in a state where a plurality of posture detecting reference objects having a known shape are arranged around the subject, wherein the plurality of posture detecting reference objects are pre-referenced. A step of obtaining a relative positional relationship between the plurality of reference objects for posture detection based on the image taken by the object photographing camera and / or the subject photographing camera; and a photographing operation of the subject photographing camera. A step of photographing at least one of the plurality of posture detection reference objects with the reference object photographing camera, and an image of the posture detection reference object at the time of measurement photographed by the reference object photographing camera And a step of calculating a posture of the measuring device based on the above, and a step of performing a predetermined measurement calculation based on the image photographed by the subject photographing camera and the posture at that time. Measurement method according to claim.