JPH1078304A - Method and device for image pickup - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a device for image pickup which simultaneously, very accurately, and simply determine three-dimensional position and direction of an image pickup unit and then easily combine images. SOLUTION: LED 102, 103, and 109 are in advance attached to three points (A, B, C) of an image pickup unit 101 as marks and blinked by an emission timing controller 104 on respective different timings. Two two-dimensional sensors 106 detect each mark on its corresponding timing. A computing mechanism 107 calculates three-dimensional positions of the three points based on the two images created by the two two-dimensional sensors, and in the basis of the results calculates simultaneously the three-dimensional position, direction, and rotating angle in relation to the azimuth axis of the image pickup unit 101 during image pickup. A recording mechanism 108 records the three- dimensional obtained position, direction, and rotating angle corresponding to an image photographed under the situation.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION The present invention relates to a three-dimensional
The present invention relates to a method and an apparatus for performing imaging while automatically obtaining a dimensional position and a direction simultaneously.

[0002]

2. Description of the Related Art When attempting to combine a real video image captured by a TV camera with a computer graphics image or the like, in addition to the lens focal length of the TV camera at the time of the imaging, the TV camera itself is used. It is necessary to record information on the three-dimensional position and orientation at the same time.

When combining two types of images, if the lens focal length, the three-dimensional position, and the azimuth of the camera at the time of capturing the images are different, it is fundamentally necessary to synthesize both images accurately. Is impossible, and the synthesized image is completely uncomfortable.

One advantage of computer graphics is that a three-dimensional model can be used to generate an image viewed at any three-dimensional position and orientation at any lens focal length. Conversely, these pieces of information are indispensable for generating a computer graphics image, and unless these are correctly provided, a correct image cannot be generated.

[0005] Therefore, in order to perform an accurate image synthesis of a real image and a computer graphics image, data of a lens focus, a three-dimensional position, and an azimuth of a TV camera at the time of imaging an image are accurately formed in correspondence with the image. It may be recorded, and a computer graphics image matched to the actual video may be generated by using the recorded image.

However, a TV camera conventionally used as a typical example of an imaging unit simply captures a two-dimensional image, and information on the three-dimensional position and orientation of the camera at the time of imaging is directly obtained. Not recorded. Therefore, T
It is necessary to measure the position and orientation of the TV camera in a three-dimensional space using some auxiliary means together with the V camera.

Conventionally, methods for obtaining the position and orientation in a three-dimensional space include (1) mechanical three-dimensional measurement, (2) gyroscope method, (3) GPS (global positioning system) method, and (4) ceiling. There is a grid pattern detection method. Hereinafter, each conventional system will be described.

(1) Mechanical three-dimensional measurement This is a method of obtaining a position in a three-dimensional space by mechanical interference. A TV camera attached to the tip of the arm is set as a measurement point, and its three-dimensional position and orientation are determined by the length of the arm,
It is determined by a combination of potentiometers, strain gauges, etc. installed for each joint. This method has the advantage of relatively high certainty.

(2) Gyroscope system This is a system in which a gyroscope is attached to a TV camera to perform imaging while measuring three-dimensional positions and orientations.

(3) GPS method This is a method for detecting a two-dimensional or three-dimensional position by receiving radio waves from GPS satellites, and is widely used in car navigation systems.

(4) Ceiling grid pattern detection method A grid pattern prepared in advance on the ceiling and used for the purpose of alignment (drawn) is mounted on an imaging TV camera with the ceiling facing slightly different directions. Three small T-cameras for positioning
In this method, the orientation of the imaging TV camera is obtained based on the ceiling grid pattern captured by the V camera.

[0012]

However, each of the conventional methods for obtaining the position and orientation in the three-dimensional space has the following problems.

(1) Mechanical three-dimensional measurement This method has a disadvantage that it takes time to mechanically move a measurement point. Further, there is a problem that not only the mechanism becomes very large but also the usable space is limited to the movable range of the arm.
In order to perform the shooting with a small mechanism, it is necessary not only to use a small TV camera, but also to shorten the arm itself, and it is only possible to realize shooting in a mini-set such as a miniature garden.

(2) Gyroscope method In this method, when a small gyroscope is used, there is a problem that it is difficult to measure the three-dimensional position and orientation with sufficient accuracy. On the other hand, if an attempt is made to use a high-precision gyroscope, the size of the apparatus becomes considerably large, and moreover, the initial setting of the gyroscope itself takes time and effort.

(3) GPS method This method has a problem that the detection accuracy is usually about 5 m, and it is difficult to perform three-dimensional measurement with higher accuracy than this. Further, there is a problem that the antenna is usually used outdoors because of receiving radio waves from a satellite, and is not suitable for use in a building.

(4) Ceiling grid pattern detection method This method is based on the premise that the imaging TV camera is used in a nearly horizontal state (that is, three positioning TV cameras face the ceiling). There is a problem that the direction of the TV camera is limited. In addition, there is a problem that the use place of the imaging TV camera is limited to a place having a ceiling pattern.

As described above, according to the conventional method of obtaining the position and orientation, the three-dimensional position and orientation of the imaging camera used in any direction can be simultaneously and simply and highly accurately and easily captured. There has been a problem that the range in which movement is possible cannot be determined without limitation.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems of the conventional system, and an object of the present invention is to simultaneously and easily and three-dimensionally position an image pickup unit with high accuracy. An object of the present invention is to provide an imaging method and apparatus capable of detecting the image.

[0019]

In order to achieve the above object, the present invention provides an imaging method and apparatus according to each of the following systems.

(First invention) A method of detecting a mark on an imaging unit with a surrounding sensor (Second invention) A method of detecting a mark on an imaging unit by a surrounding azimuth detecting mechanism (Third invention) Imaging A method of detecting a mark on the unit by a surrounding distance detection mechanism (Fourth invention) A method of detecting a surrounding mark by an azimuth detection mechanism on the imaging unit (Fifth invention) A sensor of the surrounding mark on the imaging unit (Sixth invention) Method of detecting surrounding landmarks by a distance detection mechanism on the imaging unit Hereinafter, each of the above means will be described.

<< First Invention >> In the imaging method according to the first invention of the present invention, a plurality of marks are previously attached to the image pickup unit, the marks are detected from a plurality of external directions, and the mark is obtained from the result of the detection. The three-dimensional position of the mark, or the three-dimensional position and orientation of the imaging unit calculated based on the three-dimensional position of the mark, or the three-dimensional position, orientation and rotation angle of the imaging unit, Is recorded or output in association with.

According to a first aspect of the present invention, there is provided an image pickup unit having a plurality of marks, a plurality of sensors for detecting the marks from outside, and the image pickup unit based on a detection result of the sensors. Calculating means for calculating and calculating the three-dimensional position of the unit, or the three-dimensional position and orientation of the imaging unit, or the three-dimensional position, orientation and rotation angle of the imaging unit; and the calculating means connected to the calculating means Means for recording or outputting the three-dimensional position of the imaging unit, or the three-dimensional position and azimuth of the imaging unit or the three-dimensional position, azimuth and rotation angle of the imaging unit obtained in the above step, in association with the captured image of the imaging unit. And the following.

In the first aspect of the present invention, a plurality of marks, such as marks and lamps, attached to the image pickup unit are sensed by sensors or the like from two or more directions, and at least two three-dimensional positions of the image pickup unit being picked up. By calculating the three-dimensional position and orientation in real time, and recording or outputting the image in association with the captured image of the imaging unit, the three-dimensional position and orientation of the imaging unit used in any direction can be simultaneously and accurately determined. In addition, it is easily obtained and recorded or output, thereby facilitating the subsequent generation of computer graphics and image synthesis.

<< Second Invention >> In an imaging method according to a second invention of the present invention, a plurality of marks are previously attached to an image pickup unit, and a mark direction from the plurality of external positions to the mark is detected. The three-dimensional position of the mark obtained from the result of, or the three-dimensional position and orientation of the imaging unit calculated based on the three-dimensional position of the mark or the three-dimensional position and orientation and rotation angle of the imaging unit, It is characterized in that it is recorded or output in association with the image taken by the image pickup unit.

An image pickup apparatus according to a second aspect of the present invention includes an image pickup unit having a plurality of marks, a plurality of mark azimuth detecting mechanisms for detecting the mark azimuths of the marks as viewed from a plurality of external positions. Calculating the three-dimensional position of the imaging unit, or the three-dimensional position and azimuth of the imaging unit, or the three-dimensional position, azimuth and rotation angle of the imaging unit based on the detection result of the landmark azimuth detecting mechanism. And a three-dimensional position and azimuth of the imaging unit or a three-dimensional position, azimuth and rotation angle of the imaging unit determined by the calculation unit connected to the calculation unit corresponding to the captured image of the imaging unit. And means for recording or outputting the information.

In the second invention, a plurality of marks are previously attached to the image pickup unit, and the mark orientation viewed from two or more positions outside the mark is grasped from the positions, and the three-dimensional position of the mark is determined in real time. Calculating and detecting the three-dimensional position, the imaging azimuth and the rotation angle of the imaging unit, and recording or outputting them in association with the photographed image of the imaging unit, so that the imaging unit used in any direction can be used. The three-dimensional position, azimuth, and rotation angle are easily and simultaneously obtained with high accuracy and easily, thereby facilitating the subsequent generation of computer graphics and image synthesis.
In the detection of the mark direction, the range of the space in which the image pickup unit can move is widened by movably tracking the mark.

<< Third Invention >> In an imaging method according to a third invention of the present invention, a plurality of marks are added to the imaging unit, and the plurality of marks are detected from a plurality of distance detection mechanisms installed around the imaging unit. 3D positions of the plurality of landmarks, or the three-dimensional position and orientation of the imaging unit, calculated based on the detected distances, respectively.
Alternatively, a three-dimensional position, an azimuth, and a rotation angle of the imaging unit are recorded or output in association with a captured image of the imaging unit.

An image pickup apparatus according to a third aspect of the present invention is an image pickup unit to which a plurality of marks are added, and a distance detecting mechanism installed around the image pickup unit for detecting a distance to a mark position on the image pickup unit. Calculating means for calculating the three-dimensional position of the mark, or the three-dimensional position and orientation of the imaging unit, or the three-dimensional position, orientation and rotation angle of the imaging unit based on the detected distance; 3 of the mark calculated by the calculation means connected to the means
Means for recording or outputting a three-dimensional position, or a three-dimensional position and orientation of the imaging unit, or a three-dimensional position, orientation and rotation angle of the imaging unit in association with a captured image of the imaging unit. Features.

In the third invention, a plurality of mark light sources are attached to the image pickup unit in advance, and the light emitted from the plurality of mark light sources is detected by a plurality of sensors installed in the vicinity to detect the distance between the light sources. The three-dimensional position, imaging azimuth, and rotation angle of the imaging unit are simultaneously detected by calculating the three-dimensional position based on the calculated three-dimensional position.

<< Fourth Invention >> In an imaging method according to a fourth invention of the present invention, a plurality of landmarks are prepared in advance around an imaging unit, and a landmark azimuth detecting mechanism for detecting the azimuth to the landmark is provided. A plurality of the mark orientation detection mechanisms, which are attached to the imaging unit, detect a mark orientation to the mark as viewed from the position of the mark orientation detection mechanism on the imaging unit, and calculate based on the detected mark orientation. Three-dimensional position or three-dimensional position and orientation of the imaging unit,
Alternatively, a three-dimensional position, an azimuth, and a rotation angle of the imaging unit are recorded or output in association with a captured image of the imaging unit.

An imaging apparatus according to a fourth aspect of the present invention includes a plurality of landmarks prepared in advance, a plurality of landmark direction detecting mechanisms for detecting the landmark direction of the landmarks, and the plurality of landmark direction detecting mechanisms. The attached imaging unit, the three-dimensional position of the mark azimuth detection mechanism based on the mark azimuth viewed from the mark azimuth detection mechanism detected by the mark azimuth detection mechanism, or the three-dimensional position and orientation of the imaging unit,
A calculating means for calculating a three-dimensional position, a direction and a rotation angle of the imaging unit; a calculated three-dimensional position of the landmark direction detecting mechanism; or a three-dimensional position and a direction of the imaging unit; Means for recording or outputting the dimensional position, the azimuth, and the rotation angle in association with the captured image of the imaging unit.

According to the fourth aspect of the present invention, a mark direction detecting mechanism is mounted on the image pickup unit in advance, and by using the mechanism, a direction to a mark placed at two or more known positions around the mark is detected. The three-dimensional position of the mark azimuth detecting mechanism is calculated, and the three-dimensional position of the imaging unit is calculated based on this.
The dimensional position, the imaging azimuth, and the rotation angle are simultaneously detected in correspondence with the video.

<< Fifth Invention >> In an imaging method according to a fifth invention of the present invention, a plurality of sensors are added to an imaging unit, a plurality of marks are prepared in advance around the sensor, and the marks are captured by the sensors. The three-dimensional position of the sensor with respect to the landmark, or the three-dimensional position and azimuth of the imaging unit, or the three-dimensional position, azimuth, and rotation angle of the imaging unit, calculated from the result of capturing the landmark, are captured by the imaging unit. It is characterized by recording or outputting in association with a video.

According to a fifth aspect of the present invention, there is provided an image pickup unit to which a plurality of sensors are added, a plurality of marks prepared around the sensor, and the sensor for the marks based on a result obtained by capturing the marks with the sensors. Calculating means for calculating the three-dimensional position of, or the three-dimensional position and orientation of the imaging unit, or the three-dimensional position, orientation and rotation angle of the imaging unit; and the calculated three-dimensional position of the sensor or the imaging unit Means for recording or outputting the three-dimensional position and azimuth of the imaging unit or the three-dimensional position, azimuth and rotation angle of the imaging unit in association with the photographed video of the imaging unit.

In the fifth invention, a plurality of marks, such as marks and lamps fixed around, are captured by a plurality of sensors fixed to an image pickup unit, and an image is picked up from a three-dimensional position of the plurality of sensors. The three-dimensional position, azimuth, and rotation angle of the unit are calculated in real time, and recorded or output in association with the captured image of the imaging unit, so that the three-dimensional position, azimuth, The rotation angle is simultaneously detected with high accuracy and easily and simultaneously, and recorded or output.

<< Sixth Invention >> In an imaging method according to a sixth invention of the present invention, a plurality of distance detection mechanisms for detecting a distance to a landmark are added to the imaging unit, and a plurality of landmarks are previously arranged around the imaging unit. The distance to the mark is detected from the position of the distance detecting mechanism, and the three-dimensional position of the mark, or the three-dimensional position and orientation of the imaging unit, or the three-dimensional position of the imaging unit, calculated from the result of the detection. And the azimuth and the rotation angle are recorded or output in association with the image captured by the imaging unit.

An imaging apparatus according to a sixth aspect of the present invention includes an imaging unit to which a plurality of distance detection mechanisms for detecting a distance to a mark are added, a plurality of marks installed around the imaging unit, and the distance detection mechanism having the detected distance detection mechanism. Calculating means for calculating the three-dimensional position of the mark, or the three-dimensional position and orientation of the imaging unit, or the three-dimensional position, orientation and rotation angle of the imaging unit from the distance from the position to the landmark; Means for recording or outputting the three-dimensional position of the mark, or the three-dimensional position and orientation of the imaging unit, or the three-dimensional position, orientation and rotation angle of the imaging unit in association with the captured image of the imaging unit. It is characterized by having.

According to the sixth aspect of the present invention, a mark direction detecting mechanism is installed at a plurality of surrounding known positions and is used to detect directions to a plurality of marks mounted on the image pickup unit in advance. The three-dimensional position is calculated, and based on the calculation, the three-dimensional position, the imaging direction, and the rotation angle of the imaging unit are simultaneously detected with high accuracy and easily in accordance with the image, and recorded or output.

In the image pickup apparatus according to the present invention,
Employing a plurality of illuminants as a plurality of markers, and including illuminant control means for blinking the plurality of illuminants at individual emission timings or individual frequencies, or as a plurality of markers, a plurality of illuminants having different emission colors Using a sensor that detects the mark or a distance detection mechanism that detects the distance to the mark or a mark direction detection mechanism that detects the direction,
It is practically preferable to have a color identification unit such as a color imaging unit, since each of the plurality of marks can be easily distinguished.

In the above-described imaging apparatus according to the present invention,
Newly provided detection means connected to an imaging unit and detecting one or a plurality of imaging parameters such as a lens focal length of the imaging unit, a lens aperture value, an optical filter used, a focusing position, and a shutter speed,
The recording or outputting means records or outputs the detected imaging parameter in association with the captured video of the imaging unit, which is troublesome when combining the captured video of the imaging unit and a computer graphics image. This is preferable in that the reduction is significant.

[0041]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

In each of the following embodiments, for simplicity of explanation, it is assumed that not a zoom lens with a variable focal length but a lens with a fixed focal length is used as a lens of a TV camera as an imaging unit. Will be explained.

<< Embodiment 1 >> The first embodiment of the present invention is an embodiment of the first invention of the present invention which uses a method of detecting a mark on an image pickup unit with a surrounding sensor.

FIG. 1 is a diagram for explaining this embodiment. Reference numeral 101 denotes a TV camera as an image pickup unit;
2, 103 and 109 are LEDs as markers, 10
4 is a light emission timing controller, 105 and 10
6 is a TV camera as a two-dimensional sensor, 107 is a computer as a calculation mechanism, and 108 is a magnetic video disk as a recording mechanism. Here, the imaging parameters are stored in the magnetic video disk 108 in advance. Further, the marks 102 and 103 are fixed to both ends on the TV camera 101 as an image pickup unit.

In order to operate this, first, LE as a mark is controlled by the control of the light emission timing controller 104.
D102, 103, and 109 are turned on, and this light is received by TV cameras 105 and 106 as two-dimensional sensors,
By processing the information by the calculation mechanism 107, the three-dimensional positions A, B, and C of the LEDs 102, 103, and 109 as the marks are obtained, and the azimuth and the rotation angle are also obtained therefrom.

FIG. 2 is a conceptual diagram showing the principle of a method of obtaining a three-dimensional position based on two images taken from the upper surface and the side surface. 3D position information. Note that, here, for simplification of the description, a drawing assuming parallel projection is illustrated. In an actual case, perspective projection is used, but the calculation method itself has already been established in the field of stereo photograph measurement and the like. In principle, it suffices to simply find the y and z of the mark at the corresponding x position in the top image 201 and the side image 202 respectively. As a result, three-dimensional position information 203 of the mark is obtained.

The landmarks 102, 1 in the obtained three-dimensional space
The positions of 02 and 109 are mark 102: point A (Ax, Ay, Az) mark 103: point B (Bx, By, Bz) mark 109: point C (Cx, Cy, Cz). Thus, the azimuth V of the imaging unit 101 is determined by the following equation.

V = (Vx, Vy, Vz) = (Ax-B
(x, Ay-By, Az-Bz) The point C also determines the rotation angle of the TV camera 101 as an imaging unit with the line segment AB as an axis. That is, assuming that the normal position of the point C with respect to the line segment AB is C ', the apex angle of an isosceles triangle having a point on the axis of the line segment AB as a vertex and CC' as a base is the rotation angle.

Although omitted in the above description, in practice, some measures need to be taken to prevent the LEDs 102, 103 and 109 as markers from being confused and detected.

For example, LEDs 102 and 10 as markers
3 and 109 are set in the light emission timing controller 104 so as to emit light intermittently at another timing or another frequency. This makes it possible to distinguish and detect the three.

As another method, the marks 102, 1
The two-dimensional sensors 104 and 105 are color T
There is also a method of distinguishing and detecting three persons by using a V camera.

As described above, the three-dimensional position information of the imaging unit 101 (the three-dimensional coordinate values (Ax, Ay, Bz) of the mark)
And (Bx, By, Bz)) and three-dimensional orientation information (V
x, Vy, Vz) and the rotation angle are obtained at the same time. Therefore, while performing measurement in this way, imaging is performed by the imaging unit 101, and this video information is associated with information on the three-dimensional position, azimuth, and rotation angle of the imaging unit 101 obtained by the above method. Recording mechanism 10
8 is stored. The image information stored in the recording mechanism 108, the three-dimensional position and orientation of the imaging unit 101, the rotation angle, and the information about the imaging parameters such as the lens focal length of the imaging unit 1 stored in advance are stored in a computer graphics image generation later. Computer graphics image and the captured video are synthesized.

<< Embodiment 2 >> The second embodiment of the present invention is also an embodiment of the first invention of the present invention using a method of detecting a mark on an image pickup unit with a surrounding sensor.

FIG. 3 is a diagram for explaining this embodiment. Reference numeral 301 denotes a photographing parameter detecting unit.
The other elements denoted by the same reference numerals are the same as those in the first embodiment of FIG. The difference from FIG. 1 is that this imaging parameter detection unit 301 is added.

In this embodiment, (a) the focal length of the photographing unit detected by the photographing parameter detection unit 301 when the lens is used, (b) actual aperture, (c) optical filter used, and (d) focal point At the same time, the image capturing parameters such as the alignment position, (e) shutter speed, etc.
, And are recorded together in a form corresponding to the recording information in the first embodiment of FIG. According to the present embodiment, since all information at the time of imaging is recorded simultaneously in real time in association with video information, it is easy to respond to changes in imaging parameters during imaging, and later An advantage is obtained in that the labor required for synthesizing the computer graphics image and its image can be greatly reduced.

In the first and second embodiments, three marks A, B, and C are used as marks.
By using one or more landmarks, three-dimensional position and orientation measurement can be performed with higher accuracy by using the information of landmarks that have been reliably captured.

When only the three-dimensional position and azimuth of the image pickup unit are required and the rotation angle can be ignored, only the marks A and B are required, and the mark C can be omitted. When the imaging parameters are fixed or fixed, recording and outputting of the imaging parameters can be omitted.

In the above embodiment, the TV camera as a two-dimensional sensor is also a two-dimensional sensor.
6 is used, but if three or more are used and information can be collected from multiple directions at the same time, it is hidden behind the imaging unit 101 or behind the photographer in an actual use situation. Even if there is a mark that is not detected, the intended purpose can be achieved because the mark is surely detected by the TV camera as another two-dimensional sensor instead.

Further, in the first and second embodiments, the recording mechanism is omitted, and a computer or the like for synthesizing a real photographed image and a computer graphics image to a computer or the like is used. The position, the three-dimensional position of the imaging unit, the azimuth, the rotation angle, and the imaging parameters, if necessary, may be directly output in association with each other to perform necessary processing externally.

<< Embodiment 3 >> A third embodiment of the present invention is an embodiment of the second invention of the present invention using a method of detecting a mark on an imaging unit by a surrounding azimuth detecting mechanism. is there.

FIG. 4 is a view for explaining a third embodiment of the present invention. Reference numeral 401 denotes a TV as an image pickup unit.
Cameras, 402 and 403 are LEDs as landmarks,
404 is a light emission timing controller, 405-A,
Reference numerals 405-B, 406-A and 406-B denote landmark azimuth detecting mechanisms, 407 denotes a computer as a calculating mechanism, and 408 denotes a magnetic video disk as a recording mechanism. Here, the marks 402 and 403 are fixed to both ends of the TV camera 401 as a photographing unit.

In order to operate this, first, the LE as a mark is controlled by the light emission timing controller 404.
D402 and 403 are turned on, and this light is sent to the landmark azimuth detecting mechanisms 405-A, 405-B, 406-A, and 40.
6-B, the mark azimuth detecting mechanism 40
5-A, 405-B, 406-A, and 406-B
From the position of the marker 402 and 403, respectively.
LED 402 and 4 as a mark by processing at 7
The three-dimensional positions A and B of No. 03 are obtained, and the imaging azimuth can also be obtained from this.

FIG. 5 shows an example of the configuration of the mark azimuth detecting mechanism 405-A.
Reference numeral 2 denotes a rotation mechanism having an angle detection function about two axes, 503 denotes a CCD sensor fixedly installed on the rotation mechanism 502, and 504.
Is a three-dimensional position of the mark direction detection mechanism 405-A. The same applies to the remaining three mark direction detection mechanisms 405-B, 406-A, and 406-B.

To operate this, first, the direction of the CCD sensor 503 is moved by operating the rotation mechanism 502 in accordance with a command from the controller 501, and the mark 402 attached on the image pickup unit 401 is moved to the CCD sensor 5.
Capture within the field of view of 03. Thereafter, the rotation mechanism 502 detects the amount of deviation between the center of the field of view of the CCD sensor 503 and the detection position of the mark 402, and minimizes the amount of deviation (that is, the rotation mechanism 502 so as to come to the center of the field of view of the CCD sensor 503).
Keep running. Then, a mark azimuth V1 in the direction of the mark 402 viewed from the three-dimensional position 504 of the mark azimuth detection mechanism 405 is obtained by the angle detection function around two axes incorporated in the rotation mechanism 502.

In the controller 501, the CCD sensor 50
It is determined at which position in the visual field of No. 3 (in which direction and at what distance from the center) the mark 402 is captured, and the rotating mechanism 502 is operated so that the mark 402 is always captured at the center.

The configuration and operation of the mark direction detecting mechanism 405-B are the same as those of the mark direction detecting mechanism 405-A, except that the same mark 402 is captured from a three-dimensional position different from the three-dimensional position 504 of the mark direction detecting mechanism. Find V2.

The three-dimensional position of the mark 402 is obtained from the two mark directions V1 and V2 thus obtained.

Similarly, two more mark direction detecting mechanisms 4
The three-dimensional position of the mark 403 can also be obtained by using 06-A and 406-B.

The angle detecting mechanism incorporated in the rotating mechanism can be realized by using, for example, a rotary encoder widely used in the joints of industrial robot arms.

The positions of the landmarks 402 and 403 in the obtained three-dimensional space are as follows: landmark 402: point A = (Ax, Ay, Az) landmark 403: point B = (Bx, By, Bz). Thus, the azimuth V of the imaging unit 401 is determined by the following equation.

V = (Vx, Vy, Vz) = (Ax-B
x, Ay-By, Az-Bz) Although omitted in the above description, in practical use, some measures need to be taken to prevent the LEDs 402 and 403 from being confused and detected.

For example, the marks 402 and 403 are set to emit light intermittently at another timing or another frequency. As a result, the two can be detected separately.

As another method, marks 402 and 4
There is also a method in which a color TV camera is used as the two-dimensional sensor of the landmark azimuth detecting mechanisms 405 and 406 to distinguish between the two and use different ones.

As described above, 3 of the imaging unit 401
Dimensional position information (three-dimensional coordinate values (Ax, Ay, B
z) and (Bx, By, Bz)) and imaging direction information (Vx, Vy, Vz) are obtained at the same time. Therefore, while performing such measurement, the imaging unit 40
Then, the image information is stored in the recording mechanism 408 in correspondence with the three-dimensional position of the imaging unit 401 and the position of the imaging azimuth obtained by the above method.

As described above, since the mark azimuth detecting mechanism is movable and tracks the mark of the image pickup unit, the degree of freedom of movement of the image pickup unit is increased, and the range of the space in which the image pickup unit can move can be expanded. The three-dimensional position of the mark can be detected from an image obtained by detecting two or more fixed cameras or the like as a position sensor. In this case, the range of the space in which the imaging unit can move is determined by the position sensor. Is limited within the field of view of the camera, and becomes narrow.

<< Embodiment 4 >> The fourth embodiment of the present invention is also a second embodiment of the present invention using a method of detecting a mark on an image pickup unit by a surrounding azimuth detecting mechanism. is there.

FIG. 6 is a diagram for explaining a fourth embodiment of the present invention. Reference numeral 601 denotes an imaging parameter detection unit. Other configurations of the present embodiment are the same as those of FIG. The difference from FIG. 4 is that this imaging parameter detection unit 601 is added. In the present embodiment, the imaging parameters of the imaging unit detected by the imaging parameter detection unit 601 such as: a focal length at the time of imaging of the used lens, an actual aperture, an used optical filter, a focusing position, and a shutter speed are simultaneously recorded. To the mechanism 408,
The information is recorded together in a form corresponding to the recording information in the third embodiment of FIG. As a result, all information at the time of imaging is recorded at the same time, and it is possible to greatly reduce the labor required when the computer graphics image and its video are later synthesized.

In the third and fourth embodiments, only two marks A and B are used as marks.
By using the two landmarks A, B, and C and adding the landmark azimuth detecting mechanism to detect all of them, the rotation angle about the imaging azimuth axis can be obtained in addition to the three-dimensional position and the imaging azimuth of the imaging unit. It is possible to ask.

Further, if two or more marks and a large number of mark orientation detection mechanisms are used, even if there is a mark that is hidden behind the image pickup unit 401 in an actual use situation and cannot be detected, the mark can be reliably detected. By using the information of the landmarks, other landmarks can be reliably detected instead, so that the initial purpose can be achieved, and more accurate measurement of the three-dimensional position, the imaging azimuth, and the rotation angle can be performed. It becomes possible.

Further, also in the first and second embodiments, the recording mechanism is omitted and a computer or the like for synthesizing a real photographed image and a computer graphics image to a computer or the like is used. The position, the three-dimensional position of the imaging unit, the azimuth, the rotation angle, and the imaging parameters, if necessary, may be directly output in association with each other to perform necessary processing externally.

<< Embodiment 5 >> A fifth embodiment of the present invention is a third embodiment of the present invention using a method of detecting a mark on an image pickup unit by a surrounding distance detecting mechanism. is there.

FIG. 7 is a view for explaining this embodiment. Reference numeral 701 denotes a TV camera as an image pickup unit;
2 and 703 are mark light sources, 704 is a light emission control mechanism, 705A, 705B, 705C, 706A, 706
B and 706C are distance detection mechanisms, 707 is a computer as a calculation mechanism, and 708 is a magnetic video disk as a recording mechanism. Here, the mark light source 7
02 and 703 each use a light source that radiates light uniformly to the surroundings (a light source that does not emit light only in a specific direction, or a light source that radiates light using a small diffusing lens). A light source that emits light is realized), and each is fixed to, for example, both ends of a TV camera 701 as an imaging unit.

To operate this, first, the mark light sources 702 and 703 are caused to emit light under the control of the light emission control mechanism 704, and this light is emitted to the distance detection mechanisms 705A and 705, respectively.
B, 705C, 706A, 706B, 706C,
After detecting the distance from the landmark light source to each of the distance detection mechanisms, the three-dimensional positions A and B of the landmark light sources 702 and 703 are processed by the calculation mechanism 707.
, And the imaging azimuth can also be determined from this.

The positions of the marks 702 and 703 in the obtained three-dimensional space are respectively set as mark 702: point A = (Ax, Ay, Az) mark 703: point B = (Bx, By, Bz). Thereby, the azimuth V of the imaging unit 701 is determined.

V = (Vx, Vy, Vz) = (Ax-B
x, Ay-By, Az-Bz) Although omitted in the above description, in practice, some measures need to be taken to prevent the two mark light sources 702 and 703 from being confused and detected. .

For example, the mark light sources 702 and 703 are set so as to emit light intermittently at a different timing or a different frequency so that they can be distinguished from each other. As a result, the two can be detected separately.

As another method, there is a method in which light sources having different colors (wavelengths) are used as the mark light sources 702 and 703 to distinguish the two from each other.

As described above, the image pickup unit 701-3
Dimensional position information (three-dimensional coordinate values (Ax, Ay, A
z) and (Bx, By, Bz)) and imaging direction information (Vx, Vy, Vz) are obtained at the same time. Therefore, while performing such measurement, the imaging unit 70
The image information is stored in the recording mechanism 708 in such a manner that the video information and the information on the three-dimensional position and the imaging azimuth of the imaging unit 701 obtained by the above method are associated with each other.

FIG. 8 is a diagram showing the operation of the distance detecting mechanism 705A, wherein 801 is an optical sensor, and 802 is a detecting circuit. The light emission control mechanism 704 sends a signal to the detection circuit 802 at the same time as causing the mark light source 702 to emit light. In order to secure this synchronization, the signal delay from the light emission control mechanism 704 to the optical sensor 801 and the detection circuit 802 are equalized.
Make the length of the cable connecting them equal. The optical sensor 801 detects the light emitted from the mark light source 702 and sends a signal to the detection circuit 802. The detection circuit 802 detects a time difference (delay time) T between signals transmitted from both the light emission control mechanism 704 and the detection circuit 802, and sends this to the calculation mechanism 707. This time difference (delay time) T reflects the physical distance (optical path length) between the mark light source 702 and the optical sensor 801, and the actual physical distance can be calculated from this.

The light emission control mechanism 704 sends the light sensor 8
01 and the distance detection mechanism 705A, even if the lengths of the cables are equal to each other, depending on various conditions.
Although it is assumed that the delays of the respective signals from the optical sensor 04 to the optical sensor 801 and the detection circuit 802 are not equal,
In that case, correction by calibration may be performed.

In the above example, for simplicity, the explanation has been made using a relatively simple waveform, but in practice, various methods for performing more accurate detection are used. Is possible. For example, the principle of phase difference detection, which is currently widely used as a stable technique for industrial distance measuring devices, can be applied to this. That is, the detection circuit 8
02, a phase difference detection circuit is used, and the light emission control mechanism 70
The light modulated by 4 is emitted from the mark light source 702,
After this is detected by the optical sensor 801, the phase difference between the two waveforms is detected by the detection circuit 802, and the distance information can be obtained therefrom.

In the above example, the distance detecting mechanism 705
Although only A has been described, another distance detection mechanism, namely,
Distance detection mechanisms 705B, 705C, 706A, 706
In the case of B and 706C, the situation regarding the configuration and operation is exactly the same. If the cable length for transmitting a signal from the light emission control mechanism 704A to each of the distance detection mechanisms is equal, the time and effort for correction can be reduced, and the mark light sources 702 and 702 can be omitted.
The distance from 703 to each distance detection mechanism can be accurately obtained.

<Embodiment 6> The sixth embodiment of the present invention is also a third embodiment of the present invention using a method of detecting the light source light on the imaging unit with a surrounding distance sensor. is there.

FIG. 9 is a diagram for explaining the present embodiment. Reference numeral 901 denotes an imaging parameter detection unit. FIG.
The difference from this is that the imaging parameter detection unit 901 is added to the imaging unit 701.

In this embodiment, the image pickup parameters of the image pickup unit 701 detected by the image pickup parameter detection unit 901 such as the focal length at the time of imaging of the used lens, the actual aperture, the used optical filter, the focusing position, the shutter speed, etc. At the same time, it is sent to the recording mechanism 708,
The information is recorded together in a form corresponding to the recording information in the fifth embodiment shown in FIG. As a result, all information at the time of imaging is recorded at the same time, and it is possible to greatly reduce the labor required when the computer graphics image and its video are later synthesized.

In the fifth and sixth embodiments, only two marks A and B are used as marks.
By using the three landmarks A, B, and C and adding the distance detection mechanism to detect all of them, the rotation angle about the imaging azimuth axis is obtained in addition to the three-dimensional position and the imaging azimuth of the imaging unit. It is possible.

Further, if three or more, many marks and many distance detection mechanisms are used, even if there is a mark which is hidden behind the image pickup unit 701 in an actual use situation and is not detected, the mark can be reliably detected. By using the information of the landmarks, other landmarks can be reliably detected instead, so that the intended purpose can be achieved, and more accurate measurement of the three-dimensional position, imaging azimuth, and rotation angle can be performed. Becomes possible.

Further, also in the fifth and sixth embodiments, the recording mechanism is omitted, and a computer or the like for synthesizing a real photographed image and a computer graphics image to a computer or the like is used. The position, the three-dimensional position of the imaging unit, the azimuth, the rotation angle, and the imaging parameters, if necessary, may be directly output in association with each other to perform necessary processing externally.

<< Embodiment 7 >> A seventh embodiment of the present invention is a fourth embodiment of the present invention using a method of detecting surrounding landmarks by an azimuth detecting mechanism on an imaging unit. is there.

FIG. 10 is a view for explaining the present embodiment. Reference numeral 1001 denotes a TV camera as an image pickup unit;
1002 and 1003 are LEDs (light emitting diodes) as markers, 1004 is a light emission timing controller, 1005A, 1005B, 1006A and 1006
B is a landmark azimuth detecting mechanism, 1007 is a computer as a calculating mechanism, and 1008 is a magnetic video disk as a recording mechanism. Here, landmarks 1002 and 1
003 is fixed to a known three-dimensional position in space. Also, the azimuth detecting mechanisms 1005A, 1005B, 10
06A and 1006B are fixed at known positions on the TV camera 1001 as an imaging unit.

In order to operate this, first, the L as a mark is controlled by the light emission timing controller 1004.
The EDs 1002 and 1003 are turned on, and these are respectively turned to landmark azimuth detecting mechanisms 1005A, 1005B, and 1006A.
And 1006B, tracking objectives 1002 and 1 respectively
The direction of the mark up to 003 is detected. The information of the detected landmark orientation is processed by the calculation mechanism 1007, so that the landmark orientation detection mechanisms 1005A, 1005B, 1005
Marks 1002 and 10 viewed from 6A and 1006B
03 are obtained, and the landmark azimuth detecting mechanisms 1005A, 1005B, 1006A and 10
06B can be back calculated. In addition, the imaging orientation of the imaging unit 1001 can be obtained from this.

FIG. 11 shows an example of the configuration of the landmark azimuth detecting mechanism 1005A, where 1101 is a controller, 1102 is a rotating mechanism having an angle detecting function around two axes,
1103, a CCD fixedly mounted on a rotating mechanism 1102
The sensor 1104 is a three-dimensional position of the mark direction detection mechanism 1005A. The remaining three landmark detection mechanisms 1005
The same applies to B, 1006A, and 1006B.

In order to operate the mark direction detection mechanism 1005A, first, the CCD sensor 11 is operated by operating the rotation mechanism 1102 in accordance with a command from the controller 1101.
By moving the direction of 03, a mark 1002 previously arranged around (a known three-dimensional position in space) is captured in the field of view of the CCD sensor 1103. After that, CCD sensor 11
The rotation mechanism 11 detects the shift amount between the center of the field of view 03 and the detection position of the mark 1002, and minimizes the shift amount (that is, the center of the field of view of the CCD sensor).
Keep 02 running. Then, a mark azimuth V1 in the direction of the mark 1002 viewed from the three-dimensional position 1104 of the mark azimuth detection mechanism 1005 is obtained by the angle detection mechanism around two axes incorporated in the rotation mechanism 1102.

In the controller 1101, the CCD sensor 1
It is determined at which position of the field of view 103 (in which direction and at a distance from the center) the mark 102 is captured, and the rotating mechanism 1102 is operated so that the mark 102 is always captured at the center.

The configuration and operation of the landmark azimuth detecting mechanism 1005B are the same as those of the landmark azimuth detecting mechanism 1005A, but the landmark azimuth V2 is obtained by capturing the same landmark 1002 from a three-dimensional position different from that of the landmark 1104.

The relative three-dimensional position of the mark 1002 is obtained from the two mark directions V1 and V2 thus obtained.

Similarly, two more landmark azimuth detecting mechanisms 1
006A and 1006B, the relative 3
The dimension position can also be obtained.

The angle detection function incorporated in the rotation mechanism can be realized by using, for example, a rotary encoder widely used in the joints of industrial robot arms.

Although omitted in the above description, in practice, it is necessary to take some measures to prevent the LEDs 1002 and 1003 as markers from being confused and detected.

For example, the marks 1002 and 1003 are set to emit light intermittently at another timing or another frequency. As a result, the two can be detected separately.

As another method, there is a method of distinguishing between the two by using color TV cameras as the two-dimensional sensors 1004 and 1005, and using marks of different colors as the marks 1002 and 1003, respectively.

As described above, three-dimensional position information and imaging azimuth information of the imaging unit 1001 can be obtained at the same time. Therefore, while performing measurement in this way, imaging is performed by the imaging unit 1001, and this video information and
The recording mechanism 10 associates the three-dimensional position of the imaging unit 1001 and the information of the imaging orientation obtained by the above method with each other.
08.

<< Eighth Embodiment >> An eighth embodiment of the present invention is also a fourth embodiment of the present invention using a method of detecting surrounding landmarks by an azimuth detecting mechanism on an imaging unit. is there.

FIG. 12 is a diagram for explaining the present embodiment. Reference numeral 1201 denotes an imaging parameter detection unit. The difference from FIG. 4 is that the imaging parameter detection unit 120
1 is added to the imaging unit 1001.

In this embodiment, the imaging parameters of the imaging unit detected by the imaging parameter detection unit 1201 such as the focal length at the time of imaging of the used lens, the actual aperture, the used optical filter, the focusing position, the shutter speed, etc. 10 sent to the recording mechanism
Are recorded together in a form corresponding to the recording information in the embodiment. As a result, all information at the time of imaging is recorded at the same time, and it is possible to greatly reduce the labor required when the computer graphics image and its video are later synthesized.

In the seventh and eighth embodiments, only two light sources 1002 and 1003 are used as marks. However, three marks are used according to the same principle, and the above-described method is used to detect all of these marks. By adding the mark azimuth detecting mechanism, it is possible to obtain the rotation angle around the imaging azimuth axis in addition to the three-dimensional position and the imaging azimuth of the imaging unit.

Also, if three or more, many marks and many mark direction detecting mechanisms are used, even if there is a mark which is hidden behind the image pickup unit 1001 and is not detected in an actual use situation, it is ensured. By using the captured landmark information, other landmarks can be reliably detected instead, so that the intended purpose can be achieved, and more accurate three-dimensional position, imaging azimuth and rotation angle can be obtained. Measurement becomes possible.

Also, in the seventh and eighth embodiments, the recording mechanism is omitted, and the image picked up by the image pickup unit, the mark or the mark direction is sent to a computer or the like for synthesizing the real image and the computer graphics image. The three-dimensional position of the detection mechanism, the three-dimensional position of the imaging unit, the azimuth, the rotation angle, and the imaging parameters, if necessary, may be directly output in association with each other to perform necessary processing externally.

<Embodiment 9> A ninth embodiment of the present invention is an embodiment of the fifth invention of the present invention using a method of detecting surrounding landmarks by a sensor on the image pickup unit.

FIG. 13 is a block diagram showing the present embodiment. Reference numeral 1301 denotes a TV camera as an image pickup unit;
1302-1, 1302-2, ..., 1302-n are n LEDs fixedly installed as markers, 1303-1,
1303-m are imaging units 13
M, two-dimensional sensors 1304
Is a light emission timing controller, 1307 is a computer as a calculation mechanism, and 1308 is a magnetic video disk as a recording mechanism.

To operate this, first, under the control of the light emission timing controller 1304, n LEDs 1302-1, 1302-2,.
n is turned on, and this light is used as m two-dimensional sensors.
Dimension sensors 1303-1, 1303-2, ..., 1303
−m, and the information is processed by the calculation mechanism 1307 to obtain m pieces of two-dimensional sensors 1303-1 and 1303-1.
3-2,..., 130 LED of n LEDs 1303-m
, 1302-n, and three-dimensional positions P-1, P-2,. By performing the calculation in reverse, the m two-dimensional sensors 1303-
, 1303-m are obtained, and the three-dimensional position of the imaging unit 1301 itself, and the imaging azimuth and rotation angle can be obtained from them. In order to obtain these, at least 3 points or more
What is necessary is just to determine the three-dimensional position of the dimensional sensor.

The principle of a method for obtaining a three-dimensional position based on two images has already been established in the field of stereo photograph measurement and the like. This is the same as that described in the first embodiment with reference to FIG.

<< Embodiment 10 >> A tenth embodiment of the present invention is a sixth embodiment of the present invention using a method of detecting surrounding landmarks by a distance detecting mechanism on an imaging unit. is there.

FIG. 14 is a block diagram showing the present embodiment. Reference numeral 1401 denotes a TV camera as an image pickup unit;
1402-1, 1402-2, and 1402-3 are mark light sources; 1403-1, 1403-2, and 1403-3 are distance detection mechanisms installed at different positions on the imaging unit 1401; Control mechanism, 1407
Is a computer as a calculation mechanism, and 1408 is a magnetic video disk as a recording mechanism. Here, the mark light sources 1402-1, 1402-2, and 1402-3 are:
Use a light source that radiates light uniformly to the surroundings (a light source that does not emit light only in a specific direction, or a light source that radiates light radially using a small diffusing lens) Is realized). As the distance detection mechanism, the one already described in the fifth embodiment can be used.

To operate this, first, the mark light sources 1402-1 and 1402-1 are controlled by the light emission control mechanism 1404.
2-2 and 1402-3 are made to emit light, and the lights are respectively transmitted to the distance detection mechanisms 1403-1, 1403-1 and 1403-3.
After detecting the distance from the mark light source to each of the distance detection mechanisms, the distances are processed by the calculation mechanism 1407 to obtain the mark light sources 1402-1, 1402-2, and 1
Distance detection mechanisms 1403-1 and 1403 for 402-3
The three-dimensional positions of three points 3-2 and 1403-3 are obtained, and the imaging azimuth and the rotation angle can also be obtained therefrom.

Although omitted in the above description, in practice, the light sources 1402-1, 1402-2, and 1
Some care must be taken to prevent 402-3 from being confused and detected. For example, they are set so as to emit light intermittently at a different timing or a different frequency so that they can be distinguished from each other. As a result, the two can be detected separately.

As another method, there is a method in which light sources having different colors (wavelengths) are used as the mark light sources so that the two can be distinguished from each other.

As described above, three-dimensional position information and imaging azimuth information of the imaging unit 1401 can be obtained at the same time. Therefore, while performing measurement in this way, an image is taken by the imaging unit 1401, and this video information and the information on the three-dimensional position and the imaging orientation of the imaging unit 1401 obtained by the above method are associated with each other, and the recording mechanism is used. 1408.

In the ninth and tenth embodiments, as in the second embodiment, an imaging parameter detection unit is added to the imaging unit, and the (a) used lens of the imaging unit detected thereby is used. (B)
The imaging parameters, such as the actual aperture, (c) the optical filter used, (d) the focusing position, and (e) the shutter speed, are simultaneously sent to the recording mechanism, and together with the information corresponding to the recording information in each embodiment. It may be recorded. In this case, since all information at the time of imaging is recorded simultaneously in real time in association with video information, it is easy to respond to changes in imaging parameters during imaging, and later, a computer graphics image and The advantage is that the time and labor required for synthesizing the images can be greatly reduced.

Further, at least three points have to be determined as the three-dimensional position on the imaging unit. However, the rotation angle can be ignored, and only the three-dimensional position and orientation of the imaging unit are required. It is sufficient that at least two points on the imaging unit are determined, and the number of two-dimensional sensors and azimuth detecting mechanisms on the imaging unit can be reduced accordingly.

Further, in FIG. 12, a large number of two or more three-dimensional sensors are used. However, in the actual use situation, there is a mark which is hidden behind the image pickup unit 1301 or the like and cannot be detected. Even if this is the case, it is possible to reliably capture other landmarks and to measure the three-dimensional position, the imaging direction, and the rotation angle with high accuracy.
The same applies to the case of 3.

Further, also in the ninth and tenth embodiments, the recording mechanism is omitted, and a computer or the like for synthesizing a real photographed image and a computer graphics image to a computer or the like is used. The position, the three-dimensional position of the imaging unit, the azimuth, the rotation angle, and the imaging parameters, if necessary, may be directly output in association with each other to perform necessary processing externally.

[0133]

As described above, according to the present invention,
A plurality of landmarks and a means for detecting the distance to the landmark or the landmark or the landmark are arranged on the imaging unit and its surroundings, or conversely, around the imaging unit and the imaging unit, and the result of the detection The three-dimensional position of the imaging unit, or the azimuth in addition thereto, or the rotation angle from the azimuth axis in addition thereto can be calculated. The advantage is obtained that an environment required easily and easily can be realized with high accuracy automatically at the same time as imaging. Further, there is an advantage that the degree of freedom of movement of the imaging unit is increased and the range of the space in which the imaging unit can be moved is widened.

In particular, when the imaging parameters of the imaging unit are detected during imaging and recorded or output in association with the captured image, it is possible to easily respond to changes in imaging parameters during imaging, and An advantage is obtained in that the time and labor required for image synthesis and the like can be greatly reduced.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a first embodiment according to a first invention of the present invention.

FIG. 2 is a conceptual diagram showing a method for obtaining a three-dimensional position based on two images captured by upper and side sensors in the first embodiment.

FIG. 3 is a diagram illustrating a second embodiment according to the first invention of the present invention.

FIG. 4 is a diagram illustrating a third embodiment according to the second invention of the present invention.

FIG. 5 is a diagram showing a configuration example of a landmark azimuth detecting mechanism in the third embodiment.

FIG. 6 is a diagram for explaining a fourth embodiment according to the second invention of the present invention.

FIG. 7 is a diagram illustrating a fifth embodiment according to the third invention of the present invention.

FIG. 8 is a diagram illustrating an operation example of a distance detection mechanism according to the fifth embodiment.

FIG. 9 is a view for explaining a sixth embodiment according to the third invention of the present invention.

FIG. 10 is a diagram illustrating a seventh embodiment according to the fourth invention of the present invention.

FIG. 11 is a diagram showing one configuration example of a landmark azimuth detecting mechanism of the seventh embodiment.

FIG. 12 is a diagram illustrating an eighth embodiment according to the fourth invention of the present invention.

FIG. 13 is a diagram illustrating a ninth embodiment according to the fifth invention of the present invention.

FIG. 14 is a view for explaining a tenth embodiment according to the sixth invention of the present invention.

[Explanation of symbols]

101: Imaging unit (TV camera) 101, 103: Mark (LED) 104: Light emission timing controller 105, 106: Two-dimensional sensor (TV camera) 107: Calculation mechanism (computer) 108: Recording mechanism (magnetic video disk) 201 Top image 202 Side image 203 Three-dimensional position information 301 Imaging parameter detection unit 401 Imaging unit (TV camera) 402, 403 Marks (LED) 405-A, 405-B, 406-A, 406-B ... Marking direction detection mechanism 407 ... Calculation mechanism (computer) 408 ... Recording mechanism (magnetic video disk) 501 ... Controller 502 ... Rotating mechanism 503 ... CCD sensor 504 ... 3D position of the mark direction detection mechanism 601 ... Imaging parameter detection unit 701 ... Imaging Unit (TV camera) 02,703 ... mark light source 704 ... light emission control mechanisms 705A, 705B, 705C, 706A, 706B,
706C: distance detection mechanism 707: calculation mechanism (computer) 708: recording mechanism (magnetic video disk) 901: imaging parameter detection unit 1001: imaging unit (TV camera) 1002, 1003: mark (LED) 1004: light emission timing controller 1005A, 1005B, 1006A, 1006B ... mark direction detection mechanism 1007 ... calculation mechanism (computer) 1008 ... recording mechanism (magnetic video disk) 1301 ... imaging unit (TV camera) 1302-1, 1302-2, ..., 1302-n ... mark ( 1303-1, 1303-2, ..., 1303-m ... two-dimensional sensor 1304 ... light emission timing controller 1307 ... calculation mechanism (computer) 1308 ... recording mechanism (magnetic video disk) 1401 ... imaging unit (TV camera) 1402-1, 1402-2, 1402-3 ... mark light source 1403-1, 1403-2, 1403-3 ... distance detection mechanism 1404 ... light emission control mechanism 1407 ... calculation mechanism (computer) 1408 ... recording mechanism ( Magnetic video disk)

Claims (15)

[Claims] 1. A plurality of marks are attached to an imaging unit in advance, the marks are detected from a plurality of external directions, and a three-dimensional position of the marks obtained from a result of the detection or a three-dimensional position of the marks is determined. The three-dimensional position and orientation of the imaging unit calculated based on the
An imaging method comprising: recording or outputting a dimensional position, an azimuth, and a rotation angle in association with a captured image of the imaging unit. 2. An imaging unit provided with a plurality of marks, a plurality of sensors for detecting the marks from the outside, a three-dimensional position of the imaging unit based on a detection result of the sensor, or the imaging unit Calculating means for calculating and calculating the three-dimensional position and orientation of the imaging unit or the three-dimensional position, orientation and rotation angle of the imaging unit; and the three-dimensional position of the imaging unit determined by the calculation means connected to the calculation means Means for recording or outputting the three-dimensional position and azimuth of the imaging unit or the three-dimensional position, azimuth and rotation angle of the imaging unit in association with the captured image of the imaging unit. apparatus. 3. Marking a plurality of marks in advance on an image pickup unit, detecting a mark direction to the mark as viewed from a plurality of external positions, and determining a three-dimensional position of the mark obtained from a result of the detection, or the mark. Recording or outputting the three-dimensional position and the azimuth of the imaging unit or the three-dimensional position, the azimuth and the rotation angle of the imaging unit calculated based on the three-dimensional position of the imaging unit in association with the captured image of the imaging unit; An imaging method characterized by the above-mentioned. 4. An image pickup unit having a plurality of marks, a plurality of mark azimuth detecting mechanisms for detecting a mark azimuth to the mark as viewed from a plurality of external positions, and a detection result of the mark azimuth detecting mechanism. Calculating means for calculating and calculating the three-dimensional position of the imaging unit, or the three-dimensional position and orientation of the imaging unit, or the three-dimensional position, orientation and rotation angle of the imaging unit based on Means for connecting and recording or outputting the three-dimensional position and orientation of the imaging unit or the three-dimensional position, orientation and rotation angle of the imaging unit obtained by the calculation means in association with the captured image of the imaging unit. An imaging device, comprising: 5. A method for adding a plurality of marks to an image pickup unit, detecting distances to the plurality of marks from a plurality of distance detection mechanisms installed around the image pickup unit, and based on the detected distance. The calculated three-dimensional position of the plurality of landmarks, or the three-dimensional position and orientation of the imaging unit, or the three-dimensional position, orientation and rotation angle of the imaging unit are recorded or recorded in association with the captured image of the imaging unit. Outputting an imaging method. 6. An image pickup unit to which a plurality of marks are added, a distance detection mechanism installed around the image pickup unit to detect a distance to a mark position on the image pickup unit, and a distance detection mechanism based on the detected distance. Calculating means for calculating the three-dimensional position of the mark, or the three-dimensional position and orientation of the imaging unit, or the three-dimensional position, orientation and rotation angle of the imaging unit; and calculating with the calculating means connected to the calculating means Means for recording or outputting the three-dimensional position of the mark, or the three-dimensional position and azimuth of the imaging unit, or the three-dimensional position, azimuth and rotation angle of the imaging unit in association with the captured image of the imaging unit; An imaging device comprising: 7. A plurality of landmarks are prepared in advance around an imaging unit, and a plurality of landmark azimuth detecting mechanisms for detecting azimuths to the landmarks are attached to the imaging unit, and the landmark azimuth detection on the imaging unit is performed. Detecting the mark direction from the position of the mechanism to the mark, calculated based on the detected mark direction, the three-dimensional position of the mark direction detection mechanism, or the three-dimensional position and direction of the imaging unit, or the An imaging method comprising: recording or outputting a three-dimensional position, an azimuth, and a rotation angle of an imaging unit in association with a photographed image of the imaging unit. 8. A plurality of landmarks prepared in advance around the landmark, a plurality of landmark direction detection mechanisms for detecting the landmark direction of the landmarks, an imaging unit to which the plural landmark direction detection mechanisms are attached, and the landmark direction detection A three-dimensional position of the landmark azimuth detecting mechanism, a three-dimensional position and azimuth of the imaging unit, or a three-dimensional position and azimuth of the imaging unit based on the landmark azimuth viewed from the landmark azimuth detecting mechanism detected by the mechanism; Calculation means for calculating the rotation angle, and the calculated three-dimensional position of the mark orientation detection mechanism, or the three-dimensional position and orientation of the imaging unit, or the three-dimensional position, orientation and rotation angle of the imaging unit, Means for recording or outputting in association with a captured image of the imaging unit. 9. A method according to claim 3, wherein a plurality of sensors are added to the imaging unit, a plurality of marks are prepared in advance around the mark, the mark is captured by the sensor, and a calculation is performed based on a result of capturing the mark. Recording or outputting a three-dimensional position, a three-dimensional position and orientation of the imaging unit, or a three-dimensional position, orientation and rotation angle of the imaging unit in association with a captured image of the imaging unit. Method. 10. An imaging unit to which a plurality of sensors are added, a plurality of marks prepared around the sensor, and a three-dimensional position of the sensor with respect to the mark or a result of capturing the marks by the sensor, or Calculating means for calculating a three-dimensional position and orientation, or a three-dimensional position, orientation and rotation angle of the imaging unit; a calculated three-dimensional position of the sensor, or a three-dimensional position and orientation of the imaging unit, or the imaging Means for recording or outputting a three-dimensional position, an azimuth, and a rotation angle of the unit in association with the photographed image of the imaging unit. 11. A plurality of distance detection mechanisms for detecting a distance to a landmark are added to the imaging unit, a plurality of landmarks are arranged around the imaging unit in advance, and a distance from the position of the distance detection mechanism to the landmark is detected. The three-dimensional position of the mark, calculated from the result of the detection,
Alternatively, the three-dimensional position and the azimuth of the imaging unit, or the three-dimensional position, the azimuth and the rotation angle of the imaging unit are recorded or output in association with a captured image of the imaging unit. 12. An image pickup unit to which a plurality of distance detecting mechanisms for detecting a distance to a mark are added, a plurality of marks installed around the image sensor, and a distance from the detected mark as viewed from the position of the detected distance detecting mechanism. Calculating means for calculating the three-dimensional position of the mark, or the three-dimensional position and direction of the imaging unit, or the three-dimensional position, direction and rotation of the imaging unit; and the calculated three-dimensional position of the mark or the imaging Means for recording or outputting the three-dimensional position and azimuth of the unit or the three-dimensional position, azimuth and rotation angle of the imaging unit in association with the image captured by the imaging unit. 13. The illuminant control means for using a plurality of illuminants as a plurality of marks and blinking the plurality of illuminants at individual emission timings or individual frequencies. The imaging device according to any one of claims 4, 6, 8, 10, and 12. 14. A sensor, a distance detecting mechanism, or a landmark azimuth detecting mechanism, wherein a plurality of luminous bodies having different emission colors are used as the plurality of landmarks.
The image pickup apparatus according to any one of claims 2, 4, 6, 8, 10, and 12, further comprising color identification means. 15. A new detecting means connected to an image pickup unit and detecting one or a plurality of image pickup parameters such as a lens focal length, a lens aperture value, an optical filter used, a focusing position, and a shutter speed of the image pickup unit. The apparatus according to claim 2, wherein the recording or outputting means records or outputs the detected imaging parameter in association with a captured image of the imaging unit. 8, Claim 10, Claim 12, Claim 13, Claim 1
5. The imaging device according to any one of 4.