JP5019478B2 - Marker automatic registration method and system - Google Patents

Description

  The present invention relates to a marker automatic registration method and system suitable for use in augmented reality operations and the like.

  If Augmented Reality can be used in a nuclear power plant, it will be possible to provide various support for workers working in the plant.

  In order to use augmented reality, tracking technology that measures the position and direction of the worker in real time is required, but existing tracking technologies such as GPS (Global Positioning System), magnetic sensor, ultrasonic sensor, inertial sensor, etc. Alone cannot be used inside a nuclear power plant.

  Thus, the inventors have developed a tracking method using a camera and a far / near marker as a tracking method that can be used even inside a nuclear power plant.

  The tracking method using both camera and perspective markers is a method of pasting multiple markers in the work environment, shooting them with the camera, and obtaining the relative positional relationship between the camera and the marker by image processing and geometric calculation. It is.

  This method has the advantage that it can be used indoors, is not affected by metal and noise, and does not degrade accuracy or stability over time. It is necessary to measure and store the three-dimensional position and direction.

Republished W2005-017644 JP 2004-28788 A L. Quan et al: Linear N-Point Camera Pose Determination, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 21, no. 7, pp. 774-780, 1999.

  In the tracking method using the above-described camera and perspective marker, the work of measuring and storing the three-dimensional position and direction of the attached marker in advance has been performed manually, but the working environment is a nuclear power plant. In such a case, there is a problem that the method of attaching the marker becomes complicated, which is very time-consuming and causes measurement errors.

  Accordingly, one object of the present invention is to save labor and reduce the occurrence of measurement errors by automating the measurement of the three-dimensional position and direction of the bifocal marker and the storage of the measurement results.

  Another object of the present invention is to store the position and direction of the marker in the form of a world coordinate system.

The marker automatic registration method and system according to the present invention includes a three-dimensional position and direction of a bifocal marker affixed to a work environment using a video camera, a laser distance measuring instrument, an electric pan head, etc. that can be controlled by a small computer. Is a configuration that automatically measures and registers (stores) the measurement results,
In particular,
Paste the marker serving as a reference position 置及 beauty appropriate position of the environment, represented in the world coordinate system of all of the markers in the distance shooting and laser distance measuring device with a video camera to the extent possible measurement 3 In the method of measuring and storing the dimension position and direction fully automatically,
The marker includes one large circle arranged at the center of a square mount and four small circles arranged at the four corners of the mount. By using a configuration in which the ID number of each marker is expressed by setting the white sector to 0 and the black sector to 1.
A first step of setting the focal length of the video camera to the shortest possible value;
The shooting direction of the video camera is rotated at equal intervals so that all areas of the operating range of the video camera set to the shortest settable focal length are shot at least once. A second step of recognizing and storing the distance to the marker pasted in the environment and the existing direction by executing an algorithm that captures the distance between the marker and the existing direction by photographing with a camera When,
A third step of repeating the second step by setting the focal length of the video camera to a value twice and three times the shortest value;
By executing the algorithm for measuring the position of a point on the marker using a laser distance measuring instrument for each marker stored recognized by the third step from the first step, all recognized A fourth step of measuring and storing the three-dimensional position and direction of the marker;
Applying an algorithm that converts the marker position and direction to the world coordinate system to the three-dimensional position and direction of all markers stored in the fourth step, the three-dimensional position and direction expressed in the world coordinate system The marker automatic registration method characterized by performing the 5th step which calculates | requires and memorize | stores.

In addition, among the markers that are pasted at appropriate positions and positions that serve as the reference for the environment, all markers that can be photographed with a video camera and that can be measured with a laser distance meter are represented in the world coordinate system. In a system that automatically measures and stores the three-dimensional position and direction,
Connected to a video camera that can be turned by an electric head, a laser distance measuring device that can be turned by an electric head, and the video camera and the laser distance measuring device. And a small computer that controls them,
The marker includes one large circle arranged at the center of a square mount and four small circles arranged at the four corners of the mount. The white fan shape is set to 0 and the black fan shape is set to 1, so that the ID number of each marker is expressed.
The small computer is
A first step of setting the focal length of the video camera to the shortest possible value;
The shooting direction of the video camera is rotated at equal intervals so that the entire range of the operating range of the video camera set to the shortest settable focal length is shot at least once. A second method for recognizing and storing the distance to the marker pasted in the environment and the existing direction by executing an algorithm for obtaining the distance between the marker and the existing direction by photographing using a video camera Steps,
A third step of repeating the second step by setting the focal length of the video camera to a value twice and three times the shortest value;
By executing the algorithm for measuring the position of a point on the marker using the laser distance measuring device for each marker recognized and stored by the third step from the first step, all recognized A fourth step of measuring and storing the three-dimensional position and direction of the marker;
Applying an algorithm that converts the marker position and direction to the world coordinate system to the three-dimensional position and direction of all markers stored in the fourth step, the three-dimensional position and direction expressed in the world coordinate system A program for performing the fifth step for obtaining and storing the above is provided.

  According to the present invention, it is possible to save labor and reduce the occurrence of measurement errors by automating the measurement of the three-dimensional position and direction of the near / far marker and the process of storing the measurement results in the form of a world coordinate system.

The present invention is capable of photographing with a video camera and measuring a distance with a laser distance measuring instrument among the reference position of the environment (the origin of the world coordinate system or a point on the coordinate axis) and the marker attached at an appropriate position. In a system that automatically measures and stores the three-dimensional positions and directions of all markers in the global range expressed in the world coordinate system,
Connected to a video camera that can be turned by an electric head, a laser distance measuring device that can be turned by an electric head, and the video camera and the laser distance measuring device. And a small computer that controls them,
The small computer is
A first step of setting the focal length of the video camera to the shortest possible value;
The shooting direction of the video camera is rotated at equal intervals so that all areas of the operating range of the video camera are shot at least once, and in each shooting direction, the distance from the marker A second step of recognizing and storing the distance to the marker pasted in the environment and the existing direction by executing an algorithm for determining the existing direction;
A third step of repeating the second step by setting the focal length of the video camera to a value twice and three times the shortest value;
By executing an algorithm for measuring the position of one point on the marker using the laser distance measuring device for each marker recognized and stored up to the third step, 3 of all the recognized markers A fourth step of measuring and storing the dimension position and direction;
Applying an algorithm that converts the marker position and direction to the world coordinate system to the three-dimensional position and direction of all markers stored in the fourth step, the three-dimensional position and direction expressed in the world coordinate system A program for performing a fifth step of determining and storing
The marker includes one large circle arranged at the center of a square mount and four small circles arranged at the four corners of the mount, and the great circle has ten identical fan shapes arranged in an annular shape, It is assumed that the white fan shape is 0 and the black fan shape is 1, so that the ID number of each marker is expressed.

FIG. 1 is a plan view of a perspective marker used in the first embodiment. The bifocal marker 1 includes one great circle 12 which is arranged in the center of the square white sheet 11, constituted by four small circles 13 _{0-13 3} 4 arranged in a corner of the sheet 11, the great circle 12 (a recognizable by the image processing, the point serving as a reference for tracking) small circle 13 _{0-13 3} points, wherein centers of the.

  In the great circle 12, ten identical sector shapes 12 a to 12 j are arranged in an annular shape, with the white sector shape being 0 and the black sector shape being 1 so as to express the ID number of each marker 1. To.

  The near / far marker 1 is used for tracking as follows.

  1) A plurality of perspective markers 1 are pasted in advance in the work environment, and the ID number of each perspective marker and the three-dimensional position in the work environment are measured and stored.

  2) Photograph the work environment with the far / near vision marker 1 attached with a camera worn by the worker.

  3) The camera image data obtained by shooting is processed to recognize the ID number of each perspective marker 1 and the position of the feature point on the camera shot image.

  4) A PnP problem (Perspective N-Point Problem) (see Non-Patent Document 1) is obtained by using the position of the feature point on the camera-captured image and the three-dimensional position information of each perspective marker 1 in the work environment. It solves and calculates the relative position and direction between the camera, the near / far marker 1 and the camera.

  In tracking performed in this way, in order to uniquely obtain the relative position and direction between the camera and the far / far marker 1, it is necessary to recognize at least four feature points simultaneously. In the perspective marker, the feature point at the center of the great circle of a plurality of markers is used when the distance between the camera and the marker is long, and the feature point at the center of the four small circles of one perspective marker is short when the distance is short. Is used.

  Therefore, in this method, the three-dimensional positions of all the feature points of the dual-purpose markers pasted in the work environment (or the three-dimensional position of one point on the marker and the direction of the marker) are measured and stored in advance. It is necessary to keep it.

  FIG. 2 is a perspective view showing a hardware configuration of an automatic marker registration system that automatically measures a distance-use marker pasted in such a work environment and registers a measurement result.

  2 is a video camera with a built-in electric pan head that can be controlled by a small computer, 3 is a laser distance measuring device, 4 is an electric pan head on which the laser distance measuring device 3 is mounted, and 5 is the video camera 2 and the laser distance measuring device. 3 and a small computer (portable personal computer) 7 connected to the electric camera platform 4 and the communication cable 6 to control them, 7 is attached to the work environment with the video camera 2 and the electric camera platform 4 attached. It is a tripod that is installed at a position where you can see the far and far distance marker.

  In the video camera 2, the direction, zoom value, shutter speed, and the like are controlled by a control signal from the small computer 5. The captured image data of the video camera 2 is input to the small computer 5 using a video capture device (not shown).

  The laser distance measuring device 3 is directed to the target bifocal marker 1 by directing the laser beam irradiation direction to the target bifocal marker 1 by controlling the electric head 4 by a control signal from the small computer 5. The distance between the laser distance measuring device 3 and the target bifocal marker 1 is measured based on the phase difference between the reflected light and the irradiated light, and the measurement result is transmitted to the small computer 5. .

  Specifically, for example, it can be realized by using the following devices.

Video camera 2
Manufacturer / model: Sony EVI-D30
Video output: NTSC
Focal length: 5.4 mm to 64.8 mm
Rotation range: Horizontal ± 100 degrees, vertical ± 25 degrees Control terminal: RS-232C
Laser distance measuring instrument 3
Manufacturer & Model: Leica Geosystems DISTO Pro 4a
Measurement range: 0.3m to 40m
Measurement accuracy: Standard ± 1.5mm, Maximum ± 2mm
Control terminal: RS-232C
Electric pan head 4
Manufacturer & Model: Directed Perception PTU-D46-70
Resolution: 0.012857 degrees Maximum speed: 60 degrees / second Rotation range: Horizontal ± 159 degrees, Vertical -31 degrees +48 degrees Control terminal: RS-232C
Video Capture Manufacturer / Model: IO / DATA device USB-CAP2
Image input: NTSC
Resolution: Maximum 352 x 288
Image output: USB1.1, RGB24bit
Acquisition speed: 30 fps
Small computer 5
Manufacturer & Model: ASUS M5N
CPU: Pentium M1.4GHz (Pentium is a registered trademark)
Memory: DDR333 768MB
OS: Microsoft Windows XP Home Edition (Microsoft Windows is a registered trademark)
The marker automatic registration system configured as described above is configured to automatically measure the three-dimensional position and direction of the far and near vision marker 1 attached to the work environment and store the measurement result.

  A specific method for measuring and storing (registering) the near-far marker using this marker automatic registration system is as follows.

Step S1
The near-far marker 1 is pasted at a position (the origin of the world coordinate system or a point on the coordinate axis) as a reference of the work environment and an appropriate position.

Step S2
An automatic marker registration system is installed at a position where the two-way marker 1 affixed to the work environment can be seen.

Step S3
Using the automatic marker registration system, the three-dimensional positions and directions of all the perspective markers 1 attached to the work environment are measured.

Step S4
Save the measurement results to a file and transfer it to a system that uses augmented reality.

  The measurement of the near / far marker 1 in step S3 is automatically executed as follows.

Step S31
The video camera 2 is photographed while being rotated at equal intervals over the entire operating range, and which perspective marker 1 is pasted to which position from the ID number, size, and direction of the perspective marker 1 reflected on the video camera 2. Is determined by image processing. The position information obtained here includes a relatively large error.

Step S32
More accurate three-dimensional positions and directions are measured in the following manner for all the near-far markers 1 obtained in step S31.

Step S321
Based on the position information of the near / far marker 1 obtained in step S31, the direction and zoom of the video camera 2 are adjusted and controlled so that the image of the target far / far marker 1 is captured at the center of the image captured by the video camera 2. To do.

Step S322
Based on the position information of the far / far marker 1 obtained in step S31, the electric pan head 4 is controlled so that the laser irradiation direction of the laser distance measuring device 3 is directed toward the target far / far marker 1.

Step S323
The laser irradiation position (the position of the laser irradiation spot) is recognized by comparing the captured image of the video camera 2 before laser irradiation with the captured image of the video camera 2 after laser irradiation.

Step S324
The direction of the laser distance measuring instrument 3 is finely adjusted by controlling the electric camera platform 4 so that the laser irradiation position becomes the center of the target bifocal marker 1 (the same as the center of the great circle 12).

Step S325
The distance between the target bifocal marker 1 and the laser distance measuring device 3 is measured.

Step S326
Based on the orientation of the laser distance measuring device 3 adjusted in step S324 and the distance measured in step S325, the three-dimensional position of the target bifocal marker 1 is obtained.

Step S327
The three-dimensional positions near the small circles at the four corners of the target bifocal marker 1 are obtained by the same method as in steps S322 to S326 described above.

Step S33
Using the information of the three-dimensional position of the perspective marker 1 pasted at a position serving as a reference of the work environment, the three-dimensional position and direction of each perspective marker 1 with reference to the world coordinate system are obtained by calculation.

  As described above, this marker automatic registration system is used to measure the three-dimensional position and direction of each perspective marker 1 in a fully automatic manner after being installed in the work environment where the perspective marker 1 is pasted. There is no need for the operator to operate the system, and the labor required to measure the three-dimensional positions and directions of the multiple perspective markers 1 can be greatly reduced.

  In addition, since the measurement result of each bifocal marker 1 is finally converted into a value based on the world coordinate system, the measurement is performed regardless of where the marker automatic registration system is used in the work environment. Except for errors, the same result can always be obtained.

  On the other hand, in the above-described method, since the near-far marker 1 pasted at a position invisible from the marker automatic registration system cannot be measured, all the far-far markers 1 stuck in the complicated environment are once measured. Cannot be measured. In such a case, on the condition that at least four dual-use markers 1 can be measured redundantly, the installation position of the automatic marker registration system is moved a plurality of times to measure the near-far marker 1, It is possible to integrate the measurement results by using the position information of the bifocal markers measured in duplicate.

  Here, an algorithm for realizing the above-described automatic measurement of the bifocal marker will be described. This algorithm is realized by executing a control and information processing program incorporated in the small computer 5.

  FIG. 3 shows a coordinate system handled by the marker automatic registration system. The coordinate system is all right-handed.

  The world coordinate system W is fixed in a working environment that uses augmented reality, and the center of the first perspective marker is the origin, and from the center of the first perspective marker to the center of the second perspective marker. The direction is the X axis, and the plane including the centers of the first, second, and third bifocal markers is the XY plane.

  The laser head coordinate system A has an origin at the intersection of the rotation axis in the tilt (up and down) direction and the rotation axis in the pan (left and right) direction of the electric camera platform 4 on which the laser distance measuring device 3 is mounted. The vertical direction when initialized is the Z axis, and the front direction is the negative direction of the X axis. The laser distance measuring device coordinate system L has an origin at the intersection of a straight line perpendicular to the rotary platform surface of the electric camera platform 4 passing through the origin of the laser camera platform coordinate system A and the rotary platform surface, and when the electric camera platform 4 is initialized. The vertical direction is the Z axis, and the front direction is the negative direction of the X axis.

  The photographed image (screen) coordinate system S is set on the CCD element plane of the video camera 2, and when viewing the photographing direction of the video camera 2, the upper left corner is the origin, the right is the X axis, and the downward is the Y axis.

  The camera head coordinate system B uses the intersection of the tilt direction rotation axis and the pan direction rotation axis of the electric camera platform built in the video camera 2 as the origin, and the upward direction when the electric camera platform is initialized is Z. The axis and the front direction are the negative directions of the X axis.

  The camera coordinate system C has the focal point of the video camera 2 as the origin, passes through the center of the image plane, the direction perpendicular to the image plane is the negative direction of the X axis, and the direction parallel to the X axis of the captured image coordinate system S is the Y axis. And

  The marker i coordinate system Mi of the bifocal marker 1 whose ID number is i is set by the number of types of the bifocal markers 1, the center of the bifocal marker 1 is the origin, and the normal direction of the marker surface is the Z axis. The boundary between the most significant bit and the least significant bit of the sector 12a (to 12j) representing the number of the dual marker 1 is taken as the X axis.

In this description, scalar variables are represented in lowercase letters, and vectors and matrices are represented in uppercase letters. Further, the reference coordinate system (observation coordinate system) of each variable is represented by an upper left subscript such as ^W R ₁₂ .

In this embodiment, it is approximated that the origins of the camera pan coordinate system B and the camera coordinate system C are the same, and the laser pan head coordinate system A and the camera pan coordinate system B are preliminarily set so that the respective axes are parallel to each other. It shall be fixed. Further, when the laser distance measuring instrument 3 is rotated in the tilt direction, the origin of the laser distance measuring instrument coordinate system L is assumed to rotate on a circumference that is a distance d _AL away from the rotation axis.

A marker recognition algorithm will be described. The algorithm ID number multifocal marker 1 captured in the video camera 2, the position of the feature point of the great circles _{12 (x l, y l)} , the position of the small circle ₁₃ 0-13 ₃ feature points _{(x si,} y _si ) (i = 0, 1, 2, 3), an algorithm for recognizing the size r _image of the great circle 12 (major axis radius of the ellipse). This algorithm is the same as the algorithm used when tracking using the near / far marker 1.

  Color photographed image data acquired from the video camera 2 is converted into gray image data, and the luminance value of each pixel is logarithmically converted to emphasize the change in luminance between the high luminance region and the low luminance region.

  After that, by applying a 3 × 3 Sobel filter and binarizing with a predetermined threshold value, an edge of the image (a portion with a large change in luminance) is recognized.

  Thereafter, the connectivity between pixels recognized as edges (whether they are adjacent to each other) is examined, and it is determined whether or not each edge group recognized as connected can be fitted to an ellipse. The region in the edge group determined to be able to fit to the ellipse is analyzed, and it is determined whether or not the region is the bifocal marker 1. In the case of the perspective marker 1, the ID number of the marker and the major axis of the great circle are recognized, and the centers of the great circle and the small circle are recognized as feature points.

Next, an algorithm for obtaining the distance between the distance marker and the existing direction using a video camera will be described. This algorithm, perspective radius _{r real} of dual great circle 12 of the marker 1, the center coordinates of the marker on the camera captured image _{(x l, y l),} the great circle of the major axis radius _{r image} on the camera captured image, a video camera The distance d _CM between the near-far marker 1 and the video camera 2 reflected in the video camera 2 from the internal parameters 2 (focal length f, CCD element width w _CCD , image resolution w _reso , h _reso ), and camera This is an algorithm for obtaining the pan direction ^B θ _M and the tilt direction ^B φ _M of the bifocal marker 1 represented in the pan head coordinate system B.

  The width and resolution of the CCD element of the video camera 2 can be obtained from the specifications of the video camera 2 and the video capture device, and the radius of the great circle 12 of the bifocal marker 1 is determined by checking the bifocal marker 1 in advance. Can be obtained.

Step S41
Using the marker recognition algorithm, the center coordinates (x _l , y _l ) and the major axis radius r _image of the great circle are obtained.

Step S42
The pan direction ^B θ _C , the tilt direction ^B φ _C , and the focal length f of the video camera 2 when the camera-captured image is captured by communicating with the video camera 2 are obtained.

Step S43
The distance d _CM between the video camera 2 and the near / far marker 1 is obtained by the equation (Equation 1).

ただし、ｄは、カメラ撮影画像上における撮影画像の中心と遠近両用マーカ１の中心の間の距離である。

Here, d is the distance between the center of the captured image on the camera captured image and the center of the bifocal marker 1.

Step S44
The pan direction ^B θ _M and the tilt direction ^B φ _M of the bifocal marker 1 represented by the camera platform coordinate system B are obtained by the equations (Equation 2) and (Equation 3).

ただし、ｖ_ｈ＝２arctan（ｗ_ＣＣＤ／２ｆ）は、ビデオカメラ２の水平方向の画角である。

However, v _h = 2 arctan (w _CCD / 2f) is the horizontal angle of view of the video camera 2.

Next, an algorithm for measuring the position of one point on the bifocal marker 1 using the laser distance measuring device 3 will be described. In this algorithm, the pan / tilt direction Aθ _L and the tilt direction Aφ _{L of} the electric camera platform 4 on which the laser distance measuring device 3 is mounted, and the distance d between the laser distance measuring device 3 and one point on the bifocal marker 1 are used for both perspective. This is an algorithm for _obtaining a highly accurate three-dimensional position ^A TAP of one point on the marker 1.

Step S51
The laser irradiation position is adjusted to one point on the bifocal marker 1 to be measured by an automated method described later.

Step S52
The laser distance measuring device 3 is controlled to obtain a distance d _lp between the laser distance measuring device 1 and the point to be measured.

Step S53
The pan direction ^A θ _L and the tilt direction ^A φ _L when the distance is obtained are obtained from the electric pan head 4 on which the laser distance measuring device 1 is mounted.

Step S54
The three-dimensional position ^A T _AP expressed in laser camera platform coordinate system A point on the multifocal marker 1 to be measured is obtained by equation (Equation 4).

ただし、^ＬＤ_Ｐ＝（−ｄ_ｌｐ，０，ｄ_ＡＬ）Ｔとし、Ｒ_ｚ（^Ａθ_Ｌ）およびＲ_ｙ（^Ａφ_Ｌ）は、それぞれ、Ｚ軸を中心に^Ａθ_Ｌ、Ｙ軸を中心に^Ａφ_Ｌ回転させる回転行列である。

However, ^L D _P = (− d _lp , 0, d _AL ) T, and R _z ( ^A θ _L ) and R _y ( ^A φ _L ) are ^A θ _L and Y axis centered on the Z axis, respectively. This is a rotation matrix that rotates ^A φ _L around the center.

Next, an algorithm for automatically measuring the position and direction of one perspective marker will be described. This algorithm is an algorithm for automatically measuring the accurate three-dimensional position ^A T _AM and direction ^A R _AM of the center of the perspective marker 1 from the state in which the perspective marker 1 is reflected in the video camera 2.

Step S601
A panning direction ^B θ _M and a tilt direction ^B φ _{M of the} bifocal marker 1 represented by the camera pan coordinate system B using an algorithm for determining the distance between the bifocal marker and the existing direction using a video camera. Ask for.

Step S602
Based on the information obtained in step S601, the shooting direction of the video camera 2 is adjusted so that the near-far marker 1 appears in the center of the image shot by the video camera 2.

Step S603
The focal length of the video camera 2 is adjusted so that the major axis radius r _image of the great circle 12 on the camera image of the far and near marker 1 is 30% of the vertical resolution of the video camera 2. At this time, if the long axis radius r _image before adjusting the focal length is smaller than 20% of the vertical resolution of the video camera 2, the long axis radius r _image is once the resolution of the video camera 2. After adjusting the focal length so that it becomes%, the steps S601 and S602 are repeated, and the shooting direction of the video camera 2 is adjusted so that the center of the bifocal marker 1 is again at the center of the image shot by the video camera 2, Thereafter, the focal length is adjusted so that the major axis radius r _image is 30% of the vertical resolution of the video camera 2. The focal length fa% at which the major axis radius r _image is a% of the resolution in the vertical direction of the video camera 2 is obtained by the equation (Equation 5).

ただし、ｆは、ステップＳ６０１において遠近両用マーカ１を認識したときのビデオカメラ２の焦点距離、ｒは、そのときの遠近両用マーカ１の大円１２のカメラ撮影画像上における長軸半径とする。

However, f is the focal length of the video camera 2 when the near / far marker 1 is recognized in step S601, and r is the major axis radius on the camera image of the great circle 12 of the far / far marker 1 at that time.

Step S604
Steps S601 and S602 are repeated, and the shooting direction of the video camera 2 is adjusted again so that the center of the bifocal marker 1 comes to the center of the camera shot image.

Step S605
Step S601 is executed, the pan direction ^B θ _M to the center of the perspective marker 1 represented in the camera pan coordinate system B, the tilt direction ^B φ _M , and the distance d _CM between the perspective marker 1 and the video camera 2 Ask for. At this time, the center coordinates (x ₁ , y ₁ ) of the bifocal marker 1 on the camera photographed image and the center coordinates (x _si , y _si ) of the four small circles 13 (i = 0, 1, 2, 3) The minor axis radius d of the great circle 12 is stored.

Step S606
^A three-dimensional position vector ^A T _AM at the center of the bifocal marker 1 is obtained by Expression (6).

ただし、^ＣＤ_Ｍ＝（−ｄ_ＣＭ，０，０）^Ｔとし、^ＡＴ_ＡＭは、レーザ雲台座標系Ａからカメラ雲台座標系Ｂへ向かうベクトルであり、事前に計測しておくことが可能である。

However, ^C D _M = (− d _CM , 0,0) ^T, and ^A T _AM is a vector from the laser camera platform coordinate system A to the camera camera platform coordinate system B, and can be measured in advance. Is possible.

Step S607
From the three-dimensional position vector ^A T _AM = ( ^A x _M , ^A y _M , ^A z _M ) of the center of the bifocal marker 1 obtained in step S606, using the equations (Equation 7) and (Equation 8), the laser The pan direction ^A θ _M and the tilt direction ^A φ _M of the center of the bifocal marker 1 represented by the pan-tilt coordinate system A are obtained, and the direction of the laser distance measuring device 3 is directed toward the center of the bifocal marker 1.

Step S608
By reducing the shutter speed of the video camera 2, the overall brightness of the camera-captured image is lowered. This is for facilitating recognition of the laser irradiation position in subsequent processing.

Step S609
Laser irradiation by the laser distance measuring device 3 is stopped.

Step S610
Save the camera image data as I _off .

Step S611
The laser irradiation by the laser distance measuring device 3 is resumed.

Step S612
The camera captured image data after the laser irradiation resume to save as I _on.

Step S613
The difference between the camera-captured image data I _off and I _on is taken, and the area S _diff of the pixels having a difference larger than a predetermined threshold is obtained.

Step S614
If the area S _diff is smaller than a predetermined threshold (the area of the laser irradiation spot), it is determined that the laser irradiation spot is not captured in the video camera 2 and the process proceeds to step S615. If larger, the process proceeds to step S616.

Step S615
Steps S609 to S614 are repeated while rotating the laser irradiation direction at a regular interval in a spiral manner around the three-dimensional position direction of the near-far marker 1 obtained in step S607. However, if the number of repetitions exceeds the specified number, it is determined that automatic measurement has failed.

Step S616
The barycentric coordinate G = (x _G , y _G ) of the pixel obtained by the difference calculation in step S613 is obtained.

Step S617
A vector of the difference between the center-of-gravity coordinates G and the center coordinates (x ₁ , y ₁ ) of the bifocal marker 1 recognized in step S605 is calculated, and the length is short of the great circle 12 of the bifocal marker 1 recognized in step S605. When it is smaller than 1.5 times the shaft radius b, the process proceeds to step S618, and when it is larger, the process returns to step S615.

Step S618
If the vector obtained in step S617 is smaller than a predetermined threshold value, it is determined that the laser is emitted to the center of the bifocal marker 1, and the process proceeds to step S620. If larger, the process proceeds to step S619.

Step S619
Using the vector obtained in step S617, the laser irradiation position is brought close to the center of the bifocal marker 1, and then the process returns to step S609. The direction in which the laser distance measuring device 3 should be directed is determined by the equations (Equation 9) and (Equation 10).

ただし、ｖ_ｈは、ステップＳ６１２の時点でのビデオカメラ２の水平方向の画角、^Ａθ’_Ｍと^Ａφ’_Ｍは、ステップＳ６１１の時点でのレーザ距離計測器３の向きである。

However, v _h is the angle of view in the horizontal direction of the video camera 2 at the time of step S612, and ^A θ ′ _M and ^A φ ′ _M are the directions of the laser distance measuring device 3 at the time of step S611.

Step S620
Obtain an accurate three-dimensional position ^A T _AM in the center of the bifocal marker 1 using an algorithm using a laser distance measuring device for measuring the position of a point on the bifocal marker.

Step S621
The coordinates (x _lsi , y _lsi ) (i = 0, 1, 2, 3) of the area between the great circle 12 and the small circle 13 (white area of the mount) of the bifocal marker 1 are _expressed as 11) and (Equation 12).

ただし、ｒ_{ｓｒｅａｌ}は小円１３の半径、ｄは小円１３の中心と遠近両用マーカ１の中心の間の距離である。

Here, r _real is the radius of the small circle 13, and d is the distance between the center of the small circle 13 and the center of the bifocal marker 1.

Step S622
It performs the same processing as step S609~ Step S620 with respect to four points obtained in step S621, measuring these precise three-dimensional position _{^{A T AMi (i = 0,1,2,3)}} .

Step S623
Vectors ^A X _M , ^A Y _M , and ^A Z _M in the X-axis, Y-axis, and Z-axis directions of the near-far marker 1 are obtained by equations (13) to (15).

ただし、ｉ＝０となる小円１３_０がマーカ座標系上でｘ＞０，ｙ＞０の位置にあり、遠近両用マーカ１を正面からみて時計回りに小円の番号が付けられているものとする。

However, i = 0 and becomes small circles 13 ₀ x on the marker coordinate system> 0, y> in the position of 0, which a bifocal marker 1 as viewed from the front of the number of small circle clockwise is attached And

Step S624
^From the unit vectors ^A X _Mu , ^A Y _Mu , and ^A Z _{Mu of} ^A X _M , ^A Y _M , and ^A Z _M , the direction ^A R _AM of the near-far marker 1 is obtained from the equation (Equation 16).

  In step S617 in the algorithm for automatically measuring the position and direction of this one far / far marker, the distance between the laser irradiation position and the center position of the far / far marker 1 is larger than the far / far marker 1 on the camera image. The reason that step S618 is executed only when the minor axis radius of the circle 12 is smaller than 1.5 times is that, in a complicated environment such as the inside of a nuclear power plant, the laser passes through the back of the bifocal marker 1. This is because it is impossible to determine in which direction the laser irradiation direction should be changed based only on the positional relationship between the laser imaged on the camera-captured image and the bifocal marker 1. On the other hand, when the distance between the laser irradiation position and the center position of the perspective marker 1 is smaller than 1.5 times the short axis radius of the great circle 12 of the perspective marker 1 on the camera image, the laser is Since it can be determined that the far / far marker 1 has already been irradiated and the far / far marker 1 is always flat, the laser irradiation position is always adjusted closer to the center of the far / far marker 1 in step S619. can do. Here, the reason why the radius is 1.5 times the short axis radius is that there is a blank area around the bifocal marker 1 and that area is also a plane.

  In step S621 and subsequent steps, instead of measuring the center position of the small circle 13, the white area between the small circle 13 and the great circle 12 is measured by the black center like the center of the small circle 13. This is because the laser reflection is weak, and the laser distance measuring device 3 may not be able to measure the distance when the distance measuring marker 1 is measured obliquely.

  When this algorithm is executed, it is necessary to provide a grace period (operation pause) of about 0.5 seconds to 1 second after the drive control of the electric pan head before moving to the next process. (This grace period depends on the movement of the electric head.) This is because when the electric pan head is driven and controlled, the entire system vibrates, and it is necessary to wait for the camera image data acquired by the video camera 2 and the laser irradiation destination (laser irradiation spot) to stop. is there.

  Next, an algorithm for converting the marker position and direction to the world coordinate system will be described.

  In this algorithm, the three-dimensional position and direction of the perspective marker 1 represented by the laser head coordinate system A obtained by the algorithm for automatically measuring the position and direction of the one perspective marker 1 described above This is an algorithm for converting to a three-dimensional position and direction represented by a coordinate system W. In executing this algorithm, in addition to the perspective marker 1 to be measured, the first and second perspective markers are pasted to the origin of the world coordinate system W and the point on the X axis, respectively. It is assumed that the number bifocal marker is pasted on the XY plane.

Step S701
Using the aforementioned algorithm for automatically measuring the position and direction of one perspective marker, the three-dimensional position ^A T _AMi and direction ^A R _{AMi of} the conversion target perspective marker 1 and the first to third perspectives are used. The three-dimensional position ^A T _AMj (j = 1, 2, 3) of the marker is obtained.

Step S702
The unit vectors ^A X _Wu , ^A Y _Wu , and ^A Z _Wu in the X, Y, and Z axis directions of the world coordinate system W represented by the laser head coordinate system A are expressed using the equations (17) to (19). Ask.

Step S703
A rotation matrix ^W R _WA from the world coordinate system W to the laser head coordinate system A is obtained by the equation (Equation 20).

Step S704
A parallel movement vector ^W T _WA from the world coordinate system W to the laser head coordinate system A is obtained by Expression (21).

Step S705
A homogeneous transformation matrix ^A T ⁴⁴ _AW from the laser head coordinate system A to the world coordinate system W is obtained by the equation (Equation 22).

Step S706
The three-dimensional position ^A T _AMi of the bifocal marker represented in the laser head coordinate system A is converted into a three-dimensional position ^W T _WMi represented in the world coordinate system W by the equation (Equation 23).

Step S707
Before the process in step S623 in algorithm that automatically measures the position and orientation of one bifocal markers described ^above, to convert A _T AMi the (i = 0, 1, 2, 3) in equation (23) _Thus , the direction ^W R _WMi of the bifocal marker 1 represented in the world coordinate system W is obtained.

  Next, an algorithm for automatically measuring and registering the positions and directions of all the perspective markers will be described. In this algorithm, the three-dimensional position and direction represented by the world coordinate system W of all the bifocal markers 1 within the range that can be photographed by the video camera 2 and the distance can be measured by the laser distance measuring device 3 are fully automatic. It is an algorithm for measuring and storing (registering).

Step S801
The focal length of the video camera 2 is set to the shortest possible value (the video camera 2 is set to the widest angle).

Step S802
The shooting direction of the video camera 2 is rotated at equal intervals so that the entire operating range of the video camera 2 is shot at least once, and in each shooting direction, the above-described video camera is used to By executing an algorithm for obtaining the distance between and the existing direction, the distance and the existing direction are recognized and stored in the working environment.

Step S803
Step S802 is repeated with the focal length of the video camera 2 set to a value twice and three times the shortest value.

Step S804
By executing an algorithm for measuring the position of one point on the perspective marker using the laser distance measuring instrument described above for each of the perspective markers 1 recognized and stored up to step S803, The three-dimensional position and direction of the two-way marker 1 are measured and stored.

Step S805
A three-dimensional position represented in the world coordinate system W by applying an algorithm for converting the marker position and direction to the world coordinate system to the three-dimensional positions and directions of all the perspective markers 1 stored in step S804. The direction is obtained and stored (registered).

  The three-dimensional position and direction of the bifocal marker 1 measured by the above method can be used by the following method.

  When used in a nuclear power plant, which is a work environment, the three-dimensional position and direction of a plurality of markers 1 attached to the periphery of a wall or work target device are measured in advance as described above. It is acquired (stored) in augmented reality as information on the three-dimensional position and direction represented by the world coordinate system W obtained in this way.

  When using HMD (Head Mounted Display) as equipment when using augmented reality, attach a video camera to the front of the MHD or the front of the worker's helmet, and install a small display such as a PDA (Personal Digital Assistant). If used, attach the video camera to the back of the device screen.

  When the distance between the perspective marker 1 and the worker (video camera) is long, a large number of perspective markers 1 appear small on the video camera, and the center of the great circle of the perspective marker can be recognized. Although it is possible, the centers of the four small circles are difficult to recognize. Therefore, only one feature point can be obtained from one perspective marker 1, but feature points of a plurality of perspective markers can be obtained at the same time, and tracking is performed by referring to stored information about the perspective marker 1. It is possible to recognize as many feature points as necessary.

  When the distance between the near / far marker 1 and the worker (video camera) is short, a small number of the far and far markers 1 appear on the video camera and the feature of the small circle can be recognized. All feature points on the dual-purpose marker 1 can be recognized, and the number of feature points necessary for tracking can be recognized only by one single perspective-use marker 1 being captured.

  As described above, the perspective marker 1 used in this embodiment can be effectively used for tracking at both a short distance and a long distance, and the perspective marker 1 to be pasted on the work environment can be used. There is an advantage that the number of markers 1 can be reduced.

  An experimental result relating to measurement accuracy and time required for measurement by the marker automatic registration system exemplified in this embodiment will be described.

  In this experiment, as shown in FIG. 4, using a large printer, a paper on which nine perspective markers 1 are printed is pasted on a wall, and the three-dimensional position and direction of the nine perspective markers 1 are automatically registered. The registration result and the time required for the process of automatically measuring and registering the data were recorded.

Bifocal marker 1, the radius of the great circle 12 and 50 mm, a small circle ₁₃ 0-13 ₃ 6mm radius, the distance between the center and the small circle ₁₃ 0-13 ₃ in the center of the great circle 12 and 71 mm. As shown in FIG. 5, the automatic marker registration system is set at 0 ° from the normal direction of the center 1st bifocal marker 1 at 2m and 4m, and the center 1st bifocal marker 1 as shown in FIG. , 20 ° position, and 40 ° position (6 places in total), and it was repeated 5 times at all positions and all the near-far markers 1 were automatically measured and registered.

  As a result, in all cases, the near / far marker 1 could be measured without failure.

  The time required for the measurement is an average of 195 seconds in the process of rotating and recognizing the position and direction of the bifocal marker 1 by rotating the video camera 2 over the entire movable range (the process of steps S801 to S803), and measuring the laser distance. In the processing (step S804 to step S805) of recognizing and storing the accurate three-dimensional position and direction of the bifocal marker 1 using the device 3, the average per secular marker 1 was 50 seconds.

  During the grace period after vibration control is stopped after driving the electric pan head, measures to reduce the vibration of the system by installing the video camera 2 and the laser distance measuring device 3 using different tripods or using a vibration isolation member If it does, it becomes possible to shorten a grace period and to shorten a required time.

  Table 1 shows a result of comparison between the measurement result of the near / far marker 1 and the true value (calculated from the position and direction of the near / far marker 1 set when printing the near / far marker 1 by a large printer) (coordinate standard). 1 to 3).

  As a result of calculating the root mean square error of the measurement results at all positions where the marker automatic registration system is installed, there is an error of about 2 mm in each of the X, Y, and Z axes with respect to the position. In regard to, it has been found that there is a rotation error of less than 1 ° around each axis (when all axes are combined, the position is 3.4 mm and the direction is an error of 1.1 °).

  FIG. 6 shows the accuracy when tracking is actually performed using the measurement result (registered information) of the bifocal marker 1.

  A tracking video camera with a resolution of XVGA and a horizontal angle of view of about 33 ° is moved about 5.0 m from the position of about 0.5 m in front of the first far-and-far marker 1 by about 0.5 m, and 10 times at each position. The tracking process was repeated using all of the 30 sets of marker arrangement information registered by the method described above, and the mean square error for all the results was obtained. As a result, it was found that the mean square error in the entire tracking range is less than 200 mm, and it is possible to realize an augmented reality environment equipped with a dangerous location display function and the like.

It is a top view of the bifocal marker used in Example 1 of this invention. It is a perspective view which shows the hardware constitutions of the marker automatic registration system in Example 1 of this invention which automatically measures the bifocal marker affixed on the working environment, and registers a measurement result. The coordinate system handled by this marker automatic registration system is shown. It is a front view which shows the state which affixed the paper which printed nine markers for both perspective in Example 1 of this invention on the wall. It is a top view which shows the installation position relationship of the dual-use marker pasted on the wall in Example 1 of this invention and a system. It is a characteristic view which shows the precision at the time of actually tracking using the measurement result (registration information) of the bifocal marker memorize | stored by Example 1 of this invention.

Explanation of symbols

1 ... bifocal marker, 11 ... mount, 12 ... great circle, ₁₃ 0-13 3 _... small circle, 2 ... video camera, 3 ... laser distance measuring device, 4 ... electric pan head, 5 ... small computer, 7 ... tripod .

Claims (2)

Paste the marker serving as a reference position 置及 beauty appropriate position of the environment, represented in the world coordinate system of all of the markers in the distance shooting and laser distance measuring device with a video camera to the extent possible measurement 3 It is a method of measuring and storing the dimension position and direction fully automatically,
The marker includes one large circle arranged at the center of a square mount and four small circles arranged at the four corners of the mount. By using a configuration in which the ID number of each marker is expressed by setting the white sector to 0 and the black sector to 1.
A first step of setting the focal length of the video camera to the shortest possible value;
The shooting direction of the video camera is rotated at equal intervals so that all areas of the operating range of the video camera set to the shortest settable focal length are shot at least once. A second step of recognizing and storing the distance to the marker pasted in the environment and the existing direction by executing an algorithm that captures the distance between the marker and the existing direction by photographing with a camera When,
A third step of repeating the second step by setting the focal length of the video camera to a value twice and three times the shortest value;
By executing the algorithm for measuring the position of a point on the marker using a laser distance measuring instrument for each marker stored recognized by the third step from the first step, all recognized A fourth step of measuring and storing the three-dimensional position and direction of the marker;
Applying an algorithm that converts the marker position and direction to the world coordinate system to the three-dimensional position and direction of all markers stored in the fourth step, the three-dimensional position and direction expressed in the world coordinate system The marker automatic registration method characterized by performing the 5th step which calculates | requires and memorize | stores. Of the markers that are pasted at appropriate positions and the reference position of the environment, all markers in the range that can be photographed with a video camera and that can be measured with a laser distance meter are expressed in the world coordinate system. A system that automatically measures and stores 3D positions and directions,
Connected to a video camera that can be turned by an electric head, a laser distance measuring device that can be turned by an electric head, and the video camera and the laser distance measuring device. And a small computer that controls them,
The marker includes one large circle arranged at the center of a square mount and four small circles arranged at the four corners of the mount. The white fan shape is set to 0 and the black fan shape is set to 1, so that the ID number of each marker is expressed.
The small computer is
A first step of setting the focal length of the video camera to the shortest possible value;
The shooting direction of the video camera is rotated at equal intervals so that the entire range of the operating range of the video camera set to the shortest settable focal length is shot at least once. A second method for recognizing and storing the distance to the marker pasted in the environment and the existing direction by executing an algorithm for obtaining the distance between the marker and the existing direction by photographing using a video camera Steps,
A third step of repeating the second step by setting the focal length of the video camera to a value twice and three times the shortest value;
By executing an algorithm for measuring the position of one point on the marker using the laser distance measuring device for each marker recognized and stored in the first step to the third step, A fourth step of measuring and storing the three-dimensional position and direction of the marker;
Applying an algorithm that converts the marker position and direction to the world coordinate system to the three-dimensional position and direction of all markers stored in the fourth step, the three-dimensional position and direction expressed in the world coordinate system An automatic marker registration system comprising a program for performing a fifth step for obtaining and storing

Images