JP2011252917A - Method for arranging marker in on-site operation support system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for arranging a marker to achieve a reduction in tracking error in an on-site operation support system.SOLUTION: An on-site operation support system comprises line markers 5a and 5b positioned in a work site in advance, cameras 1a to 1c carried by an operator, a HMD2 which provides the operator with an image of the work site and instruction information by displaying them thereon, an information processing apparatus 3 which measures a position and a direction of the camera (operator) based on image positions of the line markers in photographic images obtained by taking pictures of the work site with the cameras and performs information processing for displaying an information image to be given to the operator on the HMD with the image of the work site superimposed thereon. Positions of the line markers are determined as a position where line marker positions can be calculated repeatedly to reduce errors, by (1) estimating a range of errors included in a code mark obtained by the image processing and (2) estimating tracking errors in a variety of line marker arrangements using results obtained from (1), an error calculation method, and a genetic algorithm.

Description

  The present invention relates to a marker placement method in a field work support system that supports field work using augmented reality (AR) technology.

  Augmented reality technology is a technology that superimposes and displays CG (Computer Graphics) information on a real scene. It is a field work support system that provides field workers with information useful for field work and supports field work. Applied.

JP 2007-34238 A JP 2007-18188 A

  In order to support the worker's work using the augmented reality technology, high-precision tracking (technology for measuring the position and direction of the worker in the work site in real time) is required.

  As the current tracking technology, a marker is affixed to the work site, and the position and orientation of the camera (worker) are measured based on the marker in the photographed image obtained by photographing the work site with the camera carried by the worker. However, it is difficult to perform tracking with high accuracy in the entire work site, and the measurement accuracy varies depending on the location. If information (for example, a work instruction) is provided to an operator in a state where the measurement accuracy is low, there is a risk of causing a work mistake due to an incorrect work instruction.

  An object of the present invention is to propose a marker arrangement method that is preferable for realizing a reduction in tracking error in a field work support system.

The present invention relates to a line marker previously set at a work site, a camera carried by the worker, a display device that displays an image of the work site and instruction information and provides the worker, and photographs the work site with the camera. The display device measures the position and orientation of the camera (worker) based on the image position of the line marker in the captured image obtained in this manner, and superimposes the information image provided to the worker on the image of the work site. In the method of arranging line markers in a field work support system equipped with an information processing device that performs information processing to be displayed on
The line marker is arranged at a position determined by performing an optimization arrangement process using a genetic algorithm so that a measurement error of the position of the camera is minimized.

Specifically, for line markers,
(1) Estimating the size of an error included in a code mark obtained by image processing,
(2) Using the results of (1), the error calculation method and the genetic algorithm, estimate the size of tracking errors in the work site in various line marker arrangements, and gradually increase the accuracy of line marker arrangements Is obtained by iterative calculation.

  The present invention relates to a line marker previously set at a work site, a camera carried by the worker, a display device that displays an image of the work site and instruction information and provides the worker, and photographs the work site with the camera. The display device measures the position and orientation of the camera (worker) based on the image position of the line marker in the captured image obtained in this manner, and superimposes the information image provided to the worker on the image of the work site. In the arrangement of the line marker in the field work support system including an information processing apparatus that performs information processing to be displayed on the line marker, it is possible to arrange the line marker preferable for reducing the tracking error.

It is a functional block diagram of the field work support system of Example 1 in the present invention. It is a top view which shows the arrangement | sequence state of the camera in the field work assistance system of Example 1. FIG. It is a top view of the line marker in the field work support system of Example 1. It is a picked-up image for demonstrating the error which generate | occur | produces with the lens correction | amendment state of a camera. 3 is a flowchart of information processing executed by the information processing apparatus according to the first embodiment. It is a figure which shows the code mark image sequence of the line marker currently reflected in the picked-up image, and the projected mark position of the code mark image of both ends of the code mark image sequence calculated | required by calculation. It is drawing which shows the content which calculates | requires the distance between the straight line approximated from the code mark image sequence in a picked-up image, and two ends of a code mark image sequence. It is drawing which shows the content which calculates | requires the difference | error of the distance between each code mark image of the line marker calculated | required from the image recognition result. It is a figure which shows the coordinate system in the calculation algorithm which estimates the magnitude | size of a tracking error. It is a conceptual diagram of TER in the calculation algorithm which estimates the magnitude | size of a tracking error. It is a conceptual diagram of the calculation method of the error limit curve regarding x coordinate in the calculation algorithm which estimates the magnitude | size of a tracking error. It is a conceptual diagram of the calculation method of the error limit curve in the calculation algorithm which estimates the magnitude | size of a tracking error. It is a conceptual diagram of the calculation method of the error limit curve regarding y coordinate in the calculation algorithm which estimates the magnitude | size of a tracking error. It is a conceptual diagram of the method of calculating | requiring the area | region (TER) enclosed by all the error limit curves in the calculation algorithm which estimates the magnitude | size of a tracking error. It is a schematic diagram of the pseudo plant environment used in the evaluation experiment. It is a curve figure which shows an experimental result (The average value of the maximum value of the error in each generation, the best value, and the worst value). It is a perspective view which shows the line marker installation position in the example of application in three dimensions. It is a distribution chart of tracking error in an application example.

The present invention relates to a line marker previously set at a work site, a camera carried by the worker, a display device that displays an image of the work site and instruction information and provides the worker, and photographs the work site with the camera. The display device measures the position and orientation of the camera (worker) based on the image position of the line marker in the captured image obtained in this manner, and superimposes the information image provided to the worker on the image of the work site. As a preferable marker arrangement method for realizing a reduction in tracking error in the line marker arrangement method in an on-site work support system having an information processing apparatus for performing information processing to be displayed on the camera, the measurement error of the camera position is minimized. As shown in the figure, it is realized by arranging at a position determined by performing an optimized arrangement process by a genetic algorithm. For the marker,
(1) Estimating the magnitude of an error included in a code mark obtained by image processing,
(2) Using the results of (1), the error calculation method and the genetic algorithm, estimate the size of tracking errors in the work site in various line marker arrangements, and gradually increase the accuracy of line marker arrangements Is realized by repeatedly calculating.

  FIG. 1 is a functional block diagram of the field work support system.

  The camera 1 for photographing the work site (working facility) is a forward direction of the face of the worker wearing the HMD 2 on the head of an HMD (Head Mounted Display) 2 with an AR function worn by the worker (worker (Viewing direction) is attached.

  Although this embodiment will be described with an example in which the display device uses an HMD, the present invention is not limited to the HMD and can be implemented even if a tablet PC is employed. Also, the orientation of the camera is not necessarily the orientation of the operator's face because of the relationship with the position where the line marker is affixed. When installing the line marker on the ceiling surface, the camera should be pointed upward. It can also be modified so that the marker is photographed for tracking.

  As shown in FIG. 2, the camera 1 is configured such that three cameras 1a, 1b, and 1c are arranged side by side so that the work site can be photographed at a wide angle. In this embodiment, an example in which three cameras are used will be described. However, the embodiment can be modified so that two or four cameras are used.

  An information processing apparatus (portable personal computer) 3 that executes various types of information processing for on-site work support basically works by inputting a photographed image from the camera 1 and performing image processing. The position and orientation of the person is measured, and information processing is performed in which the CG image of the work site generated based on the photographed image of the work site or the design information stored in advance and the instruction information are displayed on the display screen of the HMD 2.

  Various kinds of equipment 41 (41a, 41b...) Are installed at the work site 4 based on the design information, and the line marker 5 (5a, 5b...) Is known in advance for some of the equipment 41a, 41f. Affixed to the position. The line marker 5 is used as image information for calculating the position and orientation of the camera 1 (the position and orientation of the operator) from the position in the captured image of the camera 1.

  As shown in FIG. 3, each line marker 5 is provided with 11-bit code marks 52 (52a to 52k) arranged at predetermined intervals in the length direction on the surface of an elongated belt-like base material 51, and the back surface is provided in the facility. It is configured to be attached to the surface of several devices 41a, 41f. Normally, it is desirable to install it vertically and vertically at a predetermined height position.

  In this embodiment, the line marker 5 has an example in which the code mark 52 has 11 bits, but the number of bits can be freely changed as necessary. Incidentally, if the number of bits is increased, the number of bits that can be corrected for errors can be increased, or the number of line markers that can be used at one time can be increased.

  The code mark 52 provided in each line marker 5 represents bit data in which a 30 mm × 30 mm square is “0” and a 30 mm × 60 mm length rectangle is “1”, and the 7 bits from the top are the marker ID bits, The lower 4 bits are used as error correction bits. The code data of the illustrated line marker 5 is “01010000101”.

  In this embodiment, the code mark 52 is shown as a 30 mm × 30 mm square and a 30 mm × 60 mm long rectangle, but the size of the place where the line marker 5 is pasted and between the line marker 5 and the camera 1 are shown. Thus, the size of the code mark 52 can be changed uniformly for all the code marks 52.

  On the display screen 21 of the HMD 2, the instruction information image 21 b is displayed in a state of being superimposed on the photographed image (CG image) 21 a of the work site 4. In the example shown in the figure, instructions for opening the devices (valves) 41g and 41h installed at the work site are instructed, and instruction information for instructing to prohibit the opening operations for the devices (valves) 41i to 41k is displayed. It is.

  Such image display is performed based on the image position information of the code mark image of the line marker 5 in the captured image (information) of the work site 4 input from the camera 1 by the information processing device 3. The position and orientation are measured, and the equipment 41 of the work site 4 is added to the CG image 21a of the work site generated on the basis of the photographed image of the work site 4 or the design information stored in advance on the basis of the position and orientation of the worker. This is realized by reading out and superimposing work instruction information (work instruction information image and work prohibition instruction information image) 21b1 and 21b2 in which work contents to be performed are stored in advance.

  Thus, in order to measure the position and orientation of the worker at the work site based on the code mark image position information of the line marker 5 in the captured image of the work site 4, the code mark image of each line marker 5 is accurate. Must be present in the captured image 21a in a unique shape. However, if the distortion correction of the lens of the camera 1 is insufficient, an error occurs when the captured image is processed to recognize the position of the line marker. When the position and orientation of the camera 1 (worker) are calculated based on the position information including the error, an error naturally occurs.

  In addition, the error further includes an attachment error of the line marker 5 to the equipment 41 at the work site 4 (an error between preset attachment position information and an actual attachment position) and an error in calculation processing (the information processing apparatus 3 handles the error). And other factors such as restrictions on the number of digits and errors caused by stopping optimization calculations in the middle).

  The present invention provides a measurement error caused by distortion of the lens of the camera 1, the camera resolution being finite, error or error in the information on the position where the line marker is pasted, incomplete illumination, etc. It is possible to prevent erroneous instruction information from being provided (displayed) to the worker (HMD2) due to a tracking error caused by.

  With reference to FIG. 4, an error that occurs due to insufficient lens correction of the camera 1 will be described.

  4A shows a code mark image sequence 5a of the line marker 5 in the photographed image 21a photographed by the camera 1 in which the lens correction is complete, and the code mark images 53 are arranged in a straight line. It is reflected in the state. However, the code mark images 54 of the code mark image sequence 5b of the line marker 5 in the photographed image 21a photographed by the camera 1 with insufficient lens correction are bent and arranged as shown in FIG. It is reflected in.

  The field work support system of the first embodiment is configured to control (edit) information to be provided to the worker by estimating the magnitude of such an error (tracking error) due to distortion of the lens of the camera 1. .

  FIG. 5 is a flowchart of information processing executed by the information processing apparatus 3 to realize such control.

Step S301
Captured image information of the work site (inside the facility) is acquired from the camera 1.

Step S302
By processing the acquired captured image information, the code mark image sequence of the line marker 5 is recognized, and the barycentric position of each code mark image is obtained. This process is performed on the code mark image sequences of all the line markers 5 present in the captured image.

  Specifically, the image is binarized and labeled, a square and a rectangular area are searched, and a square and a rectangular area are linearly searched for a predetermined number (the number of code marks of the line marker 5). It recognizes as a code mark image sequence (line marker 5), and recognizes the marker ID with the square of the code mark image sequence as “0” and the rectangle as “1”. Further, the positions of the center of gravity of the code mark images at both ends of the code mark image sequence are set to the positions at both ends of the code mark image sequence (line marker 5).

Step S303
The processing result of step S302 is corrected using a distortion parameter known in advance for the lens of the camera 1.

Step S304
The relative positional relationship between the line marker 5 and the camera 1 is calculated (tracked) from the position of the code mark image sequence in the captured image.

Step S305
Using the position information of the camera 1 obtained in the process of step S304, various parameters of the camera 1, and the pasting position information of the line marker 5, the position where the code mark image sequence should be reflected is calculated. (Reverse calculation).

  FIG. 6A shows the positions of the code mark images at both ends recognized as the code mark image sequence of the line marker 5 reflected in the photographed image, and FIG. 6B shows the code mark image sequence obtained by calculation. Shows the projected positions of the code mark images at both ends.

Step S306
The distance d on the photographed image between the position of the end of the code mark image sequence recognized by the image processing (FIG. 6A) and the position of the end of the code mark image sequence reversely calculated from the tracking result (FIG. 6B). ⁱ _{Determine RP} . Here, i is the serial number at the end of the code mark image sequence (the maximum value is twice the number of code mark image sequences), and the maximum value of d ⁱ _RP is d ^MAX _RP .

Step S307
The code mark image sequence is approximated by a straight line using the barycentric position of each code mark image of the line marker 5 obtained in step S302, and a distance d ⁱ _RV between the approximated straight line and the two ends of the code mark image sequence is obtained. And let d ^MAX _RP be the maximum value of d ⁱ _RV .

7A to 7D show the center of gravity of each code mark image from the code mark image sequence (a) in the photographed image in the previous step, and in this step 307, each code mark is obtained by the least square method. The content of _obtaining a straight line closest to the center of gravity of the image and then _obtaining the distance d ⁱ _RV between the approximated straight line and the two ends of the code mark image sequence is shown.

Step S308
Each code mark image that can be calculated from the distance between the code mark images obtained from the barycentric position of each code mark image of the line marker 5 obtained in step S302 and the actual dimension of the line marker 5 indicated by the marker ID of the line marker 5 A distance difference d ⁱ _{RH is obtained} for each code mark image (see FIG. 8). Here, the maximum value of d ⁱ _RH is defined as d ^MAX _RH .

Step S309
The maximum error (maximum error estimated value) d ^MAX that may be included in the position of the code mark image sequence recognized from the photographed image is obtained using the equation (1).

Step S310
Estimate the magnitude of the expected tracking error for each tracking. A calculation algorithm for estimating the magnitude of the tracking error will be described later.

Step S311
Information displayed on the HMD and provided to the operator is controlled according to the estimated tracking error. For example, when the estimated tracking error is smaller than a predetermined value, work instruction (work prohibition) information is displayed and provided to the worker, and when the estimated tracking error is greater than a predetermined value, the work instruction information is not displayed and the tracking error is displayed. Warning information for instructing to change the orientation of the camera 1 (orientation of the operator's face) is displayed in order to input photographed image information for reducing.

  The size of the tracking error in step S310 can be estimated by, for example, the following calculation algorithm.

In the tracking error calculation algorithm used in this embodiment, the method for calculating the tracking error needs to satisfy the following requirements.
(1) When using augmented reality inside the plant, if there is a possibility that an error larger than an assumed error may occur, an incorrect instruction may be given to the worker, which is dangerous. Therefore, the tracking error calculation method needs to be able to calculate an absolute result such as “maximum error is 13 mm” instead of a probabilistic result such as “measurement error within 10 mm at 90% establishment”. is there.
(2) In optimization of line markers, it is necessary to repeatedly calculate errors in various line marker arrangements. Therefore, the error calculation method needs to be sufficiently fast.

A user (worker) who uses augmented reality assumes work in a power plant. In addition, the contents of the support are also assumed to be information presentation when viewing the plant equipment, such as navigation to a specific work site and instructions on the amount and type of fluid flowing in the piping. In this case, the position of the worker's head (the position of the camera worn by the worker) is
(1) The height is almost constant (about the height of the worker).
(2) Swing rotation from side to side and up and down is performed frequently, but tilting the head is rare.
It is thought that there is a feature.

  Therefore, in the first embodiment, the tracking error calculation algorithm is simplified using these features.

Coordinate system and various definitions Figure 9 shows the definition of the coordinate system. The origin of the screen coordinate system is the center of the camera photographed image, the x axis is from the left to the right of the camera photographed image, and the y axis is the upward direction. The origin of the camera coordinate system is the focal position of the camera, the z-axis direction is the shooting field of view of the camera, and the x-axis and y-axis are parallel to the screen coordinates.

  The position and orientation of the camera is represented by a vector of 6 degrees of freedom (3 degrees of freedom at the camera position and 3 degrees of freedom at the camera orientation).

  In tracking using a camera and a line marker (code mark image sequence), there may be a case where a plurality of feature points serving as a calculation reference are recognized from one line marker and used, but in this embodiment, one line marker is used. Assuming that one feature point is recognized from the following, line marker and feature point are synonymous in the following unless otherwise noted.

Let C be the true position and orientation of the camera, and Q _i (i = 1, 2,..., N) be the three-dimensional position of the line marker. (N is the total number of markers reflected in the camera). Now, when a line marker is photographed with a camera in the position and orientation of C, it is assumed that the line marker is recognized at the position M _i (i = 1, 2,..., N) on the camera photographed image (M _i Is a two-dimensional vector). The result of estimating the position and orientation of the camera using this result is Cest. _Assume that the position Mi includes an error of Δ at the maximum due to the fact that the camera resolution is finite and the lens correction is not sufficient. At this time, a set of camera positions and orientations C ′ where the position M _i ′ (i = 1, 2,..., N) of the line marker on the camera-captured image satisfies the following formula (Equation 2) is This is an area having a certain size around Cest.

  In the first embodiment, this area is called a TER (Tracking Error Region).

  FIG. 10 shows a conceptual diagram of TER. Since the true position C of the camera is included in this TER area, the calculation algorithm for obtaining the maximum value of the tracking error finds the point farthest from Cest in the TER area. This is an algorithm for obtaining a difference in distance.

Calculation of error limit curve In the tracking error calculation algorithm used in the first embodiment, the height at which the camera moves does not substantially change depending on the height of the operator, and only the left / right / up / down rotation is performed. TER is obtained in the following flow.

(1) For all two combinations of line markers recognized by image processing, the x coordinate on the camera-captured image of the line marker always falls within the range of x ₁ ± Δ and x ₂ ± Δ, respectively. The boundary of the set of positions and orientations (referred to as error limit curves) is obtained (x ₁ and x ₂ are the x coordinates of two line markers appearing on the camera-captured image when the camera is at the true position).

  (2) For all line markers recognized by image processing, an error limit curve of a set of camera positions and orientations in which the y coordinate on the camera-captured image of the line marker always falls within the range of y ± Δ is obtained (y Is the y coordinate of the line marker that appears on the camera image when the camera is in the true position).

  (3) The region surrounded by the error limit curve obtained in (1) and (2) is defined as TER.

  In the following description, it is assumed that n line markers have been captured by the camera and their positions have been recognized (the value of n is 3 or more).

Calculation of Error Limit Curve for X Coordinate In FIG. 11, point P _A and point P _B are represented by x− of line markers Q _A (x _a , y _a , z _a ) and Q _B (x _b , y _b , z _b ). Let it be a projection point on the z plane.

Point A is the intersection of line segment CP _A and the x axis of the camera screen, and point B is the intersection of line segment CP _B and the X axis of the camera screen. The points A, A ₁ , A ₂ , B, B ₁ , B ₂ are all on the X axis of the camera screen, and ‖A ₁ A‖ = ‖AA ₂ ‖ = ‖B ₂ B‖ = ‖BB ₁ ‖ = △. At this time, the true coordinate x coordinate of the line marker on the camera-captured image is between the points A ₁ and A ₂ and between the points B ₁ and B ₂ . That is, the true position and orientation C of the camera is included in the region formed by the position and orientation C ′ that satisfies the following expression.

Here, α ₂ = ∠A ₂ CB ₂ and α ₁ = ∠A ₁ CB ₁ . These two angles can be obtained by calculating the following four equations.

Here, D is the center of the camera screen, and f is the focal length of the camera lens. _A set of positions and orientations C ′ satisfying ∠P _A C′P _B = α ₁ and ∠P _A C′P _B = α ₂ is a circle as shown in FIG. Accordingly, a set of position and orientation C ′ satisfying the equation (Equation 3) is a gray area in FIG.

Since the method for obtaining Circle 1 and Circle 2 in FIG. 12 is the same, the method for obtaining Circle 1 is shown below. First, find the center of Circle 1. It is assumed that the point F is a point on Circle 1 and satisfies ‖P _A F‖ = ‖P _B F‖. Also, let E be a point on the line segment P _A P _B and satisfy ‖P _A E‖ = ‖P _B E‖. At this time, the line segment EF is a perpendicular bisector of the line segment P _A P _B. Therefore, the center O (x _o , 0, z _o ) of Circle 1 exists on the line segment EF. Here, since ‖OF‖ = ‖OP _B ‖ = R and ∠P _A FP _B = α1 (R is the radius of Circle ₁ ), ∠OFP _B = ∠OP _B F == α _1/2 holds. . Accordingly, the radius R of Circle 1 can be obtained by the following equation.

The angle between the line segment P _A P _B and the x axis can be calculated by γ == atan {(z _b −z _a ) / (x _b −x _a )}, and the angle between the line segment P _A O and the x axis is It can be calculated by θ = − (π / 2−α ₁ −γ). Therefore, the center of Circle 1 can be calculated by the following equations (Equation 9) and (Equation 10).

  The circle obtained by the above method is hereinafter referred to as an error limit curve related to the X coordinate.

In the calculation 13 of the error limit curve for y coordinates, the point P _A, the projection points on the x-z plane of the line markers Q _A, the point A and the intersection of the line segment CQ _A and the screen. Points A ₁ and A ₂ are points on the screen that satisfy ‖A ₁ A‖ = ‖AA ₂ ‖ = Δ, and line segment A ₁ A ₂ is assumed to be parallel to the y-axis of the screen. Point D is the center of the screen (‖CD‖ = f). Point E is a projection point of point A onto the xy plane.

Calculation Method of Error Limit Curve for y Coordinate At this time, the true y coordinate on the camera-captured image of the line marker is between point A ₁ and point A ₂ . That is, the true position / posture C of the camera is always included in the region formed by the position / posture C ′ that satisfies the following expression.

である。以上の方法で求められる円を以下ではｙ座標に関する誤差限界曲線と呼ぶ。 Where β ₂ = ∠A ₂ CP _A and β ₁ = ∠A ₁ CP _A , β ₂ = ∠Q _A C′P _A and C ′ satisfying Q Q _A C′P _A = β ₁ are both points becomes a circle centered on P _a, the radius of the circle can be calculated by ‖Q _a P _a ‖ctanβ ₂ and ‖Q _a P _a ‖ctan∠β _1. here

It is. The circle obtained by the above method is hereinafter referred to as an error limit curve regarding the y coordinate.

Method of calculating tracking error from TER The true position of the camera is included in the area surrounded by all the error limit curves obtained previously. Therefore, if the distance between the point farthest between the area surrounded by all error limit curves and the true position of the camera and the true position of the camera can be obtained, this will occur when tracking is performed. The maximum possible error can be determined. FIG. 14 shows a conceptual diagram of a method for obtaining a region (TER) surrounded by all error limit curves. Hereinafter, in order to simplify the description, a case will be described in which the number of line markers recognized in the camera-captured image is three. When the number of line markers is n, only the number of circle combinations when calculating the intersection of circles is increased in the calculation method when the number of line markers is three.

As shown in FIG. 14, line markers _{Q A,} Q B, respectively _P A the projection point to the x-z plane in the camera coordinate system of _{Q C,} P B, and _{P C.} Area 1 shown in FIG. 14 is an area surrounded by an error limit curve regarding the x-coordinates of the line markers Q _A and Q _B , and the true camera position and orientation are included in this area. Q _A and _Q C, obtaining the same region with respect to the combination of _{Q B} and _{Q C.} Area 2 is an area surrounded by the error limit curve for y coordinates of the line marker Q _A, include the position and orientation of the real camera in this region. Obtaining the same region with respect to Q _B and _{Q C.}

  An area that is commonly included in the areas obtained as described above (in this case, a total of 6 areas) is TER, and the true camera position and orientation are included in this area. As described above, the distance between the position of the camera obtained using the recognized position of the line marker including an error and the position farthest in the TER area is the maximum error that can occur when tracking is executed. . Here, the property that the position in the TER region farthest from the camera position obtained by using the recognition position of the line marker including an error is necessarily one of the intersections of the error limit curves. Then, the maximum value of errors that can occur when tracking is executed can be obtained.

To summarize the above flow, when the total number of recognized line markers is n, nC ₂ × 2 error limit curves related to the x-coordinate are obtained in order to obtain the maximum value of errors that can occur when tracking is performed. It is necessary to obtain 2n circles as error limit curves with respect to the y-coordinates, and obtain intersection points with respect to combinations of all these circles (total n (n + 1)). Accordingly, the amount of calculation required to obtain the tracking error increases in proportion to the square of the number of recognized line markers, but is about 10 even when the total number of recognized line markers is large. The calculation amount is not so much. When the total number of recognized line markers is 10 or more, the number of line markers that are not used for tracking increases, which is inefficient. In this case, the number of line markers to be installed on the work site should be reduced.

  Next, an application example of this embodiment will be described with reference to FIGS.

  As shown in FIG. 17, six line markers (length: about 60 cm) 5 are attached and installed in a 10 m × 15 m room, and the camera (Dragfly B & WXGA, f = 6.37 mm) 1 is latticed at intervals of 1 m. The result shown in FIG. 18 was obtained by calculating the difference between the actual camera position (measured manually) and the tracking result as a tracking error.

  Here, optimization of the line marker arrangement in the facility (work area) will be described.

Line marker force optimization algorithm The goal of the line marker placement optimization algorithm is to adjust the location of the line marker when the augmented reality worker moves within the area, and to track within that area. Is to improve the accuracy. Finding the optimal line marker placement directly by analytical calculation based on information about the area where the operator moves and the area where the line marker can be pasted is very complicated because the calculation method becomes very complicated. Have difficulty. Therefore, in this embodiment, the line marker attachment position is determined by repeated optimization calculation using a genetic algorithm. At that time, considering that the application destination of the assumed tracking method is work support in the plant, the area where the line marker can be attached is considered to be the surface of a slender pipe, a plane such as a wall or ceiling. It is done. Therefore, here, the shape of the region where the marker can be pasted is determined to be two types, a straight line and a plane.

で定義される。ここで、ｎは環境に貼り付けるラインマーカの個数、ｆは染色体の個体番号、ｔは世代番号である。すなわち、ｍ^ｔ _ｉｋは、ｔ番目の世代として求められた、f番目のラインマーカ配置におけるｋ番目のラインマーカの貼り付け位置を表す。 Chromosome coding Each chromosome in the genetic algorithm used in this embodiment has the same number of genes as the number of line markers to be attached to the environment, and one gene has position information of one line marker. Chromosome ^M _{t i} is,

Defined by Here, n is the number of line markers to be attached to the environment, f is the individual number of the chromosome, and t is the generation number. That is, m ^t _ik represents the pasting position of the k-th line marker in the f-th line marker arrangement obtained as the t-th generation.

The evaluation function e = g (x, M) is a tracking error at the position and orientation x in the line marker arrangement M. In this embodiment, the tracking error calculation method described above is used for this calculation. For this purpose, the line marker arrangement, camera resolution, camera pixel size, lens focal length, etc. It is necessary to determine which line marker is recognized. This can be easily determined by obtaining the visual field and direction of the camera by analytical calculation and examining whether or not the line marker is in the visual field based on the arrangement information of the line marker. However, in practice, even if a line marker is in the camera's field of view, it may not be recognized because the line marker is too small on the camera image. If the distance between the markers is greater than or equal to a certain threshold value, it is necessary to determine that the line marker is not recognized. However, in the following, in order to simplify the explanation, it is assumed that all the line markers that have entered the field of view of the camera are recognized regardless of the distance.

  In using a genetic algorithm to optimize the placement of line markers, the evaluation function should be set according to optimization criteria. In this embodiment, it is considered to optimize the arrangement of line markers when using augmented reality inside the plant, but the accuracy of the information presented to the worker in that case becomes a problem. In other words, tracking accuracy is poor, and providing incorrect position and direction information to an operator can cause human error, so this should be avoided. Therefore, in this embodiment, as one example of the line marker arrangement optimization, the objective of setting the evaluation function this time is to “minimize the largest error in the entire range in which the operator moves”. To do.

  When the goal of optimizing the line marker arrangement is determined, tracking position and orientation can be evaluated with respect to all possible positions and orientations that can occur within the range of movement of the operator, and the maximum value can be obtained. Although it is ideal, for this purpose, it is necessary to repeat calculation of tracking errors in a huge position and orientation, and the calculation amount becomes enormous, which is not realistic. Therefore, in this embodiment, the tracking error is calculated with respect to a predetermined finite position and orientation, and the arrangement of the line markers is changed in the direction of decreasing the maximum value among the results. That is, in this embodiment, the evaluation function in the genetic algorithm is defined by the following formula.

Here, x _j is the position and orientation of the j-th camera, and n _p is the total number of positions and orientations used for evaluation.

  When this method is used, there is a possibility that the tracking accuracy increases at a position and orientation selected in advance, and the tracking accuracy decreases at other positions and orientations. However, in general, when the types and number of line markers appearing in the camera are the same, the tracking accuracy of adjacent positions and orientations tends to be similar, and tracking around the positions and orientations selected in advance also tends to be similar. The accuracy is expected to improve. Therefore, many positions and orientations are selected around the area where high tracking accuracy is required, depending on the characteristics of the work supported by augmented reality and the arrangement of the area. If the posture is selected, it is considered that a line marker arrangement having a desired characteristic can be obtained. However, this method cannot guarantee an accuracy such as “the maximum value of the tracking error is 10 mm”. Therefore, when the line marker optimization method using the genetic algorithm proposed in this embodiment is put into practical use, as described above, even when tracking is actually performed, the line marker arrangement information and tracking are used. Based on the obtained position and orientation, the above-described tracking error calculation algorithm should be executed to estimate the magnitude of the error and determine whether or not to present information based on the result.

Initialization In this embodiment, as the first generation of the genetic algorithm, the number of chromosome genes is the same as the number of regions to which line markers can be attached (one for each region where line markers can be placed). One line marker is arranged), and each gene has position information randomly determined within a region where the line marker can be pasted.

Re-initialization In this embodiment, in order to randomly determine the initial arrangement of line markers at the time of initialization, among the positions and orientations to be evaluated, there are positions and orientations where a sufficient number of line markers are not reflected in the camera. There is a possibility. In this case, if the initial arrangement of the line markers is repeatedly determined several times, a sufficient number of line markers may appear in the camera, but a sufficient number of line markers may not always appear. Absent. Therefore, in this embodiment, when a position and orientation that does not capture a sufficient number of line markers on the camera at the time of initialization is set, one new line marker is arranged so that it is reflected on the camera at that position and orientation. (The position is randomly determined from within the area where the line marker can be pasted). This process is called reinitialization in this embodiment. At this time, the number of genes of the chromosome in the genetic algorithm is increased by one. If a new line marker cannot be pasted so that it appears in the camera at the position and orientation in question due to restrictions on the area where the line marker can be pasted, it is determined that tracking in the target area cannot be realized. Is done. In this case, it is necessary to secure a new area where the line marker can be pasted.

Crossover Crossover is performed by selecting two parent chromosomes at random and replacing them at a certain point in the same way as a commonly used genetic algorithm. That is, when h is a random integer in the range of [1, n] Crossover is expressed by the following equation.

Mutation The mutation in this example is expressed by the following formula:

Here, h is a random integer in the range of [1, n], and m ^{t + 1} _ih is a new position candidate (a preset line marker can be pasted) for pasting the h th line marker. Select at random within the range of the area).

Selection The selection in this example uses roulette selection, i.e., the probability that the 'th chromosome is selected is calculated by the following equation.

Here, f (M _i ) and f (M _k ) are evaluation function values of the i-th and k-th chromosomes, and p is the total number (population) of chromosomes. In this embodiment, as in the case of a general genetic algorithm, excellent chromosomes are left according to the probability, and non-excellent chromosomes are deleted.

Setting the total number of chromosomes and the number of generations in the genetic algorithm The total number of chromosomes to be generated when the genetic algorithm is executed should be determined according to the total number of line markers that are attached to the environment. That is, in order to evaluate k position candidates for one line marker, if the total number of line markers is n, at least nk chromosomes should be generated.

  In general, in genetic algorithms, the more generations that perform crossover, the more appropriate results are obtained. However, when crossing up to a very large number of generations, the amount of calculation increases and the time required to obtain the result becomes longer. A detailed discussion on which generation should be cut off is not given in this example. However, considering the use of augmented reality in the actual field, it is not always necessary to find the optimal line marker arrangement. If a line marker arrangement that satisfies the accuracy required in (1) is required, it may not be optimal. Therefore, the number of generations executed by the genetic algorithm varies depending on the performance of the computer used and the time that can be spent, but basically, the required accuracy is based on the value obtained by the evaluation function. It is considered appropriate for each generation to determine whether or not a line marker arrangement that can realize tracking is obtained, and if it has already been obtained, the calculation is cut off.

Trial of the Proposed Method and its Results In order to confirm the effect of the line marker optimization method using the genetic algorithm proposed in this example, a simulation-based evaluation experiment was conducted.

Pseudo plant environment used in the experiment FIG. 15 shows a pseudo plant environment used in the evaluation experiment. This environment is a rectangular parallelepiped having a depth of 8 m, a width of 10 m, and a height of 4 m, and two types of areas, a linear area and a planar area, are set as areas where a line marker can be pasted. These areas are assumed to be the surface of a large number of pipes existing in the plant, the wall, ceiling, floor, etc. of the plant. Table 1 shows the type, size, position, and the like of the area where the line marker assumed in this experiment can be pasted. In FIG. 15, a gray circle indicates a line marker, and an arrow from the line marker indicates a movable direction of the line marker. The origin of the world coordinate system is set to the left front corner in FIG. 15, the right direction is the x axis, the upward direction is the y axis, and the front direction is the z axis.

  When tracking is actually executed, it is necessary to recognize the position and type of the line marker by image processing. Therefore, the line marker has a certain size and cannot be arranged so as to overlap each other. Therefore, in this experiment, when a new position of the line marker is generated by mutation, a restriction is set such that the minimum distance between the line markers is 50 mm.

  In Table 2, the position and orientation of the arrow in FIG. 15 showing the position and orientation of the camera used when obtaining the evaluation function in this experiment represent the position and orientation of the camera. The camera position and orientation is composed of a total of 24 positions and postures of 8 positions and 3 positions at each position, and tracking is performed with high accuracy because there are equipment and piping to be inspected nearby in the plant. Assume that the position and orientation is necessary.

  However, when the above-described genetic algorithm is executed with the goal of actually reducing the tracking error only in these 24 positions and postures, high accuracy is realized locally, and low accuracy is obtained in the adjacent region. Therefore, in this embodiment, in each generation of the genetic algorithm, a slight change is made around the 24 position / posture in which the position / posture for evaluating the tracking error is set. That is, the position and orientation to be evaluated in a range of 100 mm in the x-axis direction and the z-axis direction with respect to the position and 5 degrees in the left-right direction with respect to the orientation were changed every generation.

  Table 3 shows the parameters of the genetic algorithm used in the experiment. The camera had a resolution of 1024 × 766 pixels, the lens focal length was 3.94 mm, the pixel size was 0.00465 mm, and Δ was 0.5 pixel.

Experimental Results FIG. 16 shows the average value, the best value, and the worst value of the maximum value of the tracking error in each generation when marker placement optimization using a genetic algorithm is executed 20 times up to 200 generations. Under this setting condition, there was a position and orientation that could not capture a sufficient number of line markers when initialization was performed, so re-initialization was performed twice, and the final number of line markers was 11. In the first generation, it can be confirmed that the maximum value of the tracking error which was about 150 mm in the worst case has decreased from around the 50th generation to about 50 mm.

  Regarding the execution speed of the marker placement optimization method proposed in this embodiment, the time required for calculation for one generation was about 1 to 2 seconds when using a personal computer equipped with Core 2 Duo 2.66 GHz. (Execution software was developed using C #).

  Table 4 shows the line marker arrangement (first generation) before optimization and the line marker arrangement in the 200th generation. Looking at this, (1) the longer the distance between the line markers, the higher the accuracy, (2) the accuracy when the line markers are arranged in a straight line, (3) the z-axis direction (depth direction) of the line marker arrangement It can be confirmed that the longer the distance, the higher the accuracy.

Points to note when optimizing the line marker arrangement in the plant When applying the method proposed in this example to the optimization of the line marker arrangement in the actual nuclear power plant, the following points Need to be careful.
(1) In this embodiment, a line marker optimization method is proposed on the assumption that the distortion of the lens of the camera is appropriately corrected. If the camera lens distortion cannot be corrected appropriately for some reason, the results obtained by appropriately using the method proposed in this embodiment should be used as reference data.
(2) In this embodiment, the line marker recognition error Δ on the camera image is known. Δ is expected to vary depending on the design of the line marker, lighting conditions, etc. In order to actually use the method proposed in this research, it is necessary to actually measure Δ.
(3) In this embodiment, in optimizing the arrangement of line markers, what values are actually used as various parameters (number of generations to be calculated, mutation rate, etc.) when executing the genetic algorithm There is no strict discussion about whether to use. These parameters are expected to affect the convergence speed of the optimization and the magnitude of the final tracking error. In the future, it is desirable to consider what values should be set for these parameters.

Conclusion and Future Issues In this example, we aim to realize a new method that can obtain a line marker arrangement that can reduce tracking errors over a wide range. Lines on the camera image due to the limited camera resolution We proposed a TEC algorithm that calculates tracking error from the recognition error of the marker position, and a line marker optimization method using a genetic algorithm. Through an evaluation experiment that simulates tracking in a nuclear power plant, it was confirmed that the proposed method can determine a line marker arrangement that can reduce tracking errors within a practical time.

  DESCRIPTION OF SYMBOLS 1 (1a-1c) ... Camera, 2 ... HMD (Head Mounted Display), 3 ... Information processing apparatus, 4 ... Work site, 41 (41a, 41b ...) ... Equipment, 5 (5a, 5b) ... Line marker, 51 ... base material, 52 (52a to 52k) ... code mark.

Claims (2)

Obtained by photographing the work site with the line marker installed in advance at the work site, the camera carried by the worker, the display device for displaying the work site image and instruction information and providing it to the worker Information for measuring the position and orientation of the camera (worker) based on the image position of the line marker in the photographed image and displaying the information image provided to the worker on the work site image on the display device In the method for arranging line markers in a field work support system equipped with an information processing device for processing,
The marker placement method, wherein the line marker is placed at a position determined by performing an optimized placement process using a genetic algorithm so that a measurement error of the camera position is minimized. In claim 1, the determination of the position of the line marker is:
(1) Estimating the size of an error included in a code mark obtained by image processing,
(2) Using the results of (1), the error calculation method and the genetic algorithm, estimate the size of tracking errors in the work site in various line marker arrangements, and gradually increase the accuracy of line marker arrangements This is a marker placement method characterized in that it is obtained by repeatedly calculating.

Images