JP4858346B2 - Marker image identification device and marker image identification method - Google Patents

Description

The present invention attaches a light emitter as a marker group to an object, photographs the marker group with a pair of camera devices capable of stereo viewing, and captures a plurality of marker images photographed with one camera and the other camera. A marker image identification device and a marker image identification method for associating a set with a plurality of marker images, for example, a marker image identification device for detecting movement of an object, that is, position information and angle information (direction) of an object that changes every moment, and Used for marker image identification method and the like.

  A technique for accurately monitoring the movement of an object having a fixed shape is used in various fields. For example, in a game machine or the like, a moving image is displayed using a head-mounted display device in order to realize virtual reality (VR). At this time, it is necessary to monitor the movement of the head and change the display image in accordance with the movement of the head, and a head motion tracker (HMT) that detects the movement of the head-mounted display device mounted on the head Is attached.

As a head motion tracker (HMT), a magnetic source is used to generate a magnetic field in the surrounding space, and magnetism is measured by a magnetic sensor attached to a head-mounted display device. A magnetic measurement system that detects the position and angle of a sensor and, in turn, the position and angle of a head-mounted display device is disclosed (for example, see Patent Document 1).
Although the magnetic measurement method can be used without any limitation as long as a magnetic field is generated in the space in which the magnetic sensor moves, it is necessary to measure magnetic information in the space in which the magnetic sensor moves in advance and perform magnetic mapping. is there. In addition, if an object that causes magnetostriction such as metal exists in this space, a measurement error occurs due to the influence, and it is necessary to correct it.

On the other hand, what optically measures the position and angle of a head-mounted display device is also disclosed. For example, a reflective plate (marker group) is attached to a head-mounted display device, light is emitted from a light source, and the reflected light is photographed with a pair of camera devices. An optical measurement system that measures the above is disclosed (for example, see Patent Document 2).
The optical measurement method does not require mapping correction and is not affected by metal or the like. Therefore, the optical measurement method is easy to use in an environment or place that is easily affected by magnetostriction.

With such an optical measurement method HMT, when three-dimensional position information and angle information (direction) of an object having a fixed shape are to be measured accurately, the positions of three different points are fixed on the object. Need to know. Therefore, the position is measured by attaching markers each identifiable to three different locations on the outer surface of the object.
For example, a helmet-shaped head-mounted display device will be described in detail. LEDs (light-emitting diodes) that emit infrared light with different emission wavelengths are used as markers on the outer surface of the helmet of the head-mounted display device. These are mounted at three locations so as to be separated from each other, and the positional relationship on the helmet coordinates fixed to the helmets of these three LEDs is measured and determined in advance. These three LEDs are photographed simultaneously with two cameras that can be viewed in stereo and have a fixed installation location (that is, under a condition where the distance between the two cameras is known). After identifying each LED by the difference in emission wavelength, the absolute coordinates (coordinates determined for a pair of camera devices) of the three LEDs are measured by the so-called triangulation principle. Then, by using the positions of these three LEDs and associating the absolute coordinates with the positional relationship of the helmet coordinates fixed on the head-mounted display device, the position of the head-mounted display device with respect to the absolute coordinates, Identify the angle.

  However, in the optical measurement method as described above, since it is necessary to identify individual LEDs (markers), the LEDs themselves have some identification information that can be distinguished from other LEDs, and the detection side (camera device) Side). That is, the emission wavelength is changed for each LED, and each LED is identified by identifying the emission wavelength on the camera device side.

However, in the case where each LED has identification information that can be identified, each time the LED is replaced due to a failure or the like, it is necessary to store the identification information again, which is troublesome.
In addition, if an individual LED is to be identified with an LED having no identification information, it is only necessary to turn on the LED one by one, but in that case, all the LEDs must be turned on sequentially, Since it takes a long time to complete a cycle, it is difficult to monitor the movement that changes from moment to moment.

Therefore, when the positions of three different LEDs photographed by the camera device are optically measured and the movement (position and angle) of the object is monitored from the movements of these three LEDs, the LEDs are sequentially arranged one by one. Without turning on the LED, the LED itself does not have identification information for distinguishing it from other LEDs, and the individual LED is identified only from the position information of the LED. There has been proposed a method for monitoring the movement (position, angle) of an object to which is attached.
JP 2002-81904 A JP-T 9-506194

In the above-described method for monitoring the movement of the object, in a state where the shooting direction of the first camera and the shooting direction of the second camera are determined, the first image captured by the first camera is predicted by prediction based on epipolar geometry; Recognizing a common set of LEDs with a second image taken by a second camera is performed. 4 and 5 are diagrams for explaining prediction based on epipolar geometry.
As shown in FIG. 4, the first image A captured by the first camera 12 cannot recognize which of the three LEDs 18a, 18b, and 18c that are lit, but the three LED images 18x, 18y and 18z are projected. On the other hand, the second image B photographed by the second camera 14 cannot recognize which of the three LEDs 18a, 18b, and 18c that are lit, but three LED images 18s, 18t, and 18u are projected. It is. At this time, it cannot be recognized whether the LED images 18x, 18y, and 18z in the first image A correspond to the LED images 18s, 18t, and 18u in the second image B, respectively.

Therefore, in order to associate the sets of the LED images 18x, 18y, and 18z in the first image A with the LED images 18s, 18t, and 18u in the second image B, as shown in FIG. If each epipolar line (virtual straight line) La, Lb, and Lc connecting the LEDs 18a, 18b, and 18c is present and the epipolar lines Lx, Ly, and Lz are captured by the second camera 14, the second image B is displayed. Epipolar lines Lx, Ly, and Lz are created. The positions of the epipolar lines La, Lb, and Lc on the second image B are the positions at which the LED images 18x, 18y, and 18z are projected on the first image A, the shooting direction of the first camera 12, and the second direction. It is uniquely determined based on the relationship with the shooting direction of the camera 14.
On the other hand, in the second image B, the LED images 18s, 18t, and 18u corresponding to the three LEDs 18a, 18b, and 18c are images of the three LEDs 18a, 18b, and 18c. It exists at a position above any of the lines Lx, Ly, and Lz. Accordingly, if there is an LED image displayed on the second image B that exists on the epipolar line Lx, it can be specified that it is an LED image corresponding to the LED image 18x. Similarly, by specifying the LED images corresponding to the other two LED images 18y and 18z, the three LED images 18x, 18y and 18z displayed on the first image A and the second image B are displayed. Each of the three LED images 18s, 18t, and 18u can be associated with each other.

However, for example, as shown in FIG. 5, in the second image B, epipolar lines Ly and Lz corresponding to the two LED images 18y and 18z may exist at a very close distance. In such a case, it may be difficult to determine whether the LED image present on the epipolar line Ly is the LED image 18t or the LED image 18u among the LED images projected on the second image B. there were. Similarly, among the LED images projected on the second image B, it may be difficult whether the LED image existing on the epipolar line Lz is the LED image 18t or the LED image 18u. That is, when two epipolar lines approach, the three LED images 18x, 18y, and 18z projected on the first image A and the three LED images 18s, 18t, and 18u projected on the second image B Corresponding each set may be difficult.

  Therefore, instead of using markers that do not have identification information and measuring while sequentially identifying each marker separately, these markers are lit at the same time. Marker image identification that makes it possible to accurately and quickly associate a set of multiple marker images displayed on one camera and multiple marker images displayed on a second camera, even if the group is measured almost simultaneously. An object is to provide an apparatus and a marker image identification method.

  The marker image identification device of the present invention made to solve the above-mentioned problems is a marker image identification device for identifying at least three illuminable markers attached to the outer surface of an object, and is at least three illuminated. A first camera that captures a first image including two markers, a second camera that simultaneously captures a second image including at least three markers from a different direction from the first camera, and the first camera. An epipolar line creation unit for creating the epipolar line of each marker image on the second image, and a reference epipolar line for selecting the epipolar line farthest between the epipolar lines on the second image as the reference epipolar line Marker closest to the reference epipolar line on the line selector and the second image Is recognized by the marker image recognition unit on each of the first image and the second image, and the marker image recognition unit recognizes that the image is the same marker image as the marker image corresponding to the reference epipolar line. A marker image position determining unit that determines the positional relationship of the remaining marker image with respect to the marker image, and a remaining marker on the first image based on the positional relationship of the marker image determined by the marker image position determining unit A marker image identifying unit that associates a set of the image and the remaining marker image on the second image is provided.

  According to the marker image identification device of the present invention, first, at least three marker images are displayed on the respective images of the first camera and the second camera. And the epipolar line of each marker image projected on the 1st camera is created on the 2nd image projected on the 2nd camera. Next, on the second image, the epipolar line that is farthest between the epipolar lines is selected as the reference epipolar line. Thereby, on the second image, the marker image having the closest distance to the reference epipolar line is recognized as the same marker image as the marker image corresponding to the reference epipolar line. That is, since the furthest epipolar line is selected as the reference epipolar line, the LED image corresponding to the reference epipolar line displayed on the second image is also distant from the other LED images. The corresponding marker image can be reliably identified. Further, the positional relationship of the remaining marker image with respect to the marker image recognized by the marker image recognition unit is determined on each of the first image and the second image. For example, when three markers are lit, a vector having a marker image recognized by the marker image recognition unit as a start point and the other two marker images as an end point on each of the first image and the second image. The positional relationship of the remaining marker image with respect to the marker image recognized by the marker image recognition unit is determined by calculating and calculating the outer product. As a result, when the calculated outer product has the same sign on each of the first image and the second image, the set of marker images is associated with the same rotation direction, while the outer product is different. In the case of a code, the sets of marker images are associated with each other assuming that the rotation directions are different, and the sets of the remaining marker images on the first image and the remaining marker images on the second image are associated with each other. Therefore, for example, as shown in FIG. 5, in the second image B, even if the epipolar lines Lb and Lc corresponding to the two LED images 18 y and 18 z exist at a very close distance, the first image is obtained using the outer product. A set of LED images 18y and 18z on the upper side and LED images 18t and 18u on the second image can be associated with each other.

According to the present invention, at least three markers are lit at the same time, instead of using markers that do not have identification information and measuring each of the markers separately while sequentially identifying them. Thus, even if these marker groups are measured almost simultaneously, a set of a plurality of marker images projected on the first camera and a plurality of marker images projected on the second camera can be accurately and quickly associated with each other. it can.
(Means and effects for solving other problems)

    Further, in the present invention, the marker is positioned on the object fixed coordinate defined on the object, and the absolute coordinate defined on the space using the first camera and the second camera. Is further stored, and for each set of marker images associated by the marker image recognition unit and the marker image identification unit, the position on the absolute coordinate is calculated by triangulation to create a search marker Thus, an object monitoring unit that determines the position and the angle of the object on the absolute coordinates by associating the set of the search marker and the marker may be provided.

According to the present invention, when a pair of marker images is identified between the first image and the second image, the direction angle from the first camera and the second camera for each search marker is identified. Therefore, by performing so-called triangulation from the distance between the first camera and the second camera and these direction angles, the position of each search marker in absolute coordinates can be calculated. This is compared with the coordinates of each marker on the fixed object fixed coordinate system. Then, based on the absolute coordinates and the object fixed coordinates, the position and angle of the object with respect to the absolute coordinates are calculated. In this way, the movement of the object can be monitored by determining the position and angle of the object with respect to the absolute coordinates.

In the present invention, lines connecting at least three lit markers are not equidistant, and the object monitoring unit sets the shortest or longest inter-marker distance between the reference markers on the object fixed coordinates. The position of the search marker on the absolute coordinates is calculated by triangulation for a set of reference marker distance selection units to be selected as the distance and each marker image associated by the marker image recognition unit and the marker image identification unit. By doing this, the inter-marker distance recognition unit that recognizes the shortest or longest inter-search marker distance as the same inter-marker distance as the reference marker distance, and the marker recognized by the inter-marker distance recognition unit Associating a set of a marker attached to the outer surface of the object and a search marker with reference to the distance between them A result, may be provided with a marker identifying unit for determining the position and angle of the object on the absolute coordinate.

According to the present invention, for example, the shortest marker distance is selected as the reference marker distance on the object fixed coordinates. On the other hand, when a pair of marker images is identified between the first image and the second image, the search marker distance in absolute coordinates is calculated. Accordingly, the shortest distance between markers for search is recognized as the same distance between markers as the distance between reference markers. In other words, since the lines connecting at least three lit markers are not equidistant, if a pair of marker distances can be associated, for example, when three markers are lit, in absolute coordinates, The position of the search marker is calculated by calculating a cross product by creating a vector starting from the search marker that is not on the inter-marker distance recognized by the inter-marker distance recognition unit and ending with the other two search markers. Determine the relationship. Similarly, in absolute coordinates, a marker that is not on the marker distance recognized by the marker distance recognition unit is used as a starting point, and a vector with the other two markers as the end point is calculated to calculate the outer product. Determine the positional relationship. After that, the calculated outer products of the search marker and the marker are compared, and if the outer product has the same sign, the pair is associated as having the same rotation direction, whereas if the outer product has a different sign Associate the pairs in reverse, assuming different rotation directions. In this way, a set of the marker attached to the outer surface of the object and the search marker calculated from the first image A and the second image B can be associated with each other. Accordingly, the position and angle of the object with respect to the absolute coordinate can be calculated based on the absolute coordinate and the object fixed coordinate.

The marker image identification method of the present invention includes at least three illuminable markers attached to an outer surface of an object, a first camera that captures a first image including at least three illuminated markers, A marker image identification method in a marker image identification system, comprising: a second camera that captures a second image including at least three markers that are lit from a direction different from that of one camera, and (a) projected by the first camera. An epipolar line creation step of creating an epipolar line of each marker image on the second image projected by the second camera; and (b) an epipolar line that is farthest between the epipolar lines on the second image. A reference epipolar line selection step of selecting a line as a reference epipolar line; and (c) on the second image A marker image recognition step for recognizing that the marker image closest to the reference epipolar line is the same marker image as the marker image corresponding to the reference epipolar line; and (d) the first image and the second image On each image, a marker image position determining step for determining the positional relationship of the remaining marker images with respect to the marker image recognized in the marker image recognition step; and (e) the marker image determined in the marker image position determining step. And a marker image identifying step for associating a set of the remaining marker image on the first image and the remaining marker image on the second image based on the positional relationship of the first image and the second image.

    In the present invention, the marker is positioned on the object fixed coordinates defined on the object, and the absolute coordinates defined on the space using the first camera and the second camera. And (f) for each set of marker images associated by the marker image recognition step and the marker image identification step, the position on the absolute coordinate is calculated by triangulation to obtain a search marker. By creating, an object monitoring step of determining the position and angle of the object on the absolute coordinates by associating the set of the search marker and the marker may be included.

Further, in the present invention, lines connecting at least three lit markers are not equidistant, and (f) object monitoring step is (f-1) the shortest or longest on the object fixed coordinates. A distance between reference markers that selects a distance between markers as a distance between reference markers, and (f-2) a set of marker images associated by the marker image recognition unit and the marker image identification unit on the absolute coordinates. The inter-marker distance recognition step for recognizing that the shortest or longest inter-search marker distance is the same inter-marker distance as the reference inter-marker distance by calculating the position of the search marker at (F-3) A marker attached to the outer surface of the object with reference to the inter-marker distance recognized by the inter-marker distance recognition unit. Over and, by associating a set of the search marker may include a marker identifying step of determining the position and angle of the object on the absolute coordinate.

  Hereinafter, a marker image identification method according to an embodiment of the present invention will be described using a head motion tracker (HMT) using the marker image identification method as a specific example with reference to the drawings. However, the present invention is not limited to the specific examples described below, and can be applied without departing from the spirit of the present invention.

FIG. 1 is a diagram showing a schematic configuration of an HMT which is an embodiment of the present invention. The HMT 10 includes a pair of camera devices 11 that are installed so as to be arranged at a distance d so as to enable stereo viewing, a helmet 16 with a head-mounted display device, and outer surfaces of the helmet 16 with a head-mounted display device. The LED group 18 attached to the control unit 20 and a control calculation unit 20 configured by a computer.

The camera device 11 includes a first camera 12 and a second camera 14. Thereby, the first image A photographed by the first camera 12 is created, and the second image B photographed by the second camera 14 is created. At this time, a two-dimensional coordinate system (XaYaZa coordinates) is set in the first image A, but the origin and the direction of each coordinate axis can be arbitrarily determined. In this embodiment, as shown in FIG. The Xa axis is defined so that the left side is the Ya axis, and the origin is defined as the intersection of the bottom side and the left side. Similarly, in the second image B, a two-dimensional coordinate system (XbYbZb coordinates) is set, but the origin and the direction of each coordinate axis can be arbitrarily determined. In this embodiment, as shown in FIG. The bottom side is defined as the Xb axis, the left side is defined as the Yb axis, and the origin is defined as the intersection of the bottom side and the left side.

The first camera 12 and the second camera 14 are each directed toward the helmet 16, and at least three LEDs in the LED group 18 (the same LED for the first camera 12 and the second camera 14). Are set in accordance with the number and arrangement of the LED groups 18 so that they are simultaneously within the field of view.
In addition, you may install the 1st camera 12 and the 2nd camera 14 in the position spaced apart from the helmet 16 so that the whole image of the helmet 16 may enter into a visual field. In this case, even if the number of LEDs attached to the helmet 16 is relatively small, three LEDs can be included in the field of view of the first camera 12 and the second camera 14.
On the other hand, the first camera 12 and the second camera 14 may be brought close to the helmet 16 so that only a part of the helmet 16 enters the field of view. In this case, the number of LEDs attached to the helmet 16 must be increased so that at least three LEDs are included in the field of view, but the position of the LEDs can be accurately measured.

The helmet 16 with a head-mounted display device includes a display (not shown) and a combiner 8 that guides the eyes of the wearer by reflecting image display light emitted from the display. In addition, the wearer wearing the helmet 16 with a head-mounted display device can visually recognize the display image by the display unit and the front actual object of the combiner 8.

As shown in FIG. 2, the LED group 18 includes six heads 18a, 18b, 18c,..., 18f that emit infrared light and are separated from each other at different distances by a head mounted helmet with a display device. 16 are attached to the outer surface. The positions of the LEDs 18a, 18b, 18c,..., 18f in the helmet coordinates fixed to the helmet 16 with a head-mounted display device are stored in advance in the helmet coordinate storage unit 44 of the memory 41 described later. It is like that. Therefore, for example, when the positions of the three LEDs 18a, 18b, and 18c in the current space (absolute coordinates) are calculated by a triangulation method to be described later, the data stored in the helmet coordinate storage unit 44 is referred to. Thus, the position and angle (direction) on the space (absolute coordinates) of the helmet 16 to which the LEDs 18a, 18b, 18c are fixed can be specified.

The LED group 18 is not limited in the emission wavelength as long as the lighting state can be confirmed by the first camera 12 and the second camera 14.
Moreover, you may adjust the range which attaches LED group 18 on the helmet 16 according to the range which the helmet 16 moves. That is, when the range of movement of the helmet 16 is small, the LED group 18 may be attached only in that region. On the other hand, when the helmet 16 moves in all directions, it is necessary to attach the LED group 18 over the entire helmet 16.

The control calculation unit 20 performs various calculation processes in conjunction with the control of the camera device 11 and the LED group 18 in conjunction with them. The control and calculation contents executed by the control calculation unit 20 will be described separately for each function. As shown in FIG. 1, the control calculation unit 20 includes a marker lighting unit 21, an epipolar line creation unit 31, and a reference epipolar line selection. Unit 32, marker image recognition unit 33, marker image position determination unit 34, marker image identification unit 35, object monitoring unit 36, and video display unit 25.
The memory 41 has an area for storing various data necessary for the control calculation unit 20 to execute processing, an absolute coordinate storage unit 42 for storing absolute coordinates (XYZ coordinates), a helmet A helmet coordinate storage unit 44 that stores coordinates (X′Y′Z ′ coordinates) and positions of LEDs 18a, 18b, 18c,.

  Here, the absolute coordinates will be described. The absolute coordinate is a three-dimensional coordinate system (XYZ coordinate) fixed to the camera device 11 and can arbitrarily determine the origin and the direction of each coordinate axis. In this embodiment, as shown in FIG. The direction from the camera 14 to the first camera 12 is the X-axis direction, the direction perpendicular to the X-axis direction and perpendicular to the ceiling and downward is the Z-axis direction, the direction perpendicular to the X-axis direction and horizontal to the ceiling, and the rightward direction is the Y-axis. The origin is defined as the midpoint between the first camera 12 and the second camera 14.

The helmet coordinates are a three-dimensional coordinate system (X′Y′Z ′ coordinates) fixed to the helmet 16, and the origin and the direction of each coordinate axis can be arbitrarily determined. In this embodiment, FIG. As shown, the direction from the LED 18a to the LED 18b is the X′-axis direction, the direction perpendicular to the X′-axis direction on the surface including the LED 18c is the Y′-axis direction, and the direction perpendicular to the X′-axis direction and the Y′-axis direction. Is defined as the Z′-axis direction, and the origin (O) is defined as the LED 18a.
Further, in order to express the current position and current angle of the helmet 16 using the XYZ coordinate system, the reference point on the helmet 16 is defined as the LED 18a (the origin of the helmet coordinates), and the X′-axis direction is defined as the reference direction. It has established. Thereby, the current position and the current angle of the helmet 16 are relative to the position (x, y, z) of the LED 18a in absolute coordinates (XYZ coordinates) and the X-axis direction, the Y-axis direction, and the Z-axis direction in the X′-axis direction. Each angle (θ, φ, ψ) can be used for expression.

The marker lighting part 21 is three LEDs (markers) simultaneously photographed in the images of the first camera 12 and the second camera 14 in the LED group 18 (marker group), and connects these LEDs. Control is performed to turn on three LEDs in a positional relationship where the line does not form an equilateral triangle. At this time, the other LEDs must be turned off.

The epipolar line creation unit 31 performs control to create an epipolar line of each marker image projected by the first camera 12 on the second image B projected by the second camera 14. That is, as shown in FIG. 6A, assuming that there are respective epipolar lines (virtual straight lines) La, Lb, Lc connecting the first camera 12 and the three LEDs 18a, 18b, 18c, the epipolar lines Lx, Ly , Lz is taken by the second camera 14, epipolar lines Lx, Ly, Lz are created on the second image B. The positions of the epipolar lines La, Lb, and Lc on the second image B are the positions at which the LED images 18x, 18y, and 18z are projected on the first image A, the shooting direction of the first camera 12, and the second direction. It is uniquely determined based on the relationship with the shooting direction of the camera 14.

The reference epipolar line selection unit 32 performs control for selecting, on the second image B, the epipolar line that is farthest between the epipolar lines as the reference epipolar line.
For example, as shown in FIG. 6B, by creating a perpendicular line 1 passing through the center of the second image B and perpendicular to the Xb axis on the second image B, the perpendicular line l and the epipolar line are created. The positions of intersections lx, ly, and lz with Lx, Ly, and Lz are calculated. Furthermore, the farthest intersection lx is selected by calculating the distance between the intersections lx, ly, and lz. As a result, as shown in FIG. 6C, the epipolar line Lx corresponding to the selected intersection point lx is selected as the reference epipolar line.

The marker image recognizing unit 33 performs control for recognizing the marker image having the closest distance to the reference epipolar line on the second image B as the same marker image as the marker image corresponding to the reference epipolar line. is there. At this time, since the furthest epipolar line Lx is selected as the reference epipolar line, the LED image 18s corresponding to the reference epipolar line displayed on the second image B is also separated from the other LED images 18t and 18u. Therefore, the marker image corresponding to the reference epipolar line can be reliably identified.

The marker image position determination unit 34 performs control for determining the position of the remaining marker image with respect to the marker image recognized by the marker image recognition unit 33 on each of the first image A and the second image B. is there.
For example, as shown in FIG. 7A, on the first image A, a vector <with the marker image 18x recognized by the marker image recognition unit 33 as a start point and the other two marker images 18y and 18z as an end point <18x18y> and vector <18x18z> are generated and outer product <18x18y> × <18x18z> is calculated to determine the positional relationship of the remaining marker images 18y and 18z with respect to the marker image 18x recognized by the marker image recognition unit 33. To do. Similarly, on the second image B, a vector <18s18t> and a vector <18s18u> are generated with the marker image 18s recognized by the marker image recognition unit 33 as the start point and the other two marker images 18t and 18u as the end point. By calculating the outer product <18s18t> × <18s18u>, the positional relationship of the remaining marker images 18t and 18u with respect to the marker image 18s recognized by the marker image recognition unit 33 is determined.
As described below, <18x18y> is a vector starting from 18x and ending at 18y, and the same applies to <18x18z>, <18s18t>, and <18s18u>.

The marker image identifying unit 35 is a set of the remaining LED image on the first image A and the remaining marker image on the second image B based on the positional relationship of the marker images determined by the marker image position determining unit 34. The control which matches is performed.
For example, on the images of the first image A and the second image B, the calculated outer product <18x18y> × <18x18z> is compared with the outer product <18s18t> × <18s18u>, and the outer product has the same sign. In this case, as shown in FIG. 7B, the direction from the marker image 18y centered on the marker image 18x to the marker image 18z and the marker image 18s centered on the marker image 18s to the marker image 18t. Assuming that the direction is the same rotation direction, a pair of the marker image 18y and the marker image 18t and a pair of the marker image 18z and the marker image 18u are associated with each other. The direction from the marker image 18y centered on the marker image 18z to the marker image 18z and the marker image 18s centered on the marker image 18s As the different rotation direction and the direction to the associates set and set of the marker image 18z and the marker image 18u of the LED image 18y and the marker image 18t. Therefore, as shown in FIG. 5, in the second image B, even when the epipolar lines Lb and Lc corresponding to the two LED images 18y and 18z exist at a very close distance, Each of the LED images 18y and 18z and the LED images 18t and 18u on the second image can be associated with each other.

The object monitoring unit 36 includes a reference inter-marker distance selection unit 36a, an inter-marker distance recognition unit 36b, and a marker identification unit 36c.
The inter-reference marker distance selection unit 36a performs control to select the shortest inter-marker distance as the reference inter-marker distance on the helmet coordinates (X′Y′Z ′ coordinates) for the LEDs lit by the marker lighting unit 21. is there.
For example, based on the data stored in the helmet coordinate storage unit 44, by comparing the inter-marker distances a, b, c of the LEDs 18a, 18b, 18c lit by the marker lighting unit 21, the shortest inter-marker distance a As the distance between the reference markers.

The inter-marker distance recognition unit 36b calculates the position of the search marker on the absolute coordinates (XYZ coordinates) by triangulation for each set of LED images associated by the marker image recognition unit 33 and the marker image identification unit 35. By doing so, control is performed to recognize that the shortest distance between markers for search is the same distance between markers as the distance between reference markers.
Specifically, first, as shown in FIG. 3, the position of the LED 18l in absolute coordinates (XYZ coordinates) is a direction angle α from the first camera 12 to the LED 18l and a direction angle β from the second camera 14 to the LED 18l. And the inter-camera distance d between the first camera 12 and the second camera 14 can be uniquely calculated. The direction angle α can be determined by the shooting direction of the first camera 12 and the position of the LED image 18x in the first image A. Similarly, the direction angle β is determined by the shooting direction of the second camera 14 and the second image B. Can be determined by the position of the LED image 18s. Further, since the inter-camera distance d is known, the position of the search LED 18l in absolute coordinates (XYZ coordinates) is calculated. Similarly, the positions of the other search LEDs 18m and 18n in absolute coordinates (XYZ coordinates) are also calculated. Thereafter, by comparing the distances between the search markers, the shortest distance between the search markers is recognized as the same inter-marker distance as the reference marker distance. At this time, it cannot be recognized which of the search LEDs 18l, 18m, and 18n corresponds to each of the LEDs 18a, 18b, and 18c.

The marker identifying unit 36c is a set of LEDs 18a, 18b, 18c attached to the outer surface of the helmet 16 and search LEDs 18l, 18m, 18n based on the inter-marker distance recognized by the inter-marker distance recognizing unit 36b. , The position (x, y, z) of the helmet 16 on the absolute coordinates (XYZ coordinates) and the angle (θ, φ, ψ) are determined. At this time, since the lines connecting the three lit LEDs 18a, 18b, and 18c are not equidistant, if one pair of marker distances can be associated with each other, for example, distance recognition between markers can be performed in absolute coordinates. The positional relationship of the search LEDs is determined by calculating a cross product by creating a vector starting from the search LED that is not on the marker distance recognized by the unit 36b and ending with the other two search LEDs. To do. Similarly, in absolute coordinates, by calculating a cross product by creating a vector starting from an LED that is not on the inter-marker distance recognized by the inter-marker distance recognition unit 36b and ending with the other two LEDs, the LED is calculated. The positional relationship of is determined. Thereafter, the calculated vector products of the search LED and the LED are compared, and when the outer product has the same sign, the pair is associated with the same rotation direction, while the outer product has a different sign. Associate the pairs as having different rotational directions. In this way, it is possible to associate a set of the LEDs 18a, 18b, 18c attached to the outer surface of the helmet 16 with the search markers 18l, 18m, 18n calculated from the first image A and the second image B. It will be possible. The position (x, y, z) and the angle (θ, φ, ψ) of the helmet 16 are the position (x, y, z) of the LED 18a in absolute coordinates (XYZ coordinates) and X in the X′-axis direction. It represents using each angle ((theta), (phi), (psi)) with respect to an axial direction, a Y-axis direction, and a Z-axis direction.
The video display unit 25 performs control for emitting video display light based on the position (x, y, z) and the angle (θ, φ, ψ) of the helmet 16. As a result, the wearer can visually recognize the display image on the display.

Next, the operation of monitoring the helmet 16 by the HMT 10, that is, the operation of monitoring the position and angle every moment will be described. FIG. 8 is a diagram showing an operation flow when the position (x, y, z) and the angle (θ, φ, ψ) of the helmet 16 are monitored by the HMT 10.
First, in the process of step S101, the marker lighting unit 21 turns on three LEDs (however, the positional relationship of the three LEDs does not become an equilateral triangle) and turns off the other LEDs.

Next, in the process of step S <b> 102, the marker lighting unit 21 determines whether or not three LEDs that are lit are reflected in the first image A of the first camera 12 and the second image B of the second camera 14. Check. When it is determined that the three LEDs that are lit are not shown in the first image A of the first camera 12 and the second image B of the second camera 14, the process returns to step S101 and the three LEDs are turned off. Then, another set of LEDs is turned on. That is, the processes of steps S101 and S102 are repeated until it is determined that three LEDs that are lit are reflected in the first image A of the first camera 12 and the second image B of the second camera 14. . On the other hand, when it is determined that the three LEDs that are lit are reflected in the first image A of the first camera 12 and the second image B of the second camera 14, the process proceeds to step S103. FIG. 4 shows an image example at this time.

Next, in the process of step S103, the epipolar line creation unit 31 creates an epipolar line of each marker image projected by the first camera 12 on the second image B projected by the second camera 14 (FIG. 6 (a)).
Next, in the process of step S <b> 104, the reference epipolar line selection unit 32 selects the epipolar line farthest among the epipolar lines on the second image B as the reference epipolar line. For example, as shown in FIG. 6B, by creating a perpendicular line 1 passing through the center of the second image B and perpendicular to the Xb axis on the second image B, the perpendicular line l and the epipolar line are created. The positions of intersections la, lb, and lc with La, Lb, and Lc are calculated. Further, by calculating the distance between the intersection points la, lb, and lc, the intersection point la that is furthest away is selected. As a result, the epipolar line La corresponding to the selected intersection point la is selected as the reference epipolar line.

Next, in the process of step S105, the marker image recognition unit 33 uses the same marker image as the marker image corresponding to the reference epipolar line, on the second image B, for the marker image having the closest distance to the reference epipolar line. Recognize that there is.
Next, in the process of step S106, the marker image position determination unit 34 determines the remaining marker image for the marker image recognized by the marker image recognition unit 33 on each of the first image A and the second image B. Determine the positional relationship. For example, as shown in FIG. 7A, on the first image A, a vector <with the marker image 18x recognized by the marker image recognition unit 33 as a start point and the other two marker images 18y and 18z as an end point <18x18y> and vector <18x18z> are generated and outer product <18x18y> × <18x18z> is calculated to determine the positional relationship of the remaining marker images 18y and 18z with respect to the marker image 18x recognized by the marker image recognition unit 33. To do. Similarly, on the second image B, a vector <18s18t> and a vector <18s18u> are generated with the marker image 18s recognized by the marker image recognition unit 33 as the start point and the other two marker images 18t and 18u as the end point. By calculating the outer product <18s18t> × <18s18u>, the positional relationship of the remaining marker images 18t and 18u with respect to the marker image 18s recognized by the marker image recognition unit 33 is determined.

Next, in the process of step S107, the marker image identification unit 35, based on the positional relationship of the marker images determined by the marker image position determination unit 34, the remaining LED image and the second image B on the first image A. Corresponding pairs with the remaining marker images above. For example, on the images of the first image A and the second image B, the calculated outer product <18x18y> × <18x18z> is compared with the outer product <18s18t> × <18s18u>, and the outer product has the same sign. In this case, a pair of the marker image 18y and the marker image 18t and a pair of the marker image 18z and the marker image 18u are associated with each other. On the other hand, if the outer product has a different sign, the LED image 18y and the marker image 18t And a set of the marker image 18z and the marker image 18u are associated with each other.

Next, in the process of step S108, the reference marker distance selection unit 36a determines the shortest marker distance on the helmet coordinates (X'Y'Z 'coordinates) for the LEDs lit by the marker lighting unit 21. Select as distance.
Next, in the process of step S109, the inter-marker distance recognition unit 36b searches for the set of LED images associated by the marker image recognition unit 33 and the marker image identification unit 35 on the absolute coordinates (XYZ coordinates). By calculating the positions of the LEDs 18l, 18m, and 18n by triangulation, the shortest distance between markers for search is recognized as the same distance between markers as the distance between reference markers.

Next, in the process of step S110, the marker identifying unit 36c includes LEDs 18a, 18b, 18c attached to the outer surface of the helmet 16 with reference to the inter-marker distance recognized by the inter-marker distance recognizing unit 36b, By associating a set of search LEDs 181, 18 m, 18 n calculated from one image A and second image B, the position (x, y, z) and angle of the helmet 16 on absolute coordinates (XYZ coordinates) Control for determining (θ, φ, ψ) is performed.
Next, in the process of step S111, the video display unit 25 emits video display light based on the position (x, y, z) and the angle (θ, φ, ψ) of the helmet 16.

Next, in the process of step S112, it is determined whether or not to end this flowchart. If it is determined not to end the flowchart, the process returns to step S101. That is, the processing of step S101 to step S112 is repeated until it is determined to end the flowchart.
On the other hand, when it is determined to end the flowchart, the flowchart is ended.
Through the above arithmetic processing, the position and angle of the helmet 16 with respect to the space (absolute coordinates) can be determined, and the position and direction of the helmet can be specified. That is, the position and direction of the helmet 16 that changes from moment to moment can be specified.

Further, in the HMT 10 described above, the marker lighting unit 21 lights three LEDs, and the marker image position determination unit 34 uses the marker image recognized by the marker image recognition unit 33 as a starting point and the other two marker images. The marker lighting unit lights up four LEDs, and the marker image position determination unit uses the center of the four LED images as the center of gravity to create a marker image. The order of the other three marker images may be calculated clockwise or counterclockwise from the marker image recognized by the recognition unit 33 as a starting point.

  The present invention can be used for an optical measurement type motion tracker that uses a light emitter having no identification information.

The figure which shows schematic structure of HMT which is one Embodiment of this invention. The figure which shows schematic structure of the LED group attached to the helmet with a head mounted display apparatus which concerns on this invention. The figure explaining the absolute coordinate which concerns on this invention. The figure explaining the prediction based on epipolar geometry. The figure explaining the prediction based on epipolar geometry. The figure explaining the prediction based on the epipolar geometry which concerns on this invention. The figure explaining the prediction based on the epipolar geometry which concerns on this invention. The flowchart explaining the operation | movement by the head motion tracker of FIG.

Explanation of symbols

10 Head Motion Tracker 12 First Camera 14 Second Camera 16 Helmet (Object)
18 LED group (marker group)
DESCRIPTION OF SYMBOLS 20 Computation control part 21 Marker lighting part 31 Epipolar line creation part 32 Reference | standard epipolar line selection part 33 Marker image recognition part 34 Marker image position determination part 35 Marker image identification part

Claims (6)

A marker image identification device for identifying at least three illuminable markers attached to an outer surface of an object,
A first camera that captures a first image that includes at least three illuminated markers;
A second camera for simultaneously capturing a second image including at least three markers from a different direction from the first camera;
An epipolar line creation unit that creates an epipolar line of each marker image projected by the first camera on the second image;
On the second image, a reference epipolar line selection unit that selects, as a reference epipolar line, an epipolar line farthest between the epipolar lines;
On the second image, a marker image recognition unit that recognizes a marker image having the closest distance to the reference epipolar line as an image of the same marker as the marker image corresponding to the reference epipolar line;
On each of the first image and the second image, a marker image position determination unit that determines the positional relationship of the remaining marker image with respect to the marker image recognized by the marker image recognition unit;
A marker image identifying unit for associating a set of the remaining marker image on the first image and the remaining marker image on the second image based on the positional relationship of the marker images determined by the marker image position determining unit; A marker image identification device comprising: The marker is positioned on an object fixed coordinate defined on the object,
Using the first camera and the second camera, absolute coordinates defined in space are stored,
Further, for each set of marker images associated by the marker image recognition unit and the marker image identification unit, a search marker is created by calculating a position on the absolute coordinate by triangulation, thereby obtaining the search The marker image identification device according to claim 1, further comprising an object monitoring unit that determines a position and an angle of the object on an absolute coordinate by associating a set of the marker for use with the marker. The lines connecting at least three lit markers are not equidistant,
The object monitoring unit, on the object fixed coordinates, a distance between reference marker selection unit for selecting the shortest or the longest distance between markers as a distance between reference markers,
For each set of marker images associated by the marker image recognition unit and the marker image identification unit, the shortest or longest search marker is calculated by triangulation the position of the search marker on the absolute coordinates. An inter-marker distance recognition unit that recognizes that the inter-distance is the same inter-marker distance as the reference inter-marker distance;
The position of the object on the absolute coordinates by associating a set of the marker attached to the outer surface of the object and the marker for search with reference to the distance between the markers recognized by the inter-marker distance recognition unit The marker image identification device according to claim 2, further comprising a marker identification unit that determines the angle and the angle. At least three illuminable markers attached to the outer surface of the object;
A first camera that captures a first image that includes at least three illuminated markers;
A marker image identification method in a marker image identification system comprising a second camera that captures a second image including at least three markers that are lit from a direction different from the first camera,
(A) an epipolar line creating step of creating an epipolar line of each marker image projected by the first camera on a second image projected by the second camera;
(B) On the second image, a reference epipolar line selection step of selecting, as a reference epipolar line, an epipolar line farthest between the epipolar lines;
(C) on the second image, a marker image recognition step for recognizing that the marker image having the closest distance to the reference epipolar line is the same marker image as the marker image corresponding to the reference epipolar line;
(D) a marker image position determining step for determining a positional relationship of the remaining marker images with respect to the marker image recognized in the marker image recognition step on each of the first image and the second image;
(E) Marker image identification for associating a set of the remaining marker image on the first image and the remaining marker image on the second image based on the positional relationship of the marker images determined in the marker image position determining step A marker image identifying method comprising the steps of: The marker is positioned on an object fixed coordinate defined on the object,
Using the first camera and the second camera, absolute coordinates defined in space are stored,
Further, (f) For each set of marker images associated by the marker image recognition step and the marker image identification step, by creating a search marker by calculating the position on the absolute coordinate by triangulation 5. The marker image identification method according to claim 4, further comprising an object monitoring step of determining a position and an angle of an object on an absolute coordinate by associating a set of the search marker and the marker. The lines connecting at least three lit markers are not equidistant,
(F) The object monitoring step (f-1) a reference marker distance selection step of selecting the shortest or longest marker distance as the reference marker distance on the object fixed coordinates;
(F-2) For each set of marker images associated by the marker image recognition unit and the marker image identification unit, by calculating the position of the search marker on the absolute coordinates by triangulation, the shortest or An inter-marker distance recognition step for recognizing that the longest inter-marker distance is the same inter-marker distance as the reference inter-marker distance;
(F-3) By associating a set of a marker attached to the outer surface of the object and a search marker with reference to the inter-marker distance recognized by the inter-marker distance recognition unit, The marker image identification method according to claim 5, further comprising a marker identification step of determining an object position and angle at

Images