JP2010038847A - Motion tracker system and coordinate system setting method therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a motion tracker system that facilitates precise setting of a camera coordinate system to a photographing area of a camera device in a short time when resetting the camera coordinate system to the photographing area of the camera device. <P>SOLUTION: The motion tracker system including at least three optical markers 4 attached to a reference object 30 includes: a coordinate system setting part 27 setting the camera coordinate system to the photographing area of the camera device 2; a calibration information storage control part 31 creating calibration information that is the positions of the at least three optical markers 4 in the camera coordinate system and storing the information in a calibration information storage part 47; and a coordinate system resetting part 32 resetting the camera coordinate system to the photographing area of the camera device 2 on the basis of optical marker position information obtained by photographing the optical markers 4 attached to the reference object 30 and the calibration information stored in the calibration information storage part 47. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

The present invention relates to a motion tracker system (hereinafter also referred to as an MT system) for measuring the current position and current angle of an object to be measured with respect to the camera coordinate system by setting the camera coordinate system in the imaging region of the camera device, and its The present invention relates to a coordinate system setting method. The present invention particularly relates to a helmet with a head-mounted display device that is worn by a passenger with respect to a camera coordinate system by setting a camera coordinate system in an imaging region of a camera device disposed on a moving body such as a game machine or a vehicle. Head motion tracker system (hereinafter also referred to as an HMT system) combined with an angular velocity sensor and an acceleration sensor and its coordinate system setting in order to always measure the current position and current angle (that is, the current position and current angle of the head) Used for methods.

Techniques for accurately measuring the current position and current angle of an object (measurement object) that changes every moment are used in various fields. For example, in a game machine, in order to realize virtual reality (VR), an image is displayed by using a head-mounted helmet with a display device. At this time, it is necessary to change the image in accordance with the current position and the current angle of the head mounted display-equipped helmet. Therefore, a head motion tracker device (hereinafter also referred to as an HMT device) is used to measure the current position and current angle of the helmet with a head-mounted display device.

Also, in order to avoid losing sight of the found rescue target in rescue operations by rescue flying boats, by locking when the aiming image displayed by the helmet with head mounted display device corresponds to the rescue target, Calculating the position of the locked rescue target has been done. At this time, in order to calculate the position of the rescue target, in addition to the latitude, longitude, altitude and attitude of the flying object (rescue flying boat), the current angle and current position of the pilot's head relative to the flying object are measured. Yes. For this reason, an HMT apparatus is used.

As an HMT device used for a helmet with a head-mounted display device, an apparatus that optically measures the current position and current angle of the helmet with a head-mounted display device is disclosed (for example, see Patent Document 1). ). Specifically, by attaching a plurality of reflectors on the outer peripheral surface of the helmet with a head-mounted display device, and fixing the light source and the camera device (first camera and second camera) to the flying object, the light source The reflected light from the reflector when the light is irradiated from the camera is monitored by the camera device. There is also an optical HMT device in which light emitters are attached to a plurality of locations so as to be separated from each other (see, for example, Patent Document 2). Specifically, on the outer peripheral surface of the helmet with a head-mounted display device, LEDs (light emitting diodes), which are light emitters, are mounted at three locations so as to be separated from each other, and the positional relationship between these three LEDs is determined as HMT. Store in advance in the device. And by photographing these three LEDs simultaneously with the first camera and the second camera that are stereoscopically viewable and fixed to the flying object, the so-called triangulation principle allows the current three LEDs to be The positional relationship is measured. Thereby, the current position and the current angle of the helmet with a head-mounted display device with respect to the camera device are specified.
JP-T 9-506194 Japanese Patent Application No. 2006-284442

Furthermore, the present applicant has developed an HMT device that combines a gyro sensor (angular velocity sensor) and an acceleration sensor. As a result, the current position and current angle of the helmet with a head-mounted display device with respect to the flying object can also be measured using the measurement results of the gyro sensor and the acceleration sensor for the time interval between the first camera and the second camera. Can be calculated.
By the way, in the HMT apparatus as described above, a camera coordinate system is set in advance in the imaging region of the camera apparatus and used in order to express the current positional relationship of the three LEDs measured by the camera apparatus in spatial coordinates. ing. Here, a conventional setting method for setting the camera coordinate system in the imaging region of the camera device 2 (the first camera 2a and the second camera 2b) will be described with reference to FIGS.
The camera coordinate system is set so that the origin and the directions of the XYZ coordinate axes coincide with a reference coordinate system preset for the flying object. As a result, the positions of the reflectors and LEDs mounted on the outer peripheral surface of the head-mounted display-equipped helmet can be expressed in the camera coordinate system, and can be expressed in the reference coordinate system using the values as they are. It has become. In addition, even in an HMT device that combines a gyro sensor and an acceleration sensor, the gyro sensor and the acceleration sensor are also fixed in alignment with the reference coordinate system, so the measurement result of the gyro sensor and the acceleration sensor and the measurement by the camera device The result can be easily combined.

Therefore, in order to set a camera coordinate system that coincides with the reference coordinate system in the imaging region of the camera device 2, LEDs 61a to 61h are arranged at the vertices as shown in FIG. 9 and the sides 62a to 62l are set as the set distance. A cubic lattice 60 is produced and used. By attaching such a cubic lattice 60 to the flying object at a predetermined setting position, the direction from the position of the LED 61a to the position of the LED 61b is the X-axis direction, and the direction from the position of the LED 61a to the position of the LED 61d is the Y-axis. The direction from the position of the LED 61a to the position of the LED 61h is set as the Z-axis direction. Further, the distance from the position of the LED 61a to the position of the LED 61b, the distance from the position of the LED 61a to the position of the LED 61d, and the distance from the position of the LED 61a to the position of the LED 61h are set to be reference distances in each coordinate axis. Yes.

However, although the positional relationship between the LEDs 61a to 61h is stored using the design information (the size information of the cubic lattice such as the set distance) used when producing the cubic lattice 60, the camera coordinate system and the reference coordinates are stored. There was a problem that the system was not completely consistent.
Here, FIG. 11 is a cross-sectional view showing a schematic configuration of a general LED. The LED 61 includes a semiconductor unit 71, a reflecting plate 72, and a lens 73. As shown in FIG. 11, there is a difference that occurs during the manufacture of the LED 61 between the center obtained from the outer shape of the LED 61 and the center where the LED 61 actually emits light. And when memorize | storing the position of LED61, since the light ray radiate | emitted from LED61 with the camera apparatus 2 is detected, the center which LED61 is light-emitting is memorize | stored as a position of LED61. On the other hand, since the cubic lattice 60 cannot easily confirm the difference produced during the manufacture of the LED 61 and can easily confirm the center of the outer shape of the LED 61, the center of the outer shape of the LED 61 is cubic. It is made to be the apex of the lattice 60. Therefore, even if the positional relationship of the LED 61 is stored using the design information, the positions of the camera coordinate system and the reference coordinate system are shifted.

Therefore, the present applicant has developed a calibration device that can accurately set the camera coordinate system in the imaging region of the camera device. FIG. 3 is a diagram illustrating a schematic configuration of the calibration device, and FIG. 4 is a diagram in which the calibration device is attached to a set position of the flying object. The calibration device 50 includes an LED 51 and a stage mechanism 52 that moves the LED 51 in the XYZ directions. Accordingly, after the LED 51 of the calibration device 50 is moved by the set distance in the X direction, after being moved by the set distance in the Y direction, and after being moved by the set distance in the Z direction, the LED 51 is moved. The camera coordinate system is set in the shooting area of the camera device 2 by shooting with the camera device 2 before and after. As a result, even if there is a difference in the manufacturing process of the LED 51 between the center obtained from the outer shape of the LED 51 and the center where the LED 51 actually emits light, Since no difference occurs, the camera coordinate system can be accurately set in the shooting area of the camera device 2.

By the way, in order to set the camera coordinate system in the imaging region of the camera device 2, it is necessary to accurately attach (align) the cubic lattice 60 and the calibration device 50 to a predetermined setting position of the flying object. As a result, the flying object may be restrained for a long time.
When either the first camera 2a or the second camera 2b attached to the flying object is replaced, the camera coordinate system is reset in the imaging area of the camera device 2, but again the cubic lattice 60 or There is a problem that the calibration device 50 needs to be accurately attached to a predetermined setting position of the flying object, and the flying object is restrained for a long time.
Therefore, the present invention provides a motion tracker system capable of easily and accurately setting the camera coordinate system in the shooting area of the camera device in a short time when resetting the camera coordinate system in the shooting area of the camera device. For the purpose.

The motion tracker system of the present invention, which has been made to solve the above-mentioned problems, includes a measuring object to which at least three optical markers are attached, a first camera that photographs the optical markers, and a direction different from that of the first camera. A reference object to which a camera device having a second camera for photographing an optical marker is attached, a first image photographed by the first camera, and a photograph taken by the second camera simultaneously with the first camera photographing. A camera device control unit that acquires the second image, and an optical marker position information calculation unit that calculates optical marker position information that is a current position of the optical marker with respect to the camera device based on the first image and the second image. And a relative position including a current position and a current angle of the measurement object with respect to the reference object based on the optical marker position information. A motion tracker system including a relative information calculation unit for calculating a report, which is attachable to and detachable from a set position in the reference object, and an optical marker and a stage mechanism for moving the optical marker in the XYZ directions And at least three optical markers mounted in the reference object, based on optical marker position information obtained by photographing the optical marker of the calibration device, A coordinate system setting unit that sets a camera coordinate system in a shooting area of the camera device, and a camera coordinate system that is set in the shooting area of the camera device, and then shooting an optical marker attached to the reference object. Based on the obtained optical marker position information, at least three in the camera coordinate system A calibration information storage control unit that creates calibration information that is a coordinate position of a scientific marker and stores the calibration information in a calibration information storage unit, and the reference object when resetting the camera coordinate system in the imaging region of the camera device Based on the optical marker position information obtained by photographing the optical marker attached to the object and the calibration information stored in the calibration information storage unit, a camera coordinate system is provided in the photographing region of the camera device. And a coordinate system resetting unit to be set.

Here, examples of the “optical marker” include a light emitter (lamp, LED, etc.), a reflector, and a phosphor.
According to the MT system of the present invention, at least three optical markers are attached in the imaging region of the camera device on the reference object.
When the camera coordinate system is set for the imaging region of the camera device for the first time, first, a calibration device capable of moving the optical marker in the XYZ directions is attached to a setting position in the imaging region of the camera device. . Next, the camera device control unit acquires the first image and the second image by photographing the optical marker of the calibration device. Accordingly, the optical marker position information calculation unit calculates and stores the optical marker position information that is the current position of the optical marker with respect to the camera device. Next, after moving the optical marker by a set distance in the X direction, the camera device control unit acquires the first image and the second image by photographing the optical marker of the calibration device. Accordingly, the optical marker position information calculation unit calculates and stores the optical marker position information that is the current position of the optical marker with respect to the camera device. Next, after moving the optical marker by a set distance in the Y direction, the camera device control unit acquires the first image and the second image by photographing the optical marker of the calibration device. Accordingly, the optical marker position information calculation unit calculates and stores the optical marker position information that is the current position of the optical marker with respect to the camera device. Next, after the optical marker is moved by a set distance in the Z direction, the camera device control unit acquires the first image and the second image by photographing the optical marker of the calibration device. Accordingly, the optical marker position information calculation unit calculates and stores the optical marker position information that is the current position of the optical marker with respect to the camera device. As a result, the coordinate system setting unit sets the camera coordinate system in the imaging region of the camera device based on the stored optical marker position information. Next, after setting the camera coordinate system in the imaging region of the camera device, the calibration device is removed from the set position.

By the way, in the MT system of the present invention, when resetting the camera coordinate system in the imaging area of the camera device, it is not necessary to attach a cubic lattice, a calibration device or the like to a predetermined setting position of the reference object. ing. Therefore, in the MT system of the present invention, after removing the calibration device from the set position, the camera device controller captures the first image and the first image by photographing at least three optical markers attached to the reference object. Acquire a second image. Thereby, the optical marker position information calculation unit calculates the optical marker position information which is the current position of the optical marker with respect to the camera device, and the calibration information storage control unit calculates the optical information for the camera coordinate system based on the optical marker position information. Calibration information that is the coordinate position of the marker is calculated and stored in the calibration information storage unit.

After that, if it is necessary to reset the camera coordinate system in the shooting area of the camera device, such as when at least one of the first camera or the second camera attached to the reference object is replaced, The calibration information stored in the calibration information storage unit is used.
Here, the positions of the at least three optical markers attached in the reference object do not vary. That is, when resetting the camera coordinate system in the shooting area of the camera device, the camera coordinate system is set in the same manner as when the camera coordinate system is set for the first time, so the coordinate position of the optical marker in the reset camera coordinate system is This is the same as the coordinate position of the optical marker in the camera coordinate system set for the first time. Conversely, if the coordinate position of the optical marker in the reset camera coordinate system is the same as the coordinate position of the optical marker in the camera coordinate system set for the first time, the reset camera coordinates The system and the camera coordinate system set for the first time are set to be the same.
Therefore, in the MT system of the present invention, the camera device control unit acquires the first image and the second image by photographing at least three optical markers attached to the reference object. Thereby, the optical marker position information calculation unit calculates the optical marker position information which is the current position of the optical marker with respect to the camera device, and the coordinate system resetting unit is stored in the optical marker position information and the calibration information storage unit. Based on the calibration information, a camera coordinate system is set in the imaging region of the camera device. At this time, the camera coordinate system is set so that the coordinate position of each optical marker in the camera coordinate system set for the first time is the same as the coordinate position of each obtained optical marker.

As described above, according to the MT system of the present invention, when the camera coordinate system is reset in the imaging area of the camera device, the cubic lattice and the calibration device are accurately placed at the predetermined set position of the reference object. The camera coordinate system can be easily and accurately set in the imaging region of the camera device in a short time without the need for attachment. As a result, the reference object is not restrained for a long time.

(Means and effects for solving other problems)
Also, in the MT system of the present invention, the coordinate system setting unit moves the optical marker of the calibration apparatus in the X direction by a set distance, after moving the calibration marker in the Y direction by a set distance, and in the Z direction. After moving the camera at a set distance and before moving the optical marker, based on the optical marker position information obtained by acquiring the first image and the second image, respectively, the imaging region of the camera device You may make it set a camera coordinate system.
In the MT system of the present invention, when the coordinate system resetting unit is replaced with at least one of the first camera and the second camera attached to the reference object, The camera coordinate system may be reset.

The MT system of the present invention includes at least three optical markers attached to the measurement object, an optical marker attached to the calibration device, and at least three optical markers attached to the reference object. The optical marker may emit infrared light having the same wavelength.
In the MT system of the present invention, at least four optical markers may be attached to the reference object, and at least four optical markers may not exist on the same plane.

Furthermore, the coordinate system setting method of the present invention photographs a measurement object to which at least three optical markers are attached, a first camera that photographs the optical marker, and an optical marker from a different direction from the first camera. A reference object to which a camera device having a second camera is attached; a first image photographed by the first camera; and a second image photographed by the second camera at the same time as the first camera. A camera device control unit to acquire, an optical marker position information calculation unit that calculates optical marker position information that is a current position of the optical marker with respect to the camera device based on the first image and the second image, and the optical marker position A relative information calculation unit that calculates relative information including a current position and a current angle of the measurement object with respect to the reference object based on the information; And at least three optical markers attached to the quasi-object, the optical marker, and a stage mechanism for moving the optical marker in the XYZ directions. A coordinate system setting method for setting a camera coordinate system in an imaging region of a camera device used in a motion tracker system including a calibration device, the optical marker obtained by photographing an optical marker of the calibration device A coordinate system setting step of setting a camera coordinate system in the imaging region of the camera device based on position information; and an optical marker attached to the reference object after setting the camera coordinate system in the imaging region of the camera device Based on the optical marker position information obtained by photographing A calibration information storage control step of creating calibration information that is the coordinate positions of at least three optical markers in the standard system and storing them in the calibration information storage unit, and resetting the camera coordinate system in the imaging area of the camera device In doing so, based on the optical marker position information obtained by photographing the optical marker attached to the reference object and the calibration information stored in the calibration information storage unit, A coordinate system resetting step of setting the camera coordinate system in the imaging region.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiments described below, and it goes without saying that various aspects are included without departing from the spirit of the present invention.

The HMT system according to an embodiment of the present invention includes an HMT device 1 and a calibration device 50. FIG. 1 is a diagram showing a schematic configuration of an HMT device 1 according to the present invention, and FIG. 2 is a plan view of the helmet (measurement object) 10 with a head-mounted display device shown in FIG. 3 is a diagram showing a schematic configuration of the calibration device 50 according to the present invention, and FIG. 4 is a diagram in which the calibration device 50 is attached to the set position of the HMT device 1.
Note that the HMT device 1 includes the head position (Xh, Yh, Zh) and head angle (Θh, Φh, Ψh) of the pilot (passenger) 3 with respect to the camera coordinate system (XYZ coordinate system) set in the camera device 2. ) Including relative information is calculated. That is, the helmet coordinate system (X′Y ′) set in the helmet 10 with a head mounted display device worn by the pilot 3 with respect to the camera coordinate system (XYZ coordinate system) set in the vehicle 30 (reference object). The position and angle of the Z ′ coordinate system) are calculated.
In addition, the calibration device 50 is a camera arranged on the vehicle 30 to calculate relative information including the head position (Xh, Yh, Zh) and the head angle (Θh, Φh, Ψh) of the pilot 3. This is for setting a camera coordinate system (XYZ coordinate system) in the imaging region of the apparatus 2. In the present embodiment, the calibration device 50 is used when setting the first camera coordinate system (XYZ coordinate system), and the calibration device 50 is used when resetting the camera coordinate system (XYZ coordinate system). LED group 4 is used without using.

The HMT device 1 includes a helmet 10 with a head-mounted display device attached to the head of a pilot 3, a camera device 2 fixed to the vehicle 30, an LED group 4 attached to the vehicle 30, a computer It is comprised from the control part 20 comprised by these.
The helmet 10 with a head-mounted display device is attached to a display device (not shown), a combiner 8 that guides the eyes of the pilot 3 by reflecting image display light emitted from the display device, and an outer peripheral surface. LED group 7. In addition, the pilot 3 wearing the helmet 10 with a head-mounted display device can visually recognize the display image by the display and the front actual thing of the combiner 8.

As shown in FIG. 2, the LED group 7 includes an outer peripheral surface of the helmet 10 with a head mounted display device such that three LEDs 7a, 7b, and 7c that emit infrared light having different wavelengths are separated from each other. It is attached to the top.
Here, the helmet coordinate system (X′Y′Z ′ coordinate system) can arbitrarily determine the origin and the direction of each coordinate axis. In this embodiment, as shown in FIG. 2, the origin is the position of the LED 7a. As will be described later, the forward direction is defined as the X ′ axis direction, the forward direction is defined as the Y ′ axis direction, and the X ′ axis direction and the Y ′ axis direction are defined as the Z ′ axis direction. It is set in the data storage unit 45. The positional relationship (initial data) of the three LEDs 7a, 7b, 7c on the helmet coordinate system (X′Y′Z ′ coordinate system) is also stored in the data storage unit 45. Thereby, the position of the three LEDs 7a, 7b, and 7c at the present time is calculated by a triangulation method, which will be described later, and the initial position is referred to, so that the current position and the current angle of the head mounted display device helmet 10 are determined. It is expressed using a helmet coordinate system (X′Y′Z ′ coordinate system).

The mounted body 30 is a cockpit of a flying body on which the pilot 3 is boarded. The mounted body 30 includes a seat 30a on which the pilot 3 is seated, and a set position (described later) for attaching the calibration device 50 is determined.
The camera device 2 includes a first camera 2a and a second camera 2b. The first camera 2a and the second camera 2b are fixed to the seat 30a so as to be separated from each other by a certain distance (d1) in which the photographing direction is different and stereoscopic viewing is possible.
Here, as shown in FIG. 5, the position of the LED 7a with respect to the camera device 2 is the position of the LED 7a displayed in the first image and the second image taken by the first camera 2a and the second camera 2b. Further, the direction angle (α) from the first camera 2a and the direction angle (β) from the second camera 2b are extracted, and the distance (d1) between the first camera 2a and the second camera 2b is extracted. By using it, it can be calculated by a triangulation method. The positions of the other LEDs 7b and 7c with respect to the camera device 2 are similarly calculated.

In order to be able to express the position of each LED 7a, 7b, 7c at this time by a spatial coordinate, the camera coordinate system (in the imaging region of the camera device 2 using the calibration device 50, the LED group 4, etc.) XYZ coordinate system) is set. A setting method for setting the camera coordinate system using the calibration device 50, a resetting method for resetting the camera coordinate system using the LED group 4, a specific origin position of the camera coordinate system, and the XYZ axis directions The description of these will be described later.
If the camera coordinate system is set in the imaging area of the camera device 2, the coordinate positions of the LEDs 7a, 7b, and 7c can be expressed using the camera coordinate system by calculating with the triangulation method as described above. If the coordinate positions of the three LEDs 7a, 7b, and 7c are specified, the position and angle of the helmet 10 with a head-mounted display device to which the LEDs 7a, 7b, and 7c are positioned and attached are determined by the camera coordinate system. It can be expressed by using the position (Xh, Yh, Zh) and angle (Θh, Φh, Ψh) of the helmet coordinate system (X′Y′Z ′ coordinate system) with respect to (XYZ coordinate system). The angle (Θh) is the angle in the roll direction (rotation with respect to the X axis), the angle (Φh) is the angle in the elevation direction (rotation with respect to the Y axis), and the angle (Ψh) is in the azimuth direction ( Angle of rotation with respect to the Z-axis). Moreover, the position (Xh, Yh, Zh) of the helmet 10 with a head-mounted display device is expressed by the current coordinate position of the LED 7a which is the origin of the helmet coordinate system (X′Y′Z ′ coordinate system). To do.
In this embodiment, since the camera coordinate system is set by a setting method to be described later, the reference coordinate system set in advance for the vehicle 30 and the camera coordinate system completely match. For this reason, the position (Xh, Yh, Zh) and angle (Θh, Φh, Ψh) of the helmet coordinate system with respect to the camera coordinate system are used as they are, and the position (Xh, Yh) of the helmet coordinate system with respect to the reference coordinate system is used. , Zh) and angles (Θh, Φh, Ψh).

As shown in FIGS. 1, 4 and 7, the LED group 4 is attached to the vehicle body 30 so that nine LEDs 4 a to 4 i emitting infrared light having different wavelengths are separated from each other. It is. In addition, nine LED4a-4i is attached to the vehicle body 30 so that it may not exist in the same plane.
And the calibration information which is the positional relationship of nine LED4a-4i on a camera coordinate system (XYZ coordinate system) will be memorize | stored in the calibration information storage part 47, Thereby, LED group 4 is stored. It is possible to reset the camera coordinate system to the camera device 2 by using it.

The control unit 20 is configured by a computer including a CPU 21, a memory 41, and the like, and performs various controls and arithmetic processes. The processing executed by the CPU 21 of the control unit 20 will be described separately for each functional block. The camera device control unit 28, the optical marker position information calculation unit 22, the relative information calculation unit 23, the coordinate system setting unit 27, The driving signal generating unit 26, the video display unit 25, the calibration information storage control unit 31, and the coordinate system resetting unit 32 are included.
The memory 41 is formed with an area for accumulating various data necessary for the control unit 20 to execute processing, a reference coordinate system storage unit 43 for storing a reference coordinate system, and a camera coordinate system (XYZ). A camera coordinate system storage unit 44 that stores a coordinate system), a data storage unit 45 that stores a helmet coordinate system (X′Y′Z ′ coordinate system), and an optical marker position information storage unit that sequentially stores optical marker position information. 46 and a calibration information storage unit 47 for storing calibration information.
The data storage unit 45 stores a helmet coordinate system (X′Y′Z ′ coordinate system), and further, a positional relationship between the three LEDs 7a, 7b, and 7c on the X′Y′Z ′ coordinate system ( (Initial data) is also stored.

Here, the reference coordinate system is a coordinate system that moves together with the vehicle 30, and the origin and the direction of each coordinate axis are arbitrarily determined in advance, but in the present embodiment, a set position for attaching the calibration device 50 The reference coordinate system storage unit has an origin point as the origin, the forward direction as the X-axis direction, the forward direction as the Y-axis direction, and the X-axis direction and the Y-axis direction as the Z-axis direction. 43 is set in advance.
The camera coordinate system (XYZ coordinate system) is set so that the origin and the directions of the XYZ coordinate axes coincide with the reference coordinate system set for the vehicle 30. Thereby, it becomes possible to combine with the measurement result of the gyro sensor and acceleration sensor (not shown) aligned with the reference coordinate system set in the vehicle 30.

The camera device control unit 28 performs control to acquire a first image captured by the first camera 2a and a second image captured by the second camera 2b at the same time as the first camera 2a captures the image.
Based on the first image and the second image, the optical marker position information calculation unit 22 calculates optical marker position information that is the current position of the LED group 7, LED group 4, and LED 51 (described later) with respect to the camera device 2. The optical marker position information is sequentially stored in the optical marker position information storage unit 46.
For example, the position of the LED 7a with respect to the camera device 2 is obtained by extracting the first image captured by the first camera 2a and the second camera 2b and the position of the LED 7a displayed in the second image, and further by the first camera 2a. Triangulation method by extracting the direction angle (α) from the second camera 2b and the direction angle (β) from the second camera 2b and using the distance (d1) between the first camera 2a and the second camera 2b Calculate with

The relative information calculation unit 23 uses the optical marker position information that is the current position of the LEDs 7a, 7b, and 7c with respect to the camera device 2, and uses the current position (Xh, Yh, Zh) and control for calculating relative information including the current angle (Θh, Φh, Ψh).
Specifically, after setting the camera coordinate system (XYZ coordinate system) in the imaging region of the camera device 2 by the coordinate system setting unit 27 and the coordinate system resetting unit 32, the camera device control unit 28 sets the first image and the second image. Get images and. Since the camera coordinate system is set in the shooting area of the camera device 2, the current position of the LEDs 7a, 7b, and 7c with respect to the camera coordinate system is obtained using the optical marker position information that is the current position of the LEDs 7a, 7b, and 7c with respect to the camera device 2. Create the coordinate position of. Then, by comparing the current coordinate position of the LEDs 7a, 7b, 7c with respect to the camera coordinate system and the initial data stored in the data storage unit 45, the head mounted display device with the LED group 7 fixed thereto is provided. The position (Xh, Yh, Zh) and angle (Θh, Φh, Ψh) of the helmet 10 with respect to the camera coordinate system are calculated.
The video display unit 25 emits image display light from the display based on relative information including the position (Xh, Yh, Zh) and the angle (Θh, Φh, Ψh) of the helmet 10 with a head-mounted display device. Take control. Thereby, the pilot 3 visually recognizes the display image by the display.

Next, a setting method for setting a camera coordinate system (XYZ coordinate system) in the imaging region of the camera device 2 using the calibration device 50, or a camera coordinate system (XYZ) in the imaging region of the camera device 2 using the LED group 4. A resetting method for resetting the coordinate system will be described with reference to FIGS. 3, 4, 6 and 7.
In the present embodiment, the calibration device 50 is used when setting the first camera coordinate system (XYZ coordinate system), and the calibration device 50 is used when resetting the camera coordinate system (XYZ coordinate system). LED group 4 is used without using.

First, the calibration device 50 for setting the camera coordinate system (XYZ coordinate system) in the imaging region of the camera device 2 will be described.
The calibration apparatus 50 includes one LED 51 that emits infrared light and a stage mechanism 52 that can move the one LED 51 in the XYZ directions.
The stage mechanism 52 includes an upper plate body 52a, a middle shaft body 52b, a lower plate body 52c, and an installation member 52d. The LED 51 is fixed on the upper plate-like body 52a. The upper plate-like body 52a is capable of translational movement in the Z direction with respect to the middle shaft body 52b, and the LED 51 fixed to the upper plate-like body 52a can be moved in the Z direction. Further, the middle shaft body 62b is capable of translational movement in the X direction with respect to the lower plate body 52c together with the upper plate body 52a, and the LED 51 fixed to the upper plate body 52a is moved in the X direction. Can be moved. Further, the lower plate-like body 52c is capable of translational movement in the Y direction with respect to the installation member 52d together with the upper plate-like body 52a and the middle shaft body 62b, and the LED 51 fixed to the upper plate-like body 52a. Can be moved in the Y direction. The installation member 52d is positioned at the set position so as to be attachable / detachable.
The control of the stage mechanism 52 is executed by connecting the calibration device 50 and the computer 20 with the wiring cord 53 and receiving a drive signal output from the drive signal generator 26 of the computer 20.

When the camera coordinate system (XYZ coordinate system) is set in the imaging region of the camera device 2 by the calibration device 50, the calibration device 50 is placed in the imaging region of the camera device 2 as shown in FIG. Will be attached. At this time, the LED 51 is arranged at the origin of the reference coordinate system, the X axis direction of the reference coordinate system and the X direction of the stage mechanism 52 are aligned, and the Y axis direction of the reference coordinate system and the Y direction of the stage mechanism 52 are aligned. Alignment is performed, and the Z-axis direction of the reference coordinate system and the Z direction of the stage mechanism 52 are aligned and attached. Then, after setting the camera coordinate system (XYZ coordinate system) in the imaging region of the camera device 2 by the calibration device 50, the calibration device 50 is removed from the imaging region of the camera device 2.

The drive signal generator 26 performs control to output a drive signal to the stage mechanism 52 of the calibration device 50 by receiving a calibration signal from an input device (not shown) or the like.
For example, as illustrated in FIG. 6, by receiving a calibration signal, first, the camera device control unit 28 is caused to acquire a first image and a second image as the initial position 51 a of the LED 51. Next, after outputting a drive signal for moving the LED 51 by a set distance in the X direction to the stage mechanism 62, the camera device control unit 28 acquires the first image and the second image as the second position 51b of the LED 51. In this way, after outputting a drive signal for moving the LED 51 to a predetermined position, the camera device control unit 28 sequentially acquires the first image and the second image as each position of the LED 51, and the cubic lattice is obtained. A first image and a second image are acquired by photographing the LEDs 51 at a total of eight positions 51a to 51h corresponding to the vertices.

The coordinate system setting unit 27 performs control to set a camera coordinate system (XYZ coordinate system) in the shooting region of the camera device 2 based on the optical marker position information that is each position of the LED 51 with respect to the camera device 2.
Specifically, the direction from the initial position 51a of the LED 51 to the second position 51b of the LED 51 is the X-axis direction, the direction from the initial position 51a of the LED 51 to the fourth position 51d of the LED 51 is the Y-axis direction, The direction from the position 51a to the eighth position 51h of the LED 51 is set to be the Z-axis direction. Further, the distance from the initial position 51a of the LED 51 to the second position 51b of the LED 51, the distance from the initial position 51a of the LED 51 to the fourth position 51d of the LED 51, and the distance from the initial position of the LED 51 to the eighth position 51h of the LED 51. Are set to be a reference distance in each coordinate axis.

The calibration information storage control unit 31 sets the camera coordinate system (XYZ coordinate system) in the imaging area of the camera device 2 by the coordinate system setting unit 27 and then acquires the optical marker position information obtained by imaging the LED group 4. Based on the above, calibration information that is the coordinate positions of the nine LEDs 4a to 4i in the camera coordinate system is generated and stored in the calibration information storage unit 47.
Specifically, as shown in FIG. 7, after the camera coordinate system (XYZ coordinate system) is set in the imaging region of the camera device 2 by the coordinate system setting unit 27, the first image and the second image are transmitted to the camera device control unit 28. Get images and. Since the camera coordinate system is set in the shooting area of the camera device 2, the nine LED 4a to 4i with respect to the camera coordinate system are used by using the optical marker position information which is the position of the nine LEDs 4a to 4i with respect to the camera device 2. Create calibration information that is the coordinate position. Then, the calibration information is stored in the calibration information storage unit 47.
Thereby, when resetting the camera coordinate system in the imaging region of the camera device 2, it is not necessary to attach the cubic lattice 60, the calibration device 50, or the like to a predetermined setting position of the vehicle body 30.

The coordinate system resetting unit 31 includes optical marker position information obtained by photographing the LED group 4 and calibration information when the camera coordinate system (XYZ coordinate system) is reset in the photographing region of the camera device 2. Based on the calibration information stored in the storage unit 47, control for setting the camera coordinate system in the imaging region of the camera device 2 is performed.
For example, when one of the first camera 2a and the second camera 2b attached to the vehicle 30 is replaced, when it is necessary to reset the camera coordinate system in the shooting area of the camera device 2, first, By photographing the nine LEDs 4a to 4i, the camera device control unit 28 is caused to acquire the first image and the second image. Thereby, the optical marker position information which is a position of nine LED4a-4i with respect to the camera apparatus 2 is obtained. On the other hand, based on the calibration information stored in the calibration information storage unit 47, calibration information that is the coordinate positions of the nine LEDs 4a to 4i with respect to the camera coordinate system set for the first time is obtained. Then, the camera coordinate system is set so that the coordinate position of each LED in the camera coordinate system set for the first time is the same as the obtained coordinate position of each LED.
Accordingly, there is no need to accurately attach the cubic lattice 60 and the calibration device 50 to the predetermined setting position of the vehicle body 30, and the reset camera coordinate system and the camera coordinate system set for the first time are provided. It is set to be the same.

Here, a setting method for setting a camera coordinate system (XYZ coordinate system) in the imaging region of the camera device 2 using the calibration device 50 using the HMT system, and a camera in the imaging region of the camera device 2 using the LED group 4 A resetting method for resetting the coordinate system (XYZ coordinate system) will be described. FIG. 8 is a flowchart for explaining the setting method and the resetting method.
First, the camera device 2 is attached to the vehicle 30 in the process of step S101.
Next, in the process of step S102, the vehicle 30 leveling (aircraft leveling) is executed.

Next, in the process of step S103, the calibration device 50 is attached to the set position.
Next, in the process of step S104, the calibration device 50 mounting alignment is executed.
Next, in the process of step S105, the drive signal generator 26 receives the calibration signal from the input device or the like, thereby outputting the drive signal to the stage mechanism 52 of the calibration device 50 and the camera device controller 28. Takes a picture of the LED 51.

Next, in the process of step S <b> 106, the coordinate system setting unit 27 sets the camera coordinate system (XYZ coordinate system) in the shooting area of the camera device 2 based on the optical marker position information that is each position of the LED 51 with respect to the camera device 2. Set (coordinate system setting step).
Next, in the process of step S107, the calibration device 50 is removed from the set position.
Next, in the process of step S108, the calibration information storage control unit 31 determines the coordinate positions of the nine LEDs 4a to 4i in the camera coordinate system based on the optical marker position information obtained by photographing the LED group 4. Is created and stored in the calibration information storage unit 47 (calibration information storage control step).

Next, in the processing of step S109, the relative information calculation unit 23 uses the optical marker position information that is the current position of the LEDs 7a, 7b, and 7c with respect to the camera device 2 obtained by photographing the LED group 7 to Relative information including the current position (Xh, Yh, Zh) and the current angle (Θh, Φh, Ψh) of the helmet 10 with a head-mounted display device with respect to the coordinate system is calculated.
Next, in the process of step S110, the video display unit 25 is based on relative information including the position (Xh, Yh, Zh) and the angle (Θh, Φh, Ψh) of the helmet 10 with a head-mounted display device. Image display light is emitted from the display.
Next, in the process of step S111, it is determined whether or not the camera device 2 is to be replaced. When it is determined that the camera device 2 is to be replaced, the camera device 2 is attached to the vehicle 30 in the process of step S112. That is, it is necessary to reset the camera coordinate system in the shooting area of the camera device 2.

Next, in the process of step S113, the camera device control unit 28 images the LED group 4.
Next, in the process of step S <b> 114, the coordinate system resetting unit 31 is based on the optical marker position information obtained by photographing the LED group 4 and the calibration information stored in the calibration information storage unit 47. Then, the camera coordinate system is set in the imaging area of the camera device 2 (coordinate system resetting step), and the process returns to step S109. That is, it is not necessary to accurately attach the cubic lattice 60 and the calibration device 50 to the predetermined setting position of the vehicle body 30, and the camera coordinate system can be set in the imaging region of the camera device 2.
On the other hand, when it is determined that the camera device 2 is not to be replaced, the process returns to step S109.

As described above, according to the HMT system of the present invention, when the camera coordinate system (XYZ coordinate system) is reset in the imaging region of the camera device 2, the cubic lattice 60 and the calibration device 50 are preliminarily set in the vehicle body 30. The camera coordinate system can be easily and accurately set in the imaging region of the camera device 2 in a short time without having to be accurately attached to the set position. As a result, the vehicle body 30 is not restrained for a long time.

(Other embodiments)
(1) In the above-described HMT system, the LED group 7 emits three LEDs 7a to 7c that emit infrared light having different wavelengths, and the LED group 4 emits nine LEDs 4a to emit infrared light having different wavelengths. 4i is attached, but it may be configured to emit infrared light having the same wavelength.
(2) In the HMT system described above, the LED group 4 has a configuration in which nine LEDs 4a to 4i are attached, but may have a configuration in which three or more LEDs are attached.

The present invention can be used for a head motion tracker device and the like for measuring a head angle and a head position with respect to a reference coordinate system set for a flying object or the like.

It is a figure which shows schematic structure of the HMT apparatus which concerns on this invention. It is a top view of the helmet with a head-mounted display device shown in FIG. It is a figure which shows schematic structure of the calibration apparatus which concerns on this invention. It is the figure which attached the calibration apparatus to the setting position of the HMT apparatus. It is a figure for demonstrating the calculation method which calculates the position of LED with respect to a camera apparatus. It is a figure for demonstrating the setting method which sets a camera coordinate system to the imaging | photography area | region of a camera apparatus. It is a figure for demonstrating the resetting method which resets a camera coordinate system to the imaging | photography area | region of a camera apparatus. It is a flowchart for demonstrating a setting method and a resetting method. It is a figure for demonstrating the conventional setting method which sets a camera coordinate system to the imaging | photography area | region of a camera apparatus. It is a figure for demonstrating the conventional setting method which sets a camera coordinate system to the imaging | photography area | region of a camera apparatus. It is sectional drawing which shows schematic structure of LED.

Explanation of symbols

1 Head motion tracker device (HMT device)
2 Camera device 2a First camera 2b Second camera 3 Pilot 4, 7 LED group (optical marker group)
10 Helmet with head mounted display (measurement object)
22 Optical marker position information calculation unit 23 Relative information calculation unit 27 Coordinate system setting unit 28 Camera device control unit 30 Vehicle (reference object)
31 Calibration information storage control unit 32 Coordinate system resetting unit 46 Optical marker position information storage unit 47 Calibration information storage unit 50 Calibration device 51 LED (optical marker)
52 Stage mechanism

Claims (6)

A measurement object having at least three optical markers attached thereto;
A reference object to which a camera device having a first camera for photographing the optical marker and a second camera for photographing the optical marker from a different direction from the first camera is attached;
A camera device controller that acquires a first image captured by the first camera and a second image captured by the second camera at the same time as the first camera captures;
Based on the first image and the second image, an optical marker position information calculating unit that calculates optical marker position information that is a current position of the optical marker with respect to the camera device;
A motion tracker system comprising: a relative information calculation unit that calculates relative information including a current position and a current angle of a measurement object with respect to the reference object based on the optical marker position information;
A calibration device comprising an optical marker and a stage mechanism for moving the optical marker in the XYZ directions, which can be attached to and detached from a set position in the reference object.
And at least three optical markers mounted in the reference object,
A coordinate system setting unit that sets a camera coordinate system in an imaging region of the camera device based on optical marker position information obtained by imaging the optical marker of the calibration device;
After setting the camera coordinate system in the imaging area of the camera device, based on the optical marker position information obtained by imaging the optical marker attached to the reference object, at least three in the camera coordinate system A calibration information storage control unit that creates calibration information that is the coordinate position of the optical marker and stores the calibration information in the calibration information storage unit;
When resetting the camera coordinate system in the imaging area of the camera device, the optical marker position information obtained by imaging the optical marker attached to the reference object and the calibration information storage unit are stored. A motion tracker system comprising: a coordinate system resetting unit that sets a camera coordinate system in an imaging region of the camera device based on the calibration information. The coordinate system setting unit moves the optical marker of the calibration device in the X direction by a set distance, moves the optical marker by a set distance in the Y direction, and moves the optical marker by a set distance in the Z direction. Before setting the camera coordinate system in the imaging region of the camera device based on the optical marker position information obtained by acquiring the first image and the second image, respectively, before moving the optical marker. The motion tracker system according to claim 1, wherein: The coordinate system resetting unit resets the camera coordinate system in the shooting area of the camera device when at least one of the first camera and the second camera attached to the reference object is replaced. The motion tracker system according to claim 1, wherein the motion tracker system is characterized. At least three optical markers attached to the measurement object, an optical marker attached to the calibration device, and at least three optical markers attached to the reference object have the same wavelength. The motion tracker system according to claim 1, wherein the motion tracker system emits infrared light. At least four optical markers are attached in the reference object,
The motion tracker system according to any one of claims 1 to 4, wherein at least four optical markers do not exist on the same plane. A measurement object having at least three optical markers attached thereto;
A reference object to which a camera device having a first camera for photographing the optical marker and a second camera for photographing the optical marker from a different direction from the first camera is attached;
A camera device controller that acquires a first image captured by the first camera and a second image captured by the second camera at the same time as the first camera captures;
Based on the first image and the second image, an optical marker position information calculating unit that calculates optical marker position information that is a current position of the optical marker with respect to the camera device;
Based on the optical marker position information, a relative information calculation unit that calculates relative information including a current position and a current angle of the measurement object with respect to the reference object;
At least three optical markers mounted in the reference object;
A camera device used in a motion tracker system that is attachable to and detachable from a set position in the reference object, and that includes a calibration device that includes an optical marker and a stage mechanism that moves the optical marker in the XYZ directions. A coordinate system setting method for setting a camera coordinate system in a shooting area,
A coordinate system setting step for setting a camera coordinate system in an imaging region of the camera device based on optical marker position information obtained by imaging the optical marker of the calibration device;
After setting the camera coordinate system in the imaging area of the camera device, based on the optical marker position information obtained by imaging the optical marker attached to the reference object, at least three in the camera coordinate system A calibration information storage control step of creating calibration information which is the coordinate position of the optical marker and storing it in the calibration information storage unit;
When resetting the camera coordinate system in the imaging area of the camera device, the optical marker position information obtained by imaging the optical marker attached to the reference object and the calibration information storage unit are stored. And a coordinate system resetting step of setting a camera coordinate system in the imaging region of the camera device based on the calibration information.

Images