JP7435330B2 - Head motion tracker device and head motion tracker device for aircraft - Google Patents

Description

The present invention relates to a head motion tracker device and an aircraft head motion tracker device, and more particularly to a head motion tracker device and an aircraft head motion tracker device that include a head attachment section that is mounted on the head of an observer.

BACKGROUND ART Conventionally, head motion tracker devices and aircraft head motion tracker devices that include a head-mounted section that is mounted on an observer's head are known (for example, see Patent Document 1).

The above-mentioned Patent Document 1 discloses a head motion tracker device that includes a helmet (head attachment part) that is attached to the head of a pilot (observer). This head motion tracker device includes the helmet, a camera device installed in a passenger body (aircraft body), and a control section.

In the helmet of Patent Document 1, a plurality of head-mounted body light emitting parts that emit infrared light of different wavelengths are arranged. The control unit is configured to perform control to obtain the position and angle of the pilot's head based on position information of each of the plurality of head-mounted body light emitting units photographed by the camera device. In the head motion tracker device, a camera device is arranged at a position where a plurality of head-mounted body light emitting units can be reliably photographed in order to obtain the position and angle of the pilot's head by the control unit.

Japanese Patent Application Publication No. 2009-109319

However, in the head motion tracker device of Patent Document 1, it is necessary to install the camera device inside the aircraft body at a position where the plurality of head-mounted body light emitting parts can be reliably photographed. (Image acquisition unit) installation space must be secured. For this reason, with the head motion tracker device of Patent Document 1, it may be difficult to secure the above-mentioned installation space in an existing vehicle; It is also desired that the technology can be easily applied.

This invention has been made to solve the above problems, and one object of the invention is to provide a head motion tracker device and aircraft that can be easily applied even to existing moving objects. An object of the present invention is to provide a head motion tracker device for use in a vehicle.

In order to achieve the above object, a head motion tracker device according to a first aspect of the present invention includes a head-mounted part that is mounted on the head of an observer, a head-mounted part that is attached to the head-mounted part, and a head motion tracker device that is mounted on the head of the observer. an image acquisition unit that acquires an image in a direction along the direction of the image acquisition unit; and a control unit configured to perform control to acquire the position and angle of the observer's head based on the image acquired by the image acquisition unit. The control unit is configured to perform control to create a three-dimensional surrounding environment map around the observer based on the image acquired by the image acquisition unit, and the surrounding environment map is A first surrounding environment map made up of the first image acquired by the image obtaining unit in , a second surrounding environment map made up of the second image before the first image, and a second surrounding environment map made up of the first image acquired by the image acquisition unit; and a third surrounding environment map based on a composite image, the control unit acquires the first surrounding environment map based on the second surrounding environment map and the third surrounding environment map, and The controller is configured to perform control to estimate the position and angle of the part .

As described above, the head motion tracker device according to the first aspect of the invention includes an image acquisition section attached to the head mounting section. Further, the head motion tracker device includes a control unit configured to perform control to acquire the position and angle of the observer's head based on the image acquired by the image acquisition unit. Thereby, the position and angle of the observer's head can be acquired by the control unit based on the image of the image acquisition unit attached to the head mounted unit rather than the moving object. That is, the position and angle of the observer's head can be acquired by the control unit without installing an image acquisition unit in the moving object. As a result, there is no need to secure installation space for the image acquisition unit in the moving object, so the head motion tracker device can be easily applied even to an existing moving object.

In order to achieve the above object, a head motion tracker device for an aircraft according to a second aspect of the present invention includes a head motion tracker device worn by a pilot as an observer inside an aircraft body, and the head motion tracker device , a head-mounted section attached to the pilot's head; an image acquisition section attached to the head-mounted section that acquires an image in a direction along the direction of the pilot's head; and an image acquired by the image acquisition section. a control unit configured to perform control to acquire the position and angle of the pilot's head based on the image acquisition unit; The surrounding environment map is configured to control the creation of a three-dimensional surrounding environment map, and the surrounding environment map includes a first surrounding environment map formed from a first image acquired by the image acquisition unit at the present time, and a first The controller includes a second surrounding environment map configured by a second image before the image, and a third surrounding environment map based on an image obtained by combining the first image with the second image, and the control unit controls the second surrounding environment map. The first surrounding environment map is acquired based on the first surrounding environment map and the third surrounding environment map, and control is performed to estimate the position and angle of the observer's head .

The aircraft head motion tracker device according to the second aspect of the invention includes the image acquisition section attached to the head mounting section, as described above. Further, the head motion tracker device includes a control unit configured to perform control to acquire the position and angle of the observer's head based on the image acquired by the image acquisition unit. Thereby, the position and angle of the observer's head can be acquired by the control unit based on the image of the image acquisition unit attached to the head mounted unit instead of the moving body. That is, the position and angle of the observer's head can be acquired by the control unit without installing an image acquisition unit in the moving object. As a result, there is no need to secure installation space for an image acquisition unit on the moving object, so that the head motion tracker device for an aircraft can be easily applied even to an existing moving object. Can be done.

The head motion tracker device according to the first aspect includes a head-mounted part that is attached to the head of an observer, and a head-mounted part that is attached to the head-mounted part to acquire an image in a direction along the direction of the head of the observer. and a control section configured to perform control to acquire the position and angle of the observer's head based on the image acquired by the image acquisition section. Thereby, the position and angle of the observer's head can be acquired by the control unit based on the image of the image acquisition unit attached to the head mounted unit rather than the moving object. That is, the position and angle of the observer's head can be acquired by the control unit without installing an image acquisition unit in the moving body. As a result, there is no need to secure installation space for the image acquisition unit in the moving object, so the head motion tracker device can be easily applied even to an existing moving object.

The aircraft head motion tracker device according to the second aspect includes a head motion tracker device attached to a pilot as an observer inside the aircraft body, and the head motion tracker device includes a head motion tracker device attached to the pilot's head. an image acquisition unit that is attached to the head attachment unit and acquires an image in a direction along the orientation of the pilot's head; and a position of the pilot's head based on the image acquired by the image acquisition unit. and a control unit configured to perform control to obtain the angle. Thereby, the position and angle of the observer's head can be acquired by the control unit based on the image of the image acquisition unit attached to the head mounted unit instead of the moving object. That is, the position and angle of the observer's head can be acquired by the control unit without installing an image acquisition unit in the moving object. As a result, there is no need to secure installation space for an image acquisition unit on the moving object, so that the head motion tracker device for an aircraft can be easily applied even to an existing moving object. Can be done.

1 is a schematic diagram showing a state in which a pilot in an aircraft is wearing a head motion tracker device according to a first embodiment; FIG. FIG. 1 is a block diagram showing the configuration of a head motion tracker device according to a first embodiment. FIG. 2 is a schematic diagram showing a front view of a pilot and a pre-movement image (base image) taken by the image acquisition unit of the head motion tracker device according to the first embodiment. FIG. 3 is a schematic diagram showing a previous frame map (base map) created by the image acquisition unit of the head motion tracker device according to the first embodiment. FIG. 2 is a schematic diagram showing a state in which a lower side than the front of the pilot is photographed by the image acquisition unit of the head motion tracker device according to the first embodiment, a pre-movement image, and a post-movement image. FIG. 3 is a schematic diagram showing a current frame map created by the image acquisition unit of the head motion tracker device according to the first embodiment. FIG. 2 is a schematic diagram showing a post-movement environment map created by the head motion tracker device according to the first embodiment. FIG. 2 is a schematic diagram showing an example of the movement of the head of a pilot wearing the head motion tracker device according to the first embodiment. FIG. 3 is a schematic diagram showing a method for estimating the position and angle of a pilot's head in the control unit of the head motion tracker device according to the first embodiment. 5 is a flowchart illustrating processing for acquiring the position and angle of a pilot's head according to the first embodiment. FIG. 7 is a schematic diagram showing a state in which a head motion tracker device according to a second embodiment is worn by a pilot in an aircraft. FIG. 7 is a block diagram showing the configuration of a marker section of a head motion tracker device according to a second embodiment. FIG. 2 is a block diagram showing the configuration of a head motion tracker device according to a second embodiment. FIG. 7 is a schematic diagram showing a state in which a control unit side communication unit and a mark unit side communication unit of the head motion tracker device according to the second embodiment are communicating with each other. FIG. 7 is a schematic diagram showing an example in which the position and angle of the pilot's head to be created are reduced in estimating the position and angle of the pilot's head in the control unit of the head motion tracker device according to the second embodiment. FIG. 3 is a block diagram showing the configuration of a head motion tracker device according to a third embodiment. FIG. 7 is a schematic diagram showing a head motion tracker device and a mark section according to a third embodiment. FIG. 7 is a schematic diagram showing a head motion tracker device according to a first modification of the second embodiment. FIG. 7 is a block diagram showing the configuration of a head motion tracker device according to a second modification of the third embodiment.

DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments embodying the present invention will be described based on the drawings.

[First embodiment]
The configuration of an aircraft head motion tracker device 10 (hereinafter simply referred to as head motion tracker device 10) according to the first embodiment will be described with reference to FIGS. 1 to 9. The head motion tracker device 10 is provided in a head mounted display mounted on a pilot 11 in an aircraft 100, as shown in FIG. Specifically, the head motion tracker device 10 has a function of measuring the position and angle of the pilot's 11 head in a head mounted display. Note that the aircraft head motion tracker device 10 is an example of a "head motion tracker device" in the claims. Further, the aircraft 100 is an example of a "mobile object" in the claims. The pilot 11 is an example of an "observer" in the claims.

Specifically, as shown in FIGS. 1 and 2, the head motion tracker device 10 includes a head-mounted section 1, an image acquisition section 2, a mark section 3, and a control section 4.

The head-mounted part 1 is a member mounted on the head of the pilot 11, such as a helmet, a headset, and a belt. The image acquisition section 2 is attached to the head mounted section 1 and is configured to acquire an image in a direction along the direction of the pilot's 11 head (hereinafter referred to as direction A). The image acquisition unit 2 includes a plurality of (two) cameras 21 arranged in line. Specifically, the two cameras 21 are area sensor cameras. The two cameras 21 are sensitive to ultraviolet light and near-infrared light. The two cameras 21 are arranged at a predetermined interval. Each of the two cameras 21 is configured to take pictures in the same direction A. The two cameras 21 are equipped with the same lens. Specifically, the two cameras 21 are equipped with the same wide-angle lens or fisheye lens.

The landmark section 3 is arranged around the pilot 11, and is acquired as an image by the image acquisition section 2. That is, the landmark section 3 has a function as a marker. A plurality of (three) marker portions 3 are arranged on the aircraft 100. The landmark section 3 is arranged in the cockpit of the aircraft 100. Note that one, two, or four or more mark portions 3 may be arranged.

The mark section 3 includes a mounting section 31 and a light emitting section 32. The attachment part 31 is a base for attaching the mark part 3 to a moving body. The light emitting section 32 is attached to the surface of the mounting section 31 on the pilot 11 side. The light emitting unit 32 includes an LED (Light Emitting Diode), a VCSEL (Vertical Cavity Surface Emitting Laser), etc. that can emit ultraviolet light or near infrared light. Furthermore, a lens is attached to the light emitting unit 32 to enable wide-angle irradiation of light.

The mark section 3 is configured to be supplied with electric power by non-contact power supply, solar power generation, or the like. In this way, wiring connections for supplying power to the mark section 3 are not required.

Since it is necessary to accurately measure the position and angle of the pilot's 11's head, a plurality (two) of the marker parts 3 are arranged on the front side of the pilot 11. Thereby, when the pilot 11 faces forward, the image acquisition section 2 acquires a plurality of (two) landmark sections 3.

The control unit 4 is composed of a computer that performs various types of control and arithmetic processing. The control section 4 includes an image processing section 41, a CPU (Central Processing Unit) section 42, and a storage section 43.

The image processing unit 41 performs processing to extract feature points 12 in the image based on the image obtained by the image acquisition unit 2, processing to create a surrounding environment map 5 as a three-dimensional map based on the image by machine learning, etc. It has the function to perform The surrounding environment map 5 created by the image processing section 41 is stored in the storage section 43. Here, the feature points 12 in the image refer to the shape of the frame inside the aircraft 100, the center of gravity of parts such as switches and displays arranged inside the aircraft 100, which are acquired from the difference in contrast in the acquired image. Show points. Furthermore, the landmark portion 3 is also acquired as the feature point 12 of the image.

In the surrounding environment map 5, the relative coordinate system (XYZ coordinate system) has an axis extending in the direction in which the two cameras 21 are lined up as the X axis (see FIG. 4), and an axis extending in the direction of the optical axes of the two cameras 21 as the Y axis. (See FIG. 4), and an axis extending in a direction perpendicular to the X-axis and the Y-axis is the Z-axis (see FIG. 4). Further, the origin of the relative coordinate system is the midpoint of the two cameras 21.

Here, since the two cameras 21 are area sensor cameras, the image processing unit 41 can acquire information on the XZ plane in the surrounding environment map 5. Furthermore, the image acquisition unit 2 can acquire information in the Y-axis direction on the surrounding environment map 5 using a stereo method based on the parallax between the two cameras 21.

The surrounding environment map 5 includes a base map 51 (see FIG. 4), a current frame map 52 (see FIG. 6), and a previous frame map 53 (see FIG. 4). Note that the base map 51 and the previous frame map 53 are an example of a "second surrounding environment map" in the claims. Note that the current frame map 52 is an example of a "third surrounding environment map" in the claims.

The base map 51 is a reference map for the surrounding environment map 5. The base map 51 may be, for example, the surrounding environment map 5 based on the base image 61 photographed with the pilot 11 facing straight ahead in the A direction. The base map 51 is registered in the storage unit 43 in advance.

The previous frame map 53 is the current frame map 52 stored in the storage unit 43 before the current time. The previous frame map 53 may be, for example, a three-dimensional map based on a pre-movement image 62 taken when the pilot 11 faced directly in the A direction. Note that the pre-movement image 62 is an example of a "second image" in the claims.

The current frame map 52 is, for example, an image obtained by combining a pre-movement image 62 taken when the pilot 11 faced directly in front of the pilot 11 and an after-movement image 63 taken when the pilot 11's head moved from directly in front. It may be a three-dimensional map based on The current frame map 52 is added in accordance with the images acquired by the image acquisition unit 2. Note that the moved image 63 is an example of a "first image" in the claims.

The CPU unit 42 has a function of executing a comparison process 42a and a position angle detection process 42b. The comparison process 42a is a process of comparing the surrounding environment map 5 converted by the image processing unit 41. The position angle detection process 42b is a process of detecting the current position and angle of the pilot's 11 head based on the surrounding environment map 5 created by the image processing unit 41. The position angle detection process 42b will be explained later using an example.

The storage unit 43 is a storage medium including a RAM (Random Access Memory) and a ROM (Read Only Memory). Here, a previous frame map 53 is stored in the RAM. A base map 51 is also stored in the ROM.

(Position angle detection processing)
The control unit 4 of the first embodiment is configured to perform control to acquire the position and angle of the head of the pilot 11 based on the image acquired by the image acquisition unit 2. Note that the position and angle of the pilot's 11 head are shown as the position and angle of the midpoint of the two cameras 21.

In detail, the control unit 4 acquires the position and angle of the pilot's 11 head based on the portion of the image acquired by the image acquisition unit 2 other than the window through which the outside world inside the aircraft 100 is visible. configured to perform control. That is, the control unit 4 determines the head of the pilot 11 in the image acquired by the image acquisition unit 2 based on feature points 12 such as the shape of the frame inside the aircraft 100 and the switches and displays arranged inside the aircraft 100. The controller is configured to perform control to obtain the position and angle of the part.

With reference to FIGS. 3 to 9, position and angle detection processing 42b using a particle filter will be described as an example. In addition, although the following description shows an example in which the base map 51 and the previous frame map 53 are the same, the base map 51 and the previous frame map 53 may be different. When the base map 51 and the previous frame map 53 are different, the position angle detection process 42b may be performed using the previous frame map 53.

As shown in FIGS. 3 and 4, the control unit 4 uses machine learning to create a three-dimensional surrounding environment map 5 around the pilot 11 based on the image when the pilot 11 faces directly ahead (the pre-movement image 62). (previous frame map 53). That is, the control unit 4 acquires the distance in the direction along the direction of the head of the pilot 11 based on the parallax between the two cameras 21, and also creates a three-dimensional surrounding environment map 5 ( It is configured to control the creation of a previous frame map 53). At this time, the control unit 4 is configured to perform control to acquire a plurality of feature points 12 based on the contrast difference in the pre-movement image 62. The control unit 4 is configured to perform control to acquire a plurality of landmarks 3 based on the difference in contrast in the pre-movement image 62.

Here, the position and angle of the head of the pilot 11 are shown as (X1, Y1, Z1) and (RL1, EL1, AZ1), respectively. RL1 is an angle in the roll direction (rotation with respect to the X axis). EL1 is an angle in the elevation direction (rotation with respect to the Y axis). AZ1 is an angle in the azimuth direction (rotation with respect to the Z axis).

Next, FIGS. 5 and 6 show a case in which the head of the pilot 11 is moved downward from the front direction. In this case, the control unit 4 uses machine learning to create a three-dimensional environment around the pilot 11 based on the pre-movement image 62 and the image in the A direction after the pilot 11's head has moved (post-movement image 63). It is configured to control the creation of map 5 (current frame map 52). That is, the control unit 4 acquires the distance in the direction along the direction of the head of the pilot 11 based on the parallax between the two cameras 21, and also creates a three-dimensional surrounding environment map 5 ( It is configured to control the creation of a current frame map 52). At this time, the control unit 4 is configured to perform control to acquire the plurality of feature points 12 based on the difference in contrast in the moved image 63. The control unit 4 is configured to perform control to acquire a plurality of landmarks 3 based on the difference in contrast in the moved image 63.

The current frame map 52 is created based on an image in which the same parts (common parts) of the pre-movement image 62 and the post-movement image 63 are combined, and a part of the post-movement image 63 that is different from the pre-movement image 62 is added. It is a series of three-dimensional maps. Here, the current frame map 52 becomes the previous frame map 53 when the moved image 63 is acquired next time. Similarly, the post-movement image 63 is synthesized and added to the previous frame map 53. In this way, the current frame map 52 is used to create the surrounding environment map 5 for the current frame and subsequent frames.

Here, the position and angle of the head of the pilot 11 after being moved downward from the frontal direction are respectively shown as (X2, Y2, Z2) and (RL2, EL2, AZ2).

Next, as shown in FIG. 7, the control unit 4 is configured to perform control to obtain a post-movement environment map 54 based on a comparison between the previous frame map 53 and the current frame map 52. That is, the control unit 4 is configured to perform control to acquire a portion corresponding to the post-movement image 63 from the current frame map 52. Note that the post-movement environment map 54 is an example of a "first surrounding environment map" in the claims.

In this way, the post-movement environment map 54 is a map constructed from the post-movement images 63 acquired by the image acquisition unit 2 in the current frame (current time). The current frame map 52 is a map based on an image obtained by combining the pre-movement image 62 and the post-movement image 63. The previous frame map 53 is a map composed of the pre-movement image 62 of the frame (previous time point) before the post-movement image 63.

Then, as shown in FIGS. 7 and 8, the control unit 4 controls the head of the pilot 11 from which all the plurality of feature points 12 and the landmarks 3 as the feature points 12 included in the post-movement environment map 54 can be acquired. It is configured to perform control to estimate the position and angle as the current position and angle of the head of the pilot 11.

Specifically, as shown in FIG. 9, the control unit 4 is configured to perform control to estimate the position and angle of the head of the pilot 11 in the current frame based on the particle filter.

That is, the control unit 4 creates the head positions and angles of a plurality of pilots 11 having different head positions and angles based on the head positions and angles of the pilots 11 in frames before the current frame. The system is configured to perform control such as: The control unit 4 is configured to perform control to randomly move the head positions and angles of the plurality of pilots 11 that have been created and have different head positions and angles.

The control unit 4 selects the heads of the pilots 11 from which a map having a correlation with the post-movement environment map 54 can be obtained from the current frame map 52 from among the positions and angles of the heads of the plurality of randomly moved pilots 11. It is configured to control the selection of the position and angle of the Specifically, the control unit 4 selects all of the plurality of feature points 12 included in the post-movement environment map 54 and the landmarks 3 as the feature points 12 from among the positions and angles of the heads of the plurality of pilots 11 that have been created. It is configured to perform control to select the position and angle of the head of the pilot 11 that can be obtained.

Based on the head positions and angles of the pilots 11 (marked with circles in FIG. 9) for which correlated maps can be obtained, the control unit 4 controls the head positions and angles of the plurality of pilots 11 whose head positions and angles are different from each other. The device is configured to provide control to create the position and angle of the head. At this time, the control unit 4 is configured to perform control to delete the position and angle of the head of the pilot 11 (marked with an x in FIG. 9) that have no correlation.

As described above, the control unit 4 performs control to create a three-dimensional previous frame map 53 around the pilot 11 based on the pre-movement image 62 in the A direction (direction along the direction of the pilot's 11 head). is configured to do so. The control unit 4 performs control to create a three-dimensional current frame map 52 around the pilot 11 based on a pre-movement image 62 and a post-movement image 63 in the A direction (direction along the direction of the pilot's 11 head). is configured to do so. The control unit 4 is configured to perform control to obtain the position and angle of the pilot's 11 head based on the previous frame map 53 and the current frame map 52.

Here, the control unit 4 acquires a post-movement environment map 54 based on a comparison between the previous frame map 53 and the current frame map 52, and performs control to estimate the position and angle of the pilot's 11 head. It is configured as follows. That is, the control unit 4 is configured to perform control to obtain the moved environment map 54 from the current frame map 52.

The control unit 4 acquires the position and angle of the head of the pilot 11 based on the landmark part 3 in the three-dimensional post-movement environment map 54 based on the acquired post-movement image 63, and also acquires the position and angle of the head of the pilot 11. Control is performed to obtain the position and angle of the head of the pilot 11 based on a plurality of feature points 12 in a three-dimensional post-movement environment map 54 based on a post-movement image 63 in which feature points 12 have been obtained. It is configured. That is, the control unit 4 is configured to perform control to estimate the position and angle of the pilot's 11 head based on the plurality of feature points 12 and the landmark unit 3 included in the post-movement environment map 54.

(Head position and angle acquisition processing)
The head position and angle acquisition process will be described below with reference to FIGS. 3 to 10. The head position and angle acquisition process is a process of acquiring the position and angle of the pilot's 11 head using the camera 21 attached to the pilot's 11 head.

In step S1, the control unit 4 acquires the base map 51 from the storage unit 43 as the previous frame map 53 (see FIGS. 3 and 4).

In step S2, the control unit 4 obtains the current frame map 52 based on the image of the direction in which the pilot 11 is facing (see FIGS. 5 and 6). In step S3, the control unit 4 obtains the post-movement environment map 54 based on the comparison between the previous frame map 53 and the current frame map 52 (see FIG. 7). In step S4, the control unit 4 estimates the position and angle of the head of the observer who can acquire all the plurality of feature points 12 and landmarks 3 included in the post-movement environment map 54 in the current frame map 52. (See Figure 9).

In step S5, the control unit 4 stores the current frame map 52 in the storage unit 43 as the previous frame map 53. After step S5, the process returns to step S3 and is repeated.

(Effects of the first embodiment)
In the first embodiment, the following effects can be obtained.

In the first embodiment, as described above, the head motion tracker device 10 is provided with the image acquisition section 2 attached to the head mounted section 1. Further, the head motion tracker device 10 is provided with a control unit 4 configured to perform control to acquire the position and angle of the head of the pilot 11 based on the image acquired by the image acquisition unit 2. Thereby, the position and angle of the head of the pilot 11 can be acquired by the control unit 4 based on the image of the image acquisition unit 2 attached to the head mounted unit 1 instead of the aircraft 100. That is, the position and angle of the pilot's 11 head can be acquired by the control unit 4 without installing the image acquisition unit 2 in the aircraft 100. As a result, there is no need to secure installation space for the image acquisition unit 2 in the aircraft 100, so the head motion tracker device 10 can be easily applied even to an existing aircraft 100.

Further, in the first embodiment, as described above, the control unit 4 creates the three-dimensional surrounding environment map 5 around the pilot 11 based on the image in the direction along the direction of the pilot 11's head. It is configured to perform control and also perform control to obtain the position and angle of the pilot's head based on the surrounding environment map 5. As a result, the position and angle of the pilot's 11 head can be obtained by the calculation of the control unit 4 on the surrounding environment map 5 in the virtual space, so that the three-dimensional surroundings around the pilot 11, which differ for each aircraft 100, can be obtained. Even in the environmental map 5, the position and angle of the pilot's 11 head can be reliably acquired.

Furthermore, in the first embodiment, as described above, the surrounding environment map 5 is provided with a post-movement environment map 54 configured by the post-movement image 63 acquired by the image acquisition unit 2 in the current frame (current time). The control unit 4 is configured to perform control to estimate the position and angle of the pilot's 11 head based on the plurality of feature points 12 included in the post-movement environment map 54. Thereby, the position and angle of the pilot's 11 head are estimated with reference to the plurality of feature points 12, so the position and angle of the pilot's 11 head can be estimated more accurately.

Furthermore, in the first embodiment, as described above, the surrounding environment map 5 is provided with the previous frame map 53 configured by the before-movement image 62 that is earlier than the after-movement image 63. The control unit 4 is configured to perform control to estimate the position and angle of the pilot's 11 head based on at least the plurality of feature points 12 included in the post-movement environment map 54 and the previous frame map 53. As a result, compared to estimating the position and angle of the pilot's 11 head based only on the plurality of feature points 12 of the post-movement environment map 54, the previous frame map 53 composed of the pre-movement image 62 is also used. By doing so, changes in the position and angle of the pilot's 11 head can be more accurately grasped. As a result, the accuracy of estimating the position and angle of the pilot's 11 head can be improved.

Further, in the first embodiment, as described above, the current frame map 52 is provided in the surrounding environment map 5 based on an image obtained by combining the before-movement image 62 and the after-movement image 63. The control unit 4 is configured to obtain a post-movement environment map 54 based on a comparison between the previous frame map 53 and the current frame map 52, and perform control to estimate the position and angle of the pilot 11's head. do. As a result, the current frame map 52 is based on the image obtained by combining the image 62 before movement and the image 63 after movement. 54 can be increased. As a result, a more precise post-movement environment map 54 can be created, so that the accuracy of estimating the position and angle of the pilot's 11 head can be further improved.

In addition, in the first embodiment, as described above, the control unit 4 controls the position and angle of the head of the pilot 11 from which all the plurality of feature points 12 included in the post-movement environment map 54 can be acquired in the current frame ( The configuration is such that control is performed to estimate the position and angle of the pilot's 11's head at the current time. As a result, the position and angle of the pilot's head in the current frame (currently) can be obtained depending on whether or not all the plurality of feature points 12 included in the post-movement environment map 54 can be obtained. The method allows the position and angle of the pilot's 11 head to be estimated.

In addition, in the first embodiment, as described above, the control unit 4 controls the head position of each pilot 11 based on the position and angle of the head of the pilot 11 in the previous frame (time point before the current frame (current time)). After creating head positions and angles of a plurality of pilots 11 having different positions and angles, and changing the created head positions and angles of a plurality of pilots 11, a plurality of head positions and angles included in the post-movement environment map 54 are created. It is configured to perform control to select the position and angle of the head of the pilot 11 from which all the feature points 12 can be acquired. As a result, by creating multiple positions and angles of the head of the pilot 11, it is possible to estimate the position and angle of the head of the pilot 11 in the current frame (currently) from many variations. Eleven head positions and angles can be selected.

Further, in the first embodiment, as described above, the control unit 4 is configured to perform control to acquire a plurality of feature points 12 based on the contrast difference in the moved image 63. With this, it is possible to obtain a plurality of feature points 12 using a simple method, so it is possible to suppress an increase in the processing load on the control unit 4.

Further, in the first embodiment, as described above, the head motion tracker device 10 is provided with the mark section 3 that is arranged around the pilot 11 and is acquired as an image by the image acquisition section 2. The control unit 4 is configured to control the landmark unit 3 to acquire the position and angle of the pilot 11's head based on the landmark unit 3 in the three-dimensional surrounding environment map 5 based on the acquired image. . Thereby, the landmark section 3 can be used as the feature point 12 in the image acquired by the image acquisition section 2, so that the image acquisition section 2 can more reliably acquire the feature point 12.

Further, in the first embodiment, as described above, the image acquisition unit 2 is provided with two cameras 21 arranged side by side. The control unit 4 acquires the distance in the direction along the direction of the head of the pilot 11 based on the parallax between the two cameras 21, and also determines the three-dimensional surrounding environment based on the distance acquired by the two cameras 21. It is configured to control the creation of map 5. This makes it possible to accurately obtain the distance in the direction along the direction of the pilot's 11 head, compared to the case where the distance in the direction along the direction of the pilot's 11 head is obtained using one camera 21. Therefore, the three-dimensional surrounding environment map 5 can be created more accurately.

Further, in the first embodiment, as described above, the head motion tracker device for an aircraft is provided with the head motion tracker device 10 that is attached to the pilot 11 as an observer inside the aircraft 100. A head motion tracker device 10 is provided with an image acquisition section 2 attached to a head mounted section 1. Further, the head motion tracker device 10 is provided with a control unit 4 configured to perform control to acquire the position and angle of the head of the pilot 11 based on the image acquired by the image acquisition unit 2. Thereby, the position and angle of the head of the pilot 11 can be acquired by the control unit 4 based on the image of the image acquisition unit 2 attached to the head mounted unit 1 instead of the aircraft 100. That is, the position and angle of the pilot's 11 head can be acquired by the control unit 4 without installing the image acquisition unit 2 in the aircraft 100. As a result, there is no need to secure installation space for the image acquisition unit 2 in the aircraft 100, so the head motion tracker device 10 for an aircraft can be easily applied even to an existing aircraft 100. can be obtained.

[Second embodiment]
Next, a head motion tracker device 210 according to a second embodiment will be described with reference to FIGS. 11 to 14. Specifically, unlike the head motion tracker device 10 of the first embodiment, which includes the mark section 3 that only emits light, in the head motion tracker device 210 of the second embodiment, the mark section 203 communicates with the mark section side communication section 234. We are prepared. Note that in the second embodiment, the same components as in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

As shown in FIGS. 11 and 12, the head motion tracker device 210 includes a head mounted section 1, an image acquisition section 2, a mark section 203, and a control section 204 (see FIG. 13).

The mark section 203 is arranged around the pilot 11 and is acquired as an image by the image acquisition section 2. The mark section 203 has both a marker function and a communication function. A plurality (three) of mark sections 203 are arranged on the moving object. Note that one, two, or four or more mark portions 203 may be arranged.

The plurality of landmarks 203 enables continuous acquisition of a good current frame map 52, so that no matter which direction the pilot 11 faces, at least one or more landmarks 203 can be detected by the image acquisition unit 2. It is placed in a position where it can be photographed. Thereby, the control unit 204 can avoid a decrease in the accuracy of estimating the position and angle of the pilot's 11 head. Further, it is possible to reliably avoid a situation in which the mark section 203 is not detected by the control section 204.

The mark section 203 includes a mounting section 31, a light emitting section 203a, a light receiving section 203b, and a control section 203c.

The light emitting unit 203a is used for optical communication with the control unit 204 as well as a marker function. The light emitting unit 203a is composed of an LED, a VCSEL, etc. that can emit ultraviolet light or near-infrared light. Furthermore, a lens is attached to the light emitting unit 203a to enable wide-angle irradiation of light. The light receiving section 203b is composed of a light receiving element such as a photodiode capable of receiving ultraviolet light or near infrared light. A lens is attached to the light receiving section 203b so as to be able to receive light at a wide angle.

The control section 203c includes a CPU section 231, a storage section 232, a sensor section 233, and a mark section side communication section 234.

The CPU section 231 has a function of causing the sensor data processing 231a to execute predetermined processing. The sensor data processing 231a performs a process of converting the measured value from the sensor unit 233 into measured value information that can be transmitted to the control unit 204. The sensor data processing 231a performs a process of converting data from the control unit 204 into data that can be recognized by the control unit 203c.

The storage unit 232 is a storage medium including RAM and ROM. The storage unit 232 stores an identification number 232a of the landmark unit 203 and a body coordinate position 232b. The body coordinate position 232b indicates the coordinate position of the landmark portion 203 with respect to the reference position of the body of the aircraft 100 carrying the pilot 11. The body coordinate position 232b is input into the storage unit 232 after being measured when the landmark unit 203 is installed on the body. Note that the body coordinate position 232b is an example of the "position of a landmark" in the claims. Further, the aircraft 100 is an example of a "mobile object" in the claims.

The sensor section 233 includes an acceleration sensor 233a, a gyro sensor 233b, a tilt sensor 233c, and a direction sensor 233d. The acceleration sensor 233a measures the acceleration of the aircraft. The gyro sensor 233b measures the angular velocity of the aircraft. The tilt sensor 233c measures the tilt of the aircraft body. The orientation sensor 233d measures the orientation of the aircraft. Note that the gyro sensor 233b is an example of a "mark side gyro sensor" in the claims.

The mark unit side communication unit 234 is configured to be able to communicate with a control unit side communication unit 204f (see FIG. 13), which will be described later. The mark side communication unit 234 has a wireless communication function using light and radio waves. The landmark side communication unit 234 performs wireless communication using radio waves with the control unit side communication unit 204f when adjusting and setting the body coordinate position 232b of the storage unit 204b of the control unit 204. Further, the mark unit side communication unit 234 performs optical wireless communication with the control unit side communication unit 204f during normal use other than adjustment and setting. The marker unit side communication unit 234 transmits information on the identification number 232a of the marker unit 203, information on the body coordinate position 232b of the marker unit 203, and measurement value information of the sensor unit 233 to the control unit side communication unit 204f by light. It is composed of

Here, the mark side communication section 234 is configured to control the light emitting section 203a to emit light. That is, the marker side communication unit 234 can superimpose the light emission of the light emitting unit 203a as a marker and the light emission of the light emitting unit 203a as wireless communication using light with the control unit side communication unit 204f. The marker side communication unit 234 receives the light emitted from the light emitting unit 204c on the control unit 204 side using the light receiving unit 203b.

As shown in FIGS. 13 and 14, the control unit 204 includes a computer that performs various types of control and calculation processing. The control section 204 includes an image processing section 41, a CPU section 204a, a storage section 204b, a light emitting section 204c, a light receiving section 204d, a sensor section 204e, and a control section side communication section 204f.

The CPU section 204a has a function of causing the comparison processing 42a, the position angle detection processing 42b, and the sensor data processing 240 to execute predetermined processing. The sensor data processing 240 performs a process of converting the measured value from the sensor unit 204e into measured value information that can be recognized by the control unit 204. The sensor data processing 240 also performs processing for passing information transmitted from the landmark section 203. The sensor data processing 240 performs processing to convert data transmitted from the control unit 204 into data that can be transmitted to the control unit 203c.

The storage unit 204b is a storage medium including RAM and ROM. The storage unit 204b stores a previous frame map 53, a base map 51, and a body coordinate position 232b.

The light emitting unit 204c is composed of an LED, a VCSEL, etc. that can emit ultraviolet light or near-infrared light. The optical axis of the light emitting unit 204c is arranged in the same direction as the optical axis of the camera 21. A lens is attached to the light emitting unit 204c to enable wide-angle irradiation of light. The light emitting unit 204c can both emit light as illumination light and emit light as wireless communication by light with the marker side communication unit 234 by the control unit side communication unit 204f. It is.

The light receiving section 204d is composed of a light receiving element such as a photodiode capable of receiving ultraviolet light or near infrared light. A lens is attached to the light receiving section 204d so as to be able to receive light at a wide angle.

The sensor section 204e includes an acceleration sensor 241, a gyro sensor 242, a tilt sensor 243, and an orientation sensor 244. The acceleration sensor 241 measures the acceleration of the pilot's 11 head. The gyro sensor 242 measures the angular velocity of the pilot's 11 head. The tilt sensor 243 measures the tilt of the pilot's 11 head. The orientation sensor 244 measures the orientation of the pilot's 11 head. Note that the gyro sensor 242 is an example of a "device-side gyro sensor" in the claims.

The control unit side communication unit 204f is configured to be able to communicate with the mark unit side communication unit 234. The control unit side communication unit 204f has a wireless communication function using light and radio waves. The control unit side communication unit 204f performs wireless communication using radio waves with the landmark unit side communication unit 234 when adjusting and setting the body coordinate position 232b of the storage unit 204b of the control unit 204. Furthermore, the control unit side communication unit 204f performs optical wireless communication with the mark unit side communication unit 234 during normal use other than adjustment and setting. The control side communication unit is configured to receive information on the identification number 232a of the landmark unit 203, information on the body coordinate position 232b of the landmark unit 203, and measurement value information of the sensor unit 233 from the landmark unit side communication unit 234 using light. has been done. The control unit side communication unit 204f is configured to cause the light emitting unit 204c to emit light as illumination light.

(Position angle detection processing)
As shown in FIGS. 13 and 14. The control unit 204 of the second embodiment is configured to perform control to acquire the position and angle of the head of the pilot 11 based on the image acquired by the image acquisition unit 2. Note that the position and angle of the pilot's 11 head are shown as the position and angle of the midpoint between the two cameras 21.

If a sufficient number of feature points 12 cannot be extracted from the moved image 63 acquired by the image acquisition unit 2, the control unit 204 uses the measurement value information of the sensor unit 233 and the measurement value information of the sensor unit 204e. Based on this information, control is performed to obtain the position and angle of the pilot's 11 head. Further, the control unit 204 is configured to perform similar control based on the previous frame map 53 and the current frame map 52 even when the angle of the observer's head cannot be obtained within a predetermined time. At this time, the control unit 204 is configured to perform control to transmit a signal requesting measurement value information of the sensor unit 204e to the mark unit 203 acquired by the image acquisition unit 2 using the light from the light emitting unit 204c. .

Specifically, if the angle of the pilot's 11 head cannot be obtained within a predetermined time based on the previous frame map 53 and the current frame map 52, the control unit 204 controls the angle of the pilot's 11's head measured by the gyro sensor 242. It is configured to perform control to obtain the angle of the pilot's 11 head based on the difference between the angular velocity and the angular velocity of the aircraft 100 measured by the gyro sensor 233b. That is, the information on the angular velocity of the aircraft 100 measured by the gyro sensor 233b is used to offset the information on the angular velocity of the aircraft 100 included in the angular velocity of the head measured by the gyro sensor 242. Thereby, since the control unit 204 can acquire information on the angular velocity of the aircraft 100 through optical communication, there is no need for the control unit 204 to use a dedicated cable to acquire the information on the angular velocity of the aircraft 100.

In addition, in detail, if the angle of the head of the pilot 11 cannot be obtained within a predetermined time based on the previous frame map 53 and the current frame map 52, the control unit 204 controls the angle of the head of the pilot 11 measured by the acceleration sensor 241. It is configured to perform control to obtain the position of the pilot's 11 head based on the difference between the acceleration of the aircraft 100 and the acceleration of the aircraft 100 measured by the acceleration sensor 233a. That is, information on the acceleration of the aircraft 100 measured by the acceleration sensor 233a is used to offset information on the acceleration of the aircraft 100 included in the head acceleration measured by the acceleration sensor 241. Thereby, since the control unit 204 can acquire information on the acceleration of the aircraft 100 through optical communication, there is no need for the control unit 204 to use a dedicated cable to acquire the information on the acceleration of the aircraft 100.

In addition to optical communication, the pilot 11 may input the attitude, azimuth, etc. of the aircraft 100 as needed.

The control unit 204 compares the angle of the head of the pilot 11 acquired by the gyro sensor 233b and the gyro sensor 242 with the angle of the head of the pilot 11 estimated based on the previous frame map 53 and the current frame map 52. The system is configured to perform control such as: Furthermore, the control unit 204 determines the position of the head of the pilot 11 acquired by the acceleration sensor 233a and the acceleration sensor 241, and the position of the head of the pilot 11 estimated based on the previous frame map 53 and the current frame map 52. It is configured to perform control to compare the If the angle and position of the pilot 11's head match in both of the above two comparison results, the control unit 204 combines the previous frame map 53 and the base map 51, or combines the base map 51 with the previous frame map 53 and the base map 51. It is configured to control updating to the previous frame map 53.

The control unit 204 uses a post-movement environment map to prevent the accuracy of the position and angle of the pilot 11 from decreasing due to the similarity between the previous frame map 53 and the current frame map 52 and disturbances. The position and angle of the head of the pilot 11 are estimated based on the body coordinate position 232b of the landmark section 203 rather than the coordinate position of the feature point 12 on the 54. Specifically, when the image acquisition unit 2 acquires the landmark 203, the control unit 204 determines whether the pilot The device is configured to perform control to obtain the position and angle of the head of No. 11. This makes it possible to reset (reset) the position and angle of the pilot 11's head each time the image acquisition unit 2 acquires the landmark section 203. Here, when the position and angle of the pilot 11's head are reset, the control unit 204 can perform control to correct the coordinate position of the previous frame map 53 based on the body coordinate position 232b of the landmark unit 203. preferable. Note that the body coordinate position 232b is an example of the "position of a landmark" in the claims.

As shown in FIG. 15, the control unit 204 controls the head positions of a plurality of pilots 11 having different head positions and angles based on the head positions and angles of the pilots 11 in frames before the current frame. When creating the positions and angles, control is performed to limit the positions and angles of the heads of the plurality of pilots 11 to be created. That is, the control unit 204 is configured to perform control to limit the positions and angles of the heads of the plurality of pilots 11 to be created based on the measurement value information of the tilt sensor 243 and the direction sensor 244. As a result, the positions and angles of the heads of the plurality of pilots 11 to be referenced and compared are reduced, so that the processing of the control unit 204 can be reduced. Note that the other configurations of the second embodiment are similar to those of the first embodiment.

(Effects of the second embodiment)
In the second embodiment, the following effects can be obtained.

In the second embodiment, as described above, the head motion tracker device 210 is provided with the image acquisition section 2 attached to the head mounted section 1. Further, the head motion tracker device 210 is provided with a control unit 204 configured to perform control to acquire the position and angle of the head of the pilot 11 based on the image acquired by the image acquisition unit 2. Thereby, the head motion tracker device 210 can be easily applied even to the existing aircraft 100.

Further, in the second embodiment, as described above, the control unit 204 is provided with the control unit side communication unit 204f. The landmark section 203 is provided with a landmark section side communication section 234 that can communicate with the control section side communication section 304f, and a storage section 232 that stores the position of the landmark section 203 with respect to the reference position of the aircraft 100 carrying the pilot 11. When the image acquisition unit 2 acquires the landmark 203, the control unit 204 determines the position of the pilot 11's head and Configure to perform control to obtain angle. As a result, the accuracy of the position of the landmark part 203 can be increased compared to the case where the position of the landmark part 203 is acquired from the image acquired by the image acquisition part 2. The accuracy of the position of the mark portion 203 can be improved. As a result, the control unit 204 can obtain a more accurate position and angle of the pilot's 11 head.

Furthermore, in the second embodiment, as described above, the head motion tracker device 210 is provided with the gyro sensor 242 that measures the angular velocity of the pilot's 11 head. A gyro sensor 233b that measures the angular velocity of the aircraft 100 carrying a pilot is provided in the marker section 203. If the angle of the head of the pilot 11 cannot be obtained within a predetermined time based on the surrounding environment map 5, the control unit 204 controls the angular velocity of the head of the pilot 11 measured by the gyro sensor 242 and the angular velocity of the head of the pilot 11 measured by the gyro sensor 233b. The control is configured to perform control to obtain the angle of the pilot's 11 head based on the difference between the angular velocity of the aircraft 100 and the angle of the pilot's 11's head. As a result, when the angle of the pilot's 11 head cannot be obtained based on the surrounding environment map 5, the angle of the pilot's 11's head can be supplemented by the gyro sensor 242 and the gyro sensor 233b. The angle of the part can be reliably obtained. Note that other effects of the second embodiment are similar to those of the first embodiment.

[Third embodiment]
Next, a head motion tracker device 310 according to a third embodiment will be described with reference to FIGS. 16 and 17. Specifically, unlike the head motion tracker device 210 of the second embodiment that includes a mark section 203 having a light emitting section 203a, in the head motion tracker device 310 of the third embodiment, the mark section 304 emits light by reflection. Note that, in the third embodiment, the same components as in the second embodiment are given the same reference numerals, and the description thereof will be omitted.

Specifically, as shown in FIGS. 16 and 17, the head motion tracker device 310 includes a head mounted section 1, an image acquisition section 302, a light emitting section 303, a marker section 304, and a control section 305. We are prepared.

The image acquisition unit 302 includes one camera 21 and a light receiving optical system (not shown). In detail, camera 21 is an area sensor camera. The camera 21 is sensitive to ultraviolet light and near-infrared light. Reflected light emitted from the light emitting unit 303 and reflected on the inner surface of the aircraft body enters the camera 21 via the light receiving optical system. The light receiving optical system includes a filter, a lens, etc. that allows only light in a specific wavelength range to pass through, in order to prevent the camera 21 from receiving light in a specific wavelength range.

The image acquisition unit 302 has a function of calculating distance. Specifically, the image acquisition section 302 is configured to be able to acquire the distance based on the difference between the wavelength of the light emitted by the light emitting section 303 and the wavelength of the reflected light from the light emitting section 303 .

The light emitting unit 303 includes an LED, a VCSEL, and the like that can emit ultraviolet light or near-infrared light. The optical axis of the light emitting unit 303 is arranged in the same direction as the optical axis of the camera 21. Here, it is preferable that the light emitting unit 303 emits light of a wavelength that is not easily affected by surrounding light. A lens is attached to the light emitting unit 303 to enable wide-angle irradiation of light.

Here, in order to improve the distance calculation function, the image acquisition unit 302 is provided with a mechanism (such as a motor) that can change the optical axis direction angle of the camera 21 and the optical axis direction angle of the light emitting unit 303. Preferably.

The landmark section 304 is arranged around the pilot 11, and is acquired as an image by the image acquisition section 302. The mark section 304 has a function as a marker. The mark section 304 is made of a retroreflector or a reflector having reflection characteristics in a specific wavelength region in order to facilitate feature point extraction processing. A plurality (three) of mark sections 304 are arranged on the moving object. Note that one, two, or four or more mark portions 304 may be arranged.

The plurality of landmarks 304 are arranged in areas where estimation precision of the position and angle of the pilot's 11 head is particularly required, areas where it is difficult to extract the feature points 12, and the like.

The control unit 305 is composed of a computer that performs various types of control and calculation processing. The control section 305 includes an image processing section 41, a CPU section 305a, a storage section 305b, and a sensor section 305c.

The CPU unit 305a has a function of executing comparison processing 42a, position angle detection processing 42b, and sensor data processing 350. The sensor data processing 350 performs a process of converting the measured value from the sensor unit 305c into measured value information that can be recognized by the control unit 305.

The storage unit 305b is a storage medium including RAM and ROM. The previous frame map 53, the base map 51, and the body coordinate position 306 are stored in the storage unit 305b.

The sensor section 305c includes an acceleration sensor 351, a gyro sensor 352, a tilt sensor 353, and a direction sensor 354. Note that the gyro sensor 352 is an example of a "device-side gyro sensor" in the claims.

(Position angle detection processing)
As shown in FIGS. 16 and 17. The control unit 305 of the third embodiment is configured to perform control to acquire the position and angle of the head of the pilot 11 based on the image acquired by the image acquisition unit 302. Note that the position and angle of the pilot's 11 head are shown as the position and angle of the midpoint of the tip of one camera 21.

Here, since one camera 21 is an area sensor camera, the image processing unit 41 can acquire information on the XZ plane in the three-dimensional map. Furthermore, the image acquisition unit 302 can acquire information in the Y-axis direction on the three-dimensional map using a distance measurement function.

Specifically, the control unit 305 is configured to perform control to acquire the position and angle of the pilot's 11 head based on the portion inside the aircraft 100 of the image acquired by the image acquisition unit 302. There is. That is, the control unit 305 determines the head of the pilot 11 in the image acquired by the image acquisition unit 302 based on the feature points 12 such as the shape of the frame inside the aircraft 100 and the switches and displays arranged inside the aircraft 100. The controller is configured to perform control to obtain the position and angle of the part. Furthermore, the landmark portion 304 is also a feature point 12 of the image.

The control unit 305 is configured to perform control to acquire a plurality of feature points 12 using reflected light from the switch, display, mark unit 304, etc. with respect to the light emitted from the light emitting unit 303. Here, the control unit 305 determines information in the image due to the angle and reflectance of the light from the light emitting unit 303 with respect to the surface of the interior of the aircraft, the area where the light from the light emitting unit 303 does not reach, and the contrast in the image. In order to suppress the occurrence of defects, control is performed to complement information in the image using the distance calculated by the image acquisition unit 302. Note that the other configurations of the third embodiment are similar to those of the second embodiment.

(Effects of the third embodiment)
In the third embodiment, the following effects can be obtained.

In the third embodiment, as described above, the head motion tracker device 310 is provided with the image acquisition section 302 attached to the head mounted section 1. Further, the head motion tracker device 310 is provided with a control unit 305 configured to perform control to acquire the position and angle of the head of the pilot 11 based on the image acquired by the image acquisition unit 302. Thereby, the head motion tracker device 310 can be easily applied even to the existing aircraft 100. Note that other effects of the third embodiment are similar to those of the second embodiment.

[Modified example]
Note that the embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims rather than the description of the embodiments described above, and further includes all changes (modifications) within the meaning and range equivalent to the claims.

For example, in the first to third embodiments described above, the aircraft 100 is an example of a "mobile body" in the claims, but the present invention is not limited to this. In the present invention, the moving object may be a vehicle.

Furthermore, in the first to third embodiments described above, the position and angle detection processing 42b using a particle filter has been described as an example, but the present invention is not limited to this. In the present invention, the position angle detection process may be performed using an extended Kalman filter or the like.

Further, in the first to third embodiments described above, an example was shown in which the control unit 4 (204, 305) performs the position angle detection process 42b on both the feature point 12 and the landmark unit 3, but the present invention is not limited to this. Not limited. In the present invention, the control unit may perform position and angle detection processing using either the feature point or the landmark. For example, as in the first modification shown in FIG. 18, in a region where the landmark section 3 is not arranged, the control section performs the position angle detection process 42b only using the feature points 12 based on the frame of the aircraft 100, etc. It may be configured as follows.

Further, in the first to third embodiments described above, the control unit 4 (204, 305) compares the previous frame map 53 (second surrounding environment map) and the current frame map 52 (third surrounding environment map). Although an example is shown in which the position and angle detection processing 42b is performed based on the above, the present invention is not limited to this. In the present invention, the control unit may be configured to perform the position angle detection process based on a comparison between the base map and the third surrounding environment map. Further, the control unit may be configured to perform the position angle detection process based on a comparison between the base map, the second surrounding environment map, and the third surrounding environment map.

Furthermore, in the first to third embodiments described above, the head motion tracker device 10 (210, 310) is provided with the mark section 3 (203, 304), but the present invention is not limited to this. . In the present invention, the head motion tracker device does not need to include a mark section.

Further, in the first to third embodiments described above, an example was shown in which there is one pre-movement image 62 (second image), but the present invention is not limited to this. In the present invention, there may be a plurality of second images.

Furthermore, in the first and second embodiments described above, an example was shown in which the mark section 3 was configured to emit light, but the present invention is not limited to this. In the present invention, the mark section may have a configuration in which the shape of the attachment section changes depending on the angle observed from the image acquisition section, or a configuration in which the shape of the light from the light emitting section changes. You may do so.

Further, in the second embodiment, the control unit 204 acquires the identification number 232a and the body coordinate position 232b of the landmark unit 3 through optical communication from the landmark unit 203, but the present invention is not limited to this. I can't. In the present invention, the control unit may cause the image acquisition unit to acquire the position and identification number of the landmark by providing an AR (Augmented Reality) tag (QR (Quick Response) code, AprilTAG, etc.) on the landmark. .

Further, in the third embodiment described above, the head motion tracker device 310 has shown an example including one image acquisition unit 302, but the present invention is not limited to this. In the present invention, the head motion tracker device 410 may include two image acquisition units 402, as in a second modification shown in FIG. In this case, the head motion tracker device 410 may include a stereo image processing unit 403 that acquires the distance based on the parallax between the two image acquisition units 402 by synchronizing the two image acquisition units 402 to capture images. good.

Further, in the first to third embodiments described above, the control unit 4 (204, 305) performs control to create a three-dimensional environmental map 5 around the pilot 11 (observer) using machine learning. Although an example of the configuration has been shown, the present invention is not limited to this. In the present invention, the control unit may create a three-dimensional surrounding environment map using a method other than machine learning.

Further, in the first to third embodiments described above, an example was shown in which only the pilot 11 (observer) was on board the aircraft 100 (mobile body), but the present invention is not limited to this. In the present invention, a plurality of observers may be inside the moving object. Each of the plurality of observers may be equipped with a head motion tracker device. In this case, the plurality of head motion tracker devices may each include an image acquisition unit capable of receiving different lights (ultraviolet light and near-infrared light). This allows multiple head motion tracker devices to be operated simultaneously.

1 Head Mounted Part 2, 302, 402 Image Acquisition Part 3, 203, 304 Marker Part 4, 204, 305 Control Part 5 Surrounding Environment Map 10, 210, 310, 410 Aircraft Head Motion Tracker Device (Head Motion Tracker Device)
11 Pilot (observer)
12 Feature points 21 Camera 52 Current frame map (third surrounding environment map)
53 Previous frame map (second surrounding environment map)
54 Environmental map after movement (first surrounding environment map)
62 Image before movement (second image)
63 Image after movement (first image)
100 Aircraft 204f Control unit side communication unit 232 Storage unit 233b Gyro sensor (marker side gyro sensor)
234 Marker side communication unit 242 Gyro sensor (device side gyro sensor)

Claims (12)

a head-mounted part that is mounted on the observer's head;
an image acquisition unit that is attached to the head-mounted unit and acquires an image in a direction along the direction of the observer's head;
a control unit configured to perform control to acquire the position and angle of the observer's head based on the image acquired by the image acquisition unit ;
The control unit is configured to perform control to create a three-dimensional environmental map around the observer based on the image acquired by the image acquisition unit,
The surrounding environment map is
a first surrounding environment map constituted by a first image acquired by the image acquisition unit at the current time;
a second surrounding environment map configured by a second image before the first image;
a third surrounding environment map based on an image obtained by combining the first image with the second image,
The control unit acquires the first surrounding environment map based on the second surrounding environment map and the third surrounding environment map, and controls estimating the position and angle of the observer's head. A head motion tracker device configured to perform . 2. The control unit according to claim 1 , wherein the control unit is configured to perform control to estimate the position and angle of the observer's head based on a plurality of feature points included in the first surrounding environment map. The head motion tracker device described. The control unit controls estimating the position and angle of the observer's head based on at least the plurality of feature points included in the first surrounding environment map and the second surrounding environment map. The head motion tracker device according to claim 2 , wherein the head motion tracker device is configured. The control unit obtains the first surrounding environment map and estimates the position and angle of the observer's head based on a comparison between the second surrounding environment map and the third surrounding environment map. The head motion tracker device according to claim 3 , wherein the head motion tracker device is configured to perform control to perform the following. further comprising a plurality of landmarks arranged around the observer and acquired as the image by the image acquisition unit,
The control unit determines the position and angle of the observer's head from which all of the plurality of feature points can be obtained from the plurality of landmarks included in the first surrounding environment map, based on the position and angle of the observer's head at the current moment. The head motion tracker device according to any one of claims 2 to 4 , wherein the head motion tracker device is configured to perform control to estimate the position and angle of. The control unit creates a plurality of head positions and angles of the observer whose head positions and angles are different from each other based on the position and angle of the observer's head at a time point before the current time. and, after changing the positions and angles of the heads of the plurality of observers that have been created, the position of the head of the observer at which all of the plurality of feature points included in the first surrounding environment map can be obtained. The head motion tracker device according to claim 5 , wherein the head motion tracker device is configured to perform control to select a direction and an angle. The head according to any one of claims 2 to 6 , wherein the control unit is configured to perform control to acquire the plurality of feature points based on a contrast difference in the first image. Motion tracker device. further comprising a landmark section arranged around the observer and acquired as the image by the image acquisition section,
The control unit is configured to perform control to acquire the position and angle of the observer's head based on the landmark portion in the three-dimensional surrounding environment map based on the image in which the landmark portion is acquired. A head motion tracker device according to any one of claims 1 to 7 , comprising: The control unit includes a control unit side communication unit,
The landmark portion is
a marker side communication unit capable of communicating with the control unit side communication unit;
a storage unit that stores the position of the landmark with respect to a reference position of the moving object carrying the observer;
When the image acquisition unit acquires the landmark, the control unit determines the position of the observer's head based on the position of the landmark acquired from the storage unit via the landmark side communication unit. The head motion tracker device according to claim 8 , wherein the head motion tracker device is configured to perform control to obtain the angle and angle. further comprising a device-side gyro sensor that measures the angular velocity of the observer's head;
The landmark section includes a landmark section-side gyro sensor that measures the angular velocity of the moving object carrying the observer,
If the angle of the observer's head cannot be obtained within a predetermined time based on the surrounding environment map, the control unit may obtain the angular velocity of the observer's head measured by the device-side gyro sensor and the landmark. The head according to claim 8 or 9 , wherein the head is configured to perform control to obtain the angle of the observer's head based on a difference between the angular velocity of the moving body and the angular velocity measured by the side gyro sensor. Motion tracker device. The image acquisition unit includes two cameras arranged side by side,
The control unit acquires a distance in a direction along the direction of the observer's head based on the parallax between the two cameras, and also acquires the three-dimensional distance based on the distance acquired by the two cameras. The head motion tracker device according to any one of claims 1 to 10 , wherein the head motion tracker device is configured to perform control to create a surrounding environment map. Equipped with a head motion tracker device attached to the pilot as an observer inside the aircraft body,
The head motion tracker device includes:
a head mounting section attached to the pilot's head;
an image acquisition unit that is attached to the head mounted unit and acquires an image in a direction along the direction of the pilot's head;
a control unit configured to perform control to acquire the position and angle of the pilot's head based on the image acquired by the image acquisition unit ,
The control unit is configured to perform control to create a three-dimensional environmental map around the observer based on the image acquired by the image acquisition unit,
The surrounding environment map is
a first surrounding environment map constituted by a first image acquired by the image acquisition unit at the current time;
a second surrounding environment map configured by a second image before the first image;
a third surrounding environment map based on an image obtained by combining the first image with the second image,
The control unit acquires the first surrounding environment map based on the second surrounding environment map and the third surrounding environment map, and controls estimating the position and angle of the observer's head. An aircraft head motion tracker device configured to perform .

Images