JP2022028894A - Optical information processing apparatus, optical information processing method, and program for optical information processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently monitor a flying UAV.

SOLUTION: A total station 100 comprises: a target position calculation unit 107 that measures the distance to an unmanned aerial vehicle while optically tracking the unmanned aerial vehicle to measure the position of the unmanned aerial vehicle; an external orientation element calculation unit 122 that calculates an external orientation element of a head-mounted display; a reticule position calculation unit 123 that, based on the position of the unmanned aerial vehicle and the external orientation element of the head-mounted display, calculates the position of the unmanned aerial vehicle on a display screen of the head-mounted display; and an image creation unit 115 that creates an image displaying guide display indicating the position of the unmanned aerial vehicle on the display screen.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to a technique for monitoring an aircraft such as a UAV.

Techniques for using unmanned aerial vehicles (UAVs) for photography, surveying, etc. are known. This technique requires visual monitoring of the flying UAV. As a method of monitoring a flying UAV, there is a method of tracking a flying UAV by a TS (total station) (see, for example, Patent Document 1). In this method, the UAV is tracked using the target tracking function of the TS, and since the TS has a telescope and an imaging function, it is possible to visually track the UAV using this optical system. .. It is also possible to specify the position of the flying UAV by using the laser ranging function provided in the TS.

US2014 / 0210663 Gazette

In the above-mentioned method of tracking a UAV with a TS, the automatic target tracking function of the TS is used. In this technique, the UAV is captured and tracked with a search laser beam. The UAV is provided with a reflecting prism that reflects the search laser beam in the incident direction, and the TS tracks the UAV by detecting the reflected light from the reflecting prism on the TS side.

By the way, in the tracking of UAV by TS, it becomes a problem that TS loses sight of UAV. For example, when a bird, electric wire, utility pole, steel tower, tall tree, building, etc. enters between the TS and UAV, the search laser light (tracking light) from the TS is blocked and the TS loses sight of the UAV. There is.

When the TS loses sight of the UAV, the operator or the observer performs an operation to search for the UAV using the TS telescope, but it is difficult to put the lost UAV into the field of view of the telescope. Although the field of view can be widened by lowering the magnification of the telescope, it is difficult to find a UAV that flies up to several hundred meters away from the wide field of view.

If the operator loses sight of the UAV, an accident such as a collision with a building or a crash may occur, and it is desirable to prevent such an accident from occurring. Against this background, it is an object of the present invention to provide a technique for effectively monitoring a flying UAV.

The present invention provides a position measuring unit for an unmanned aerial vehicle that measures the position of the unmanned aerial vehicle by measuring the distance to the unmanned aerial vehicle while optically tracking the unmanned aerial vehicle, and an external directing element for a head-mounted display. A position calculation unit that calculates the position of the unmanned aerial vehicle on the display screen of the head-mounted display based on the position of the unmanned aerial vehicle and the external positioning element of the head-mounted display. An optical information processing apparatus including an image creating unit that creates an image displaying a guide display indicating the position of the unmanned aerial vehicle on the display screen.

The present invention measures the position of the unmanned aerial vehicle by measuring the distance to the unmanned aerial vehicle while optically tracking the unmanned aerial vehicle, calculates the external orientation element of the head-mounted display, and calculates the external control element of the unmanned aerial vehicle. Based on the position and the external orientation element of the head-mounted display, the position of the unmanned aerial vehicle on the display screen of the head-mounted display is calculated, and a guide display indicating the position of the unmanned aerial vehicle on the display screen is displayed. This is an optical information processing method for creating an image.

According to the present invention, a technique for effectively monitoring a flying UAV can be obtained.

It is a conceptual diagram of an embodiment. It is a block diagram of TS (total station) using the invention. It is a flowchart which shows an example of the processing procedure. It is a flowchart which shows an example of the processing procedure. It is a figure which shows an example of the display screen. It is a figure which shows an example of the display screen. It is a figure which shows an example of the display screen. It is a block diagram of an embodiment. It is a principle diagram which shows the principle of finding the position of a reticle on a screen. It is a figure which shows an example of the display screen. It is a figure which shows an example of the display screen. It is a figure which shows an example of the display screen. It is a figure which shows an example of the display screen. It is a principle diagram which shows the principle of the backward association method.

1. 1. First Embodiment (Structure)
FIG. 1 shows a block diagram of a TS (total station), which is an example of a position measuring device using the invention. The TS100 measures the three-dimensional position of the UAV200 using distance measuring light while tracking the flying UAV200. The tracking of the UAV200 is performed by searching the reflection prism 202 provided in the UAV200 using the search light, and the measurement of the position of the UAV200 is performed by irradiating the reflection prism 202 with the ranging light and detecting the reflected light. It is done in. Further, the TS100 functions as an optical information processing apparatus using the present invention. With this function, various image information is displayed on the smart glasses hung by the operator 500.

The UAV 200 is remotely controlled by the operator 500. The operator 500 wirelessly controls the UAV 200 using the controller 501. Of course, it is also possible to make the UAV 200 autonomously fly according to a predetermined flight plan. The UAV200 is a commercially available model and is not a special one. The UAV200 includes a GNSS position measuring unit, an IMU, a wireless communication device, a storage device for storing flight plans and flight logs, and an altimeter.

A reflection prism 202, which is a target for surveying by the TS100, is attached to the lower part of the UAV200. Further, the UAV 200 is equipped with a camera 201, and can shoot from the air. The operator 500 hangs the smart glasses 503, and the smart glasses 503 display various information to be described later processed by the TS 100.

The TS100 uses its own position measuring device using GNSS, a camera that acquires an image of the object to be measured (UAV200), a laser scanning function for searching to search for a target (reflection prism 202 of UAV200), and a laser beam for distance measurement. Laser ranging function to measure the distance to the target (reflection prism 202), function to measure the direction of the target measured by laser (horizontal angle and vertical angle (elevation angle or depression angle)), distance to the target and direction It has a function to calculate the three-dimensional position of the target, a function to communicate with an external device, and a laser scan function to obtain point group data.

By measuring the distance and direction to the target, the position of the target with respect to the TS100 can be measured. Here, if the position of the TS100 is known, the position (latitude / longitude / altitude or XYZ coordinates on the Cartesian coordinate system) of the target (UAV200 in this case) in the map coordinate system can be known. This function is a function of a commercially available TS and is not special. Examples of the technology related to TS are described in JP-A-2009-229192, JP-A-2012-202821, and the like. The map coordinate system is a coordinate system that handles map information (for example, latitude, longitude, altitude (elevation)), and for example, the position information obtained by GNSS is usually described in the map coordinate system.

Hereinafter, an example of the TS (total station) 100 used in the present embodiment will be described. FIG. 2 shows a block diagram of the TS100. The TS100 includes a camera 101, a target search unit 102, a distance measuring unit 103, a horizontal / vertical detection unit 104, a horizontal / vertical drive unit 105, a data storage unit 106, a position measurement unit 107, a communication device 108, and a target position calculation unit. 109, UAV tracking control unit 111, laser scanner 112, control microcomputer 113, image acquisition unit 114, image creation unit 115, flight altitude calculation unit 116, flight speed calculation unit 117, distance calculation unit 119 to obstacles, display magnification It includes an adjustment unit 120, a three-dimensional model creation unit 121, an external control element calculation unit 122, a reticle display position calculation unit 123, and a UAV direction calculation unit 124.

Each functional unit shown in FIG. 2 may be configured by dedicated hardware, or those that can be configured by software by a microcomputer may be configured by software. The hardware used to realize the configuration of FIG. 2 includes various electronic devices (for example, a camera module constituting the camera 101, a wireless module constituting the communication device 108, etc.), various drives using a motor, and the like. Examples include mechanisms, sensor mechanisms, optical components, various electronic circuits, CPUs (Central Processing Units), ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and the like. The above hardware configuration is the same for the UAV200.

The camera 101 captures a moving image or a still image of a surveying object such as a UAV 200 or a target. The camera 101 is configured by using a camera module using a CCD or a CMOS sensor, takes a picture of a positioning target (for example, UAV200) through a telescope, and obtains image data thereof. Normally, the operator roughly collimates the target to be surveyed using the image taken by the camera 101 through the telescope, and then the target is precisely collimated by the autonomous operation by the automatic tracking function. ..

The optical axis of the camera 101 coincides with the optical axis of the ranging unit 103 (the optical axis of the laser beam for ranging), which will be described later, and the target for specifying the position is captured in the center of the captured image of the camera 101. .. The TS100 is provided with a small display (for example, a liquid crystal display or the like), and an image taken by the camera 101 can be displayed there. The image data taken by the camera 101 can be output to the outside and can be displayed on an appropriate display. In this example, the image taken by the camera 101 is displayed on the smart glasses 502 (glasses having the function of a head-mounted display) worn by the operator 500 who controls the UAV 200.

The smart glasses 502 is a transmissive head-mounted display, also called a glasses-type wearable terminal, and displays various information related to the flight of the UAV 200. The smart glasses 502 can also see through the screen and look far away, and the image output from the TS100 is displayed overlaid on the background visible in the field of view. Smart glasses are glasses-type devices in which a small display is mounted at the position where the lens of the glasses is located. This display is transmissive and can be seen through the display on the other side.

A person wearing smart glasses looks at an object through his glasses and at the same time, a screen that is virtually formed in front of him (as if the screen were expanded from several tens of centimeters to several meters in front of him). ) Displays various images like a normal display. The standard of the image displayed on the smart glass can be the same as the image displayed on a PC, a smartphone, or the like. Various types of smart glasses are sold by multiple manufacturers.

The magnification of the image taken by the camera 101 and displayed on the display equipped with the TS100 and the external display can be changed. The zoom function of this image is realized by a method by an optical method, a method by digital processing, a method in which both are combined, and the like. It is also possible to use a commercially available digital camera as the camera 101 and attach it externally to the TS100. The image data taken by the camera 101 can be stored in an appropriate storage area in association with data such as the measurement time, the measurement direction, the measurement distance, and the position of the measurement target related to the distance measurement target.

The target search unit 102 searches for a target (reflection prism 202 of UAV200) using a search laser beam having a triangular pyramid shape or a fan-shaped beam. The target search is performed with the TS100 as a reference position. The search and tracking of a target (reflection prism) by TS is described in, for example, Japanese Patent No. 512439.

The distance measuring unit 103 measures the distance to the target using the distance measuring laser beam. The range-finding unit 103 includes a light-emitting element for range-finding laser light, an irradiation optical system, a light-receiving optical system, a light-receiving element, a range-finding calculation unit, and an optical path for reference light. The distance to the object is calculated from the phase difference between the distance measurement light reflected from the object and the reference light. The method of calculating the distance is the same as that of normal laser ranging.

The horizontal / vertical detection unit 104 measures the horizontal and vertical angles (elevation angle and depression angle) of the target measured by the distance measuring unit 103. The housing portion including the target search unit 102, the optical system of the distance measuring unit 103, and the camera 101 is capable of horizontal rotation and elevation angle (depression angle) control, and the horizontal and vertical angles are measured by the encoder. .. The output of this encoder is detected by the horizontal / vertical direction angle detection unit 104, and the horizontal direction angle and the vertical direction angle (elevation angle and depression angle) are measured.

The horizontal / vertical drive unit 105 is a motor that performs horizontal rotation and elevation control (and depression angle control) of the housing portion including the target search unit 102, the optical system of the distance measuring unit 103, and the camera 101, and the drive circuit of the motor. , The control circuit of the drive circuit is provided. A laser scanner 112, which will be described later, is also arranged in this housing portion. The data storage unit 106 stores control programs, various data, survey results, and the like necessary for the operation of the TS 100.

The position measuring unit 107 measures the position of the TS100 using GNSS. The position measuring unit 107 has a function of performing both relative positioning and independent positioning. In an environment where relative positioning can be performed, it is preferable to measure the position of the TS100 using relative positioning, but when relative positioning is difficult, the position of the TS100 is measured by independent positioning. In the positioning of the UAV200 by the TS100, it is necessary to acquire the position of the TS100 in the map coordinate system as known data in advance. In actual operation, the TS100 may be installed at a position where the coordinates are known in advance, or the position where the TS100 is installed may be positioned by the position measuring unit 107. In the former case, positioning by the position measuring unit 107 becomes unnecessary.

The communication device 108 communicates with an external device. The TS100 can be operated by an external terminal (dedicated terminal, PC, tablet, smartphone, etc.), and communication at this time is performed using the communication device 108. Further, the communication device 108 receives various data necessary for the operation of the TS 100 and outputs various data acquired by the TS 100 to the outside. For example, the TS100 can access a data server that handles map information and topographical information via an Internet line, and the TS100 can acquire various types of information related to surveying. Further, it is possible to configure a part of the functions shown in FIG. 2 to be performed by an external device, and in that case, various data are exchanged via the communication device 108. Further, the communication device 108 receives the control signal transmitted from the UAV 200 from the controller 501, and based on this control signal, the control stick operation instruction display illustrated in FIG. 5 is performed.

The target position calculation unit 109 calculates the position (coordinates) of the target with respect to the TS100 from the distance and direction to the target (in this case, the reflection prism 202 mounted on the UAV200). Here, the distance to the target is obtained by the ranging unit 103, and the direction of the target is obtained by the horizontal / vertical detection unit 104. Since the position of the TS100 as the reference position is specified by the position measuring unit 107, the position of the target in the map coordinate system can be obtained by obtaining the position of the target with respect to the TS100.

The UAV tracking control unit 111 controls to track the supplemented UAV. That is, the direction of the TS is controlled according to the incident direction of the tracking light (tracking light reflected from the UAV) detected by the target search unit 102, so that the optical axis of the TS100 always faces the UAV moving in the air. Control is performed by the UAV tracking control unit 111. Specifically, the incident direction of the tracking light reflected from the target (reflection prism 202) with respect to the TS100 is detected, and based on this, the optical axis of the TS100 (the optical axis of the distance measurement light from the distance measuring unit 103) is located at the position of the reflection prism 202. ) Is always facing, the UAV tracking control unit 111 performs a process of outputting a control signal to the horizontal / vertical drive unit 105.

The laser scanner 112 uses the range-finding laser light as the scan light to obtain point cloud data. Point cloud data is data that captures an object as a set of points whose three-dimensional coordinates are known. In this example, the target search unit 102 and the laser scanner 112 have different configurations, and the laser scanner 112 operates separately from the target search unit 102. Here, the wavelength of the laser light handled by the laser scanner is selected to be different from the wavelength of the laser light used by the target search unit 102 so as not to interfere with the laser light used by the target search unit 102. A laser scanner for obtaining point cloud data is described in, for example, Japanese Patent Application Laid-Open No. 2010-151682, Japanese Patent Application Laid-Open No. 2008-268004, US Pat. No. 8,67,190, US Pat. No. 7,969558 and the like. As the laser scanner 112, an electronic laser scanner that scans electronically (Solid state optical phased array method) can also be used. This technique is described, for example, in US Publication No. US2015 / 0293224.

Hereinafter, the laser scanner 112 will be described. The laser scanner 112 detects the distance to the object based on the irradiation unit of the laser beam for distance measurement, the light receiving unit that receives the laser light for distance measurement reflected from the object, and the flight time of the laser light for distance measurement. A unit and a distance measuring direction detecting unit for detecting the irradiation direction (distance measuring direction) of the laser beam for distance measuring are provided. Further, the laser scanner 112 is a distance measuring target point position calculation unit that calculates a three-dimensional position of a reflection point of a distance measuring laser beam based on its own position, a range measuring distance and a distance measuring direction, and a range measuring laser light. It is provided with a scan control unit that controls the irradiation direction of the laser and the light receiving direction of the reflected light (optical axis of the ranging light).

The range-finding laser beam is pulsed at a specific repetition frequency, and is irradiated in a specific range while being scanned. The distance from the flight time of the laser beam for distance measurement to the reflection point is calculated. Normally, the distance to the object is calculated from the phase difference between the reference light flying through the reference optical path provided in the device and the ranging light reflected from the reference light irradiated from the reference light path. From the distance measured, the irradiation direction of the laser beam for distance measurement, and the position of the laser scanner 112, a three-dimensional position with the position of the laser scanner 112 at the reflection point as the origin is calculated. Point cloud data can be obtained by measuring the positions of many reflection points. Here, the position and orientation of the laser scanner 112 in the TS100 are acquired as known information in advance, and the position of the laser scanner 112 in the map coordinate system is calculated based on the positioning data of the position measuring unit 107. Therefore, it is possible to acquire the three-dimensional coordinates of each scanned point (reflection point) in the map coordinate system.

By scanning and irradiating the range-finding laser beam at a specific oscillation frequency, the three-dimensional coordinates of each of a large number of reflection points on the object are acquired. The set of a large number of reflection points in this object becomes point cloud data. In the point cloud data, the object is captured as a set of points whose three-dimensional positions are specified.

The laser scanner 112 can acquire the RGB intensity of the reflection point. The RGB intensity is obtained by selecting the reflected light with an R filter, a G filter, and a B filter, and detecting the light intensity of each color. Therefore, the data regarding the RGB intensity of each point of the obtained point cloud data can also be obtained. In addition, it is not limited to RGB, and it is also possible to acquire the intensity of one or a plurality of specific color information. It is also possible to detect the intensity of the reflected light of the scan light separately (or in addition to the color information) from the color information. By utilizing at least one of the color intensity and the reflection intensity of the scan light, the UAV 200 may be distinguished (distinguished) from other flying objects such as birds.

The control microcomputer 113 controls the processing procedure of FIGS. 3 and 4, which will be described later, and controls the operation of the entire TS100. The image acquisition unit 114 acquires image data of an image taken by the camera 101. The image creation unit 115 creates an image to be displayed on the smart glasses 502. 5 to 7 and 10 to 13 show an example of an image created by the image creating unit 115 and displayed on the smart glasses 502.

The flight altitude calculation unit 116 calculates the ground altitude of the UAV200 positioned by the TS100. The elevation altitude of the UAV200 is obtained by subtracting the elevation of the ground surface at the longitude and latitude of the position of the UAV200 from the elevation of the position of the UAV200 positioned by the TS200. The altitude of the ground surface is obtained from the map information, but if the map information cannot be obtained, the value of the altitude of the installation position of the TS100 is adopted.

The flight speed calculation unit 117 calculates the speed (speed and direction) of the UAV200 tracked by the TS100. Since the position of the TS100 is measured every moment, a velocity vector can be obtained by calculating the tangent vector of the movement path, and the movement direction (θx, θy, θz) and speed (Vx) of the UAV200 are obtained from this velocity vector. , Vy, Vz).

The distance calculation unit 119 to the obstacle calculates the distance from the UAV 200 to the obstacle when there is an obstacle that obstructs the flight of the UAV 200. Obstacles are buildings, trees, towers, birds, etc. The three-dimensional data related to the obstacle is acquired from the existing three-dimensional data (for example, three-dimensional map coordinate data) or the point cloud data obtained from the laser scanner 112. Since the bird data is obtained from the existing data, it is obtained from the point cloud data obtained from the laser scanner 112.

The display magnification adjustment unit 120 adjusts the display magnification of the image displayed on the smart glasses 502. For example, if the UAV 200 is likely to interfere with an obstacle, the display magnification is adjusted so that both are within the field of view.

The three-dimensional model creation unit 121 creates a model that captures the UAV 200 and obstacles in three dimensions. This model is used when displaying the positional relationship between the UAV 200 and an obstacle on smart glasses. This point will be described later.

The reticle display position calculation unit 123 calculates the display position of the reticle (guide display of Δ, □, ○, ◎, ×, etc. indicating the position on the screen) on the display screen of the smart glasses 502. The display position of the reticle is a position on the display screen of the UAV200 (position specifying object) that can be seen through the smart glasses 502. Therefore, the reticle display position calculation unit 123 calculates the position on the display screen of the UAV200 (position specifying object) that can be seen through the smart glasses 502. The direction calculation unit 124 calculates the direction of the UAV 200 out of the field of view of the smart glasses 502. Details of the processing of the reticle display position calculation unit 123 and the direction calculation unit 124 will be described later.

The operation of the TS100 and the smart glasses 502 is performed using a dedicated terminal, a smartphone or tablet on which scanning application software is installed. For example, switching of various displays displayed on the smart glasses 502, which will be described later, is performed using a smartphone or tablet. Of course, a dedicated controller may be prepared to operate the TS100 and the smart glasses 502. Further, a form in which the operation function of the TS100 is added to the controller 501 of the UAV200, a form in which the UAV200 and the TS100 are operated by a smartphone, and the like are also possible.

(Example of processing)
Hereinafter, an example of the processing performed by the TS100 will be described. 3 and 4 are flowcharts showing an example of the processing procedure. The program that executes the processes of FIGS. 3 and 4 is stored in an appropriate storage area such as the data storage unit 106, and is read out from the appropriate storage area and executed. It is also possible to store the program in an appropriate storage medium, data server, or the like, read it from the storage medium, data server, or the like, and execute the program.

First, a process of capturing the UAV 200 by the TS100 and specifying the position by laser ranging will be described. FIG. 3 shows an example of this process. In this process, first, the search for the reflection prism 202 of the UAV 200 is performed (step S101). This process is performed by the target search unit 102. Further, a laser scan for acquiring point cloud data by the laser scanner 112 is started.

For example, the TS100 captures the UAV200 while it is landing on the ground before it takes off. Then, the laser scan for the captured UAV200 is started by the laser scanner 112, and the acquisition of the point cloud data of the UAV200 and its surroundings is started. After that, the UAV200 starts flying, the tracking of the UAV200 by the TS100 is performed, and at the same time, the laser scan of the UAV200 and its surroundings is continuously performed. Further, the UAV200 may be tracked by the following method. In this case, when the flight of the UAV200 is started, the UAV200 first rises from the ground to a specified height and then stops there. This stop position is defined in advance in the flight plan and is also known on the TS100 side. Therefore, the TS100 captures the UAV200 by aiming at the position of the UAV200 that rises and stops. Further, the laser scan in which the UAV200 is within the scan range is started by the laser scanner 112, and the acquisition of the point cloud data of the UAV200 and its surroundings is started. As the range of the laser scan at this stage, for example, a range having a radius of about 20 to 50 m centered on the UAV200 is selected.

After capturing the target reflection prism 202, control is performed to direct the optical axis of the TS100 (the optical axis of the ranging unit 103) toward the reflection prism 202 (step S102). After capturing the reflection prism 202, the UAV tracking control unit 111 controls the optical axis of the ranging unit 103 to be directed to the reflection prism 202.

Next, the distance measuring unit 103 irradiates the distance measuring light toward the reflecting prism 202 to measure the position of the UAV 200 (step S103). The measurement of the position of the UAV 200 is performed by the target position calculation unit 109.

The processes of steps S101 → S102 → S103 are repeated, and tracking of the moving UAV200 is continuously performed. As a result, the position (three-dimensional coordinates) of the UAV200 is acquired every moment, and the position and the movement path of the UAV200 are grasped on the side of the TS100.

The UAV200 is photographed by the camera 101 while the UAV200 is being tracked by the TS101. The image taken by the UAV 200 by the camera 101 is displayed on the smart glasses 502. Hereinafter, the processing related to the image displayed on the smart glasses 502 will be described. This process is the same when displaying an image taken by the camera 101 on a display of a smartphone, tablet, PC, or the like.

FIG. 4 shows an example of processing related to the image displayed on the smart glasses 502. The process of FIG. 3 is performed, the UAV200 is captured by the TS100, and the process of FIG. 4 is executed in a state where the positioning is continuously performed. First, the altitude above ground level of the UAV200 is calculated by the altitude calculation unit 116 (step S111). Next, the flight speed calculation unit 117 calculates the speed (direction and speed) of the UAV 200 (step S112).

Next, the image of the UAV 200 taken by the camera 101 is displayed on the display screen of the smart glasses 502 (step S113). Next, the flight state information of the UAV200 obtained at this stage is displayed on the display screen of the smart glasses 502 (step S114). Examples of flight state information include information on altitude above ground level and speed (direction of flight and speed of flight). The flight state information includes an ascending rate, a descending rate, a turning radius, a deviation from the flight plan route, a distance from the TS100, and the like. The processing of steps S113 and S114 is performed by the image creating unit 115. FIG. 5 shows an example of the field of view seen by the operator 500 through the smart glasses 502 at the stage of step S114.

Next, the calculation of the distance between the UAV 200 and the obstacle is performed by the "distance calculation unit 119 between the obstacle" (step S115). Next, it is determined whether or not the distance calculated by the stap S115 is equal to or less than a predetermined value (for example, 15 m or less) (step S115), and if this distance is equal to or less than the specified value, step S117 is performed. If not, the process of step S111 or less is repeated. By repeating the process of step S111 and the like, the altitude and speed information displayed on the smart glasses 502 is continuously updated every moment.

In step S117, the display magnification on the smart glasses 502 is adjusted so that both the UAV 117 and the target obstacle are displayed on the screen. This process is performed by the magnification adjusting unit 120. FIG. 6 shows an example of the display screen obtained as a result of this processing. By displaying the UAV 200 and the obstacle on the screen at the same time as shown in FIG. 6, the operator 500 can grasp the positional relationship between the UAV 200 and the obstacle.

Further, when the distance between the UAV 200 and the obstacle is equal to or less than the distance determined to be dangerous (for example, 10 m or less), a warning display screen is displayed on the smart glasses 502 (step S118). This process is performed by the image creation unit 115. In this display, the UAV200 may collide with an obstacle, and the distance to the obstacle, the direction, the expected contact time, the direction suitable for avoidance, etc. are displayed. .. FIG. 7 shows an example of the display screen at this time.

2. 2. Second Embodiment When the UAV200 and an obstacle are present on the optical axis of the TS100, the UAV200 and the obstacle may overlap each other in the image obtained by the camera 101 of the TS100, and it may be difficult to understand the sense of distance between the two. In this case, the viewpoint change display described below is displayed on the smart glasses 502. The viewpoint change display is created by the image creation unit 115 based on the three-dimensional model created by the three-dimensional model creation unit 121.

The three-dimensional model creation unit 121 creates a dummy model of the UAV 200 and a three-dimensional model of an obstacle. The dummy model of the UAV200 is a dummy model converted into three-dimensional data that imitates the UAV200 arranged at the three-dimensional position calculated by the target position calculation unit 109. The orientation of this dummy model is not accurate, but it is not important here.

There are two methods for acquiring a three-dimensional model of an obstacle, one is to acquire from a database of three-dimensional map information, and the other is to acquire from point cloud data obtained from a laser scanner 112. However, the point cloud data obtained from the laser scanner 112 has an occlusion problem, and the data of the part that becomes the blind spot from the TS100 is missing, so the missing part was modeled by complementing the data. Image Create an image.

The image creation unit 115 creates an image of the three-dimensional model including the UAV 200 and the obstacle viewed from a direction perpendicular to or horizontal to the line connecting the UAV 200 and the obstacle. In this image, the UAV200 is modeled, and obstacles are also modeled to be an unnatural image in some cases, but that point is acceptable because it is an image for grasping the sense of distance between the two.

Here, since the three-dimensional positions of the UAV200 and the obstacle are known, the separation distance between the two is known. Therefore, the distance between the two is displayed in the above image. In addition, the moving direction of the UAV and other various information shown in FIGS. 5 to 7 are also displayed. In this image, even when the UAV200 and the obstacle overlap on the optical axis of the TS100, the positional relationship between the UAV200 and the obstacle can be visually grasped.

3. 3. Third Embodiment FIG. 5 shows the field of view of the smart glasses 502, that is, the state in which the UAV looks small through the smart glasses 502. In the present embodiment, a reticle that serves as a guide display (display of a mark) is displayed at a position on the screen of the UAV that is visible through the smart glasses 502. By displaying this reticle, the position of the UAV200 on the screen of the smart glasses is shown, and it is displayed that the TS100 is capturing the UAV200.

Further, in FIG. 5, the UAV200 is visible, but when the flight position of the UAV200 is far away, the UAV200 may not be visible (or difficult to see) visually. In addition, it may be difficult to distinguish it from a bird. In such a case, it is convenient to display a reticle that guides and displays the position of the UAV 200 on the screen seen by the operator 500 on the smart glasses 502, in other words, in the field of view of the operator 500 through the smart glasses 502. be.

FIG. 10 shows an example in which a reticle that guides the position of the UAV 200 on the screen of the smart glass 502 is displayed. In the case of FIG. 10, an example in which the UAV200 looks relatively large is shown, but if the distance to the UAV200 is long and it is difficult to see or distinguish it from a bird, the reticle display is used to pass through the smart glass 502. By guiding the position of the UAV200 in the visible field of view, it becomes easier to directly see the UAV200.

Hereinafter, a mechanism for causing the smart glasses 502 to display FIG. 10 will be described. The process of causing the smart glasses 502 to display FIG. 10 is performed by the reticle display position calculation unit 123 of FIG.

The reticle display position calculation unit 123 performs the following processing. First, the external orientation element calculation unit 122 calculates the external orientation element (direction and position) of the smart glasses 502. The calculation of the external control element of the smart glass 502 will be described later.

The orientation of the smart glasses 502 is defined by the orientation in front of them. The position of the smart glasses 502 is a virtual viewpoint position of the person wearing the smart glasses 502 (in this case, the operator 500). Since this viewpoint position is set at the time of designing the smart glass 502, that position is used. This position is described in the coordinate system handled by the TS100 (for example, the map coordinate system).

If the position of the smart glasses 502 is known, the position of the UAV 200 is determined by the TS100 and is known. Therefore, a direction line connecting the smart glasses 502 and the UAV 200 is set, and the direction line and the display screen of the smart glasses 502 (operator 500). The intersection with the screen that is virtually displayed in front of you) is calculated as the reticle display position. This reticle display position is a position where the UAV200 can be seen if it can be visually recognized. In this way, the position of the reticle in the field of view of the smart glass 502 is calculated.

For example, in FIG. 9, the point P1 is the three-dimensional position of the UAV200, and the point O is the position of the viewpoint of the person (operator 502) who is wearing the smart glasses 502. Here, the intersection p1 between the direction line connecting the point P1 and the point O and the display screen of the smart glasses 502 virtually arranged is calculated as the display position of the reticle.

Based on the above calculation result, the image creation unit 115 performs a process of displaying the reticle on the screen of the smart glasses 502. In this way, the image illustrated in FIG. 10 is displayed on the smart glasses 502.

Next, the calculation of the external control element of the smart glass 502 performed by the external control element calculation unit 122 will be described. First, when the UAV200 is locked by the TS100 at the start of flight of the UAV200, a calibration process is performed to acquire the initial value of the external control element. At this time, the UAV200 is stopped at a specified altitude, and the UAV200 is locked (captured) by the TS100.

At this time, the display of the smart glasses 502 is set to the calibration mode, and the screen shown in FIG. 11 is displayed. The screen of FIG. 11 shows an example in which a guide display for collimation is displayed at a predetermined position (in this case, the center) in the screen (in the field of view). The operator 500 moves his head so that the UAV200 can be seen in the center of the guide display for collimation, and captures the image of the UAV200 in the center of the guide display for collimation. The calibration process is executed in this state. This process is performed, for example, by operating a smartphone used as an operation terminal.

Here, the operator 500 carries a position specifying device 303 using GNSS (for example, the GPS function of a smartphone is used), and the positioning data is sent to the TS100. The external orientation element calculation unit 122 performs relative positioning using the positioning data sent from the operator 500 and the positioning data of the position measurement unit 107, and calculates the three-dimensional position of the viewpoint of the operator 500.

For example, when using the GPS function of a smartphone, prepare correction values for cases such as putting the smartphone in the chest pocket or putting it in the bottom pocket of the pants in advance, and using the correction values, the viewpoint position of the operator 500. Is calculated.

In this way, data on the position of the viewpoint of the operator 500 and the position of the UAV 200 as seen from the position of this viewpoint can be obtained. Here, the direction from the position of the viewpoint of the operator 500 toward the position of the UAV 200 is the direction of the smart glasses 502. As described above, the initial values of the external orientation elements (position and orientation) of the smart glasses 502 are calculated.

This external orientation element changes as the operator 500 moves and moves his head. The external control element calculation unit 122 corrects the external control element of the smart glasses 502 corresponding to this change. Modifications to the external orientation elements of the smart glasses 502 are detected based on the outputs of the 3D electronic compass 301 and the 3D accelerometer 302 in FIG.

The three-dimensional electronic compass 301 and the three-dimensional acceleration sensor 302 are attached to the smart glasses 502 and detect a change in the orientation of the smart glasses 502. The three-dimensional electronic compass 301 detects the three-dimensional direction of the smart glasses 502. The three-dimensional acceleration sensor 302 detects a three-dimensional component of acceleration applied to the smart glasses 502. The three-dimensional electronic compass 301 and the three-dimensional acceleration sensor 302 use parts used in smartphones.

The outputs of the three-dimensional electronic compass 301 and the three-dimensional acceleration sensor 302 are input to the TS100. A wireless LAN line or the like is used for data transmission. Upon receiving the outputs of the three-dimensional electronic compass 301 and the three-dimensional acceleration sensor 302, the external orientation element calculation unit 122 corrects the orientation of the smart glasses 502 at any time. Further, the position of the smart glasses 502 is corrected at any time from the change of the positioning data of the GNSS position measuring device 303 carried by the operator 500. In this way, the external locating element of the smart glass 502 is modified at any time, and the process related to the display of the reticle illustrated in FIG. 10 is performed using the external locating element modified at any time.

The above-mentioned correction of the external control element of the smart glass 502 gradually accumulates errors, and the accuracy is lost in the process of the correction. Hereinafter, the process of maintaining the accuracy of the external control element of the smart glass 502 will be described. In this process, TS100 is used as the collimation target. That is, as shown in FIG. 12, the operator 500 moves his / her head so that the TS 100 is located at the center of the cross guide for collimation. In this state, the external control element 122 is recalculated.

At this time, the external control element calculation unit 122 performs the following processing. In this process, information on the position of the smart glasses (viewpoint position of the operator 500) and the position of the TS100 is acquired, and information is obtained.
The direction from the position of the smart glass 502 toward the TS100 is calculated. In this way, the external orientation element of the smart glasses 502 is recalculated. The external orientation elements of the smart glasses 502 change as the operator 502 moves and moves his head, the modifications being made from time to time as described above.

The calculation of the external control element of the smart glasses 502 with the TS100 as the collimation may be used for the calculation of the initial value at the first stage. Further, when the UAV 100 is in a position where it is easy to collimate, the external control element of the smart glass 502 may be recalculated (calibrated) by the method described in relation to FIG. Further, the target whose position is specified may be placed in a range visible to the operator 500, and the external control element may be recalculated (or initially set) by collimating the target.

By recalculating the external control element of the smart glass 502 at any time, it is possible to suppress the decrease in the accuracy of the reticle display in FIG. 10 due to the decrease in the accuracy of the external control element. It is also possible to mount a GNSS position specifying device and an IMU (inertial measurement unit) on the smart glasses 502 and measure the external orientation elements (position and attitude) of the smart glasses 502 in real time. If the smart glass 502 cannot afford to mount the GNSS positioning device and IMU, attach the GNSS positioning device and IMU to the head-mounted helmet or head-mounted auxiliary device, and attach the GNSS positioning device and IMU to the head-mounted helmet or head. It is also possible to attach a part-mounted auxiliary device to the head and acquire an external control element of the smart glass 502.

4. Fourth Embodiment It is also possible to fix the camera to the smart glass and calculate the external orientation element of the smart glass based on the image taken by the camera. In this case, a plurality of targets whose positions are specified, a small camera attached to smart glasses, and a target extraction unit that extracts the target from the image are required, and the external orientation element calculation unit 122 requires a smart by the backward interaction method using the target. Calculation of the external targeting element of the glass is performed. The calculation of the external orientation element of the camera using the backward association method is described in, for example, Japanese Patent Application Laid-Open No. 2013-186816.

Hereinafter, the calculation of the external control element of the smart glass using the image taken by the camera will be described. In this case, a camera that shoots a moving image is attached to the smart glasses. The relationship between the position and orientation of the smart glasses and the camera is acquired in advance as known data. In addition, three or more identifiable targets whose positions (three-dimensional coordinates) are predetermined are placed within a range visible to a person wearing smart glasses. At this time, the targets are arranged so that three or more targets are always photographed by the camera even if the human moves or the direction of the head is changed. As a target, a form of identification by a number or a color code can be used. Examples of the color code target are described in JP-A-2011-053030 and JP-A-2011-053031.

Since the main purpose of the camera is to shoot the target, set the direction so that the target can be shot effectively. Further, the orientation of the camera in the horizontal direction does not need to be adjusted to the orientation of the smart glasses as long as the target can effectively shoot, and the camera may be directed to the side or the rear of a person wearing the smart glasses, for example.

In calculating the external orientation element, first, three or more targets are photographed by the camera in the initial state. Then, the target is identified and extracted from the captured image by image processing. Since the three-dimensional coordinates of each target are known, the external orientation element of the camera can be calculated by the backward orientation method. When the smart glasses (camera) move, the external orientation element of the camera changes, but if three or more targets are shown, the external orientation element of the camera can be calculated by the backward association method. In this way, the calculation of the fluctuating external control element of the camera is continuously performed.

FIG. 14 shows the principle of the backward association method. In this case, the point O is the position of the camera (the position of the shooting viewpoint), the points P1, P2, and P3 are the positions of the target, and the points p1, p2, and p3 are the positions of the target image on the screen of the image taken by the camera. Is. Here, points P1, P2, and P3 are identified (positioned) in advance and are known, and p1, p2, and p3 are obtained from captured images. Then, the position O of the camera is obtained by setting the direction line connecting P1 and p1, the direction line connecting P2 and p2, and the direction line connecting P3 and p3, and finding the intersection O of these three direction lines.

Further, the optical axis of the camera can be mathematically obtained by obtaining the direction line connecting the point O and the center of the screen. By obtaining the optical axis of this camera, the direction of the optical axis from the point O can be obtained as the direction of the camera. In this way, the position and orientation of the camera fixed to the smart glasses, that is, the external orientation element is calculated. Since the relative positional relationship and the orientation relationship between the camera and the smart glass are known, the external orientation element of the smart glass can be obtained by obtaining the external orientation element of the camera.

By continuously taking pictures with the camera and continuing the above processing, the fluctuating external orientation elements of the smart glasses are sequentially calculated and continuously updated.

For example, a camera for photographing the ground surface around the operator 500 is fixed to the smart glass 502 in FIG. 1. Further, a plurality of targets are arranged around the operator 500 and their positions are measured. Further, the relationship between the position and orientation of the smart glasses 502 and the camera is acquired in advance. Here, while the operator 500 is manipulating (or monitoring) the UAV 200, the targets are arranged so that three or more targets are always photographed by the camera.

Even if the operator 502 moves his head and the external orientation element of the smart glass 502 fluctuates, the external orientation element of the smart glass 502 is calculated using the above target, and the external orientation of the smart glass 502 at that time is performed. The element is retrieved. This process continues and the external orientation elements of the smart glasses 502 are constantly updated. Using this dynamically updated external orientation element, for example, the process related to the display of FIG. 10 is performed.

In the case of this method, an image processing unit that extracts targets from the camera and captured images is required, and positioning work for multiple targets is required in advance. It can be obtained with high accuracy.

The method for obtaining the external orientation element of the smart glasses 502 described in the present specification can be used in combination of a plurality. The reticle display in FIG. 10 requires the position data of the UAV200, but the positioning data of the GNSS position measuring device mounted on the UAV200 can also be used as the position data.

Using a stereo camera as a camera, the position of the feature points extracted from the stereo image is obtained by the forward interaction method to obtain point cloud data, and the point cloud data is used as the control point to obtain the external control element of the camera (external control element of smart glass). ) Can also be calculated. In addition, using one camera, a stereo image is acquired by changing the viewpoint, point cloud data is obtained in the same manner as in the above case, and the external control element of the camera based on this point cloud data (external control element of smart glasses). ) Can also be calculated. In these cases as well, a target whose position is specified is required to acquire the initial value, but since the point cloud extracted from the image is also used for the standardization, the number can be reduced as compared with the case where only the above-mentioned target is used. .. Japanese Patent Application Laid-Open No. 2013-186816 describes a technique for obtaining point cloud data from a stereo image and a technique for obtaining a point cloud data from a stereo image obtained with one camera.

5. Fifth Embodiment Here, an example will be described in which when the operator 500 loses sight of the UAV 100, a display that guides the direction of the UAV 100 as seen from the operator 500 is displayed on the display screen of the smart glasses 502. The TS100 is tracking the UAV200, but the operator 500 may lose sight of the UAV100. In this case, the function of the UAV direction calculation unit 124 provides a display that guides the direction of the UAV 100 in the display screen of the smart glasses 502 (the field of view that the operator 500 is visually recognizing through the smart glasses).

FIG. 13 shows an example of the display screen at that time. In this case, the UAV 100 can be captured by directing the line of sight in the upper left direction. Hereinafter, the function of the UAV direction calculation unit 124 will be described. First, the TS100 captures it. The position of the UAV 200 calculated by the target position calculation unit 109 is acquired. Next, the external control element of the smart glass 502 calculated by the external control element calculation unit 122 is acquired.

Next, a vector orthogonal to the optical axis (front direction) of the smart glasses 502 is obtained, and this vector is orthographically projected onto the display surface of the smart glasses 502, and this vector is displayed on the smart glasses 502 as an arrow. In this way, the display (guide display) of the arrow indicating the direction of the UAV 200 shown in FIG. 13 is displayed on the smart glasses 502. Looking at the image of FIG. 13, the operator 502 can see the direction of the UAV200 and capture the UAV200 by turning his face in the direction of the arrow. At this time, when the UAV200 is in the field of view, the reticle illustrated in FIG. 10 is displayed, so that the UAV200 can be easily visually recognized even if the UAV200 is far away and difficult to visually recognize.

(others)
The smart glasses 502 may be worn by a person other than the UAV200 operator 500. As a method of guiding the position of the UAV on the display screen, it is also possible to display an arrow or highlight which split screen of the 4-split or 9-split screen the UAV is located on. .. As the target for displaying the reticle in FIG. 10, it is also possible to select a point or an object whose position other than the UAV is specified. The technique exemplified in FIG. 10 can be used for monitoring, searching, confirmation, and the like of a position-specified object. The invention disclosed in the present specification can be used in a technique for effectively visually recognizing an object.

100 ... TS (total station), 200 ... UAV, 201 ... camera, 200 ... reflective prism, 500 ... UAV operator, 501 ... controller, 502 ... smart glasses.

Claims (2)

An unmanned aerial vehicle position measuring unit that measures the position of the unmanned aerial vehicle by measuring the distance to the unmanned aerial vehicle while optically tracking the unmanned aerial vehicle.
An external control element calculation unit that calculates the external control element of the head-mounted display,
A position calculation unit that calculates the position of the unmanned aerial vehicle on the display screen of the head-mounted display based on the position of the unmanned aerial vehicle and the external orientation element of the head-mounted display.
An image creation unit that creates an image displaying a guide display indicating the position of the unmanned aerial vehicle on the display screen.
An optical information processing device equipped with. The position of the unmanned aerial vehicle is measured by measuring the distance to the unmanned aerial vehicle while optically tracking the unmanned aerial vehicle.
Calculate the external orientation element of the head-mounted display,
Based on the position of the unmanned aerial vehicle and the external orientation element of the head-mounted display, the position of the unmanned aerial vehicle on the display screen of the head-mounted display is calculated.
An optical information processing method for creating an image displaying a guide display indicating the position of the unmanned aerial vehicle on the display screen.

Images