JP6730764B1 - Flight route display method and information processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging method and an information processing apparatus capable of confirming the suitability of a flight route by checking the situation of an actual flight range and the flight route of a flying object. The present invention relates to a flight route display method for displaying a flight route of an air vehicle on a user terminal, including position information and shooting angle of view information, shooting angle information, and shooting direction information of the user terminal. Obtaining shooting state information, generating a flight route image including the flight route depicted in the virtual space based on the flight route information regarding the flight route, and capturing obtained based on the shooting state information Generating a composite image obtained by superimposing the flight path image on the image, the flight path display method. [Selection diagram] Fig. 7

Description

The present invention relates to a flight path display method for an air vehicle and an information processing device.

In recent years, aircrafts such as drones and unmanned aerial vehicles (UAVs) (hereinafter collectively referred to as “aircraft”) have begun to be used in industry. Under such circumstances, Patent Document 1 discloses a flight route display method for displaying a flight route for the purpose of inspection and the like.

Japanese Patent Laid-Open No. 2015-058758

However, in the technique disclosed in Patent Document 1 above, the flight route R is shown in a bird's-eye view on the map M. However, in the technique, it is possible to confirm what flight route the flight vehicle flies from a three-dimensional viewpoint. There wasn't. Especially when the target of inspection is a three-dimensional object, it is not possible to confirm what flight route the flight body will fly around the three-dimensional object from a three-dimensional point of view. Was difficult.

In addition, it was possible to obtain images of the flight range from various viewpoints in advance and check them by comparing them with the flight route, or by actually flying locally, it was possible to confirm it. In this case, the flight path is not superposed on the real space, so accuracy is poor, and it is difficult to cope with a change from the time of image acquisition. In the latter case, there may be a risk that the user's effort to fly locally or the flight route may not be properly confirmed.

The present invention has been made in view of such a background, and by comparing the situation of the actual flight range with the flight path of the flying object, the flight path display method and the flight path display method capable of confirming the suitability of the flight path and An object is to provide an information processing device.

A main invention of the present invention for solving the above-mentioned problems is a flight route display method for displaying a flight route of a flying object on a user terminal, which is position information and shooting angle information, shooting angle information of the user terminal. A step of obtaining shooting state information including shooting direction information; a step of generating a flight route image including the flight route depicted in a virtual space based on flight route information related to the flight route; Generating a composite image obtained by superimposing the flight path image on the captured image acquired based on the flight path display method.

According to the present invention, it is possible to provide a flight route display method and a server in which the suitability of the flight route can be confirmed by comparing the actual flight range situation with the flight route of the flying object.

It is a figure which shows the structure of the management system concerning embodiment of this invention. It is a block diagram which shows the hardware constitutions of the management server of FIG. It is a block diagram which shows the hardware constitutions of the user terminal of FIG. It is a block diagram which shows the hardware constitutions of the flying body of FIG. It is a block diagram which shows the function of the management server of FIG. 6 is a block diagram showing a structure of a parameter information storage unit of FIG. 5. FIG. 3 is a flowchart of a flight route display method according to an embodiment of the present invention. It is a figure which shows an example of the picked-up image and flight route image concerning embodiment of this invention. It is a figure which shows an example of the synthetic image G concerning embodiment of this invention. It is a figure which shows an example of the picked-up image and flight route image concerning embodiment of this invention. It is a figure which shows an example of the synthetic image G concerning embodiment of this invention. It is a figure which shows an example of the picked-up image and flight route image concerning embodiment of this invention. It is a figure which shows an example of the synthetic image G concerning embodiment of this invention. It is a figure which shows an example of the display screen of the user terminal concerning embodiment of this invention. It is a figure which shows an example of the picked-up image and flight route image concerning embodiment of this invention. It is a figure which shows an example of the synthetic image G concerning embodiment of this invention.

The contents of the embodiments of the present invention will be listed and described. The flight route display method and the information processing device according to the embodiment of the present invention have the following configurations.
[Item 1]
A flight route display method for displaying a flight route of an air vehicle on a user terminal,
Acquiring shooting condition information including position information and shooting angle information of the user terminal, shooting angle information, and shooting direction information;
Generating a flight route image including the flight route depicted in a virtual space based on flight route information related to the flight route;
Generating a composite image obtained by superimposing the flight route image on the captured image acquired based on the capturing state information.
A flight path display method characterized by the above.
[Item 2]
The flight route display method according to item 1,
The image showing the flight path further includes information about waypoints,
A flight path display method characterized by the above.
[Item 3]
The flight route display method according to item 1 or 2,
The image showing the flight path further includes a three-dimensional model of a virtual air vehicle,
A flight path display method characterized by the above.
[Item 4]
The flight route display method according to items 1 to 3,
The image showing the flight path further includes a three-dimensional model of a virtual imaging target,
A flight path display method characterized by the above.
[Item 5]
The flight route display method according to items 1 to 4,
The user terminal is a wearable device having an AR function,
A flight path display method characterized by the above.
[Item 6]
The flight route display method according to items 1 to 5,
Further comprising the step of editing the flight path while displaying the composite image,
A flight path display method characterized by the above.
[Item 7]
The flight route display method according to items 1 to 6,
Further comprising the step of displaying an image acquired from the aircraft.
A flight path display method characterized by the above.
[Item 8]
The flight route display method according to item 7,
It is possible to switch the image acquired from the flying object so as to be displayed larger than the composite image,
A flight path display method characterized by the above.
[Item 9]
An information processing device for displaying a flight path of a flying object on a user terminal,
A shooting state information acquisition unit that acquires shooting state information including position information and shooting angle information of the user terminal, shooting angle information, and shooting direction information;
A flight route image generation unit that generates a flight route image including the flight route depicted in the virtual space based on flight route information regarding the flight route;
A composite image generation unit that generates a composite image obtained by superimposing the flight route image on the captured image acquired based on the shooting state information,
An information processing device characterized by the above.

<Details of the embodiment>
Hereinafter, an embodiment of a method for imaging a flying object and an information processing apparatus according to an embodiment of the present invention will be described. In the accompanying drawings, the same or similar reference numerals and names are given to the same or similar elements, and redundant description regarding the same or similar elements may be omitted in the description of each embodiment. Further, the features shown in each embodiment can be applied to other embodiments as long as they do not contradict each other.

<Structure>
As shown in FIG. 1, the management system according to the present embodiment includes a management server 1, one or more user terminals 2, one or more flying bodies 4, and one or more flying body storage devices 5. ing. The management server 1, the user terminal 2, the flying object 4, and the flying object storage device 5 are communicably connected to each other via a network. Note that the illustrated configuration is an example, and the present invention is not limited to this, and for example, the configuration may be such that the user does not have the aircraft storage device 5 and is carried by the user.

<Management server 1>
FIG. 2 is a diagram showing a hardware configuration of the management server 1. The illustrated configuration is an example, and other configurations may be included.

As shown in the figure, the management server 1 is connected to the plurality of user terminals 2, the flying body 4, and the flying body storage device 5, and constitutes a part of this system. The management server 1 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

The management server 1 includes at least a processor 10, a memory 11, a storage 12, a transmission/reception unit 13, an input/output unit 14, and the like, which are electrically connected to each other via a bus 15.

The processor 10 is an arithmetic unit that controls the overall operation of the management server 1, controls the transmission and reception of data between each element, and executes information processing required for execution of applications and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and/or a GPU (Graphics Processing Unit), and executes a program or the like for the system stored in the storage 12 and expanded in the memory 11 to execute each information processing. ..

The memory 11 includes a main memory configured by a volatile storage device such as a DRAM (Dynamic Random Access Memory) and an auxiliary storage configured by a non-volatile storage device such as a flash memory or a HDD (Hard Disc Drive). .. The memory 11 is used as a work area or the like of the processor 10, and also stores a BIOS (Basic Input/Output System) executed when the management server 1 is started, and various setting information.

The storage 12 stores various programs such as application programs. A database that stores data used for each process may be built in the storage 12.

The transmission/reception unit 13 connects the management server 1 to the network and the block chain network. The transmission/reception unit 13 may include Bluetooth (registered trademark) and BLE (Bluetooth Low Energy) short-range communication interfaces.

The input/output unit 14 is an information input device such as a keyboard and a mouse, and an output device such as a display.

The bus 15 is commonly connected to each of the above elements and transmits, for example, an address signal, a data signal, and various control signals.

<User terminal 2>
The user terminal 2 shown in FIG. 3 also includes a processor 20, a memory 21, a storage 22, a transmission/reception unit 23, an input/output unit 24, a photographing unit 26, a photographing state information acquisition unit 27, and the like, which are mutually connected via a bus 25. It is electrically connected. Since the function of each element can be configured similarly to the management server 1 described above, detailed description of the similar configuration will be omitted.

The image capturing unit 26 is, for example, a camera and acquires a captured image. The shooting state information acquisition unit 27 stores, for example, sensors such as a GPS, a gyro sensor, an atmospheric pressure sensor, and a temperature sensor, and information about the shooting unit 26 of the terminal (for example, shooting angle of view information) (memory 21 or storage). 22), etc.) and the like, and includes the shooting angle information, shooting angle information, shooting direction information, and user terminal position information (for example, latitude/longitude information and altitude information) of the user terminal when the shot image is acquired. Etc.) is acquired as the shooting state information. These acquired images and information are transmitted to the management server 1 and stored in the storage unit 160 described later. The altitude information included in the user terminal position information may be altitude information calculated based on the above-described pressure sensor or temperature sensor, or altitude information set by the user, for example, height information set by the user, Alternatively, the height information may be a value obtained by vertically offsetting the average height information corresponding to the gender of the user by a predetermined height according to the assumed user terminal position.

<Aircraft 4>
FIG. 4 is a block diagram showing the hardware configuration of the flying body 4. The flight controller 41 can have one or more processors, such as programmable processors (eg, central processing units (CPU)).

The flight controller 41 has a memory 411 and can access the memory. Memory 411 stores logic, code, and/or program instructions executable by the flight controller to perform one or more steps. The flight controller 41 may also include sensors 412 such as an inertial sensor (acceleration sensor, gyro sensor), GPS sensor, proximity sensor (e.g., rider).

The memory 411 may include, for example, a separable medium such as an SD card or a random access memory (RAM) or an external storage device. The data obtained from the cameras/sensors 42 may be directly transmitted to and stored in the memory 411. For example, still image/moving image data captured by a camera or the like may be recorded in the internal memory or the external memory, but the present invention is not limited to this, and at least the management server 1 or the internal memory from the camera/sensor 42 or the internal memory may be used. It may be recorded in either the user terminal 2 or the aircraft storage device 5. The camera 42 is installed on the flying body 4 via a gimbal 43.

The flight controller 41 includes a control module (not shown) configured to control the state of the air vehicle. For example, the control module may adjust the spatial arrangement, velocity, and/or acceleration of a vehicle having six degrees of freedom (translational motions x, y and z, and rotational motions θ _x , θ _y and θ _z ). , ESC44 (Electric Speed Controller) to control the propulsion mechanism (motor 45, etc.) of the air vehicle. The propeller 46 is rotated by the motor 45 supplied from the battery 48, and lift of the flying object is generated. The control module can control one or more of the states of the mount and sensors.

The flight controller 41 is configured to send and/or receive data from one or more external devices (eg, transceiver (propo) 49, terminal, display, or other remote controller). It is possible to communicate with the unit 47. The transceiver 49 can use any suitable communication means such as wired communication or wireless communication.

For example, the transmission/reception unit 47 uses one or more of a local area network (LAN), a wide area network (WAN), infrared, wireless, WiFi, a point-to-point (P2P) network, a telecommunication network, cloud communication, and the like. can do.

The transmitting/receiving unit 47 transmits and/or receives one or more of data acquired by the sensors 42, processing results generated by the flight controller 41, predetermined control data, a user command from a terminal or a remote controller. be able to.

The sensors 42 according to the present embodiment may include an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, a proximity sensor (eg, rider), or a vision/image sensor (eg, camera).

<Management server functions>
FIG. 5 is a block diagram illustrating functions implemented in the management server 1. In the present embodiment, the management server 1 includes a communication unit 110, a flight mission generation unit 120, a captured image/shooting state information reception unit 130, a flight route image generation unit 140, a composite image generation unit 150, and a storage unit 160. ing. The flight mission generation unit 120 includes a flight path generation unit 121. The storage unit 160 also includes a flight route information storage unit 162, a flight log storage unit 164, and a shooting information storage unit 166. Note that the storage unit 160 may further include a storage unit that stores information necessary for performing imaging, and includes, for example, information regarding flight conditions (for example, flight speed, waypoint intervals, etc.) and imaging targets. Even if each has a storage unit (not shown) that stores information about the object (for example, position coordinates and height information) and information about the surrounding environment of the object (for example, information about the terrain and surrounding structures) Good.

The communication unit 110 communicates with the user terminal 2, the flying object 4, and the flying object storage device 5. The communication unit 110 also functions as a reception unit that receives a flight request from the user terminal 2.

The flight mission generation unit 120 generates a flight mission. The flight mission is information including at least an imaging point (so-called waypoint, including, for example, latitude and longitude information and flight altitude information) information, an imaging direction information, and a flight route including imaging date and time information. The flight path may be set by a known method, for example, referring to one in which the imaging point and the imaging direction are manually set, or setting the position coordinates of the imaging object and the imaging distance from the imaging object. For example, the flight path generation unit 121 may automatically calculate and set the imaging point and the imaging direction. The generated information regarding the flight route may be stored in the flight route information storage unit 162.

Note that the flight path may be configured such that the position where the aircraft is carried by the user is set as the flight start position without the aircraft storage device 5, or the user collects the aircraft at the flight end position. Then, based on the information (for example, position information, storage state information, storage body information, etc.) of the flying body storage device 5 managed by the management server 1, it is selected as the flight start position, the intermediate stopover or the flight end position. It may be configured to be generated as a flight path including the position of the flying object storage device 5.

The picked-up image/shooting state information receiving unit 130 receives the picked-up image and shooting state information transmitted from the user terminal 2, and stores them in the shooting information storage unit 166 as necessary.

The flight route image generation unit 140 constructs, for example, a virtual three-dimensional coordinate space (so-called VR space), and draws a flight route in the VR space based on the flight route information stored in the flight route information storage unit 162 or the like. To do. Then, based on the shooting state information, the flight route drawn by the virtual camera in the VR space is the same as the shooting state of the shooting unit 26 (for example, shooting angle of view, shooting angle, shooting direction, shooting position, etc.). ), and a flight route image is acquired (see, for example, the left diagram in FIG. 8). It should be noted that not only the flight path is drawn in the VR space, but also the three-dimensional model of the imaging target, the three-dimensional model of the flying object 4, and the surrounding environment (for example, buildings, trees, etc.). , Information about a flight route such as a waypoint number may be drawn, and these drawn items may be photographed in the flight route image.

The composite image generation unit 150 may generate a composite image by, for example, acquiring the captured image and the flight route image of the capturing unit 26 and superimposing the flight route image on the captured image.

The flight route information storage unit 162 stores the image capturing point information, the image capturing direction information, and the image capturing date/time information of the flight object generated by the flight mission generating unit 120. The flight log storage unit 164, for example, on the flight route set in the flight mission, information acquired by the flying body 4 (for example, information on the position passed from takeoff to landing, a still image, a moving image, Voice and other information).

As shown in FIG. 6, the shooting information storage unit 166 includes at least a shooting view angle information storage unit 1661, a shooting angle information storage unit 1662, a shooting direction information storage unit 1663, and a user terminal position information storage unit 1664, and a shooting image The shooting state information acquired by the shooting state information receiving unit 130 is stored.

In the present embodiment, the functions of the information processing device on the management server 1 include a captured image/shooting state information reception unit 130, a flight route image generation unit 140, a composite image generation unit 150, and a storage unit 160. However, the present invention is not limited to this, but these configurations (instead of the photographed image/photographing state information receiving unit 130, the photographing unit 26 and the photographing state information acquiring unit 27) are used. It may be provided as a function of its own information processing device, and may obtain flight route information set on the user terminal 2 or flight route information generated by the management server 1 to generate the above-described composite image. As a result, the processing load on the user terminal 2 increases, but the amount of data communication and the frequency of communication with the management server 1 can be reduced.

<Example of flight route display method>
The flight route display method according to the present embodiment will be described with reference to FIGS. FIG. 7 illustrates a flowchart of the flight route display method according to the present embodiment. FIG. 8-13 is an example illustrating generation of a composite image for displaying a flight route according to the embodiment of the present invention.

First, the information processing apparatus acquires the captured image S captured by the image capturing unit 26 of the flying object and the capturing state information acquired by the capturing state information acquiring unit 27 (SQ101).

Next, the information processing device draws the flight route in the VR space based on the flight route information generated in the management server 1 or the user terminal 2, and draws it by the virtual camera in the VR space based on the shooting state information. The flight route is captured to obtain the flight route image H (SQ102).

Next, the information processing apparatus causes the synthetic image generation unit 150 to superimpose the flight route image H on the captured image S to generate a synthetic image G (SQ103), and displays the synthetic image G on the screen of the user terminal 2. (SQ104).

Next, the information processing apparatus determines whether or not the shooting condition has been changed (SQ105), and if the shooting condition has been changed, starts again from acquisition of the shot image and shooting state information (SQ101). If it is determined that the shooting conditions have not been changed, it is determined whether the shooting is finished (SQ106). If not, the captured image and the shooting state information are acquired again (SQ106). Although the process starts from SQ101), the photographing state information may not be acquired at this time.

Note that the operation related to this flow may be performed at any time according to a predetermined sampling rate, for example, a flight route is displayed on a screen captured by the user terminal 2, and functions as a so-called AR. In addition, when an imaging target is present, for example, when the imaging target is imaged, a composite image in which a flight path is superimposed on the acquired still image or moving image may be acquired.

<Composite image G example 1>
As illustrated in FIG. 8, the flight route image H includes, for example, the flight route FR, the waypoint WP, information related thereto (eg, waypoint number), and the virtual flying object VD. Then, as illustrated in FIG. 9, the captured image S and the flight route image H are combined to obtain a combined image G. As a result, the flight route FR and the like within the capturing range where the captured image S is acquired are virtually superimposed on the space of the real world, so that it becomes easy to confirm the suitability of the flight route FR.

Further, by arranging the virtual flying object VD in the VR space by further using the shooting date/time information, it becomes easy to confirm which position the flying object 4 will fly at the time designated by the user. At this time, in order to easily distinguish between the part where the flight has ended (the solid line part of the flight route FR in FIGS. 8 and 9) and the planned flight part (the dotted line part of the flight route FR in FIGS. 8 and 9), The shape and thickness of the line may be changed, or the colors may be divided.

<Composite image G example 2>
As illustrated in FIG. 10, a captured image S including the imaging target T may be acquired. In this case, as illustrated in FIG. 11, the positional relationship between the imaging target T and the flight route FR can be visually recognized, so that the suitability of the flight route FR can be easily confirmed.

<Composite image G example 3>
As illustrated in FIG. 12, a captured image S including the imaging target T and the flying body 4 may be acquired. In this case, as illustrated in FIG. 13, the positional relationship between the flying body 4 and the flight route FR can be visually recognized, so that it is particularly easy to confirm the suitability of the flight route FR after the present point. Further, as illustrated in FIG. 14, when the composite image G is displayed, by providing the display area W for displaying the image acquired from the flying body 4, the suitability of the flight route in consideration of the suitability of imaging. May be made identifiable. Note that in FIG. 14, the composite image G is displayed larger than the display area W, but instead of this, the display area W may be displayed larger than the composite image G, and these can be switched. You may comprise.

<Composite image G example 4>
As illustrated in FIG. 15, a flight route image H including the virtual imaged object VT may be acquired. In this case, as illustrated in FIG. 16, the virtual imaging target VT and the actual imaging target T can be superimposed and compared, so that, for example, the imaging target T has an unexpected configuration O. In this case, it is possible to easily compare the planned flight route FR with the actual configuration of the imaging target T, and thus it becomes easier to confirm the suitability of the flight route FR.

Furthermore, the flight route FR may be edited while displaying the composite image G on the user terminal 2. This makes it easier to set a more appropriate flight route FR while confirming the flight route with the composite image G.

In addition, although the case where the imaging target is a steel tower is illustrated and described in this example, the imaging target is not limited to this and may be any object as long as it can be captured by the camera 42. It may be a condominium, a house, a chimney, an antenna tower, a lighthouse, a windmill, a tree, a statue of a Kannon, a living thing such as a person or an animal, or smoke such as a fire. Further, the object to be imaged is not limited to one that actually exists, and an arbitrary three-dimensional model such as an object to be constructed can be arranged.

Although the user terminal 2 exemplifies a mobile communication terminal such as a smartphone, the user terminal 2 is not limited to this, and may be a wearable device having an AR function (for example, a wearable device such as AR glasses). As a result, the visibility of the flight route is improved, and the suitability of the flight route is more easily confirmed.

In this way, the suitability of the flight route can be confirmed by comparing the actual flight range situation with the flight route of the flying body.

The above-described embodiments are merely examples for facilitating the understanding of the present invention, and are not for limiting the interpretation of the present invention. It goes without saying that the present invention can be modified and improved without departing from the spirit thereof and that the present invention includes equivalents thereof.

1 Management Server 2 User Terminal 4 Aircraft

Claims (10)

A flight route display method for displaying a flight route of an air vehicle on a user terminal,
Acquiring shooting condition information including position information and shooting angle information of the user terminal, shooting angle information, shooting direction information;
Generating an image showing a flight path including the flight path rendered in a virtual space based on flight path information about the flight path;
Including a step of generating a composite image obtained by superimposing the flight route image on the captured image acquired based on the capturing state information,
The image showing the flight path further includes a three-dimensional model of a virtual imaging target,
A flight path display method characterized by the above. The flight route display method according to claim 1, wherein
The image showing the flight path further includes information about waypoints,
A flight path display method characterized by the above. A flight path display method according to claim 1 or 2,
The image showing the flight path further includes a three-dimensional model of a virtual air vehicle,
A flight path display method characterized by the above. The flight route display method according to any one of claims 1 to 3,
The user terminal is a wearable device having an AR function,
A flight path display method characterized by the above. It is the flight route display method according to any one of claims 1 to 4,
Further comprising the step of editing the flight path while displaying the composite image,
A flight path display method characterized by the above. The flight route display method according to any one of claims 1 to 5,
Further comprising the step of displaying an image acquired from the aircraft.
A flight path display method characterized by the above. The flight route display method according to claim 6,
It is possible to switch the image acquired from the flying object so as to be displayed larger than the composite image,
A flight path display method characterized by the above. An information processing device for displaying the flight route of a flying object on a user terminal,
A shooting state information acquisition unit that acquires shooting state information including position information and shooting angle information of the user terminal, shooting angle information, and shooting direction information;
A flight route image generation unit that generates an image showing a flight route including the flight route drawn in the virtual space based on flight route information about the flight route;
A captured image acquired based on the captured state information, a composite image generation unit that generates a composite image obtained by superimposing the flight path image,
The image showing the flight path further includes a three-dimensional model of a virtual imaging target,
An information processing device characterized by the above. A flight route display system for displaying a flight route of an air vehicle on a user terminal,
A shooting state information acquisition unit that acquires shooting state information including position information and shooting angle information of the user terminal, shooting angle information, and shooting direction information;
A flight route image generation unit that generates an image showing a flight route including the flight route drawn in the virtual space based on flight route information about the flight route;
A captured image acquired based on the captured state information, a composite image generation unit that generates a composite image obtained by superimposing the flight path image,
The image showing the flight path further includes a three-dimensional model of a virtual imaging target,
A flight route display system characterized in that A program for causing an information processing device to execute a flight route display method for displaying a flight route of a flying object on a user terminal,
The flight route display method,
Acquiring shooting condition information including position information and shooting angle information of the user terminal, shooting angle information, shooting direction information;
Generating an image showing a flight path including the flight path rendered in a virtual space based on flight path information about the flight path;
Including a step of generating a composite image obtained by superimposing the flight route image on the captured image acquired based on the capturing state information,
The image showing the flight path further includes a three-dimensional model of a virtual imaging target,
A program characterized by that.