WO2022070851A1

WO2022070851A1 - Method, system, and program

Info

Publication number: WO2022070851A1
Application number: PCT/JP2021/033465
Authority: WO
Inventors: 瑛理香田中
Original assignee: 株式会社Clue
Priority date: 2020-09-30
Filing date: 2021-09-13
Publication date: 2022-04-07
Also published as: JPWO2022070851A1

Abstract

[Problem] To make it easy to set the flight path of a flight vehicle on site. [Solution] A method relating to the control of a flight vehicle, said method including: acquiring information about an image captured by an uncrewed flight vehicle 20; displaying the image on a touch panel 12; acquiring input information generated on the basis of an operation on the image displayed on the touch panel 12; and outputting to the uncrewed flight vehicle 20 information about flight control for flying to a position in actual space corresponding to a position on the image, the input information including information relating to the position on the image.

Description

Methods, systems and programs

This disclosure relates to methods, systems and programs.

There is an increasing need for inspections and photography using unmanned aerial vehicles such as drones. Therefore, the development of technology for easily operating an unmanned aircraft is underway. For example, Patent Document 1 discloses a technique for operating an unmanned flying object using a touch panel.

Japanese Unexamined Patent Publication No. 2019-85041

In order to take pictures using a flying object at the site of inspection or construction management, it is required to set an appropriate flight route on the spot.

This disclosure has been made in view of this background, and an object thereof is to provide a method, a system and a program capable of easily setting a flight path of an air vehicle in the field. ..

According to the present disclosure, it is a method relating to control of an air vehicle, which is to acquire information of an image captured by the air vehicle, to display the image on a display unit, and to display the image on the display unit. To acquire input information generated based on an operation on the image, and the input information includes information on a position on the image, and a flight control for flying a position in real space corresponding to the position on the image. Information is output to the flying object, and a method including is provided.

Further, according to the present disclosure, the system is related to the control of the flying object, the image acquisition unit for acquiring the information of the image captured by the flying object, the display control unit for displaying the image on the display unit, and the display unit. An input information acquisition unit that acquires input information including information about a position on the image, which is generated based on an operation on the image displayed on the image, and a position in real space corresponding to the position on the image. A system including an output control unit for outputting flight control information to the flying object is provided.

Further, according to the present disclosure, it is a program for making a computer function as a control device for an air vehicle, and displays an image acquisition unit for acquiring information of an image captured by the air vehicle and the image on a display unit. Corresponds to the display control unit, the input information acquisition unit that acquires input information including information about the position on the image, which is generated based on the operation on the image displayed on the display unit, and the position on the image. A program is provided that functions as an output control unit that outputs flight control information for flying a position in real space to the flying object.

According to the present invention, it is possible to easily set the flight path of the flying object in the field.

It is a figure which shows the outline of the system 1 which concerns on one Embodiment of this disclosure. It is a block diagram which shows the structure of the information processing terminal 10 which concerns on the same embodiment. It is a block diagram which shows an example of the functional structure of the unmanned aircraft 20 which concerns on the same embodiment. It is a block diagram which shows the functional structure of the control part 11 which concerns on the same embodiment. It is a flowchart which concerns on a series of control in the system 1 which concerns on the same embodiment. It is a figure which shows the example of the 1st situation which concerns on the control method by the system 1 which concerns on the same embodiment. It is a figure which shows the 1st screen example displayed on the touch panel 12 which concerns on the same embodiment. It is a figure which shows the 2nd screen example displayed on the touch panel 12 which concerns on the same embodiment. It is a figure which shows the 3rd screen example which is displayed on the touch panel 12 which concerns on the same embodiment. It is a figure which shows the 4th screen example displayed on the touch panel 12 which concerns on the same embodiment. It is a figure which shows the modification of the 4th screen example displayed on the touch panel 12 which concerns on the same embodiment. It is a figure which shows the 5th screen example displayed on the touch panel 12 which concerns on the same embodiment. It is a figure which shows the sixth screen example displayed on the touch panel 12 which concerns on the same embodiment. It is a figure which shows the example of the 2nd situation which concerns on the control method by the system 1 which concerns on the same embodiment. It is a figure which shows the 7th screen example displayed on the touch panel 12 which concerns on the same embodiment. This is an example of a control method by the system 1 according to a modification of the same embodiment.

The preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings below. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

<System overview>
FIG. 1 is a diagram showing an outline of a system 1 according to an embodiment of the present disclosure. As shown in the figure, the system 1 includes an information processing terminal 10 and an unmanned flying object 20. The system 1 according to the present embodiment can be used, for example, for construction management and inspection of the building S1 which is an object of photography by the unmanned flying object 20. In such a system 1, U using the information processing terminal 10 operates the touch panel of the information processing terminal 10 so as to trace the flight path of the unmanned flying object 20. Such an operation may be a continuous or intermittent operation such as a slide or a swipe. The unmanned aircraft 20 can be controlled to fly at a position in real space estimated from the position information on the image obtained by such a tracing operation.

The information processing terminal 10 according to this embodiment is mounted by a so-called tablet-shaped small computer. In another embodiment, the information processing terminal 10 may be realized by a portable information processing terminal such as a smartphone or a game machine, or may be realized by a stationary information processing terminal such as a personal computer. Further, the information processing terminal 10 may be realized by a plurality of hardware and may have a configuration in which the functions are distributed to them.

FIG. 2 is a block diagram showing the configuration of the information processing terminal 10 according to the present embodiment. As shown in the figure, the information processing terminal 10 includes a control unit 11 and a touch panel 12 which is an example of a display unit.

The processor 11a is an arithmetic unit that controls the operation of the control unit 11, controls the transmission and reception of data between each element, and performs processing necessary for program execution. In the present embodiment, the processor 11a is, for example, a CPU (Central Processing Unit), and executes a program stored in the storage 11c described later and expanded in the memory 11b to perform each process.

The memory 11b includes a main storage device composed of a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary storage device composed of a non-volatile storage device such as a flash memory and an HDD (Hard Disc Drive). .. While the memory 11b is used as a work area of the processor 11a, the BIOS (Basic Input / Output System) executed when the control unit 11 is started, various setting information, and the like are stored.

The storage 11c stores programs and information used for various processes. For example, when the user operates the flying object for capturing the image information of the roof 101 via the information processing terminal 10, the storage 11c may store a program for controlling the flight of the flying object. ..

The transmission / reception unit 11d connects the control unit 11 to a network such as an Internet network, and may be provided with a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

The input / output unit 11e is an interface to which input / output devices are connected, and in the present embodiment, the touch panel 12 is connected.

The bus 11f transmits, for example, an address signal, a data signal, and various control signals between the connected processor 11a, memory 11b, storage 11c, transmission / reception unit 11d, and input / output unit 11e.

The touch panel 12 is an example of a display unit, and includes a display surface on which acquired images and images are displayed. In the present embodiment, this display surface receives information input by contact with the display surface, and is implemented by various techniques such as a resistance film method and a capacitance method.

For example, an image captured by the unmanned flying object 20 can be displayed on the display surface of the touch panel 12. Further, on the display surface, buttons, objects, and the like for flight control of the unmanned vehicle 20 and control of the image pickup apparatus may be displayed. Further, the user can input input information to the image, the button, or the like displayed on the display surface via the touch panel 12. The input information will be described in detail later, but the operations related to the input of the input information include, for example, a touch (tap) operation for a button, an object, etc., a tap operation for determining the flight path of the unmanned aircraft 20, and a slide operation. , Swipe operation, etc.

FIG. 3 is a block diagram showing an example of the functional configuration of the unmanned aircraft 20 according to the present embodiment. As shown in FIG. 3, the unmanned vehicle 20 according to the embodiment includes a transmission / reception unit 22, a flight controller 23, a battery 24, an ESC 25, a motor 26, a propeller 27, and a camera 28 in the main body 21. The unmanned flying object 20 is an example of an flying object. The type of the flying object is not particularly limited, and may be, for example, a so-called multi-rotor type drone as shown in FIG.

The flight controller 23 can have one or more processors 23A such as a programmable processor (eg, central processing unit (CPU)).

The flight controller 23 has a memory 23B and can access the memory 23B. Memory 23B stores logic, code, and / or program instructions that the flight controller can execute to perform one or more steps.

The memory 23B may include, for example, a separable medium such as an SD card or a random access memory (RAM) or an external storage device. The data acquired from the sensors 23C may be directly transmitted and stored in the memory 23B. For example, still image / moving image data taken by the camera 28 is recorded in the built-in memory or the external memory.

The flight controller 23 includes a control module configured to control the state of the flying object. For example, the control module may adjust the spatial placement, velocity, and / or acceleration of an air vehicle with 6 degrees of freedom (translation x, y and z, and rotational motion θ _x , θ _y and θ _z ). The propulsion mechanism (motor 26, etc.) of the flying object is controlled via the ESC (Electric Speed Controller) 25. The control module can control one or more of the camera 28, the sensors 23C, and the like.

The flight controller 23 is a transmitter / receiver configured to transmit and / or receive data from one or more external devices (eg, a terminal such as an information processing terminal 10, a display device, or another remote controller). It is possible to communicate with 22. For example, the transmission / reception unit 22 uses one or more of a local area network (LAN), a wide area network (WAN), infrared rays, wireless, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communication, and the like. can do.

The transmission / reception unit 22 is one or more of data acquired by the camera 28 and sensors 23C, processing results generated by the flight controller 23, predetermined control data, user commands from the information processing terminal 10 or a remote controller, and the like. Can be sent and / or received.

Sensors 23C according to this embodiment may include an inertial sensor (acceleration sensor, gyro sensor), GPS sensor, proximity sensor (eg, rider), or vision / image sensor (eg, camera).

The battery 24 can be a known battery such as a lithium polymer battery. The power for driving the unmanned vehicle 20 is not limited to the electric power supplied from the battery 24 or the like, and may be, for example, the power of an internal combustion engine or the like.

The camera 28 is an example of an image pickup device. The type of the camera 28 is not particularly limited, and may be, for example, an ordinary digital camera, an omnidirectional camera, an infrared camera, an image sensor such as a thermography, or the like. The camera 28 may be connected to the main body 21 so as to be independently displaceable by a gimbal or the like (not shown).

FIG. 4 is a block diagram showing a functional configuration of the control unit 11 according to the present embodiment. As shown in FIG. 4, the control unit 11 includes an image acquisition unit 111, a display control unit 112, an input information acquisition unit 113, a flight control information generation unit 114, a camera control information generation unit 115, and an output control unit 116. Each of these functional units can be realized by the processor 11a reading a program stored in the storage 11c into the memory 11b and executing the program.

The image acquisition unit 111 has a function of acquiring information on an image captured by the unmanned flying object 20. The image acquisition unit 111 appropriately acquires an image captured by the camera 28 mounted on the unmanned flying object 20. The image to be acquired may be a moving image obtained in real time, or may be a still image captured at an arbitrary timing. The acquired image information is output to the display control unit 112.

The display control unit 112 has a function of displaying the acquired image on the touch panel 12. Further, the display control unit 112 includes information such as buttons, objects, and texts for providing information to the user who uses the system 1 and for acquiring input information based on the operation by the user in the image. It has a function to display. Further, it may also have a function for displaying the display based on the input information obtained by the input information acquisition unit 113, which will be described later, on the touch panel 12.

The input information acquisition unit 113 has a function of acquiring input information generated based on an operation on an image displayed on the touch panel 12. The input information referred to here includes, for example, information regarding a position on an image displayed on the touch panel 12. The position on the image is, for example, the position of the pixels constituting the image. That is, the input information includes information indicating to which position on the image the user has performed an operation.

The information about the position on the image includes, for example, the information of a line segment consisting of a continuous set of positions on the image. The information of such a line segment is obtained when the user performs an operation such as tracing the touch panel 12. For example, when an image of one space captured by the unmanned vehicle 20 from the sky is displayed on the touch panel 12, the user operates so as to trace the position where the unmanned vehicle 20 wants to fly. The trajectory obtained by the tracing can be the flight route of the unmanned flying object 20. The tracing operation does not necessarily have to be a continuous operation, but may be an intermittent operation. That is, even when the user's tracing operation is performed at a plurality of locations, a flight route can be formed based on such a trajectory. Of these, the route corresponding to the flight route can be complemented as appropriate for the portion where the tracing operation is interrupted.

Further, the input information may include information regarding the imaging target position on the image of the camera 28 loaded on the unmanned flying object 20. The image pickup target position on the image of the camera 28 is a position for determining the image pickup direction of the camera 28 during flight of the unmanned flying object 20. The user can determine at least one of the images of the space displayed on the touch panel 12 as the imaging target position.

Further, the input information may include information regarding the position (altitude) in the height direction of the unmanned aircraft 20 in real space during flight by flight control. The position of the camera 28 in flight can be determined from the information of the position in the height direction, the position in the real space (horizontal direction) of the unmanned flying object 20 during flight, and the image pickup target position of the camera 28. can.

Further, the input information may include information regarding the speed of the unmanned flying object 20 during flight by flight control. In addition, the information input via the touch panel 12 or the like can be appropriately acquired by the input information acquisition unit 113 as input information.

The flight control information generation unit 114 has a function of generating information (flight control information) for controlling the flight of the unmanned aircraft 20 based on the input information. The flight control information includes, for example, the flight path of the unmanned flight object 20, the start and end points of the flight path, the orientation of the unmanned flight object 20 (around the pitch axis, the roll axis, and the yaw axis), the flight altitude of the unmanned flight object 20, and the flight altitude of the unmanned flight object 20. Contains information on the flight speed of the unmanned aircraft 20. The flight control referred to here means that the flight of the unmanned flying object 20 is controlled according to the flight control information generated based on the input information according to the present embodiment input to the touch panel 12.

The flight control information generation unit 114 calculates, for example, a position in real space corresponding to a position on an image included in the input information. The position information in the real space is, for example, latitude information and longitude information. Such latitude information and longitude information can be calculated based on the image pickup position information of the camera 28 (or the unmanned vehicle 20) of the unmanned vehicle 20. Specifically, the flight control information generation unit 114 refers to the angle of view of the camera 28 and the height direction of the unmanned flying object 20 at the time when the camera 28 takes an image while hovering with reference to the pixel at the center of the captured image. Based on the position of, the position in real space corresponding to each pixel of the image can be calculated.

After calculating the flight position (flight path) in the real space, the flight control information generation unit 114 determines the flight speed, direction, etc. of the unmanned aircraft 20 in flight at the flight position, and generates it as flight control information. .. The flight speed during flight may be determined, for example, by input based on an operation from the user. The orientation during flight may be determined, for example, according to the orientation of the camera 28 determined by the camera control information generation unit 115 described later, or if the camera 28 is connected to the main body 21 by a gimbal or the like, the flight may be determined. The inside orientation may be constant. The control of the behavior of the unmanned vehicle 20 during such flight can be realized by a known technique.

The camera control information generation unit 115 has a function of generating information (camera control information) for controlling the operation of the camera 28 based on the input information. The camera control information includes, for example, information regarding the orientation of the camera 28, the imaging timing, the imaging process, and the like.

For example, during the flight of the unmanned flying object 20 flying based on the flight control information, the camera control information generation unit 115 determines the direction (imaging direction) of the camera 28 based on the position in the real space corresponding to the imaging target position. Generate information to control. The information for controlling the orientation of the camera 28 is, for example, the information of the camera 28 at the flight position calculated based on the image pickup target position acquired by the input information acquisition unit 113 and the position of the unmanned vehicle 20 in flight. Contains information about orientation. At least the horizontal orientation of the camera 28 can be calculated by the relationship between the imaging target position and the flight position. When there are a plurality of imaging target positions, for example, an imaging target position corresponding to a flight position at a certain timing may be separately determined. Further, as the imaging target position, for example, only the position in the horizontal direction may be specified. In this case, the angle around the pitch axis of the camera 28 can be adjusted as appropriate as described later. Further, the orientation of the camera 28 can be appropriately controlled during flight other than flight control, for example, in response to an input to the touch panel 12 by the user. Further, the imaging target position may include a position in the height direction in the real space. If such a position in the height direction is set, the angle around the pitch axis of the camera 28 can be automatically determined.

The output control unit 116 has a function of outputting various information generated by the flight control information generation unit 114 and the camera control information generation unit 115 to the unmanned vehicle 20. The unmanned flying object 20 controls the flight of the unmanned flying object 20 and the operation of the camera 28 based on the acquired flight control information and the camera control information.

Next, an example of the control method of the unmanned aircraft 20 using the system 1 according to the present embodiment will be described with reference to the flowchart. FIG. 5 is a flowchart of a series of controls in the system 1 according to the present embodiment.

First, the image acquisition unit 111 acquires image information from the unmanned flying object 20 (step SQ101). At this time, the unmanned flying object 20 is hovering over the object to be imaged.

FIG. 6 is a diagram showing an example of the first situation relating to the control method by the system 1 according to the present embodiment. As shown in FIG. 6, the unmanned aircraft 20 is hovering over the building S1 which is the object to be imaged at the height of the altitude H1. The image pickup direction of the camera 28 is directly downward. Although the altitude H1 is the altitude of the camera 28 here, the altitude H1 may be the altitude of the unmanned flying object 20.

FIG. 7 is a diagram showing a first screen example displayed on the touch panel 12 according to the present embodiment. As shown in FIG. 7, the information bar D11, the main screen D12, the sub screen D13, the plurality of buttons, and the objects 101 to 109 are displayed on the screen V1 of the touch panel 12. The information bar D11 is an area mainly for presenting information about the touch panel 12 and the unmanned flying object 20. In the example shown in FIG. 7, the information bar D11 displays the radio wave condition of the touch panel 12, the remaining battery level, and the current altitude of the unmanned vehicle 20. The captured image or the like obtained by the camera 28 may be displayed on the main screen D12. On the main screen D12, other information may be displayed superimposed on the captured image. The sub screen D13 is an area for displaying the captured image captured by the camera 28. In the present embodiment, the sub-screen D13 may be used, for example, to supplementally display a captured image used for determining a flight path. For example, by tapping the sub screen D13, the flight route setting can be started. Specific examples will be described later.

Hereinafter, the buttons 101 to 107 and the

objects

108 and 109 will be described. The button 101 is a button used for landing the unmanned aircraft 20. The button 102 is, for example, a button for moving the unmanned aircraft 20 to the starting point of the set flight path. The button 103 is a button for starting control of flight along the flight path of the unmanned flying object 20. The button 104 is a button for performing an image pickup process by the camera 28. The button 105 is a button for adjusting the altitude of the unmanned aircraft 20. The button 106 is a button for adjusting the horizontal position of the unmanned aircraft 20. The button 107 is a button for displaying or editing the set route. The object 108 is an object for adjusting the horizontal direction (around the yaw axis) of the unmanned aircraft 20. The object 109 is an object for adjusting the angle around the pitch axis of the camera 28. The captured image displayed on the main screen D12 includes images of the building S1, the plaza S2, and the road S3, which are the imaging target portions.

Next, when the user taps the sub screen D13, the process related to the input of the flight path is started (step SQ103). FIG. 8 is a diagram showing a second screen example displayed on the touch panel 12 according to the present embodiment. The screen shown in FIG. 8 is a screen for inputting the flight path of the unmanned flying object 20. The captured image and the buttons 201 to 204 are displayed on the screen V1. Button 201 is a button for setting a flight path. The button 202 is a button for setting the center point (position to be imaged) of the captured image. The button 203 is a button for setting the flight speed of the unmanned flying object 20. Button 204 is a button for saving the settings.

When the button 201 is selected on the screen V1 shown in FIG. 8, points and lines corresponding to the flight path can be input to the screen V1. For example, the user operates the screen V1 with a finger, a stylus pen, or the like to specify a position on the image corresponding to the flight path. Such an operation may be an intuitive slide operation such as drawing with a pen.

FIG. 9 is a diagram showing a third screen example displayed on the touch panel 12 according to the present embodiment. As shown in FIG. 9, a locus 210 can be drawn on the screen V1 by a user operation. In addition, the point 211 indicating the start point and the end point of the locus 210 may be displayed by another aspect. The point 211 indicating the start point and the end point can be, for example, a start point and a goal point in the flight control of the unmanned aircraft 20. The start point and the end point of the locus (line segment) may or may not match. Further, when the start point and the end point are different positions when the locus 210 is drawn, one of the start point and the end point may be adjusted to be the same position as the other. By matching the start point and the end point, the images (frames) taken during the flight of the unmanned vehicle 20 match at the start time and the end time. This makes it possible to obtain a more useful image. Further, either the start point or the end point may be provided so that its location can be changed on the locus 210. Further, the locus 210 once drawn may be appropriately modified.

Next, when the user taps the button 202, the process related to the input of the imaging target position is started (step SQ105). FIG. 10 is a diagram showing a fourth screen example displayed on the touch panel 12 according to the present embodiment. As shown in FIG. 10, when the user taps the button 202, the imaging target position (also referred to as a center point) 212 on the image can be selected. The image pickup target position 212 on the image can be adjusted by, for example, a tap operation or a slide operation. The imaging target position 212 on the image determined here corresponds to at least a horizontal position in real space. That is, during flight control, the camera 28 of the unmanned flying object 20 continues to face the direction in which the image pickup target position in the real space corresponding to the image pickup target position 212 on the image exists in the horizontal direction.

In the example shown in FIG. 10, the imaging target position 212 was set to one point, but this technique is not limited to such an example. FIG. 11 is a diagram showing a modified example of the fourth screen example displayed on the touch panel 12 according to the present embodiment. As shown in FIG. 11, as in the image pickup target position 213, the image pickup target position may be selected not by a point but by a line segment or the like consisting of a plurality of points. Further, the start point and the end point of the imaging target position 213 may be set as the point 214, similarly to the point 211 indicating the start point and the end point of the locus 210. This makes it possible to dynamically change the direction of the camera 28 of the unmanned flying object 20 under flight control toward the image pickup target.

Next, when the user taps the button 203, the process related to the input of the flight speed during flight by the flight control of the unmanned aircraft 20 is started (step SQ107). FIG. 12 is a diagram showing a fifth screen example displayed on the touch panel 12 according to the present embodiment. As shown in FIG. 12, when the user taps the button 203, the flight speed at which the unmanned vehicle 20 flies according to the flight path corresponding to the trajectory 210 can be selected. In the example shown in FIG. 12, the flight speed can be selected by, for example, selecting one of the low speed button 215, the medium speed button 216, and the high speed button 217. The method of setting the flight speed is not limited to this example, and for example, the flight speed can be specified numerically.

Tap button 204 to complete the setting of the flight path and the position to be imaged by the camera.

Next, when returning to the screen V1 as shown in FIG. 7, the process related to the input of the flight altitude during flight by the flight control of the unmanned aircraft 20 is started by the user's selection (step SQ109). FIG. 13 is a diagram showing a sixth screen example displayed on the touch panel 12 according to the present embodiment. The screen V1 shown in FIG. 13 displays a form 120 in which the user taps the button 105 on the screen V1 shown in FIG. 7 to set the altitude during flight by controlling the flight of the unmanned vehicle 20. By inputting the altitude for the form 120, the altitude during flight by flight control is determined. The altitude during flight may be fixed or variable during flight. When the altitude is variable, the altitude may be appropriately set by a form other than the form 120. The altitude during flight may be predetermined.

The information obtained by the user's operation on the touch panel 12 in steps SQ103 to SQ109 can be output to the flight control information generation unit 114 and the camera control information generation unit 115 as appropriate input information.

Next, the flight control information generation unit 114 generates flight control information for flying a position in the real space corresponding to the trajectory 210 based on the obtained input information (step SQ111). Further, the camera control information generation unit 115 controls the camera so that the camera 28 continuously faces the image pickup target position in the real space corresponding to the image pickup target position 212 on the image based on the obtained input information. Generate information (step SQ113). This information is output to the unmanned aircraft 20 by the output control unit 116.

Next, when the user taps the button 102, the unmanned aircraft 20 starts moving to the start point of the flight path based on the flight control information (steps SQ115, SQ117). FIG. 14 is a diagram showing an example of a second situation relating to the control method by the system 1 according to the present embodiment. Here, the unmanned aircraft 20 makes a round flight so as to make a turn over the building S1 at a height H2 according to the flight control information. At that time, the unmanned flying object 20 can control the attitude during flight so that the image pickup target position C1 is in the image pickup direction of the camera 28 according to the camera control information.

Next, when the user taps the button 103, the unmanned aircraft 20 starts flying based on the flight control information (step SQ119). At that time, the camera 28 acquires the captured image during flight while facing the direction of the imaging target position C1 (step SQ121). FIG. 15 is a diagram showing a seventh screen example displayed on the touch panel 12 according to the present embodiment. On the main screen D12 of the screen V1 shown in FIG. 15, an image captured by the camera 28 of the unmanned vehicle 20 in flight based on flight control is appropriately displayed. The captured image may be a moving image or a still image captured at a predetermined interval. Further, the image captured during hovering may be displayed on the sub screen D13. Further, on the sub screen D13, the locus 130 corresponding to the flight path previously set by the user's input may be superimposed and displayed. Further, the object 131 corresponding to the unmanned flying object 20 may be displayed on the sub screen D13 at the position corresponding to the current position of the unmanned flying object 20 flying in the flight path. This makes it possible to confirm in real time (or when viewing the image after flight) from which point on the flight path the image displayed on the main screen D12 was captured.

The example of the control method of the unmanned aircraft 20 by the system 1 according to the present embodiment has been described above. In the above embodiment, an example of a control method in the actual flight of the unmanned vehicle 20 has been described, but the present technology is not limited to such an example. For example, this technology can be used for simulation in the case of flying a virtual drone in VR (Virtual Reality) or AR (Augmented Reality) space. By simulating the flight behavior by controlling the flight of the unmanned vehicle 20 by VR or AR, it is possible to confirm in advance the image pickup direction of the camera 28 when the unmanned vehicle 20 is actually flown. As the simulation technique using the VR space or the AR space, a known technique can be used.

As described above, according to the system 1 according to the present embodiment, it is possible to intuitively input the position information corresponding to the flight path to the touch panel 12 to the user interface. Further, the unmanned flight object 20 can be made to fly a desired flight path only by inputting the position information on the screen of the touch panel 12. This makes it possible to easily set the flight path of the flying object at the site. Further, by setting the image pickup target position for determining the orientation of the camera with respect to the touch panel 12, the orientation of the camera during flight can be easily determined. This facilitates imaging using an unmanned flying object of an imaging object such as a building. The flight control information and the camera control information once set may be stored in the storage 11c or the like. As a result, it is possible to repeatedly fly an unmanned vehicle under the same conditions in inspections and the like, and to image an image to be imaged at the same position and angle.

In the above embodiment, the image captured by the unmanned flying object 20 during hovering is an image obtained by imaging the image pickup direction of the camera 28 loaded on the unmanned flying object 20 as the ground direction. Not limited to examples. FIG. 16 is an example of a control method by the system 1 according to a modification of the present embodiment. As shown in FIG. 16, in such a system 1, the image pickup direction of the camera 28 when the unmanned aircraft 20 is hovering may be horizontal with respect to the structure S4. At this time, the unmanned flying object 20 may be controlled so that the camera 28 takes an image of the structure S4 while the unmanned flying object 20 is flying on a plane parallel to the height direction with respect to the height H3. In this case, the imaging direction of the camera 28 may be determined, for example, with the position of the height H3 in the structure S4 as the imaging target position. As described above, the system 1 according to the present embodiment can be applied regardless of the positional relationship between the unmanned flying object 20 and the structure that is the imaging target.

Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that anyone with ordinary knowledge in the art of the present disclosure may come up with various modifications or amendments within the scope of the technical ideas set forth in the claims. Is, of course, understood to belong to the technical scope of the present disclosure.

The device described in the present specification may be realized as a single device, or may be realized by a plurality of devices, etc., which are partially or wholly connected by a network. For example, the control unit and the storage of the information processing terminal 10 may be realized by different servers connected to each other by a network.

The series of processes by the apparatus described in the present specification may be realized by using any of software, hardware, and a combination of software and hardware. It is possible to create a computer program for realizing each function of the information processing terminal 10 according to the present embodiment and implement it on a PC or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed, for example, via a network without using a recording medium.

Further, the processes described in the present specification using the flowchart diagram do not necessarily have to be executed in the order shown in the figure. Some processing steps may be performed in parallel. Further, additional processing steps may be adopted, and some processing steps may be omitted.

Further, the effects described in the present specification are merely explanatory or exemplary and are not limited. That is, the technique according to the present disclosure may exert other effects apparent to those skilled in the art from the description of the present specification, in addition to or in place of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(Item 1)
It ’s a method of controlling an aircraft.
Acquiring the information of the image captured by the aircraft and
Displaying the image on the display unit and
Acquiring input information generated based on an operation on the image displayed on the display unit, and the input information includes information on a position on the image.
Outputting flight control information for flying a position in real space corresponding to the position on the image to the flying object, and
How to include.
(Item 2)
The method described in item 1.
The input information includes information of a line segment consisting of a continuous set of positions on the image.
A method of outputting flight control information for flying a route in real space corresponding to a line segment on an image to the flying object.
(Item 3)
The method described in item 2.
The information of the line segment includes the information of the start point and the end point of the line segment.
A method in which, when the position of the end point acquired by an operation on the image is different from the position of the start point, the position of the end point is adjusted so that the position of the end point coincides with the position of the start point.
(Item 4)
The method according to any one of items 1 to 3.
The input information includes information about an image pickup target position on the image of the image pickup apparatus loaded on the flight object.
During the flight of the flying object that flies based on the flight control information, the flight is provided with information for controlling the orientation of the image pickup device based on the image pickup target position in the real space corresponding to the image pickup target position. A method that further includes outputting to the body.
(Item 5)
The method described in item 4.
A method in which the imaging target position is designated by a plurality or consecutively on the image.
(Item 6)
The method according to item 4 or 5.
The method, wherein the imaging target position includes a position in the height direction in real space.
(Item 7)
The method according to any one of items 1 to 6.
Obtaining information about the position of the flying object in the height direction at the time when the image was taken,
A method, wherein the position in real space corresponding to the position on the image is determined based on the position in the height direction of the flying object at the time when the image is taken.
(Item 8)
The method according to any one of items 1 to 7.
The input information includes information about the height position of the flying object in real space during flight by the flight control.
(Item 9)
The method according to any one of items 1 to 8.
A method further comprising displaying an image captured by an image pickup apparatus loaded on the flying object flying based on the flight control on the display unit.
(Item 10)
The method according to any one of items 1 to 9.
A method further comprising displaying an image previously displayed on the display unit in another area of the display unit during flight of the flying object flying under the flight control.
(Item 11)
The method according to item 10.
Further, it is possible to display the object corresponding to the flying object at the position on the image in the other area corresponding to the flight position in the real space of the flying object flying under the flight control in the other area. Including, method.
(Item 12)
The method according to any one of items 1 to 11.
A method further comprising displaying the result of a simulation of flight behavior by the flight control of the flying object on the display unit.
(Item 13)
The method according to any one of items 1 to 12.
The flight control information includes information relating to the speed at which the flight object is flown.
A method, wherein the operation on the image displayed on the display unit includes an operation for determining the speed.
(Item 14)
The method according to any one of items 1 to 13.
The image captured by the flying object during hovering is an image obtained by imaging the image pickup direction of the image pickup apparatus loaded on the flying object as the ground direction.
(Item 15)
It is a system related to the control of the aircraft.
An image acquisition unit that acquires information on the image captured by the aircraft,
A display control unit that displays the image on the display unit,
An input information acquisition unit that acquires input information including information about a position on the image, which is generated based on an operation on the image displayed on the display unit.
An output control unit that outputs flight control information for flying a position in real space corresponding to the position on the image to the flying object.
A system equipped with.
(Item 16)
A program for making a computer function as a control device for an air vehicle.
An image acquisition unit that acquires information on the image captured by the aircraft,
A display control unit that displays the image on the display unit,
An input information acquisition unit that acquires input information including information about a position on the image, which is generated based on an operation on the image displayed on the display unit.
An output control unit that outputs flight control information for flying a position in real space corresponding to the position on the image to the flying object.
A program that functions as.

1 System 10 Information processing terminal 11 Control unit 12 Touch panel 20 Unmanned flying object 28 Camera 111 Image acquisition unit 112 Display control unit 113 Input information acquisition unit 114 Flight control information generation unit 115 Camera control information generation unit

Claims

It ’s a method of controlling an aircraft.
Acquiring the information of the image captured by the aircraft and
Displaying the image on the display unit and
Acquiring input information generated based on an operation on the image displayed on the display unit, and the input information includes information on a position on the image.
Outputting flight control information for flying a position in real space corresponding to the position on the image to the flying object, and
How to include.
The method according to claim 1.
The input information includes information of a line segment consisting of a continuous set of positions on the image.
A method of outputting flight control information for flying a route in real space corresponding to a line segment on an image to the flying object.
The method according to claim 2.
The information of the line segment includes the information of the start point and the end point of the line segment.
A method in which, when the position of the end point acquired by an operation on the image is different from the position of the start point, the position of the end point is adjusted so that the position of the end point coincides with the position of the start point.
The method according to any one of claims 1 to 3.
The input information includes information about an image pickup target position on the image of the image pickup apparatus loaded on the flight object.
During the flight of the flying object that flies based on the flight control information, the flight is provided with information for controlling the orientation of the image pickup device based on the image pickup target position in the real space corresponding to the image pickup target position. A method that further includes outputting to the body.
The method according to claim 4.
A method in which the imaging target position is designated by a plurality or consecutively on the image.
The method according to claim 4 or 5.
The method, wherein the imaging target position includes a position in the height direction in real space.
The method according to any one of claims 1 to 6.
Obtaining information about the position of the flying object in the height direction at the time when the image was taken,
A method, wherein the position in real space corresponding to the position on the image is determined based on the position in the height direction of the flying object at the time when the image is taken.
The method according to any one of claims 1 to 7.
The input information includes information about the height position of the flying object in real space during flight by the flight control.
The method according to any one of claims 1 to 8.
A method further comprising displaying an image captured by an image pickup apparatus loaded on the flying object flying based on the flight control on the display unit.
The method according to any one of claims 1 to 9.
A method further comprising displaying an image previously displayed on the display unit in another area of the display unit during flight of the flying object flying under the flight control.
The method according to claim 10.
Further, it is possible to display the object corresponding to the flying object at the position on the image in the other area corresponding to the flight position in the real space of the flying object flying under the flight control in the other area. Including, method.
The method according to any one of claims 1 to 11.
A method further comprising displaying the result of a simulation of flight behavior by the flight control of the flying object on the display unit.
The method according to any one of claims 1 to 12.
The flight control information includes information relating to the speed at which the flight object is flown.
A method, wherein the operation on the image displayed on the display unit includes an operation for determining the speed.
The method according to any one of claims 1 to 13.
The image captured by the flying object during hovering is an image obtained by imaging the image pickup direction of the image pickup apparatus loaded on the flying object as the ground direction.
It is a system related to the control of the aircraft.
An image acquisition unit that acquires information on images captured by the aircraft,
A display control unit that displays the image on the display unit,
An input information acquisition unit that acquires input information including information about a position on the image, which is generated based on an operation on the image displayed on the display unit.
An output control unit that outputs flight control information for flying a position in real space corresponding to the position on the image to the flying object.
A system equipped with.
A program for making a computer function as a control device for an air vehicle.
An image acquisition unit that acquires information on images captured by the aircraft,
A display control unit that displays the image on the display unit,
An input information acquisition unit that acquires input information including information about a position on the image, which is generated based on an operation on the image displayed on the display unit.
An output control unit that outputs flight control information for flying a position in real space corresponding to the position on the image to the flying object.
A program that functions as.