JP6582269B1 - Information display method for control of flying object - Google Patents

Abstract

An object of the present invention is to optimize the display of information including a plurality of images transmitted from a plurality of flying objects. An information display method for displaying information including images provided from a plurality of flying objects, wherein the image display area receives images transmitted from the plurality of flying objects and displays the received images. Generating an image display area corresponding to each of the plurality of flying objects, and displaying an image corresponding to the flying object selected by the user from among the image display areas corresponding to each of the plurality of flying objects. Control to display in the first image display area. [Selection] Figure 1

Description

  The present invention relates to an information display method related to control of an aircraft.

  In recent years, various services using a flying body such as a drone and an unmanned aerial vehicle (UAV) used for various purposes (hereinafter simply referred to as “aircraft”) have been provided. .

  Aircraft data transmitted from the flying object is received by the ground station located remotely when the flying object is flying, and the aircraft information, map information, and aircraft operation panel obtained from the aircraft data on the monitor screen of a personal computer installed on the ground station For example, Patent Document 1 discloses a technique for displaying an airframe-mounted camera operation panel and the like.

  Further, for example, Patent Document 2 discloses a technique for displaying flight data of each unmanned air vehicle by switching or dividing on one screen when operating a plurality of air vehicles.

WO2006-098469 JP-A-6-312700

  However, the technique disclosed in the above-mentioned patent document focuses on the operation of a single or a plurality of flying objects, and aims to optimize the monitor screen display of necessary information in a remotely provided device. A technique for displaying a plurality of pieces of image information transmitted from a plurality of flying objects and optimizing the image display has not yet been disclosed.

  Therefore, an object of the present invention is to optimize display of information including a plurality of images transmitted from a plurality of flying objects.

  One embodiment of the present invention is an information display method for displaying information including images provided from a plurality of flying objects, receives images transmitted from the plurality of flying objects, and displays the received images. An image display area corresponding to each of the plurality of flying objects is generated, and an image corresponding to the flying object selected by the user corresponds to each of the plurality of flying objects. Control is performed so that the image is displayed in the first image display area of the image display area.

  According to the present invention, it is possible to realize optimization of display of information including a plurality of images transmitted from a plurality of flying objects.

It is a system block diagram which consists of a management terminal and a flying body by 1st Embodiment by this invention. It is a system configuration figure consisting of a management terminal and a user terminal by a 1st embodiment by the present invention. It is a figure which shows the hardware constitutions of the management terminal by the 1st Embodiment by this invention. 1 is a functional block diagram of a flying object according to a first embodiment of the present invention. It is a functional block diagram of a control part and a storage part among management terminals by a 1st embodiment by the present invention. It is a flowchart figure concerning information display by a 1st embodiment by the present invention. It is an example of the user interface displayed on the user terminal by 1st Embodiment by this invention. It is another example of the user interface displayed on the user terminal according to the first embodiment of the present invention. It is an example of the user interface displayed on the user terminal by the 2nd Embodiment by this invention. It is a functional block diagram of a control part and a storage part among management terminals by a 3rd embodiment by the present invention. It is a flowchart figure concerning information display by a 3rd embodiment by the present invention. It is an example of the screen display of the system by embodiment of this invention.

The contents of the embodiment of the present invention will be listed and described. An aircraft according to an embodiment of the present invention has the following configuration.
[Item 1]
An information display method for displaying information including images provided from a plurality of flying objects,
Receiving images transmitted from the plurality of aircraft,
An image display area for displaying the received image, and generating an image display area corresponding to each of the plurality of flying objects;
An information display method for controlling to display an image corresponding to a flying object selected by a user in a first image display area among image display areas corresponding to each of the plurality of flying objects.
[Item 2]
The information display method according to claim 1,
The information display method according to claim 1, wherein the first image display area is larger than an image display area corresponding to each of the plurality of other flying objects.
[Item 3]
The information display method according to claim 1,
An information display method for generating a second image region for displaying predetermined information related to the predetermined aircraft.
[Item 4]
The information display method according to claim 3,
The information display method, wherein the predetermined information includes any one of a flight path, position information, flight position information, flight altitude, flight speed, and battery remaining amount of a corresponding aircraft.
[Item 5]
The information display method according to claim 1,
An information display method for providing a control panel area for controlling the predetermined aircraft.
[Item 6]
The information display method according to claim 1,
An information display method for recording an image transmitted from the predetermined aircraft.
[Item 7]
The information display method according to claim 1,
An information display method for transmitting the displayed image to a plurality of users.
[Item 8]
The information display method according to claim 7,
Receiving message input from each of the plurality of users;
An information display method for generating a message display area for displaying the received message.

(First embodiment)
Hereinafter, an aircraft control method, an aircraft, and a flight program according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a system configuration diagram including a management terminal and a flying object according to the first embodiment of the present invention.

  As shown in FIG. 1, the management terminal 1 is connected to a plurality of flying bodies 2A, 2B, 2G, etc. via a network such as the Internet (hereinafter, for convenience, an arbitrary flying body is simply referred to as a flying body. 2 ”). Examples of networks include local area networks (LAN), wide area networks (WAN), infrared, wireless, WiFi, point-to-point (P2P) networks, telecommunications networks, cloud communications, and the like. Here, examples of the flying object 2 include an unmanned flying object (for example, a drone) that autonomously flies or can be operated and controlled remotely by a user terminal described later. In addition to predetermined information from the flying object 2 to the management terminal 1, for example, still image / moving image data taken by the camera of the flying object 2, predetermined control data such as flight position information (latitude, longitude, altitude Etc.), information such as flight path, battery usage / remaining amount, flight speed, flight time, acceleration, tilt, and other equipment operation status are transmitted automatically or in response to a request from the user.

FIG. 2 is a system configuration diagram including a management terminal and a user terminal according to the first embodiment of the present invention. As shown in FIG. 2, the management terminal 1 is connected to a plurality of user terminals 3A, 3B, 3G, etc. via a network such as the Internet (hereinafter, for convenience, an arbitrary user terminal is simply referred to as “user terminal”). 3 ”). Examples of networks include local area networks (LAN), wide area networks (WAN), infrared, wireless, WiFi, point-to-point (P2P) networks, telecommunications networks, cloud communications, and the like. Here, as the user terminal 3, for example, a general-purpose computer such as a workstation or a personal computer may be used, or a mobile device terminal such as a smartphone or a tablet may be used. The user terminal 3 includes a display device such as a display panel, and the display device displays predetermined information including image and moving image data transmitted from the management terminal 1 by display software such as a built-in browser.

  FIG. 3 is a diagram illustrating a hardware configuration of the management server 1. In addition, the structure shown in figure is an example and may have a structure other than this.

  As shown in the figure, the management server 1 is connected to a database (not shown) and forms part of the system. The management server 1 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

  The management server 1 includes at least a control unit 10, a memory 11, a storage 12, a transmission / reception unit 13, an input / output unit 14, and the like, which are electrically connected to each other through a bus 15.

  The control unit 10 is an arithmetic device that controls the operation of the entire management server 1 and performs data transmission / reception control between elements, information processing necessary for application execution and authentication processing, and the like. For example, the processor 10 is a CPU (Central Processing Unit), and executes each information process by executing a program stored in the storage 12 and expanded in the memory 11.

  The memory 11 includes a main memory composed of a volatile storage device such as a DRAM (Dynamic Random Access Memory) and an auxiliary memory composed of a non-volatile storage device such as a flash memory or an HDD (Hard Disc Drive). . The memory 11 is used as a work area of the processor 10 and stores a BIOS (Basic Input / Output System) executed when the management server 1 is started up, various setting information, and the like.

  The storage 12 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 12.

  The transmission / reception unit 13 connects the management server 1 to a network and a block chain network. The transmission / reception unit 13 may include a Bluetooth (registered trademark) and a BLE (Bluetooth Low Energy) short-range communication interface.

  The input / output unit 14 is an information input device such as a keyboard / mouse and an output device such as a display.

  The bus 15 is commonly connected to the above-described elements, and transmits, for example, an address signal, a data signal, and various control signals.

FIG. 4 is a functional block diagram of the flying object according to the first embodiment of the present invention. In addition, the following functional block diagram is described as a concept stored in a single device (aircraft) for the sake of simplicity of explanation, but for example, a part of its functions can be exhibited by an external device, It may be logically configured by using cloud computing technology.

  The flight controller 21 can include one or more processors such as a programmable processor (eg, a central processing unit (CPU)).

  The flight controller 21 has a memory 22 and can access the memory. The memory 22 stores logic, code, and / or program instructions that can be executed by the flight controller 21 to perform one or more steps.

  The memory 22 may include a separable medium such as an SD card or a random access memory (RAM) or an external storage device. Data acquired from the camera and sensors 27 may be directly transmitted to and stored in the memory. For example, still image / moving image data shot by a camera or the like is recorded in a built-in memory or an external memory. The camera is installed on the flying object via the gimbal 28.

The flight controller 21 includes a control module configured to control the state of the aircraft. For example, the control module may adjust the spatial arrangement, velocity, and / or acceleration of an aircraft that has six degrees of freedom (translational motion x, y, and z, and rotational motion θ _x , θ _y, and θ _z ). The aircraft propulsion mechanism (motor 30 and the like) is controlled via the ESC 29. The propeller 31 is rotated by the motor to generate lift of the flying object. The control module can control one or more of the states of the mounting unit and sensors.

  The flight controller 21 is configured to send and / or receive data from one or more external devices (eg, a transceiver 25 (propo), terminal, display device, or other remote controller). The communication with the unit 26 is possible. The transceiver 25 can use any appropriate communication means such as wired communication or wireless communication.

  For example, the transmission / reception unit 26 uses one or more of a local area network (LAN), a wide area network (WAN), infrared, wireless, WiFi, point-to-point (P2P) network, telecommunication network, cloud communication, etc. can do.

  The transmission / reception unit 26 transmits and / or receives one or more of data acquired by the sensors 27, a processing result generated by the flight controller 21, predetermined control data, a user command from a terminal or a remote controller, and the like. be able to.

  The sensors 27 according to the present embodiment may include an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, a proximity sensor (eg, a rider), or a vision / image sensor (eg, a camera).

FIG. 5 is a functional block diagram of a control unit and a storage unit in the management terminal according to the first embodiment of the present invention. FIG. 6 is a flowchart according to the information display process executed in the management terminal according to the first embodiment of the present invention. This information processing can be realized by executing a program for displaying information stored in the storage unit in the control unit.

As shown in FIG. 6, first, the information receiving unit 41 of the control unit 10 receives related information such as image / moving image data and / or predetermined control data from the flying object 2 (S101). The control unit 10 stores the information in the image storage unit 51 or the information storage unit 52 of the storage unit 11.

  Subsequently, the display control unit 44 of the control unit 10 receives the image received from the flying object 2 for the preset layout of the display area and the common elements arranged in accordance with the UI template stored in the template storage unit 53. A user interface (UI) screen file to be displayed on the display device of the user terminal 3 is generated by arranging information such as moving image data and text (S102).

  FIG. 7 shows an example of a user interface displayed on the user terminal according to the first embodiment of the present invention. The user interface screen 60 has a plurality of display areas 61 and 62A-62E for displaying image / moving image data transmitted from each aircraft 2. Each display area is provided for displaying the image / moving image data of the corresponding flying object. For example, the display area 61 displays the image / moving picture data transmitted from the drone 1, and the display area 62 </ b> A displays the image / moving picture data transmitted from the drone 2. Similarly, the display area 62 </ b> B displays the image / moving image data transmitted from the drone 3. Here, the display area 61 is an area for displaying the flying object selected by the user. The display area 61 has a display area larger than that of the other display areas 62A to 62E. For example, in FIG. 7, the display area 61 displays image / moving image data received from the drone 1 selected by the user via the flying object selection unit 64. Thereby, the user can confirm the image / moving image data received from the selected flying object more easily. The flying object selector 64 can select another drone 2-6 by pull-down. The flying object operation panel 63 is arranged so as to be overlaid on the display area 61, and is prepared for the flying object flight control (for example, control of the flying direction of the flying object) and the flying object according to a request from the user. Camera control (for example, switching between still image / moving image shooting, zoom-in / out control, camera shooting direction control, etc.). The flight control of the flying object is realized by executing the flying object control program stored in the flying object control program storage unit 57 of the storage unit 11 in the flying object control unit program execution unit 48 of the control unit 10.

  Further, the user interface screen 60 displays the position information and flight path transmitted from the GPS of the currently selected aircraft on a map, a map display area 65, and various control data (e.g., latitude) transmitted from the aircraft. , Longitude, altitude, direction, speed, remaining battery level, etc.). Further, the user interface screen 60 can have a recording button 67 for recording moving image data transmitted from the selected flying object and displayed in the display area 61. Moreover, it can have the share button 68 for starting the chat program for sharing the user interface screen 60 to another user terminal. Each user can check a comment received from another user while inputting a comment through the chat interface while referring to the shared user interface screen 60. The chat program is realized by executing the chat program stored in the chat program storage unit 55 of the storage unit 11 by the chat program execution unit 47 of the control unit 10. In addition, each user can control the aircraft operation panel 63 displayed in the display area 61 on the shared user interface screen. At this time, the original user who shared the user interface screen (host user) provides a selection button (not shown) for designating the authority for operating the operation panel 63 to a specific other user. It can also be provided on the screen 60.

Returning to FIG. 6, the determination unit 43 of the control unit 10 determines whether or not the management terminal 1 has received a request from the user terminal 3 to select a desired flying body among a plurality of flying bodies (S103). . As described above, for example, the user terminal 3 selects a desired flying object by a pull-down operation of the flying object selection unit 64 on the user interface screen 60. When the determination result is YES, the display control unit 45 displays the image / moving image data transmitted from the flying object selected by the user terminal 3 in the display area 61 (S104). For example, as illustrated in FIG. 8, when the user terminal 3 makes a request to select the drone 2, the image / moving image data transmitted from the drone 2 is displayed in the display area 61. The drone 1 selected so far is displayed in the display area 62A where the image / moving image data of the drone 2 has been displayed. By providing a user interface screen as in this embodiment, it is possible to view the image / movie data of a plurality of flying objects from a bird's-eye view and to display a large image / movie data of the flying object to be focused. It is possible to provide a user with a display that is easier to see. In addition, by providing an operation panel that enables aircraft operation and camera operation of the flying object to a display area that displays image / movie data of the selected flying object, a user interface with higher operability is provided. be able to.

As can be understood by comparing FIGS. 7 and 8, in the map display area 65, the drone 1 relating to the image currently displayed in the display area 61 is displayed in the center (d1), and other drones (for example, Drone 2: d2) may be displayed on the same screen. In this case, an icon that can indicate the traveling direction of each drone (for example, a figure that is asymmetric with respect to the traveling direction) and a flight history (flight path) may be written together. Furthermore, information for avoiding a collision may be presented by displaying a planned route (not shown).

In the above description, each drone is premised on autonomous flight (flight based on a flight control program set or stored in advance). However, flight control and camera operation are also included. For example, the following operation operation can be performed.
i) Flight control: Local operator / Camera operation: Local operator ii) Flight control: Local operator / Camera operation: Operation panel for this system iii) Flight control: Operation panel for this system / Camera operation: Local operator iv) Flight Control: Operation panel of this system / Camera operation: Operation panel of this system In the case of iii) and iv) above, when the flight control is performed on the operation panel of this system, the local operator It is preferable not to accept the operation of the operation prop. However, in an emergency or the like, it is more preferable to accept a forced interrupt operation from a local operator by operating a predetermined button or the like.

(Second Embodiment)
FIG. 9 is an example of a user interface displayed on the user terminal according to the second embodiment of the present invention. In the present embodiment, when the user selects a desired flying object, the image / movie data transmitted from the selected flying object is the image / moving picture data of another flying object that has been selected so far. Rather than being displayed in the predetermined display area (display area 61 in FIGS. 7 and 8), the display area where the image / movie data of the aircraft selected this time has been displayed so far is enlarged. It is characterized by being. For example, in FIG. 9, when the user terminal 3 selects the drone 2 in the flying object selection unit 64, the display area 62A where the image / moving image data of the drone 2 has been displayed so far is enlarged and displayed. The image / video data display area 61 of the adjacent drone 1 is reduced and displayed, and the display position of the image / video data display area 62B of the drone 3 is changed.

(Third embodiment)
FIG. 10 is a functional block diagram of a control unit and a storage unit in the management terminal according to the third embodiment of the present invention. FIG. 11 is a flowchart for information display according to the third embodiment of the present invention.

  The functional block diagram of the control unit 10 and the storage unit 11 of the management terminal 1 shown in FIG. 10 is basically the same as the functional block diagram of the control unit 10 and the storage unit 11 in the first embodiment shown in FIG. However, as a feature of the present embodiment, the control unit 10 includes an object detection unit 49 that detects a predetermined object stored in the search object storage unit 58 of the storage unit 11.

  As shown in FIG. The information receiving unit 41 receives information such as image / moving image data and / or related information such as predetermined control data from the flying object 2 (S201). The control unit 10 stores the information in the image storage unit 51 or the information storage unit 52 of the storage unit 11.

  Subsequently, the display control unit 44 of the control unit 10 receives the image received from the flying object 2 for the preset layout of the display area and the common elements arranged in accordance with the UI template stored in the template storage unit 53. A user interface (UI) screen file to be displayed on the display device of the user terminal 3 is generated by arranging information such as moving image data and text (S202).

Next, the determination unit 43 of the control unit 10 determines whether or not any of the plurality of flying objects 2 has detected a predetermined object by the management terminal 1 (S203). For example, when a plurality of flying objects are searching for a victim in a mountainous area, each flying object transmits image / video data to the management terminal 1, and the management terminal 1 then searches the search object storage unit 58. The image data related to the search target person stored in is referred to and it is determined whether or not the search target person is included in the image / movie data received from the flying object. When the determination result is YES, the display control unit 45 controls to display the image / moving image data of the flying object in which the search target person is detected in the display area 61 in FIG. 7 (S204). By providing a user interface screen as in this embodiment, it is possible to enlarge the image / movie data of a row object in which a specific object is detected while checking the image / movie data provided from a plurality of flying objects from a bird's-eye view. By displaying, the user can be alerted.

In the above-described embodiment, the camera image of a plurality of flying objects is displayed in the display area. However, for example, the image is obtained by a fixed camera or communication software via a network such as a so-called conference call. The acquired image and the image acquired by the moving body including the camera unit such as the ground moving body and the underwater moving body can also be displayed. In this case, the map display area 65 (see FIG. 7) may indicate the position of each camera (= mobile body, fixed camera, electronic conference terminal, etc.). In addition, when remote operation is performed from multiple points using the operation panel, a display that enables the terminals having the initiative to perform the operation to recognize each other is displayed, and a display for switching the initiative is also provided. It is preferable to display them together.

  The flying object of the present invention can be expected to be used as an industrial flying object in surveying, surveying, observation and the like. In addition, the flying object of the present invention can be used in an aircraft related industry such as a multicopter drone, and further contributes to the improvement of safety related to the flight of these flying objects and the flying object through the present invention. be able to.

  The above-described embodiments are merely examples for facilitating understanding of the present invention, and are not intended to limit the present invention. The present invention can be changed and improved without departing from the gist thereof, and it is needless to say that the present invention includes equivalents thereof.

DESCRIPTION OF SYMBOLS 1 Management terminal 2A-2G Aircraft 3A-3G User terminal

Claims (2)

An image provided by at least one flying body camera, an image provided by a fixed camera, an image provided by communication software via a network using the camera, an image provided by a camera of a ground mobile object, Or an information display method for displaying a plurality of information including any of images provided from a camera of an underwater vehicle,
Receiving an image provided from the camera;
Generating a plurality of image display areas including an image display area for displaying the received image;
Controlling an image corresponding to the selected camera to be displayed in a first image display area of the plurality of image display areas ;
An information display method for performing display so that terminals having the initiative to perform remote operation can recognize each other when performing remote operation from each terminal at a plurality of points . The information display method according to claim 1,
The information display method according to claim 1, wherein the first image display area is larger than the other image display areas.