JP2023057120A - Method for displaying information concerning control of flying body - Google Patents

Abstract

PROBLEM TO BE SOLVED: To optimize the display of information including multiple images transmitted from multiple flying bodies.

SOLUTION: A method for displaying information, including images provided from multiple flying bodies, includes receiving images transmitted from the multiple flying bodies, generating an image display area corresponding to each of the multiple flying bodies, in which the received images are displayed, and controlling to display an image corresponding to a flying body selected by a user in a first image display area of the image display area corresponding to each of the multiple flying bodies.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to an information display method for controlling an aircraft.

In recent years, drones and unmanned aerial vehicles (UAV: Unmanned Aerial Vehicles) have been used for various purposes.
Various services using flying vehicles such as anned aerial vehicles (hereinafter collectively referred to as "flying vehicles") are provided.

During the flight of the aircraft, the aircraft data transmitted from the aircraft is received at a remote ground station, and the aircraft information, map information, and aircraft control panel obtained from the aircraft data on the monitor screen of the computer installed at the ground station. For example, Patent Document 1 discloses a technique for displaying an operation panel for a camera mounted on an aircraft.

Further, for example, Patent Document 2 discloses a technique for displaying flight data of each unmanned aircraft by switching or dividing them on a single screen when operating a plurality of aircraft.

WO2006-098469 JP-A-6-312700

However, the technique disclosed in the above-mentioned patent document focuses on the operation of a single or a plurality of flying objects, and aims at optimizing the monitor screen display of necessary information in a device provided remotely. However, a technique for optimizing image display by displaying a plurality of pieces of image information transmitted from a plurality of flying objects has not yet been disclosed.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to optimize display of information including a plurality of images transmitted from a plurality of aircraft.

One embodiment of the present invention is an information display method for displaying information including images provided from a plurality of flying objects, comprising receiving images transmitted from the plurality of flying objects and displaying the received images. generating an image display area corresponding to each of the plurality of flying objects;
The image corresponding to the aircraft selected by the user is controlled to be displayed in the first image display area among the image display areas corresponding to each of the plurality of aircraft.

Advantageous Effects of Invention According to the present invention, it is possible to optimize display of information including a plurality of images transmitted from a plurality of aircraft.

It is a system block diagram which consists of a management terminal and an aircraft by 1st Embodiment by this invention. 1 is a system configuration diagram including a management terminal and user terminals according to a first embodiment of the present invention; FIG. 3 is a diagram showing the hardware configuration of a management terminal according to the first embodiment of the present invention; FIG. 1 is a functional block diagram of an aircraft according to a first embodiment of the present invention; FIG. FIG. 4 is a functional block diagram of a control unit and a storage unit in the management terminal according to the first embodiment of the present invention; FIG. 4 is a flow chart for displaying information according to the first embodiment of the present invention; It is an example of a user interface displayed on the user terminal according to the first embodiment of the present invention. 4 is another example of a user interface displayed on the user terminal according to the first embodiment of the present invention; It is an example of a user interface displayed on the user terminal according to the second embodiment of the present invention. FIG. 10 is a functional block diagram of a control unit and a storage unit in the management terminal according to the third embodiment of the present invention; FIG. 11 is a flow chart for displaying information according to the third embodiment of the present invention; It is an example of a screen display of the system according to the embodiment of the present invention.

The contents of the embodiments of the present invention are listed and explained. An aircraft according to an embodiment of the present invention has the following configuration.
[Item 1]
An information display method for displaying information including images provided from a plurality of flying objects,
receiving images transmitted from the plurality of flying objects;
generating an image display area for displaying the received image, the image display area corresponding to each of the plurality of flying objects;
An information display method for controlling an image corresponding to a flying object selected by a user to be displayed in a first image display area among image display areas corresponding to each of the plurality of flying objects.
[Item 2]
The information display method according to claim 1,
The information display method, wherein the first image display area is larger than an image display area corresponding to each of a plurality of other flying objects.
[Item 3]
The information display method according to claim 1,
An information display method, comprising generating a second image area displaying predetermined information related to the predetermined air vehicle.
[Item 4]
The information display method according to claim 3,
The information display method, wherein the predetermined information includes any one of flight path, position information, flight position information, flight altitude, flight speed, and remaining battery capacity of the corresponding aircraft.
[Item 5]
The information display method according to claim 1,
A method of displaying information, providing a control panel area for controlling the predetermined aircraft.
[Item 6]
The information display method according to claim 1,
An information display method for recording an image transmitted from the predetermined flying object.
[Item 7]
The information display method according to claim 1,
An information display method, wherein the displayed image is transmitted to a plurality of users.
[Item 8]
The information display method according to claim 7,
receiving message input from each of the plurality of users;
An information display method for generating a message display area for displaying the received message.

(First embodiment)
A control method for an aircraft, an aircraft, and a flight program according to a first embodiment of the present invention will be described below with reference to the drawings. FIG. 1 shows, according to a first embodiment according to the invention,
It is a system block diagram which consists of a management terminal and an aircraft.

As shown in FIG. 1, the management terminal 1 is connected to a plurality of flying objects 2A, 2B, 2G, etc. via a network such as the Internet (hereinafter, for convenience, an arbitrary flying object among these is simply referred to as "flying object 2”). Examples of networks include local area networks (LAN
), wide area networks (WAN), infrared, wireless, WiFi, point-to-point (P2P) networks, telecommunication networks, cloud communications, and the like. Here, the flying object 2 includes, for example, an unmanned flying object (such as a drone) that can fly autonomously or that can be remotely operated and controlled by a user terminal, which will be described later. In addition to predetermined information, such as still image/video data taken by the camera of the aircraft 2, predetermined control data, such as flight position information (latitude, longitude, altitude etc.), flight path, battery usage/remaining capacity, flight speed, flight time, acceleration, tilt, and other information such as the operating status of the device, automatically or upon request from the user.

FIG. 2 is a system configuration diagram consisting of a management terminal and user terminals according to the first embodiment of the present invention. As shown in FIG. 2, the management terminal 1 is connected to a plurality of user terminals 3A, 3B, 3G, etc. via a network such as the Internet (hereinafter, for convenience, an arbitrary user terminal among these is simply referred to as "user terminal"). 3”). Examples of networks include local area networks (LAN), wide area networks (WAN), infrared, wireless,
WiFi, point-to-point (P2P) networks, telecommunication networks, cloud communications, and the like. Here, the user terminal 3 may be, for example, a general-purpose computer such as a workstation or a personal computer, or may be a mobile device terminal such as a smart phone or a tablet. The user terminal 3 includes a display device such as a display panel, and the display device displays predetermined information including image and moving image data transmitted from the management terminal 1 using display software such as a built-in browser.

FIG. 3 is a diagram showing the hardware configuration of the management server 1. As shown in FIG. Note that the illustrated configuration is an example, and other configurations may be employed.

As shown, the management server 1 is connected to a database (not shown) and constitutes part of the system. The management server 1 may be, for example, a general-purpose computer such as a workstation or personal computer, or may be logically realized by cloud computing.

The management server 1 includes at least a control unit 10, a memory 11, a storage 12, a transmission/reception unit 1
3. It has an input/output unit 14 and the like, which are electrically connected to each other through a bus 15 .

The control unit 10 is an arithmetic unit that controls the overall operation of the management server 1, controls transmission and reception of data between elements, executes applications, and performs information processing necessary for authentication processing. For example, the processor 10 is a CPU (Central Processing Unit
), and executes each information processing by executing a program or the like stored in the storage 12 and developed in the memory 11 .

The memory 11 is a DRAM (Dynamic Random Access Memory
y) and other volatile storage devices, and flash memory and HDD (Hard
Auxiliary memory composed of a non-volatile memory device such as a Disc Drive). The memory 11 is used as a work area for the processor 10, and stores a BIOS (Basic Input/Output System) executed when the management server 1 is started, various setting information, and the like.

The storage 12 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 12 .

The transmitter/receiver 13 connects the management server 1 to the network and the blockchain network. Note that the transmission/reception unit 13 is compatible with Bluetooth (registered trademark) and BLE (Blu
(etooth Low Energy) short-range communication interface.

The input/output unit 14 is an information input device such as a keyboard and a mouse, and an output device such as a display.

A bus 15 is commonly connected to the above elements and transmits, for example, address signals, data signals and various control signals.

FIG. 4 is a functional block diagram of the aircraft according to the first embodiment of the present invention. In order to simplify the explanation, the functional block diagram below is described as a concept stored in a single device (aircraft). It may be configured logically by using cloud computing technology.

Flight controller 21 may have one or more processors, such as programmable processors (eg, central processing units (CPUs)).

The flight controller 21 has a memory 22 and can access the memory. Memory 22 stores logic, code, and/or program instructions that are executable by flight controller 21 to perform one or more steps.

Memory 22 may include, for example, separable media or external storage devices such as SD cards and random access memory (RAM). Data acquired from cameras and sensors 27 may be communicated directly to and stored in memory. For example, still image/moving image data captured by a camera or the like is recorded in a built-in memory or an external memory. A camera is installed on the flying vehicle via a gimbal 28 .

Flight controller 21 includes a control module configured to control the state of the vehicle. For example, the control module may be configured to adjust the spatial orientation, velocity, and/or acceleration of a vehicle having six degrees of freedom (translational motions x, y, and z, and rotational motions _θx , _θy , and _θz ). , ESC 29 to control the propulsion mechanism (motor 30, etc.) of the aircraft. The propeller 31 is rotated by the motor to generate the lift of the aircraft. The control module can control one or more of the states of the mount, sensors.

Flight controller 21 is configured to transmit and/or receive data from one or more external devices (e.g., transceiver 25 (propo), terminal, display, or other remote controller). It can communicate with the unit 26 . Transceiver 25 may use any suitable means of communication, such as wired or wireless communication.

For example, transceiver 26 utilizes one or more of a local area network (LAN), wide area network (WAN), infrared, wireless, WiFi, point-to-point (P2P) networks, telecommunications networks, cloud communications, and the like. can do.

The transmitting/receiving unit 26 transmits and/or receives one or more of data obtained by the sensors 27, processing results generated by the flight controller 21, predetermined control data, user commands from a terminal or a remote controller, and the like. be able to.

Sensors 27 according to the present embodiment include inertial sensors (acceleration sensors, gyro sensors),
It may include a GPS sensor, a proximity sensor (eg lidar), or a vision/image sensor (eg camera).

FIG. 5 is a functional block diagram of the control unit and storage unit in the management terminal according to the first embodiment of the present invention. Also, FIG. 6 is a flow chart for information display processing executed in the management terminal according to the first embodiment of the present invention. This information processing can be realized by having the control unit execute an information display program stored in the storage unit.

As shown in FIG. 6, first, the information reception unit 41 of the control unit 10 receives related information such as image/moving image data and/or predetermined control data from the aircraft 2 (S101). The control unit 10 stores the information in the image storage unit 51 or the information storage unit 52 of the storage unit 11 .

Subsequently, the display control unit 44 of the control unit 10, according to the UI template stored in the template storage unit 53, displays the image received from the aircraft 2 with respect to the layout of the preset display area and the arranged common elements. By arranging information such as video data and text, a user interface (UI) screen file for display on the display device of the user terminal 3 is generated (S102).

FIG. 7 shows an example of a user interface displayed on the user terminal according to the first embodiment of the present invention. The user interface screen 60 has a plurality of display areas 61 and 62A-62E for displaying image/video data transmitted from each aircraft 2. FIG.
Each display area is provided to display image/moving image data of the corresponding aircraft. For example, the display area 61 displays image/video data transmitted from the drone 1, and the display area 62
A displays image/video data transmitted from the drone 2 . Similarly, the display area 62B
displays the image/video data transmitted from the drone 3 . Here, the display area 61 is
It is characterized by being an area for displaying the flying object selected by the user. again,
The display area 61 is characterized by having a larger display area than the other display areas 62A-62E. For example, in FIG. 7, the display area 61 displays image/moving image data received from the drone 1 selected by the user via the aircraft selection unit 64 . This allows the user to more easily check the image/video data received from the selected aircraft. The flying object selection unit 64 can select another drone 2-6 by pull-down. The aircraft action panel 63 is arranged so as to overlay the display area 61,
Flight control of the aircraft (for example, control of the flight direction of the aircraft) in response to a request from the user
Also, it is possible to control the camera provided on the flying object (for example, switching between still image/moving image shooting, control of zoom-in/out, control of camera shooting direction, etc.). The flight control of the aircraft is realized by executing the aircraft control program stored in the aircraft control program storage section 57 of the storage section 11 in the aircraft control section program execution section 48 of the control section 10 .

The user interface screen 60 also includes a map display area 65 for displaying on a map the position information and flight path of the currently selected flying object transmitted from the GPS, and various control data (for example, latitude) transmitted from the flying object. , longitude, altitude, direction, speed, remaining battery capacity, etc.). Furthermore, the user interface screen 60
can have a record button 67 for recording the video data transmitted from the selected aircraft and displayed in the display area 61 . It may also have a share button 68 for launching a chat program for sharing the user interface screen 60 with other user terminals. While referring to the shared user interface screen 60, each user can check comments received from other users while inputting comments through the chat interface. The chat program is realized by executing the chat program stored in the chat program storage section 55 of the storage section 11 with the chat program execution section 47 of the control section 10 . In addition, each user can operate the aircraft operation panel 6 displayed in the display area 61 on the shared user interface screen.
3 can be controlled. At this time, the original user (host user) who shared the user interface screen pushes a selection button (not shown) for specifying the authority for operating the operation panel 63 to a specific other user on the user interface. It can also be provided on the screen 60 .

Returning to FIG. 6, the determination unit 43 of the control unit 10 determines whether or not the management terminal 1 has received from the user terminal 3 a request to select a desired flying object from among a plurality of flying objects (S103). . As described above, for example, the user terminal 3 selects the aircraft selection section 64 of the user interface screen 60
A desired flying object is selected by the pull-down operation of . If the determination result is YES, the display control unit 45 displays the image/moving image data transmitted from the aircraft selected by the user terminal 3 in the display area 61 (S104). For example, as shown in FIG. 8 , when the user terminal 3 makes a request to select the drone 2 , image/video data transmitted from the drone 2 is displayed in the display area 61 . Drone 1, which has been selected so far, is the image of Drone 2 so far.
It is displayed in the display area 62A where the moving image data was displayed. By providing a user interface screen like this embodiment, it is possible to check the image/video data provided from a plurality of flying objects in a bird's-eye view and to display the image/video data of the flying object to be focused on in a large size. , it is possible to provide the user with an easier-to-see display. In addition, by providing an operation panel that enables aircraft operation and camera operation in the display area for displaying image and video data of the selected aircraft, a user interface with higher operability is provided. be able to.

As can be understood by comparing FIGS. 7 and 8, the current display area 6 is displayed in the map display area 65.
Drone 1 related to the image displayed in 1 may be displayed in the center (d1), and other drones (for example, drone 2: d2) may be displayed in the same screen. In this case, an icon that can indicate the direction of travel of each drone (for example, a graphic that is asymmetrical with respect to the direction of travel) and a flight history (flight route) may be written together. Furthermore, information for collision avoidance may be presented by displaying a planned course (not shown).

In addition, in the above explanation, it is assumed that each drone will fly autonomously (flying based on a preset or stored flight control program, etc.), but including such a form, flight control and camera operation For example, the following operations can be used.
i) Flight control: local operator/camera operation: local operator ii) Flight control: local operator/camera operation: operation panel of this system iii) Flight control: operation panel of this system/camera operation: local operator iv) Flight Control: Operation panel of this system / Camera operation: Operation panel of this system In the case of iii) and iv) above, when flight control is performed by the operation panel of this system, the on-site operator It is preferable not to accept the operation of the operation propo. However, in an emergency or the like, it is more preferable to accept a forced interrupt operation from the local operator by operating a predetermined button or the like.

(Second embodiment)
FIG. 9 is an example of a user interface displayed on the user terminal according to the second embodiment of the present invention. In this embodiment, when the user selects a desired flying object, the image/moving image data transmitted from the currently selected flying object is replaced by the image/moving image data of another flying object that has been selected so far. Instead of being displayed in the predetermined display area (display area 61 in FIGS. 7 and 8), the display area in which the image/movie data of the flying object selected this time has been displayed is displayed enlarged. characterized by being For example, in FIG. 9, when the user terminal 3 selects the drone 2 in the flying object selection unit 64, the display area 62A in which the image/moving image data of the drone 2 has been displayed is enlarged. The image/moving image data display area 61 of the adjacent drone 1 is reduced and displayed, and the display position of the image/moving image data display area 62B of the drone 3 is changed.

(Third Embodiment)
FIG. 10 is a functional block diagram of the control unit and storage unit in the management terminal according to the third embodiment of the present invention. FIG. 11 is a flow chart for information display according to the third embodiment of the present invention.

The functional block diagram of the control unit 10 and the storage unit 11 of the management terminal 1 shown in FIG. 10 is basically equivalent to the functional block diagram of the control unit 10 and the storage unit 11 in the first embodiment shown in FIG. However, a feature of this embodiment is that the control unit 10 has an object detection unit 49 that detects a predetermined object stored in the search object storage unit 58 of the storage unit 11 .

As shown in FIG. 11, the . The information reception unit 41 receives information such as image/moving image data and/or related information such as predetermined control data from the aircraft 2 (S201).
The control unit 10 stores the information in the image storage unit 51 or the information storage unit 52 of the storage unit 11 .

Subsequently, the display control unit 44 of the control unit 10, according to the UI template stored in the template storage unit 53, displays the image received from the aircraft 2 with respect to the layout of the preset display area and the arranged common elements. - By arranging information such as video data and text, a user interface (UI) screen file for displaying on the display device of the user terminal 3 is generated (S202).

Next, the determination unit 43 of the control unit 10 determines whether or not the management terminal 1 has detected a predetermined object in any one of the plurality of flying objects 2 (S203). For example, when a plurality of flying objects are searching for victims in a mountainous area, each flying object transmits image/moving image data to the management terminal 1 . , and determines whether or not the image/moving image data received from the aircraft includes the search target. If the determination result is YES, the display control unit 45
controls to display the image/video data of the aircraft that detected the search target in the display area 61 in FIG. 7 (S204). By providing a user interface screen like this embodiment, it is possible to view the image/video data provided by a plurality of flying objects from a bird's-eye view and enlarge the image/video data of the body that detected a specific object. By displaying, it is possible to call attention to the user.

In the above-described embodiment, camera images of a plurality of flying objects are displayed in the display area. It is also possible to display captured images and images captured by moving objects equipped with a camera unit, such as ground moving objects and underwater moving objects. In this case, the map display area 65 (see FIG. 7) may indicate the position of each camera (=mobile body, fixed camera, electronic conference terminal, etc.). In addition, when remote control is performed from multiple locations using the operation panel, the display should be such that the terminals that have the initiative to perform the operation can recognize each other. It is preferable to display them together.

The flying object of the present invention can be expected to be used as an industrial flying object for investigation, surveying, observation, and the like. In addition, the flying object of the present invention can be used in aircraft-related industries such as multicopter drones, and through the present invention, contributes to the improvement of safety related to the flight of these flying objects and flying objects. be able to.

The above-described embodiments are merely examples for facilitating understanding of the present invention, and are not intended to limit and interpret the present invention. It goes without saying that the present invention can be modified and improved without departing from its spirit, and that equivalents thereof are included in the present invention.

1 Management terminal 2A-2G Aircraft 3A-3G User terminal

Claims (8)

An information display method for displaying information including images provided from a plurality of flying objects,
receiving images transmitted from the plurality of flying objects;
generating an image display area for displaying the received image, the image display area corresponding to each of the plurality of flying objects;
An information display method for controlling an image corresponding to a flying object selected by a user to be displayed in a first image display area among image display areas corresponding to each of the plurality of flying objects. The information display method according to claim 1,
The information display method, wherein the first image display area is larger than an image display area corresponding to each of the plurality of other flying objects. The information display method according to claim 1,
An information display method, comprising generating a second image area displaying predetermined information related to the predetermined air vehicle. The information display method according to claim 3,
The information display method, wherein the predetermined information includes any one of flight path, position information, flight position information, flight altitude, flight speed, and remaining battery capacity of the corresponding aircraft. The information display method according to claim 1,
A method of displaying information, providing a control panel area for controlling the predetermined aircraft. The information display method according to claim 1,
An information display method for recording an image transmitted from the predetermined flying object. The information display method according to claim 1,
An information display method, wherein the displayed image is transmitted to a plurality of users. The information display method according to claim 7,
receiving message input from each of the plurality of users;
An information display method for generating a message display area for displaying the received message.

Images