KR101662032B1

KR101662032B1 - UAV Aerial Display System for Synchronized with Operators Gaze Direction

Info

Publication number: KR101662032B1
Application number: KR1020150060031A
Authority: KR
Inventors: 김병수
Original assignee: 주식회사 유브이코어
Priority date: 2015-04-28
Filing date: 2015-04-28
Publication date: 2016-10-10

Abstract

[0001] The present invention relates to a flight control and image capturing technology of a UAV according to a direction of a pilot's eyes, and more particularly, to a technique of synchronizing a 3-axis rotational motion direction of a HMD and a camera photographing angle with a gimbals- And at the same time to the flight control of the UAV.
The UAV aerial photographing system synchronized with the direction of the pilot's eyes according to the present invention includes a gimbal structure ball camera mounted on the bottom of the cockpit of the UAV or UAV that is the subject of the flight, a pilot operating the UAV and the gimbal structure ball camera on the ground, An HMD worn on the head of the eyeball and the front face of the eyeball, an motion detection sensor mounted on the HMD for detecting the head movement of the pilot, a control device for receiving the radio signal including the image information of the UAV, And an integrated control / measurement system for transmitting a radio wave signal in which information and image information are collected to the HMD.

Description

[0001] The present invention relates to a UAV aerial photographing system synchronized with a pilot's gaze direction,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a flight control and an aerial image capturing technology of an unmanned aerial vehicle (UAV) according to the direction of a pilot's eyes, and more particularly, to a gimbal structure ball camera device To a 3-axis rotational motion direction of an HMD (helmet / head mounted display) and a camera photographing angle while at the same time performing flight control of a UAV.

In recent years, avionics, which is one of the most vigorously converged aviation ICT technologies, has been developed as a military technology because it can create high added value due to the nature of the industry, Spin-off, which transfers technology to private companies; spin-ons, which apply civilian technology to military applications; spin-offs, which develop technologies for joint use by private companies and the military; -up). In the field of aviation ICT, which requires cutting-edge technology, it is often applied to the military industry due to the flow of the civilian industry. The technology field, which has high expectation for demand and is actively developing overseas advanced defense industry, is augmented reality and HMD. HMD is a type of display mounted on the head, and various forms exist depending on the view mode. Helmets, instrumentation and optical / signal processing technologies are integrated, and many applications are compatible with HMD. HMD is a combination of fusion technologies such as optical technology, material technology, hardware and software. HMD is developed for military purposes to overcome the disadvantage that existing head-up display (HUD) can process and provide line-of-sight data in a limited range ahead of the aircraft fuselage First, Through this, it is possible to provide immediate flight information and target information about the direction of the pilot's gaze by linking the HMD with the weapon sensor. Developed for military purposes, HMD has recently invested heavily in the development of commercial HMDs by leading companies such as GOOGLE and MICROSOFT due to the rapid growth of mobile devices and the increasing demand for portable display devices have. A key technology in aircraft HMDs is see-through optical technology that allows real-world and virtual images to be viewed at the same time. A see-through type optical system is a system in which an image is displayed by a wave guide or a projection system is used to display an image on a visor or a corresponding transparent lens, and a method of displaying on a flat panel such as an LCD and an OLED . Another key technology is precise helmet tracking. Accurate helmet attitude tracking technology enables accurate flight information of various navigation sensors to be provided to the pilot's gaze direction. Helmet attitude tracking techniques include electromagnetic, optical, and inertial methods, and a hybrid method in which two methods can be simultaneously applied to increase accuracy. The last key technology is augmented reality technology. Currently, sensor based tracking technology is widely used in HMD for aircraft. The data transmitted through many sensors attached to the aircraft are transferred to the HMD by the main computer, and the information desired by the user is displayed on the HMD as augmented reality. The technology fusion of HMD and Augmented Reality can be regarded as the point of wearable device. The HMD, which displays only the images transmitted from various sensors, can be regarded as a simple display, but the flight efficiency of the pilots can be improved by realizing the information of the aircraft through fusion with the augmented reality technology.

A UAV is a flight program that is piloted remotely from the ground without a pilot, or pre-entered mission program, in which the aircraft recognizes and judges the surrounding environment and flies by itself. It can be used for collecting information on wide-area video and signal information, surveillance and reconnaissance, communication relaying, and information on fires or natural disasters in polluted areas or large areas where human access is difficult. UAV can be divided into aviation and ground systems. The main task of the aviation sector is to collect information by cameras and various sensors and transmit them to the ground sector in real time. The ground sector is used for ground communication equipment, Equipment and so on.

Similar prior arts for constructing an aerial image capturing system by integrating the HMD and the UAV described above include the 10-2011-0004185, 10-2013-0069600, and 10-0931029 filed in the Korean Intellectual Property Office. Previously, however, the control of the 3-axis rotation direction of the roll, pitch and yaw of the UAV, HMD and camera angles, and the implementation of augmented reality including the HUD in the HMD image Technology was not provided.

KR 10-2011-0004185 KR 10-2013-0069600 KR 10-0931029

The present invention aims to satisfy the technical needs required from the background of the above-mentioned invention.

More specifically, the present invention aims at synchronizing a camera device mounted on a UAV with a three-axis rotational motion direction of an HMD and a camera photographing angle, and implementing the flight control technology of the UAV on the basis thereof, And to provide a photographing system.

The technical objects to be achieved by the present invention are not limited to the above-mentioned problems, and other technical subjects not mentioned can be clearly understood by those skilled in the art from the following description. There will be.

In order to achieve the above object, the UAV aerial photographing system synchronized with the direction of the pilot's eyes according to the present invention includes a UAV, which is the subject of flight; A gimbal structure ball camera mounted at the bottom of the cockpit portion of the UAV; Pilots operating UAV and gimbal structure ball cameras on the ground; HMD worn on pilot's head and front of eyeball; A head tracker mounted on the HMD for sensing head movement of the pilot; A control unit for receiving data on which UAV flight information and image information of a gimbal structure type ball camera are combined; And an integrated control / measurement system for transmitting the flight information and the image information data to the HMD.

As described above, according to the present invention, when performing a task for collecting information on wide-area image and signal information collection, surveillance and reconnaissance, communication relay, information on fires and natural disasters that are difficult for human access, It is possible to integrate the personnel and the aerial photographing system operation personnel into a one-person system, thereby maximizing the efficiency of performing the above-mentioned special purpose mission.

It is to be understood that the technical advantages of the present invention are not limited to the technical effects mentioned above and that other technical effects not mentioned can be clearly understood by those skilled in the art from the description of the claims There will be.

FIG. 1 is a schematic block diagram of a UAV aerial photographing system synchronized with a pilot's gaze direction according to an embodiment of the present invention.
FIG. 2 is an exemplary view showing a synchronized state of a UAV flight direction, a gimbal structure type ball camera shooting direction, and a pilot sight line direction of a UAV aerial photographing system synchronized with a sight line direction of a pilot according to an embodiment of the present invention.
FIG. 3 is an exemplary view of UAV flight information and gimbal structure type ball camera photographing information that the pilot can visually confirm through the HMD of the UAV aerial image capturing system synchronized with the sight direction of the pilot according to the embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings, It is not. In the following description of the present embodiment, the same components are denoted by the same reference numerals and symbols, and further description thereof will be omitted.

Referring to FIG. 1, the UAV aerial photographing system synchronized with the pilot's direction according to the present invention includes a UAV 100 as a subject of unmanned aerial photographing for aerial photographing; A gimbal structure ball camera 200 mounted at the lower end of the cockpit of the UAV 100; A pilot 300 for piloting and operating the UAV 100 and the gimbal structure ball camera 200 on the ground; An HMD 400 to be worn on the head of the pilot 300 and the eyeball; A head tracker 410 attached to the HMD 400 to sense head movement of the pilot 300; A control unit (500) for receiving the radio wave signal A (W10) in which the flight information of the UAV (100) and the image information of the ball camera (200) of the gimbal structure type are combined; And an integrated control / measurement system (600) for transmitting the propagation signal A (W10) for the collected flight information and image information to the HMD. The information acquired by the motion detection sensor 410 for sensing the head movement of the pilot 300 is transmitted to the integrated control and control system 600 and is transmitted to the control device 500 again, (100) and the gimbal structure type ball camera (200).

Referring to FIG. 2, the operating direction of the UAV 100 operating in the sky above can be divided into three types of three-axis rotational motion with respect to the moving direction. First, the roll (U100), which is a rotational motion around an axis on a parallel horizontal plane, is to rotate the stick to the left if it moves to the left and to the right if it moves to the right. The second is the pitch (U110), which is the rotational motion around the axis in a horizontal plane perpendicular to the direction of travel; when the pilot pushes the joystick forward, the nose falls down and pulls back. The third is the yaw (U120), which is a rotational movement about an axis in a vertical plane perpendicular to the direction of travel; the nose is left when the pilot presses the left pedal, and the nose is right when the right pedal is depressed. The motion sensing sensor 410 attached to the HMD 400 worn by the pilot 300 is configured to detect movement of the roll 300 in relation to the head movement of the pilot 300, Axis rotation movement coordinate system and transmits the position change information to the integrated control / measurement system 600 to calculate the coordinate change and transmit the coordinate change to the controller 500 through the radio wave signal B (W20) The control device 500 transmits the radio wave signal A (W10) to the UAV 100 to control the three-axis rotary motion of the roll U100, the pitch U110 and the yaw U120, ) Can be controlled.

The gimbal structure type ball camera 200 is also used as the aerial image 300 in the same manner as the 3-axis rotation direction U100 (U110; U120) of the UAV 100 and the 3-axis rotation direction H300 (H310; H320) A roll (C200) and a pitch (C210) which are three-axis rotation motions for shooting; (C220) are divided into three directions. The gimbal structure type ball camera 200 is installed at the lower end of the UAV 100 to acquire image information related to the operation. The gimbal structure type ball camera 200 is designed in a globular shape, Enables rapid switching to 3-axis rotation direction (C200; C210; C220) and minimizes the dead zone of the image capturing angle of view. The camera lens for image photographing should be mounted on the barrel section C230 and the three axes rotational directions C200 and C210 of the gimbals type ball camera 200 and the three axes of the UAV 100, It is preferable that an attitude sensor capable of providing reference data for synchronizing the rotational motion directions U100 (U110; U120) and the three-axis rotational directions H300 (H310; H320) Do. When the UAV 100 is in flight through the synchronization process based on the attitude sensor, the pilot image is taken in accordance with the direction of the pilot 300 when capturing an image, To maximize the efficiency of UAV and camera operation.

Here, it is preferable that the pilot 300 is dualized in a form that can selectively activate or deactivate the function of the posture sensor according to the operation status of the UAV 100 and the gimbal structure ball camera 200. The reason is that if only the unidirectional three-axis rotational motion direction is realized by always synchronizing the sight line direction of the pilot 300, the aerial image capturing direction of the gimbal structure type camera 200 and the UAV 100, There may be a problem in the take-off and landing of the UAV 100 and a stall phenomenon may occur in the flight, so that the pilot 300 may fall regardless of the flight intention of the pilot 300.

FIG. 3 is a block diagram of the UAV 100 in which the pilot 300 can visually confirm the pilot 300 through the HMD 400 in the intelligent flight control and aerial image capturing system of the UAV synchronized with the direction of the pilot according to the embodiment of the present invention. ) Flight information, and photographing information of the gimbal structure type camera 200. FIG. The image information 420 captured by the gimbal structure type ball camera 200 mounted on the lower end of the cockpit portion of the UAV 100 and the image information 420 transmitted by the aerial image capturing and transmitting operation, When the control unit 500 receives the integrated navigation information 430 of the UAV 100 for the pilot 300, the pilot 300 transmits the integrated navigation information 430 to the integrated control / measurement system 600. The pilot 300 transmits the integrated navigation information 430 to the HMD 400 So that a visual confirmation can be made in a unified screen form.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, will be. Accordingly, the true scope of the present invention should be determined only by the appended claims.

100: UAV
200: Gimbal structure ball camera
300: Pilot
400: HMD
500: Control device
600: Integrated Control / Instrumentation System

Claims

In a UAV aerial photographing system synchronized with a pilot's gaze direction,
UAV (100), the subject of unmanned aerial flight for aerial photographing;
A gimbal structure ball camera 200 mounted at the lower end of the cockpit of the UAV 100;
A pilot 300 for piloting and operating the UAV 100 and the gimbal structure ball camera 200 on the ground;
An HMD 400 to be worn on the head of the pilot 300 and the eyeball;
An operation detection sensor 410 mounted on the HMD 400 for detecting head movement of the pilot 300;
A control unit (500) for receiving the radio wave signal A (W10) in which the flight information of the UAV (100) and the image information of the ball camera (200) of the gimbal structure type are combined;
An integrated control / measurement system (600) for transmitting to the HMD (400) the radio wave signal A (W10) in which the combined flight information and image information are embedded;
The UAV 100 transmits the information obtained by the motion detection sensor 410 that senses head movement of the pilot 300 through the integrated control and measurement system 600 to the control device 500, And a gimbal structure ball camera (200) interlocked with each other to remotely control the UAV aerial image capturing system synchronized with the pilot's gaze direction.

The method according to claim 1,
The motion detection sensor 410 attached to the HMD 400 detects a three-axis rotational motion coordinate system of the roll H300, pitch H310, and yaw H320 with respect to head movement of the pilot 300 The position change information is calculated in the integrated control / measurement system 600 by the coordinate change of the motion detection sensor 410 and transmitted to the control apparatus 500 through the radio wave signal B (W20) 500 are transmitted to the UAV 100 through the radio wave signal A10 to control the three-axis rotational motion of the roll U100, the pitch U110 and the yaw U120, A UAV aerial photographing system synchronized with the pilot's gaze direction, which controls the navigation direction.

The method according to claim 1,
The gimbal structure type ball camera 200 is configured to receive the aerial image U110 of the pilot 300 as well as the three axes of rotation directions H300 and H310 of the pilot 300, A roll (C200) and a pitch (C210) which are three-axis rotation motions for shooting; (C220) are divided into a three-axis rotational motion direction of the UAV and a C220.

The method of claim 3,
The gimbal structure type ball camera 200 is installed at the lower end of the UAV 100 to acquire aerial image information. By designing the aerial image information in a spherical shape, (C200; C210; C220), and minimizes the blind spot of the aerial image capturing angle. The UAV aerial photographing system synchronized with the pilot's gaze direction.

The method of claim 3, wherein
The lens barrel C230 of the gimbal structure type ball camera 200 is equipped with a camera lens for capturing an aerial image and has a three-axis rotation direction C200, C210, C220 of the gimbals type ball camera 200 and a UAV 100 Axis direction of rotation of the HMD 400 mounted on the pilot 300 and the three-axis rotational motion directions H300 and H310 of the HMD 400 mounted on the pilot 300 are provided Wherein the position sensor is mounted on the UAV.

The method according to claim 1,
The UAV 100 in flight can synchronize the view direction of the pilot 300 with the photographing direction of the gimbal structure camera 200 through the HMD 400 and the motion detection sensor 410, 300), which is synchronized with the direction of the pilot's eyes.

6. The method of claim 5,
In the process of synchronizing the three-axis rotation direction U100 (U110; U120) of the UAV 100 and the three-axis rotation direction C200 (C210) C220 of the gimbal structure type ball camera 200 to the posture sensor, Wherein the controller is capable of selectively activating or deactivating the function of the attitude sensor according to the operation status of the UAV 100 and the gimbal structure ball camera 200. [ UAV aerial imaging system synchronized with gaze direction.

The method according to claim 1,
The image information 420 captured by the gimbal structure type ball camera 200 mounted on the lower end of the cockpit portion of the UAV 100 and the image information 420 transmitted by the aerial image capturing and transmitting operation, The pilot 300 can be visually confirmed in the form of a screen in which the pilot 300 is unified through the HMD 400 with respect to the flight information 430 of the UAV 100, Shooting system.