KR20190123095A - Drone-based omni-directional thermal image processing method and thermal image processing system therefor - Google Patents

Abstract

The present invention relates to a drone-based omnidirectional thermal image processing method and a thermal image processing system therefor. The drone-based omnidirectional thermal image processing method includes a step of allowing a drone to photograph an image from a thermal imaging camera having a wide-angle lens; a step of allowing the drone to correct temperature and shape distortion with respect to the distortion of the image generated by the wide-angle lens; a step of transmitting the corrected image to a terrestrial thermal image processing apparatus; and a step of allowing the terrestrial thermal image processing apparatus to output the image received from the drone as an omnidirectional image. The correction of the temperature distortion and the image distortion is performed by inverse conversion mapping. It is possible to provide a wide field of view.

Description

Drone-based omni-directional thermal image processing method and thermal image processing system therefor}

The present invention relates to a drone-based omnidirectional thermal image processing method and a thermal image processing system therefor, and more particularly, to rapidly process an omnidirectional movie image based on an image photographed by a drone equipped with a thermal imaging camera. And a thermal image processing system therefor.

Drone is a dictionary meaning "low hum". Here, various meanings are derived and are also used to mean "submersible" or "instrumental bass." Recently, however, it is mainly used to mean a small unmanned aerial vehicle (unmanned aerial vehicle).

In recent years, drones are equipped with a function of maintaining a gas balance using a plurality of micro propellers (Rotors) mounted, and taking a picture of a surrounding situation by mounting a camera or the like.

Such drones are used in various industrial fields. More specifically, in the past, it has been used for military and hobby, but recently, the utilization of the transportation industry, the film, and the broadcasting industry has become very wide, and various aircrafts having various sizes and performances have been developed according to the purpose of use. . In particular, drones are put into operation in areas not accessible to humans, such as jungles, remote areas, volcanic areas, natural disaster areas, and nuclear power plant accident areas.

In addition, drones can move freely and quickly along preset routes, allowing remote monitoring of specific areas through onboard cameras or sensors. In addition, the drone may be connected to the coordinating device or coordinator terminal by wire or wirelessly, and may monitor a specific area by performing a function such as flight and photography according to a command transmitted from the coordinating device or coordinator terminal.

However, until now, search and detection using drones caused blind spots when searching due to the angle of view of the mounted camera, so that the drone had to fly several times in the search area, resulting in a long search and search time. .

Therefore, for the purpose of searching, monitoring or detecting using a drone, a drone having a function capable of detecting and identifying a target while providing a wide field of view is required.

In order to solve the above problems, the present specification provides a wide field of view and a drone-based omnidirectional thermal image processing method having a camera capable of detecting and identifying a target, and a thermal image processing system therefor. The purpose is to provide.

According to an embodiment of the present disclosure, a drone-based omnidirectional thermal image generation method includes: generating a thermal image raw image by a drone thermal imaging camera having a wide-angle lens; Correcting the temperature and shape distortion of the drone by the wide-angle lens with respect to the distortion of the thermal image; The drone transmitting the corrected image to a terrestrial thermal image processing apparatus; And outputting, by the terrestrial thermal image processing apparatus, the image received from the drone as an omnidirectional image, wherein the temperature distortion and the image distortion are corrected by comparing the images before and after the wide-angle lens is mounted on the thermal imaging camera. It is characterized by applying an area ratio.

The temperature distortion and image distortion correction are performed by inverse conversion mapping.

The area ratio is defined as the ratio of the IFOV area of the two-dimensional coordinate system to the corresponding area of the surface of the three-dimensional sphere.

According to an embodiment of the present disclosure, a drone-based omnidirectional thermal image generation method includes: generating a thermal image raw image by a drone by a plurality of thermal imaging cameras; The drone arranging a plurality of thermal image raw images adjacent to each other where the angles of view overlap; Transmitting the drone-prepared image to a terrestrial thermal image processing apparatus; And stitching, by the terrestrial thermal image processing apparatus, the image received from the drone to output the omnidirectional image.

According to an embodiment of the present disclosure, the plurality of thermal imaging cameras are disposed so that angles of view overlap each other, and between the disposing and transmitting, the drone performs temperature correction of an area where the angles of view overlap. It further comprises.

According to the exemplary embodiment of the present specification, a drone-based omnidirectional thermal image generation method includes a step of generating a panoramic still image by capturing a search area by a thermal imaging camera equipped with a drone rotated 360 degrees, wherein the drone is a panorama Photographing a portion of an image of each cut so as to overlap in a rotational direction; Transmitting, by the drone, the panoramic still image to a terrestrial thermal image processing apparatus; And outputting the omnidirectional image by correcting the superimposed image from the panoramic still image received from the drone.

According to an embodiment of the present disclosure, the step of outputting the omnidirectional image by correcting the overlapped image in the panoramic still image, characterized in that for correcting the temperature by the average value of the temperature of the overlapping first image and the second image.

According to an embodiment of the present disclosure, a drone-based omnidirectional thermal image generation system includes a drone equipped with a thermal imaging camera having a wide-angle lens and capturing an image by the thermal imaging camera to generate a thermal image low image; And a thermal image processing apparatus for outputting an image received from the drone as an omnidirectional image, wherein the drone corrects temperature and shape distortion with respect to the distortion of the thermal image low image, and processes the corrected image into the thermal image image. The temperature distortion and the image distortion are transmitted to the device, and the area ratio calculated by comparing the images before and after the wide-angle lens of the thermal imaging camera is applied.

According to an embodiment of the present disclosure, a drone-based omnidirectional thermal image generation system includes: a drone equipped with a plurality of thermal imaging cameras and generating a thermal image low image by capturing an image by the thermal imaging camera; And a thermal image processing apparatus for stitching an image received from the drone to output the omnidirectional image.

As described above, according to the present specification, it is possible to provide an omnidirectional thermal image of the drone, and it can be observed regardless of clouds, bad weather, and fog with excellent transmission characteristics that are distinguished from visible light, and can observe daytime at night. Let's do it.

Drone-based search, search time and drone flight time can be shortened to ensure the fastest time in case of an accident.

By providing an omnidirectional image, the blind spots in the search can be solved, reducing the number of drones and operators.

1 is a block diagram of a drone-based omnidirectional thermal image processing system according to an embodiment of the present invention.
2 is a block diagram illustrating a schematic configuration of a drone according to a first embodiment of the present invention.
3 is a view for explaining distortion correction by a wide-angle lens according to the first embodiment of the present invention.
4 is a flowchart illustrating a drone-based omnidirectional thermal image generation method according to a first embodiment of the present invention.
5 is a block diagram showing a schematic configuration of a drone according to a second embodiment of the present invention.
6 is a flowchart illustrating a drone-based omnidirectional thermal image generation method according to a second embodiment of the present invention.
7 is a diagram for describing an arrangement of an image according to a second exemplary embodiment of the present invention.
8 is a diagram illustrating an example of a reliability map created according to the second embodiment of the present invention.
9 is a diagram illustrating an example of a screen displaying image information of an ROI according to a second exemplary embodiment of the present invention.
10 is a block diagram illustrating a schematic configuration of a drone according to a third exemplary embodiment of the present invention.
11 is a flowchart illustrating a drone-based omnidirectional thermal image generation method according to a third embodiment of the present invention.
12 is a block diagram of a search system using a drone-based omnidirectional thermal image processing method.

It is to be noted that the technical terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In addition, the technical terms used in the present specification should be interpreted as meanings generally understood by those skilled in the art unless they are specifically defined in this specification, and are overly inclusive. It should not be interpreted in the sense of or in the sense of being excessively reduced. In addition, when the technical terms used herein are incorrect technical terms that do not accurately represent the spirit of the present invention, it should be replaced with technical terms that can be understood correctly by those skilled in the art. In addition, the general terms used in the present invention should be interpreted as defined in the dictionary or according to the context before and after, and should not be interpreted in an excessively reduced sense.

Also, the singular forms used herein include the plural forms unless the context clearly indicates otherwise. In the present application, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various components, or various steps described in the specification, wherein some of the components or some of the steps It should be construed that it may not be included or may further include additional components or steps.

In addition, the suffixes "module" and "unit" for the components used herein are given or mixed in consideration of ease of specification, and do not have meanings or roles that are distinguished from each other.

In addition, terms including ordinal numbers, such as first and second, as used herein may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

The thermal imaging camera described in the present invention is a device that tracks, detects, and displays a screen by using infrared wavelengths (0.9 to 14 μm). The thermal imaging camera displays the relative or absolute temperature in black and white.

Thermal imaging cameras have a relatively low resolution compared to ordinary cameras. For example, 80 * 60, 160 * 120, 320 * 240, 640 * 480 and 1920 * 1080 are conventionally used.

The detectors of thermal imaging cameras are made of a variety of materials that are sensitive to infrared wavelengths and employ a focal plane array (FPA) consisting of a micrometer-sized grid.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, and the same or similar components will be given the same reference numerals regardless of the reference numerals, and redundant description thereof will be omitted.

In addition, in describing the present invention, when it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it is to be noted that the accompanying drawings are only for easily understanding the spirit of the present invention and are not to be construed as limiting the spirit of the present invention by the accompanying drawings.

1 is a block diagram of a drone-based omnidirectional thermal image processing system according to an embodiment of the present invention.

Referring to FIG. 1, a drone-based omnidirectional thermal image processing system according to the present invention is a system for outputting an omnidirectional thermal image by using an image of a thermal imaging camera mounted on a drone. Apparatus 200 and thermal image processing apparatus 300 and display apparatus 400 are included.

The drone 100 is remotely controllable, and a thermal imaging camera is installed to capture a thermal image raw image for generating an omnidirectional thermal image during flight. The drone 100 transmits the photographed image to a terminal or a device connected through a network.

The detailed configuration of the drone 100 will be described later in detail. The detailed configuration of the drone 100 may differ in some embodiments.

The remote controller 200 controls the drone 100 through a network remotely based on a user input.

To this end, the remote control apparatus 200 is a device for receiving at least one of a user's command, selection, and information, and includes a plurality of input keys and function keys for receiving numeric or text information and setting various functions. Can be. For example, the remote controller 200 may include a hold button for controlling the movement and stop of the drone 100, a rise button for commanding the rise of the drone 100, and commanding the drone 100 to descend. It may include a down button for.

In addition, the remote control device 200 is a device for measuring the speed, direction, gravity, and acceleration of a moving object, and includes a wearable display device having an inertial measurement unit (IMU) sensor such as an accelerometer, an angometer, a geomagnetic machine, and an altimeter. And a smartphone, and may include a joystick, a key pad, a dome switch, a touch pad (static pressure / capacitance), a touch screen, a jog wheel, a jog switch, a jog Various devices such as a shuttle, a mouse, a stylus pen, and a touch pen may be included.

The remote control apparatus 200 may include a wired / wireless communication module. Here, the wireless Internet technology, Wireless LAN (WLAN), Wi-Fi (Wi-Fi), Wireless Broadband (Wibro), WiMAX (World Interoperability for Microwave Access (Wimax), High-speed downlink packet access (High) Speed Downlink Packet Access (HSDPA), IEEE 802.16, Long Term Evolution (LTE), and Wireless Mobile Broadband Service (WMBS), and the like, and may include the short range communication. The technology may include Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and the like. The wired communication technology may include universal serial bus (USB) communication.

The remote control apparatus 200 may have a plurality of functions including a manual operation mode, an automatic takeoff and landing function, an automatic route point flight, and an emergency return. Manual operation mode is a mode that is operated by the manager's control command, the aircraft can be operated within a limited speed and tilt angle, and emergency actions can be performed in a sudden situation. The auto takeoff and landing feature can be automatically taken off and landed by simply pressing the remote controller switch or the Geographic Coordinate System (GCS) screen button. Automatic route point flight, by inputting a plurality of route points to plan a variety of movement paths, it is possible to edit and save the route, reducing the manager's work intensity and easy operation. Emergency return is a function to automatically return to the safe position by the judgment of the drone controller or the operator in emergency mode such as sudden malfunction or low battery condition.

The thermal image processing apparatus 300 receives an image and information related to the image (hereinafter, referred to as image information) from the drone 100. Here, the image includes at least one of a moving image and a still image. The thermal image processing apparatus 300 stores the image and the image information received from the drone 100 in the storage device 150 or converts the image and the image suitable for the display device 400 based on the received image.

The thermal image processing apparatus 300 includes a general multimedia processor for converting a corrected image into an image file for display, and a memory for storing such image file.

In addition, when there is a region overlapping the image received from the drone 100, the thermal image processing apparatus 300 may output the stitching process.

The thermal image processing apparatus 300 transmits one or more of the converted image and flight information to the display apparatus 400.

The display apparatus 400 shows the omnidirectional thermal image and flight information processed by the thermal image processing apparatus 300 to the user through a screen. The user designates a region of interest (ROI) of the 360-degree (360-degree) image displayed on the screen using a user interface device (not shown), for example, a mouse, and the information (eg, Average temperature, maximum temperature, minimum temperature, coordinate information) can be checked.

In the exemplary embodiment of the present invention, the remote controller 200, the thermal image processing apparatus 300, and the display apparatus 400 are illustrated as separate components, but the present invention is not limited thereto. Although functionally divided for the sake of convenience, it will be apparent that each device shown above may be implemented as one or two devices according to the intention of those skilled in the art.

2 is a block diagram illustrating a schematic configuration of a drone according to a first embodiment of the present invention.

Referring to FIG. 2, the drone 100 may include a photographing unit 110 including a thermal imaging camera, a wireless communication unit 120, a control unit 130, an image processing unit 140, a starting unit 150, and the like. have.

The photographing unit 110 includes a thermal imaging camera having a wide angle lens. The wide angle lens may be a fisheye type lens.

Since the photographing unit 110 uses infrared wavelengths (0.9 to 14 μm), the wide-angle lens may use any one or more of silicon (Si) or germanium (Ge) capable of passing a long wavelength.

The number of cameras mounted on the photographing unit 110 is adaptively adjusted according to the angle of view of the wide-angle lens. For example, the photographing unit 110 may be equipped with two thermal imaging cameras having a wide-angle lens having an angle of view of 180 degrees. In this case, the first thermal imaging camera and the second thermal imaging camera are disposed to face each other to photograph opposite directions. The first thermal image and the second thermal image are generated by the first thermal image camera and the second thermal image camera, respectively.

An embodiment of the present invention has been described in the case of a wide-angle lens having an angle of view of 180 degrees, but is not limited thereto. If the angle of view is 120 degrees, three or four cameras can be used to shoot.

The control unit 130 receives control information from the remote controller 200 through the wireless communication unit 120, and controls the starter 150 for adjusting the movement of the drone 100 according to the control information. Can fly. At this time, the wireless communication unit 120 may support a communication method according to the network.

In addition, the controller 130 stores the image photographed through the photographing unit 110 during flight of the drone 100 in the data storage unit 160, or transmits the image to the image processor 140, and transmits the image processor 140. The corrected image may be transmitted to the thermal image processing apparatus 300 through the wireless communication unit 120.

In addition, when a gyro sensor or an acceleration sensor is provided in the drone 100, the controller 130 generates posture information based on the sensing value measured by the sensor according to the control information, and then heats it through the wireless communication unit 120. The image may be transmitted to the image processing apparatus 300.

In addition, when the Global Positioning System (GPS) or the like is provided, the controller 130 generates position information on the current position of the drone 100 based on the measured position measurement value, and thermal image through the wireless communication unit 120. The image processing apparatus 300 may transmit the image.

The image processor 140 performs image correction on the distortion of the thermal image low image generated by the wide-angle lens.

As shown in (a) of FIG. 3, the image incident through the wide-angle lens is larger in size and more curved as it is moved away from the center. As a result, the image incident through the wide-angle lens has a larger radial distortion as it moves away from the center point.

The image processor 140 detects an image incident through the image sensor as an electric signal image, and corrects the electric signal image.

The image processor 140 corrects the image through a reverse mapping method.

The reverse mapping method calculates coordinate values of a corresponding distorted image by using coordinate values of a corrected image. The reverse mapping method is performed by applying an area ratio calculated by comparing an area of an image before and after mounting a wide-angle lens of a thermal imaging camera.

The area ratio is defined as the ratio of the IFOV (resolution) area of the two-dimensional coordinate system to the corresponding area of the surface of the three-dimensional sphere.

IFOV area can be obtained according to the following equation (1).

[Equation 1]

IFOV Angle = (camera horizontal full FOV / horizontal FPA array, camera vertical full FOV / vertical FPA array)

Next, referring to FIG. 3 (b), four points [IF (U1, V1), (U2, V2), (U3, V3), (U4, V4) of the IFOV pixel having two-dimensional coordinates (u, v) values] From the three-dimensional spherical coordinates (r, θ, φ) according to the refraction of the wide-angle lens from the three-dimensional coordinate system ((x1, y1, z1), (x2, y2, z2), (x3, y3, z3), ( x4, y4, z4)]. The spatial area consisting of four points of the obtained three-coordinate system is obtained, and the area on the spherical surface corresponding to the IFOV pixel can be approximated.

ρ * IFOV area = approximation of area on spherical surface

The area ratio ρ on the UV map for all the IFOVs (FPAs) can be obtained and prepared in a table form to generate an area ratio table.

The area ratio table generated as described above is stored in the data storage unit 160.

In the case of a thermal image, each pixel contains corresponding temperature (radiation energy) information. When distortion occurs in an area of a pixel, distortion occurs in radiation energy information. As described above, the distortion (temperature distortion) of the radiation energy information includes a lot of radiation energy in a narrow FPA pixel area from the center of the image to the outside, similarly to the area distortion, and the temperature distortion becomes more severe toward the outside. Therefore, in the thermal image low image obtained from the thermal camera of the wide-angle lens, correction of the temperature distortion is essential.

The image processor 140 corrects the temperature and shape distortion of the distortion on the thermal image low image generated by the wide-angle lens by inverse conversion mapping.

In more detail, the total radiant energy input to the camera can be expressed by Equation 2.

[Equation 2]

W _tot = e * t * W _obj + (1-e) * t * W _amb + (1-t) * W _atm

Where e * t * W _obj = Emission energy of the object, e is the emissivity of the object and t is the transmittance of the atmosphere.

(1-e) * t * W _amb = radiation emitted from the surrounding environment is the value reflected from the surface of the object, and (1-e) is the reflectance of the object.

(1-t) * W _atm = atmospheric emission energy, (1-t) is the emissivity of the atmosphere.

The total radiation energy of the wide-angle lens satisfies Equation 3 below.

[Equation 3]

W _tot = ρ (e * t * W _obj + (1-e) * t * W _amb + (1-t) * W _atm )

In this way, the distortion ratio can be corrected with a small amount of calculation by applying the area ratio p.

The data storage unit 160 stores the data photographed by the photographing unit 110 and the data analyzed by the image processor 140. In addition, the data storage unit 160 stores a coordinate system area ratio table and a distortion correction program or data using the same. Storage media such as flash memory type, hard disk type, multimedia card micro type, card type memory (eg SD or XD memory, etc.), RAM ( RAM, ROM, or at least one type of storage medium.

4 is a flowchart illustrating a drone-based omnidirectional thermal image generation method according to a first embodiment of the present invention.

The drone photographs the search area with a thermal imaging camera having a wide-angle lens in flight to generate a thermal image raw (S110). As described above with reference to FIGS. 2 and 3, when the image is received by the wide-angle lens, since the radiation distortion necessarily follows, the thermal image low image generates distortion in the image and the temperature data.

The image processor 140 corrects the distortion of the thermal image low image generated by the wide-angle lens by using the pre-stored area ratio for each coordinate (S120).

More specifically, the temperature correction can be easily performed by multiplying the area value of the coordinates by the temperature value of the coordinates based on the area ratio of the coordinates stored in advance for each unit area of the three-dimensional coordinates of the thermal image having the distortion occurring. The correction of the temperature by applying the pre-stored area ratio for each coordinate is not large, and thus can be performed in the drone itself.

The corrected image is transmitted to the terrestrial thermal image processing apparatus (S130).

Image information may be transmitted together with the image transmission. The image information may include at least one of actual position information, average temperature information, maximum temperature information, and lowest temperature information of the captured image.

The thermal image processing apparatus outputs the image received from the drone as an omnidirectional image through the multimedia processor (S140).

Next, a method of displaying an omnidirectional thermal image by using a drone equipped with a plurality of thermal imaging cameras according to the second embodiment will be described with reference to FIGS. 5 to 9.

5 is a block diagram showing a schematic configuration of a drone according to a second embodiment of the present invention, Figure 6 is a view for explaining a drone-based omnidirectional thermal image generation method according to a second embodiment of the present invention 7 is a flowchart illustrating an arrangement of an image according to a second embodiment of the present invention. FIG. 8 is a diagram illustrating an example of a reliability map created according to the second embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a screen displaying image information of an ROI according to a second exemplary embodiment of the present invention.

First, a schematic configuration of a drone according to a second embodiment will be described with reference to FIG. 5.

The photographing unit 112 includes a plurality of thermal imaging cameras. Each thermal imaging camera is arranged such that the angles of view overlap one another in one direction.

The resolution of a thermal imaging camera according to an embodiment of the present invention is 320 * 240, and four or more are arranged in an area of 1920 * 1080.

In this case, when applied to a 1080P transmission modem it is possible to transmit more than 12 thermal image resolution 320.

The control unit 130 receives control information from the remote controller 200 through the wireless communication unit 120, and controls the starter 150 for adjusting the movement of the drone 100 according to the control information. Can fly.

In addition, the controller 130 arranges the image photographed by the photographing unit 110 during the flight of the drone 100 and the image photographed by the image processor 340, and the thermal image processing apparatus 300 through the wireless communication unit 120. ) Can be sent.

In addition, when a gyro sensor or an acceleration sensor is provided in the drone 100, the controller 130 generates posture information based on the sensing value measured by the sensor according to the control information, and then heats it through the wireless communication unit 120. The image may be transmitted to the image processing apparatus 300.

The image processor 142 arranges and transmits a thermal image row image generated by the plurality of thermal cameras corresponding to each arrangement column.

At this time, the image processing unit 142 does not perform the stitch operation of the image corresponding to the overlapping angles of view of the thermal imager, that is, the overlapping image, because a large amount of calculation is performed, and simply arranges the thermal image low image and transmits it through the wireless communication unit 120. do.

Next, a drone-based omnidirectional thermal image generation method according to a second embodiment of the present invention will be described with reference to FIG. 6.

A thermal image low image is generated by photographing the plurality of thermal imaging cameras equipped with drones (S210).

As shown in FIG. 7, the drone arranges images captured from a plurality of thermal imaging cameras in correspondence with actual arrangement positions (S220). The arrangement follows a set of alignment rules according to the arrangement position of the pre-stored camera. Arrange the images of overlapping angles of view next to each other.

The arranged image is transmitted in real time to the thermal image processing apparatus on the ground without stitching (S230).

The terrestrial thermal image processing apparatus stitches the image received from the drone and outputs the omnidirectional image (S240).

In order to perform a stitching operation based on a general RBG image, a black-and-white step is required. However, when using a thermal image low image as in the present embodiment, the black-and-white step may be omitted and a matching process may be performed immediately.

More specifically, a matching point map is generated by finding the same region using the SIFT, SIRF, and FAST techniques in the input thermal image.

However, temperature correction is performed on the overlapping area of the thermal image low image. Temperature correction can be performed by averaging the temperatures of the two zones.

In addition, as shown in Equation 4, reliability can be obtained by averaging the difference between the temperature values of the matching points of the two images.

[Equation 4]

% Confidence = (1-abs (A-B) / pixel maximum) 100

A and B are temperatures obtained from different thermal imaging cameras for the same matched region. The greater the difference in temperature value, the lower the reliability. As shown in FIG. 8, a reliability map in which reliability is calculated for all pixels can be generated.

In addition, as illustrated in FIG. 9, when a certain ROI is selected by the user in the output image, the average temperature, the reliability, the maximum temperature, and the minimum temperature of the selected ROI may be displayed on the screen. .

Next, a method of displaying an omnidirectional thermal image by using a drone equipped with a single thermal camera according to a third embodiment will be described with reference to FIGS. 10 to 11.

FIG. 10 is a block diagram illustrating a schematic configuration of a drone according to a third embodiment of the present invention. FIG. 11 is a view illustrating a method of generating a drone-based omnidirectional thermal image according to a third embodiment of the present invention. It is a flow chart.

First, a schematic configuration of a drone according to a third embodiment will be described with reference to FIG. 10.

The photographing unit 114 includes one thermal imaging camera. The photographing unit 114 stores a thermal image raw image captured by the provided thermal imaging camera in the data storage unit 164 or transmits the image to the image processing unit 144.

The control unit 134 receives the control information from the remote controller 200 through the wireless communication unit 120, and controls the starter 154 to adjust the movement of the drone 100 according to the control information drone 100 Can fly.

The controller 134 controls the starter 154 according to the control information, and controls the photographing unit 114 while performing a rotational motion in a horizontal stable state in the air, thereby performing a thermal imaging panorama. In the thermal imaging panorama, the faster the drone's rotational speed, the more distorted the image is.

In addition, the controller 134 captures the search area through the photographing unit 114 during the flight of the drone 100 and the thermal image processing apparatus (120) through the wireless communication unit 120 to process the image processed by the image processor 144 ( 300).

In addition, the control unit 134 generates the posture information based on the sensing value measured by the sensor only when the gyro sensor, the acceleration sensor, etc. are provided in the drone 100, and heats through the radio communication unit 120. The image may be transmitted to the image processing apparatus 300.

The image processor 144 generates a panoramic image by using each still image generated by the photographing unit 114. Since the generation of the panorama image is a general technique for those skilled in the art, detailed description thereof will be omitted.

The starter 154 has a horizontal gimbal.

The horizontal gimbal refers to a device that electronically prevents the shake when the drone is photographing the thermal imaging camera while rotating the horizontal. Usually, the gimbal is composed of two axes or three axes to transmit tilt information from the gyro sensor to the motor. It is operated by operating the motor to be in place.

Next, a method of generating a drone-based omnidirectional thermal image according to a third embodiment of the present invention will be described with reference to FIG. 11.

First, the controller 134 controls the drone, rotates 360 degrees during the flight, photographs the search area using the mounted thermal imaging camera, and captures a panoramic thermal still image. The controller 134 controls the photographing unit 114 to photograph a part of the image of each cut of the panorama so as to overlap in the rotation direction (S310).

The controller 134 transmits the panoramic image to the terrestrial thermal image processing apparatus (S320).

The terrestrial thermal image processing apparatus outputs an omnidirectional image by correcting an overlapped image in the panoramic image received from the drone (S330).

As described above, in the third exemplary embodiment of the present invention, a single thermal imaging camera may acquire an omnidirectional thermal image.

In the third embodiment of the present invention, since only one thermal imaging camera is required, it is suitable to be applied to a micro drone.

12 is a block diagram of a search system using a drone-based omnidirectional thermal image processing method.

Referring to FIG. 12, the search system using the drone-based omnidirectional thermal image processing method according to the present invention includes the drone 100, the remote controller 200, and the thermal image processing apparatus described above with reference to FIG. 1. 300 and the display device 400, and further includes a search server (500).

In FIG. 12, the same components as in FIG. 1 are denoted by the same reference numerals, and redundant description thereof will be omitted, and different portions from FIG. 1 will be mainly described.

The search server 500 transmits and receives information to and from the remote control apparatus 200 and the image information processing apparatus 300, and stores the received information, or controls the search request path, search area information, search target information, and the like. 200 and the image information processing apparatus 300.

The search server 500 stores route information, map information, singularity information, image information received from the drone 100 and flight information of the searched area.

The search server 500 monitors the flight so that the drone 100 does not go to a dangerous area, and may generate control information when the abnormality of the drone 100 occurs according to an immature operation of the user and transmit it to the drone 100. Through this, the drone 100 may control the flight based on the control information of the search server 500. In this case, the search server 500 may receive input information from an operation expert of the drone 100, and generate control information for operating the drone 100 based on the input information.

The remote controller 200 may control the drone 100 not to be out of the flight range of the selected course based on the search request path received from the search server 500.

 The image information processing apparatus 300 may provide an alarm when an object similar to the search target received from the search server 500 is found.

The alarm can be guided by texture or voice.

The database unit (not shown) of the search server 500 according to an embodiment may store information on at least one search server. Information about the search server includes identification of the search base (ID), the location of the search base, the type of drone that can land at the search base, the number and performance of charging facilities used to charge the battery of the drone, the current search. Information about the number and type of drones being charged at the base or waiting to take off may be included. The search server 500 may receive information about the ground base from the ground base in real time and update the information.

The database unit (not shown) of the search server 500 according to an embodiment may store information about at least one unmanned aerial vehicle. The information on the unmanned aerial vehicle includes identification information (ID) of the unmanned aerial vehicle, current location, speed, windproof performance, waterproof performance, flight time at battery fully charged, battery remaining time, flight time at current battery status, and reference distance. Information about the amount of battery required to move as much as the type of sensor mounted on the drone may be included. Each of the unmanned aerial vehicles may report to the search server 500 information about the unmanned aerial vehicle, such as a current position or a current battery status, periodically or in real time.

The database unit (not shown) of the search server 500 according to an embodiment may store geographic information about at least one area that needs to be monitored through an unmanned aerial vehicle. For example, geographic information about the at least one area may include map information including the altitude of the area that the drone is monitoring or information about a forest area that may be burned in the event of a fire.

The database unit (not shown) of the search server 500 according to an embodiment may store information related to a fire occurrence history or a fire occurrence history for at least one region. For example, the area where the fire is in progress, the area where the combustion is over, the area divided to perform surveillance operations by drone, the drone assigned to each divided area, the fire progress, At least one of the information on the result may be stored in the fire information unit.

As such, in the embodiment of the present invention, since the drone provides the omnidirectional thermal image, the drone-based search, the search time and the drone flight time can be shortened, and the number of operating drones and the number of operators can be reduced.

In an embodiment of the present invention, it is possible to search an area where people cannot go in real time.

In addition, in the embodiment of the present invention, it is not necessary to adjust the angle of the camera in the drone by applying a 360-degree omnidirectional camera capable of 360-degree image recording to the drone can reduce the weight of the drone.

In addition, in the embodiment of the present invention, by providing the recorded omnidirectional thermal image, temperature information and flight information together, the service (for example, search status notification, Terrain information description, voice guidance, and display of screen text).

The aforementioned method can be implemented through various means. For example, embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

In the case of a hardware implementation, a method according to embodiments of the present invention may include one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), and Programmable Logic Devices (PLDs). And field programmable gate arrays (FPGAs), processors, controllers, microcontrollers and microprocessors.

In the case of an implementation by firmware or software, the method according to the embodiments of the present invention may be implemented in the form of a module, a procedure, or a function that performs the functions or operations described above. The software code may be stored in a memory unit and driven by a processor. The memory unit may be located inside or outside the processor, and may exchange data with the processor by various known means.

Embodiments disclosed herein have been described with reference to the accompanying drawings. As described above, the embodiments shown in each drawing should not be construed as limiting, but may be combined with each other by those skilled in the art, and some components may be omitted.

Here, the terms or words used in the present specification and claims should not be construed as being limited to the ordinary or dictionary meanings, but should be interpreted as meanings and concepts corresponding to the technical spirit disclosed in the present specification.

Therefore, the embodiments described in the present specification and the configuration shown in the drawings are merely exemplary embodiments disclosed herein, and do not represent all of the technical ideas disclosed herein, and various equivalents may be substituted for them at the time of the present application. It should be understood that there may be water and variations.

100: drone
110: shooting unit
120: wireless communication unit
130: control unit
140: the image processing unit
150: starting unit
200: remote control device
300: thermal image processing device
400: display device

Claims (12)

Generating, by the drone, a thermal image by capturing a thermal imaging camera having a wide-angle lens;
Correcting the temperature and shape distortion of the drone by the wide-angle lens with respect to the distortion of the thermal image;
The drone transmitting the corrected image to a terrestrial thermal image processing apparatus; And
Outputting the image received from the drone as an omnidirectional image by a terrestrial thermal image processing apparatus;
Including,
The method of claim 1, wherein the correction of the temperature distortion and the image distortion is performed by applying an area ratio calculated by comparing images before and after mounting the wide-angle lens of the thermal imaging camera. The method of claim 1, wherein the correction of the temperature distortion and the image distortion is performed by inverse conversion mapping. The method according to claim 2, wherein the area ratio is defined as the ratio of the IFOV area of the two-dimensional coordinate system to the corresponding area of the surface of the three-dimensional sphere. A drone generating a thermal image by capturing a plurality of thermal cameras;
The drone arranging a plurality of thermal image raw images adjacent to each other where the angles of view overlap;
Transmitting the drone-prepared image to a terrestrial thermal image processing apparatus; And
Stitching the image received from the drone and outputting it as an omnidirectional image by the terrestrial thermal image processing apparatus;
Drone-based omnidirectional thermal image generation method comprising a. The method of claim 4, wherein the plurality of thermal imaging cameras are arranged so that the angle of view overlap each other,
Between the disposing step and the transmitting step,
And a drone-based temperature correction of the region where the angles of view overlap. Generating a panoramic still image by capturing a search area by a thermal imaging camera equipped with a drone rotating by 360 degrees, wherein the drone photographs a part of an image of each cut of the panorama in a rotational direction;
Transmitting, by the drone, the panoramic still image to a terrestrial thermal image processing apparatus; And
Outputting the omnidirectional image by correcting the overlapped image in the panoramic still image received from the drone
Drone-based omnidirectional thermal image generation method comprising a. The method of claim 6, wherein the correcting the overlapped image in the panoramic still image and outputting the omnidirectional image comprises:
A drone-based omnidirectional thermal image generation method according to claim 1, wherein the temperature is corrected by an average value of temperatures of the overlapped first and second images. A drone mounted to a thermal imaging camera having a wide-angle lens, the drone photographing an image by the thermal imaging camera to generate a thermal image low image;
Thermal image processing apparatus for outputting the image received from the drone as an omnidirectional image
Including,
The drone corrects temperature and shape distortion for distortion of the thermal image raw image, and transmits the corrected image to the thermal image processing apparatus,
And the area ratio calculated by comparing images before and after mounting a wide-angle lens of the thermal imaging camera is used to correct the temperature distortion and the image distortion. A drone equipped with a plurality of thermal imaging cameras and generating an image of a thermal image by capturing an image by the thermal imaging camera;
A thermal image processing apparatus for stitching an image received from the drone and outputting it as an omnidirectional image
Including,
And a drone to arrange a plurality of thermal image raw images adjacent to each other, and transmit the disposed image to a terrestrial thermal image processing apparatus. 10. The system of claim 9, wherein the drone performs temperature correction of an area where the angle of view overlaps. A drone equipped with a single thermal imaging camera and rotating 360 degrees to take a panoramic still image;
A thermal image processing apparatus for outputting an omnidirectional image by correcting a superimposed image from a panoramic still image received from the drone
Including,
The drone photographs a part of the image of each cut of the panorama so as to overlap in a rotational direction, and transmits the panoramic image to the thermal image processing apparatus on the ground. 12. The system of claim 11, wherein the drone includes a horizontal gimbal.

Images