KR102298645B1 - 3D modeling system of underwater surfaces using infrared thermal imaging camera and drone - Google Patents

Abstract

An embodiment relates to an underwater surface three-dimensional modeling system using an infrared thermal imaging camera and an underwater drone. More specifically, the system includes: an underwater drone including sonar and a heating element, and measuring underwater surface information while being driven in an investigation area; an infrared thermal imaging camera acquiring a thermal image information by photographing the whole investigation area during the driving of the underwater drone; and a control server processing the thermal image information acquired by the infrared thermal imaging camera and mapping coordinate information of the underwater drone derived from a position of the heating element with the underwater surface information measured by the sonar to three-dimensionally model an underwater surface in a set area. Therefore, accurate position information of the underwater drone acquired through the infrared thermal imaging camera and the heating element attached on the underwater drone is mapped with underwater image information measured by the sonar of the underwater drone, and, as a result, precise shape information of submarine topography or an underwater surface can be acquired.

Description

3D modeling system of underwater surfaces using infrared thermal imaging camera and drone}

The content disclosed in the present specification relates to a three-dimensional modeling system for an underwater surface, and more specifically, a water drone that measures the underwater surface and an infrared thermal imaging camera that captures the operation area of the underwater drone. Accurate shape information of the underwater surface It relates to an underwater surface three-dimensional modeling system using an infrared thermal imaging camera and underwater drone that can be realized in three dimensions.

Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application, and inclusion in this section is not an admission that it is prior art.

Small airplanes, helicopters, unmanned submersibles, and unmanned waters that are not normally carried by humans

Jung and others are called drones. Of these, those that float above the water surface or operate underwater

A drone is called a surface drone or an underwater drone.

However, by mounting a sonar on a surface drone or an underwater drone, the seabed, underwater surface, or terrain

to measure or explore.

In particular, surface drones or underwater drones are used in nuclear power plants as underwater structures or underwater charts.

It also measures the surface and monitors the condition of structures such as dams, dams, ports, and weirs or the seabed environment.

It is used for multiple purposes, such as turing.

And even if topographical and geographical information of the seabed or underwater surface is collected,

Because erosion and deposition are frequent due to flowing water currents,

update is required

Conventionally, water or underwater drones are used to collect information on the seabed or underwater surface.

By mounting a sonar on the pole, it emits sound waves, receives the reflected sound waves, and analyzes the terrain,

Geographic information was collected.

However, in the case of underwater drones, GPS is basically used to determine the location.

However, in the case of a large range, the GPS error is not very important, but it is possible to explore a narrow range or area.

In this case, the position error becomes a big problem.

In the case of GPS, the accuracy is usually only 5 ~ 10m, and the very good performance is also

Accuracy up to 2m is possible. Therefore, GPS does not matter where the error of several meters is not important.

It can be used to acquire location information of drones in open areas or in the sea over a wide range, but

When accurately modeling a narrow range of underwater surfaces, these errors become a major problem.

cause

In particular, the subsea topography of a narrow area within a subsea structure, such as a nuclear power plant, is

Due to the various facilities, equipment, and structure geometries, the use of GPS for underwater drones is accurate.

It is difficult to obtain location information.

This will eventually lead to the acquisition of incorrect underwater surface or topographic information, which may lead to the maintenance of the power plant.

may cause problems with the

(Patent Document 1) KR100950979 Y1

(Patent Document 2) KR101119400 Y1

(Patent Document 3) KR1020130096854 A

The disclosed content is the precise shape information of the seabed topography or the underwater surface by mapping the precise location information of the underwater drone obtained using the heating element attached to the underwater drone and the infrared thermal imaging camera, and the underwater image information measured from the sonar of the underwater drone. The purpose of this study is to provide an underwater surface 3D modeling system using an infrared thermal imaging camera and underwater drone.

An underwater surface three-dimensional modeling system using an infrared thermal imaging camera and an underwater drone according to an embodiment,

A sonar and a heating element are provided, and the underwater surface information is measured while operating within the survey area.

Underwater drones to be determined;

While the underwater drone is operating, the entire investigation area is photographed and deteriorated in real time.

an infrared thermal imaging camera for acquiring image information;

Process the thermal image information obtained from the infrared thermal imaging camera and

Coordinate information of the underwater drone derived from the location of the heating body and the number measured from the sonar

a control information processing device for modeling the underwater surface in a preset area in three dimensions by mapping the intermediate surface information; is made, including

The control information processing device,

a) The position of the heating element by processing the image information obtained from the infrared thermal imaging camera

is detected and the position coordinates of the underwater drone are calculated from the position information of the detected heating element.

out,

b) processing the underwater surface image information measured in the sonar,

c) modeling the underwater surface as a three-dimensional image by mapping the derived position coordinates of the underwater drone and the processed underwater surface image information,

(a)) to calculate and derive the position coordinates of the underwater drone,

a-1) storing the thermal image information transmitted from the infrared thermal imaging camera;

a-2) displaying the stored thermal image information on a screen;

a-3) processing the thermal image information of a preset area of the screen to detect an area above a reference temperature, and obtaining location information of the heating element from the detected area;

a-4) Displaying the processed image information on a colored screen according to temperature distribution,

a-5) calculating the actual position coordinates of the underwater drone from the obtained position information of the heating element; is characterized by

According to the embodiments, it is possible to solve the problem that a large error occurs in the position coordinates of the underwater drone when the conventional GPS is used in a narrow or closed survey area. That is, the precise position coordinates of the underwater drone can be derived while tracking the heating element with an infrared thermal imaging camera, so the precision of the underwater surface shape of the irradiation area is greatly improved.

1 is a configuration diagram showing an underwater surface three-dimensional modeling system using an infrared thermal imaging camera and an underwater drone according to an embodiment;
FIG. 2 is a diagram showing a schematic configuration of the control information processing apparatus of FIG. 1; FIG.
3 is a block diagram illustrating the configuration of the underwater drone of FIG. 1 as an example;
4 is a block diagram showing the configuration of a PLC controller applied to the underwater drone of FIG.
5 is a photograph showing a display screen of a thermal image according to an embodiment;
6 is a photograph showing a display screen of a location of a heating element according to an embodiment;
7 is a photograph showing a display screen of an underwater surface image according to an embodiment;
8 is a photograph showing a display screen of 3D modeling according to an embodiment;

FIG. 1 is a configuration diagram illustrating an underwater surface three-dimensional modeling system using an infrared thermal imaging camera and an underwater drone according to an embodiment, and FIG. 2 is a diagram illustrating a schematic configuration of the control information processing device shown in FIG. 1 .

1 and 2, the system according to one embodiment is largely composed of an underwater drone 100, an infrared thermal imaging camera 200, and a control information processing device (300).

The underwater drone 100 is a device that can be operated remotely in a floating state on the surface or underwater. The underwater drone 100 is provided with a sonar 110 and a heating element 120 . In this case, the sonar 110 radiates and receives ultrasonic waves to the underwater surface to measure the three-dimensional shape of the underwater surface. To this end, the sonar may be composed of a transmitting module for generating and emitting an ordinary ultrasonic wave, a receiving module for receiving the reflected ultrasonic wave, and a transmitting module for converting the received information into data and transmitting it to the control information processing device. And at this time, the sonar 110 is installed on the lower surface of the underwater drone (100). In addition, in this case, the heating element 120 should have a relatively higher temperature than the ambient temperature such as the underwater drone or water to be easily recognized by the infrared thermal imaging camera 200 . Therefore, the heating element 120 is suitable for a waterproof-treated lamp, it will be preferable to use an LED lamp. In addition, the heating element 120 is preferably attached to the upper surface of the underwater drone (100). And, the underwater drone 100 is provided with an inertial measurement sensor. The inertial measurement sensor (IMU, Inertial Measurement Unit) is a 3-axis accelerometer sensor, a 3-axis gyro sensor, and a 3-axis geomagnetic sensor, and is for measuring the speed, acceleration, direction, gravity, etc. of the underwater drone. Information measured by the inertial measurement sensor may also be transmitted to the control information processing device 300 .

The infrared thermal imaging camera 200 generates a thermal image capable of tracking the underwater drone 100 by photographing the entire area corresponding to the area of the underwater surface to be irradiated. In general, infrared cameras can be divided into infrared night vision cameras and infrared thermal imaging cameras. An infrared thermal imaging camera is a device that tracks and detects heat of a target object in a non-contact manner and displays the temperature distribution on the screen. And, it is an equipment that converts the frequency response sensed from infrared into an electrical signal and implements it as an image system. A general camera has the same structure as the human eye, so it captures images similar to what our eyes see, but a thermal imaging camera is a special equipment that uses only heat to take pictures. At this time, it is possible to identify the object according to the amount of heat generated, so the object can be identified regardless of the presence or absence of an obstacle or light. In order to solve the problem that a large error occurs when measuring the position of an underwater drone with GPS in an existing narrow space or a closed space, the position error can be greatly reduced by using the infrared thermal imaging camera in one embodiment. Specifically, the infrared thermal imaging camera 200 is installed so as to photograph a surface area sufficiently including a preset irradiation area. To this end, a support can be built on one side of the irradiation area and the infrared thermal imaging camera 200 can be installed on the upper end of the support. However, other methods are acceptable as long as the entire investigation area can be photographed. In this state, when the underwater drone 100 is photographed with the infrared thermal imaging camera 200 while operating within the irradiation area, a thermal image of the underwater drone 100 appears, and the heating element 120 is relatively high temperature. Since it appears in a color that is clearly recognized, the position of the underwater drone 100 can be accurately tracked and recognized.

The infrared thermal imaging camera 200 includes a transmission module that acquires photographed thermal image information in real time, converts it into data, and transmits it to the control information processing device 300 .

The control information processing device 300 is an underwater drone 100 obtained by processing the underwater surface image information measured by the sonar 110 of the underwater drone 100 and the image information obtained from the infrared thermal imaging camera 200 ) Finally, a three-dimensional underwater surface image is generated by mapping the position coordinates of the This control information processing device processes the image information transmitted from the infrared thermal imaging camera 200 to detect the position of the heating element 120 and the position coordinates of the underwater drone 100 from the detected position information of the heating element 120 . is calculated and derived. In detail, the control information processing device 311 stores the thermal image information transmitted from the transmission module of the infrared thermal imaging camera 200 in real time. Then, the stored image information is displayed on the screen. Through this, the manager can actually monitor the operation status of the underwater drone 100 . In addition, the control information processing device processes the thermal image image information within the irradiation area (range) set by the administrator on the screen of the thermal image image output unit 312 and converts it into temperature distribution information, and the processed temperature distribution information A region above a specific temperature value is detected from among the regions, and the position of the detected region is recognized and acquired as position information of the heating element. In other words, image warping of the thermal image information is performed, and after binarizing the thermal image, a region above a specific temperature value is detected. And the coordinates corresponding to the detected values are determined as the positional coordinates of the heating element 120 . Also, the temperature distribution image information may be displayed on a colored screen. That is, the manager can monitor the position of the heating element 120, that is, the operating condition and position of the underwater drone 100 on the screen. And, the control information processing device 315 finally derives the actual position coordinates of the underwater drone 100 by processing the obtained position information of the heating element 120 . Next, the control information processing device stores the underwater surface image information measured and transmitted by the sonar 110, and displays the stored underwater surface image information on the screen. That is, the manager can monitor the shape of the underwater surface or the seabed surface measured by the sonar 110 in real time. The underwater surface image information is also three-dimensionally mapped. For reference, the control information processing device 300 receives and processes the inertial measurement value information measured by the inertial measurement sensor, and then derives the correct position coordinates of the underwater drone 100 . Next, the actual position coordinates of the underwater drone 100 derived in this way and the stored underwater surface image information are received and mapped to each other, and the mapped information is converted into a three-dimensional image and displayed. That is, the specific position coordinate value of the underwater drone 100 and the shape of the underwater surface measured using the sonar 110 at the specific position coordinate value are matched to form a point cloud. As a specific example, the point cloud formation preferentially models the processed underwater surface image information by a preset three-dimensional model format based on modeling for the body surface of the underwater surface. And, it consists of an operation of mapping the calculated actual position coordinates of the underwater drone and the modeled underwater surface image information to form a point cloud. Additionally, also in this case, first, the matching operation is performed in advance by calculating the deep learning parameters for the object region through deep learning for 2D object region segmentation from only a plurality of different infrared thermal images in advance. Build the model. Then, object 2D region segmentation is performed from the corresponding input infrared thermal image by the object 2D region segmentation deep learning model. Next, by performing logical AND between the object 2D region division result and depth data, depth information of the object is obtained. So, by performing the 3D object region segmentation of the object by applying the perspective transformation to the depth information of the object, the xyz of the object derived therefrom is mapped with the position coordinates of the underwater drone above to form a point cloud. In this case, it is possible to convert and show a certain number of such point cloud-formed data into a three-dimensional three-dimensional image shape. At this time, it is possible to clearly distinguish the color according to the height of the underwater surface. Since three-dimensional modeling with a point cloud is a known technique, a detailed description thereof will be omitted.

3 is a block diagram illustrating the configuration of the underwater drone 100 of FIG. 1 as an example.

As shown in Figure 3, the underwater drone 100 according to an embodiment is a sonar, a sensor unit for detecting the operating state, a sensor fusion device for estimating the operating state as a result of the detection of the operating state, and to control each unit It includes a PLC controller and a driving unit of the underwater drone.

In this case, the PLC controller includes an IOT module that communicates with the control information processing device the position coordinates by the heating element of the underwater drone and the underwater surface information by the sonar as IOT data.

The sonar measures the underwater surface information of the area desired by the manager, for example, in the survey area during operation.

When the underwater drone is operated, the sensor unit includes at least one sensor for detecting a running state. In this case, the sensor unit includes any one or more of a pressure sensor, a three-axis acceleration sensor, a three-axis gyro sensor, and a geomagnetic sensor to measure data necessary for driving and performing a mission.

The sensor fusion device estimates the driving state by combining the driving state results sensed by the sensor unit. For example, the flight state is estimated by calculating the combination of pressure and acceleration during operation.

The PLC controller provides the coordinate information of the underwater drone derived from the location of the heating element and the underwater surface information measured in the sonar to a pre-registered control information processing device, and operates according to the operating state estimated by the sensor fusion device to control the action.

The driving unit drives the driving operation by the control of the PLC controller, for example, by the PWM control of the PLC controller. In this case, the driving unit includes a transmission, a motor, a propeller, and the like, and drives the driving operation by differently changing the rotational speed and the rotational direction according to the estimated flight state.

4 is a block diagram illustrating the configuration of a PLC controller applied to the underwater drone 100 of FIG. 3 .

As shown in FIG. 4, the PLC controller applied to the underwater drone according to an embodiment includes a part for signal input and output such as the sonar and the sensor part, a part for signal processing, and a part for communication with the control information processing device. It is made to include all inclusively.

To this end, the PLC controller according to an embodiment includes a power module, an A/I module for signal input/output and signal processing of the sonar and sensor units, an A/O module, an A/D module, a D/A module, and a D/I It includes all of the module, the D/O module, the IOT module for communication with the control information processing device, and the CPU module for controlling each module.

The power module is for supplying power to the underwater drone itself.

The A/I module is to integrally receive analog information including an analog measurement result by the sonar (analog sonar) and an analog driving state detection result by the sensor fusion device.

The A/O module integrally outputs analog information including analog control information to a corresponding driving device.

The A/D module integrates digital conversion of analog information including an analog measurement result by the sonar (analog sonar) and an analog driving state detection result such as acceleration and geomagnetism by the sensor fusion device.

The D/A module converts digital information by a digital unit to analog.

The D/I module receives digital information including a digital measurement result by the sonar (digital sonar) and a digital flight state detection result by the sensor fusion device as an integrated input.

The D/O module outputs digital information including digital control information according to a control signal of the control information processing device to a driving unit or the like.

The IOT module communicates with the control information processing device as IOT data for the coordinate information of the underwater drone derived from the location of the heating element and the underwater surface information measured by the sonar. In this case, the IOT module has its own wired/wireless communication module, for example, a LoRa module, and provides the measurement result to the control information processing device on the ground.

The CPU module is a basic CPU module of the PLC controller and controls each module.

In addition, if it is further explained in this regard, modules with various functions are required for the existing PLC system used in various environments, and accordingly, PLC manufacturers provide various modules that satisfy the requirements of users.

For example, a module having various functions, such as a digital input/output module, an analog input/output module, and a communication module, is used in a PLC system, and a system desired by a user is built through these various modules.

For example, the technology of Patent Document KR101778333 Y1 is an invention registered as such a technology, and specifically relates to a PLC system having a diagnostic module for diagnosing whether an output module of the PLC is defective in operation.

The above-described IOT module according to an embodiment uses these points to provide a PLC that provides an IOT function from the above-described IOT module, and through this, further optimized monitoring and control from the device side is made easy. .

Accordingly, the above-described PLC controller according to an embodiment basically includes an input module that receives all signals from various digital and analog sensor units, a CPU module that is a central control unit, and an output module that outputs a control signal to a control object. include

In addition, the PLC controller includes an IOT module that additionally performs an IOT function in the PLC itself, so that the CPU module interworks with the input/output module by such an IOT module to perform an IOT function from the corresponding control logic. , and communicates with the control information processing device on the ground.

Additionally, in addition to the above, the CPU module performs an alarm to the control information processing device by the IOT module when the operating state estimated by the sensor fusion device corresponds to a preset dangerous driving state. .

Hereinafter, a three-dimensional modeling process of an underwater surface according to an embodiment will be described with reference to FIGS. 5 to 8 together.

Specifically, FIG. 5 is a photograph showing a display screen of a thermal image, and FIG. 6 is a photograph illustrating a display screen of a location of a heating element. And, Figure 7 is a photograph showing the display screen of the underwater surface image, Figure 8 is a photograph showing the display screen of the three-dimensional modeling.

Here, the operator is an operator who controls the underwater drone 100 and an engineer who operates the control server 300 .

First, in the three-dimensional modeling process of the underwater surface according to an embodiment, the infrared thermal imaging camera 200 is installed at a position and height to designate an irradiation target area (generally a rectangular), and to photograph the entire area including the irradiation area. do.

Then, the engineer checks the thermal image, designates the irradiation area as shown in the dotted rectangle in FIG. 5, and measures the horizontal and vertical actual distance of the irradiation area.

In addition, the irradiation area is divided into one or more grids for the location of the heating element. When the engineer enters the number of grids, it is automatically partitioned as shown in FIG. 6 .

After the setting as described above, the operator operates the underwater drone 100 to which the heating element 120 is attached within the irradiation area.

The operator may operate the underwater drone 100 along a predetermined path to scan all areas within the irradiation area. For example, the underwater drone can be operated over the entire area of the irradiation area by moving it to the left or right while reciprocating in the front-rear direction from one side of the rectangular irradiation area.

At this time, the underwater surface image information measured by the sonar 110 and the thermal image image information photographed by the infrared thermal imaging camera 200 are synchronized while being transmitted to the control information processing device 300 in real time.

And the thermal image image information among the information transmitted to the control information processing device 300 undergoes underwater drone location recognition, and the real-time location coordinates of the underwater drone 100 are derived and mapped.

Next, the underwater surface image information is mapped to the two pieces of information after the underwater surface information processing, and provides the underwater surface shape of the irradiation area as a three-dimensional color image as shown in FIG. 8 .

100: underwater drone 110: sonar
120: heating element 200: infrared thermal imaging camera
300: control information processing device

Claims (5)

An underwater drone equipped with a sonar and a heating element, and measuring underwater surface information while operating within the investigation area;
an infrared thermal imaging camera that captures the entire irradiation area while the underwater drone is operating to acquire thermal image information in real time;
By processing the thermal image information obtained from the infrared thermal imaging camera and mapping the coordinate information of the underwater drone derived from the location of the heating element and the underwater surface information measured in the sonar, the underwater surface within a preset area is three-dimensionally Modeling control information processing device; is made, including
The control information processing device,
a) by processing the image information obtained from the infrared thermal imaging camera to detect the position of the heating element, and calculate and derive the position coordinates of the underwater drone from the detected position information of the heating element,
b) processing the underwater surface image information measured in the sonar,
c) mapping the derived position coordinates of the underwater drone and the processed underwater surface image information to model the underwater surface as a three-dimensional image, and calculating and deriving the position coordinates of the underwater drone (a)),
a-1) storing the thermal image information transmitted from the infrared thermal imaging camera;
a-2) displaying the stored thermal image information on a screen;
a-3) processing the thermal image information of a preset area of the screen to detect an area above a reference temperature, and obtaining position information of the heating element from the detected area;
a-4) Displaying the processed image information on a colored screen according to temperature distribution,
a-5) Calculate the actual position coordinates of the underwater drone from the obtained position information of the heating element,

Modeling with the three-dimensional image (c))
c-1) Map the calculated actual position coordinates of the underwater drone and the processed underwater surface image information to form a point cloud,
c-2) converting the mapped point cloud into a three-dimensional color image and outputting it; An underwater surface three-dimensional modeling system using an infrared thermal imaging camera and an underwater drone, characterized in that delete The method according to claim 1,
(c-1)) by mapping the calculated actual position coordinates of the underwater drone and the processed underwater surface image information to form a point cloud
c-1-1) preferentially modeling the processed underwater surface image information by a preset three-dimensional model format based on modeling for the body surface of the underwater surface;
c-1-2) mapping the calculated actual position coordinates of the underwater drone with the modeled underwater surface image information to form a point cloud; An underwater surface three-dimensional modeling system using an infrared thermal imaging camera and an underwater drone, characterized in that The method according to claim 1,
The underwater drone,
a sonar measuring the underwater surface information in the survey area;
a sensor unit including at least one sensor for detecting a driving state when the driving is performed;
a sensor fusion device for estimating the driving state by combining the driving state results sensed by the sensor unit;
Providing the coordinate information of the underwater drone derived from the location of the heating element and the underwater surface information measured by the sonar to a pre-registered control information processing device, and controlling the operation operation according to the operation state estimated by the sensor fusion device PLC controller; and
a driving unit for driving the driving operation under the control of the PLC controller; contains,
The PLC controller,
power module of the underwater drone itself;
An A/I module that integrally receives analog information including an analog measurement result by an analog sonar and an analog driving state detection result by the sensor fusion device;
An A/O module that integrally outputs analog information including analog control information;
an A/D module for integrally converting analog information including an analog measurement result by the analog sonar and an analog driving state detection result by the sensor fusion device;
D/A module for converting digital information by digital unit to analog;
a D/I module that integrally receives digital information including a digital measurement result by a digital sonar and a digital driving state detection result by the sensor fusion device;
a D/O module for integrally outputting digital information including digital control information according to a control signal by the control information processing device;
IOT module for communicating as IOT data with the control information processing device for the coordinate information of the underwater drone derived from the location of the heating element and the underwater surface information measured by the sonar; and
CPU module for controlling each module; An underwater surface three-dimensional modeling system using an infrared thermal imaging camera and an underwater drone, characterized in that it comprises a. 5. The method according to claim 4,
The CPU module is
when the driving state estimated by the sensor fusion device corresponds to a preset dangerous driving state, performing an alarm to the control information processing device by the IOT module; An underwater surface three-dimensional modeling system using an infrared thermal imaging camera and an underwater drone, characterized in that

Images