KR102517267B1 - System and method for inspecting marine bridge - Google Patents

Abstract

The present invention relates to a system and method for inspecting a marine bridge, which inspect a marine bridge using a drone and artificial intelligence technology. The system includes: a communication unit which receives marine bridge information obtained through sensors mounted on a drone; and a processing unit which analyzes the marine bridge information using an artificial intelligence model and checks surface abnormalities of the sea bridge.

Description

Marine bridge inspection system and method {SYSTEM AND METHOD FOR INSPECTING MARINE BRIDGE}

The present invention relates to a marine bridge inspection system and method for inspecting marine bridges using drones and artificial intelligence technology.

Marine bridges require professional safety inspection and maintenance skills, and related local governments manage marine bridges by consigning safety inspection and glass management tasks to specialized institutions. However, marine bridges have a large structure, and there are sections where visual inspection is impossible due to special regional conditions of the sea, so even safety inspections by specialized institutions may have realistic blind spots. In particular, the high pylons of marine bridges are very difficult to access, so in most cases they cannot be inspected even during a precise safety inspection. can suffer

In the present invention, after obtaining marine bridge information using sensors mounted on a drone, the acquired marine bridge information is analyzed with an artificial intelligence model to detect surface anomalies of marine bridges, and to determine the type and location of surface abnormalities of the detected marine bridges. It is intended to provide a marine bridge inspection system and method that can be recorded and managed.

A marine bridge inspection system according to an embodiment of the present invention includes a communication unit that receives marine bridge information obtained through a sensor mounted on a drone, and analyzes the marine bridge information with an artificial intelligence model to check surface abnormalities of the marine bridge. Characterized in that it includes a processing unit.

The processing unit is characterized in that the first artificial intelligence model is used to select surface abnormality information of the marine bridge from among the marine bridge information.

The first artificial intelligence model is characterized in that it is implemented as a classification model based on a convolutional neural network (CNN) or a vision transformer (ViT).

The processing unit is characterized in that the selected surface abnormality information of the marine bridge is analyzed with a second artificial intelligence model to determine the type and location of the surface abnormality of the marine bridge.

The second artificial intelligence model is characterized in that it is implemented as a fully convolutional-based segmentation model, a 1-stage object detection model, or a 2-stage object detection model.

The processing unit may display a surface abnormality type and a surface abnormality location of the marine bridge based on a 3D marine bridge model.

The drone is characterized in that the marine bridge information is obtained using at least one of an image sensor and lidar.

A marine bridge inspection method according to an embodiment of the present invention includes the steps of receiving marine bridge information obtained through a sensor mounted on a drone, and analyzing the marine bridge information with an artificial intelligence model to check surface abnormalities of the marine bridge. It is characterized by including steps.

The checking step may include selecting surface abnormality information of the marine bridge from information on the surface of the marine bridge using a first artificial intelligence model, and analyzing the selected surface abnormality information of the marine bridge with a second artificial intelligence model to analyze the surface of the marine bridge and determining the type of abnormality and the location of the surface abnormality.

The first artificial intelligence model is characterized in that it is implemented as a classification model based on a convolutional neural network (CNN) or a vision transformer (ViT).

The second artificial intelligence model is characterized in that it is implemented as a fully convolutional-based segmentation model, a 1-stage object detection model, or a 2-stage object detection model.

The marine bridge inspection method may further include displaying a surface abnormality type and a surface abnormality location of the marine bridge based on a 3D marine bridge model.

The receiving of the marine bridge information includes acquiring the marine bridge information using at least one of an image sensor and a lidar by the drone, and transmitting the acquired marine bridge information by the drone. to be

The present invention acquires marine bridge information using sensors mounted on drones, analyzes it with an artificial intelligence model, detects surface anomalies of marine bridges, and provides the detection results as a 3D modeling-based platform, so that even non-professional personnel can build marine bridges. It can be managed, and the inspection method can be objectified to improve inspection work efficiency.

1 is a configuration diagram showing a marine bridge inspection system according to an embodiment of the present invention.
2 is a diagram illustrating an AI-based maritime bridge information processing process according to an embodiment of the present invention.
3 is a flowchart illustrating a marine bridge inspection method according to an embodiment of the present invention.

Terms such as "comprise", "comprise", and "having" described in this specification mean that the corresponding component may be inherent unless otherwise specified, and therefore do not exclude other components but constitute other components. This means that it can contain more elements.

In addition, terms such as “… unit”, “… unit”, and “module” described in this specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software. can In addition, articles such as “one,” “one,” and “the” are meant to include both the singular and plural in the context of describing the present invention, unless otherwise indicated herein or clearly contradicted by the context. can be used

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram showing a marine bridge inspection system according to an embodiment of the present invention.

Referring to FIG. 1 , the marine bridge inspection system may include a drone 100 and a server 200. The drone 100 refers to an unmanned aerial vehicle (UAV). The drone 100 may move to the sea bridge according to the instructions of the server 200 and obtain sea bridge information through at least one sensor mounted thereon. The server 200 may analyze the marine bridge information acquired by the drone 100 with an artificial intelligence model to check the abnormal state of the marine bridge.

The drone 100 may include a communication unit 110, a positioning unit 120, a driving unit 130, a detection unit 140, a storage unit 150, a power supply unit 160, and a control unit 170.

The communication unit 110 may support communication between the drone 100 and the server 200. The communication unit 110 may be a wireless communication circuit (eg, a cellular communication circuit, a short-distance wireless communication circuit, or a global navigation satellite system (GNSS) communication circuit) or a wired communication circuit (eg, a local area network (LAN) communication circuit, or power line communication). circuit) can be used to transmit and receive data.

The positioning unit 120 may measure the current location of the drone 100, that is, the drone location. The positioning unit 120 may be implemented as a Global Positioning System (GPS) receiver. The positioning unit 120 may calculate the current location of the drone 100 using signals transmitted from three or more GPS satellites.

The driving unit 130 may control flight of the drone 100 according to a control command (control signal) of the server 200 received through the communication unit 110 . For example, the driving unit 130 may control the motor output, that is, the rotational speed of the motor. The driving unit 130 may be implemented as an electronic speed controller (ESC). The motor is driven under the control of the driving unit 130 and may be combined with the propeller to rotate together. The driving unit 130 may control the flight of the drone 100 using a difference in rotation speed of the propeller.

The detection unit 140 may obtain (acquire) marine bridge information (marine bridge surface information) using various sensors mounted on the drone 100 . The detection unit 140 may obtain image information, that is, image data of the bridge by using an image sensor (or camera) mounted on the drone 100. For example, the detection unit 140 may capture an image of the surface of the pylon of the marine bridge using an image sensor.

In addition, the detection unit 140 may acquire marine bridge information through Light Detection And Ranging (LiDAR) or the like. The detection unit 140 may obtain lidar data for the bridge over the sea using lidar. In this embodiment, although the detection unit 140 uses an image sensor and / or LIDAR to obtain marine bridge information, it is described, but it is not limited thereto, and a radar (Radio Detecting And Ranging, radar) and / or an ultrasonic sensor, etc. It may be implemented to obtain marine bridge information by additionally using it.

The storage unit 150 may store marine bridge information detected by the detection unit 140 . The storage unit 150 may store the flight path of the drone 100 received through the communication unit 110 . The flight path is path information for reaching a specific point on the sea bridge, and may be provided from the server 200 . The storage unit 150 may store programs for operation of the control unit 170 .

The storage unit 150 includes a flash memory, a hard disk, a solid state disk (SSD), a secure digital card (SD card), a random access memory (RAM), a static random access memory (SRAM), It may be implemented as at least one of storage media such as read only memory (ROM), programmable read only memory (PROM), electrically erasable and programmable ROM (EEPROM), and erasable and programmable ROM (EPROM).

The power supply unit 160 may supply power necessary for the operation of each of the components mounted on the drone 100. The power supply unit 160 may receive power from a battery or fuel cell installed in the drone 100 and supply it to each component.

The control unit 170 transmits motion information obtained through various sensors (eg, gyro, acceleration sensor, air pressure sensor, etc.) mounted on the drone 100 and position information obtained through the positioning unit 120 to the drive unit 130. send (forward) to In addition, the control unit 170 may receive a control signal transmitted from the server 200 through the communication unit 110 and transmit it to the driving unit 130 .

The control unit 170 may obtain marine bridge information by using the detection unit 140 after bringing the drone 100 close to the main tower of the marine bridge located at a high altitude along the flight path.

Specifically, the control unit 170 may capture an image of the surface of the pylons of the bridge by using the image sensor of the detection unit 140 . In addition, the control unit 170 may acquire lidar data on the surface of the main tower of the marine bridge by using the lidar of the detection unit 140. The control unit 170 may measure the location of the drone (ie, GPS location data) through the positioning unit 120 when acquiring marine bridge information.

The control unit 170 may transmit marine bridge information obtained by the detection unit 140, for example, a pylon surface image, a pylon surface lidar data, and a drone location (or detection location) to the server 200 through the communication unit 110. there is. At this time, the control unit 170 may transmit the marine bridge information obtained by the detection unit 140 to the server 200 in real time or at a predetermined transmission period.

The control unit 170 may store the marine bridge information obtained through the detection unit 140 in the storage unit 150 when the connection with the communication network through the communication unit 110 is not smooth. Thereafter, a method of manually transferring the marine bridge information stored in the storage unit 150 to the server 200 may be taken.

The controller 170 may include at least one processor. The at least one processor includes an application specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a microcontroller, and a microprocessor. It may be implemented with at least one of processing devices such as a microprocessor.

The server 200 may inspect the marine bridge and manage the inspection result. The server 200 may also serve as a ground control system for tracking the flight trajectory of the drone 100 and controlling the flight of the drone 100 . The server 200 may include a communication unit 210, a memory 220, a user input unit 230, an output unit 240, and a processing unit 250.

The communication unit 210 may support wired communication or wireless communication between the server 200 and the drone 100 . The communication unit 210 may transmit and receive data using a wireless communication circuit and/or a wired communication circuit. The communication unit 210 may transmit a control command of the processing unit 250 or receive marine bridge information transmitted from the drone 100 . The communication unit 210 may store the received marine bridge information in the memory 220 or transmit it to the processing unit 250.

The memory 220 may store artificial intelligence (AI) models, marine bridge information, marine bridge inspection information, 3D marine bridge models, and the like. The memory 220 may store instructions executed by the processing unit 250 .

The memory 220 includes a flash memory, a hard disk, a solid state disk (SSD), a secure digital card (SD card), a random access memory (RAM), a static random access memory (SRAM), and a ROM. It may include at least one of storage media such as read only memory (Read Only Memory), programmable read only memory (PROM), electrically erasable and programmable ROM (EEPROM), and erasable and programmable ROM (EPROM).

The user input unit 230 may generate data according to manipulation by a user (eg, a facility manager, etc.). The user input unit 230 may include a keyboard, a touch pad, buttons, and/or a touch screen. The user input unit 230 may transmit maintenance related information according to the user input to the processing unit 250 .

The output unit 240 may output the marine bridge inspection progress status and result. The output unit 240 may be a Liquid Crystal Display (LCD), a Thin Film Transistor-Liquid Crystal Display (TFT-LCD) and/or an Organic Light-Emitting Diode (OLED) display, etc. may include the same display.

The processing unit 250 may control overall operations of the server 200 . The processing unit 250 includes an application specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a microcontroller, and a microprocessor. It may be implemented with at least one of processing devices such as a microprocessor.

The processing unit 250 may obtain marine bridge information through the drone 100 . The processing unit 250 may analyze the acquired marine bridge information using an artificial intelligence model to determine whether the surface of the marine bridge is abnormal. In addition, the processing unit 250 may determine the type of surface abnormality and the location of the surface abnormality (location where the surface abnormality occurs) of the marine bridge.

The processing unit 250 may match the determination result with the GPS location data and store it again in the memory 220 . In other words, the processing unit 250 may store the surface abnormality type (eg, concrete crack, peeling, efflorescence, etc.) and the location of the surface abnormality of the marine bridge in the memory 220 .

The processing unit 250 may output the surface anomaly type and surface anomaly location of the marine bridge to the output unit 240 based on the 3D marine bridge model. The processing unit 250 may differently define colors and/or icons according to types of surface anomalies. The processing unit 250 may position and display a color or icon matching the surface anomaly type at the surface anomaly position of the 3D marine bridge model. At this time, the processing unit 250 may position a color mark or icon at any one point or central position in the surface abnormality area.

The processing unit 250 may receive a selection signal indicating selection of a color mark or icon displayed on the 3D marine bridge model from the user input unit 230 . When the selection signal is received, the processing unit 250 may display detailed information on the surface abnormality type matched to the selected color mark or icon in a separate pop-up window. Detailed information may include detection date and time, detection detailed photo source, photo after artificial intelligence model inference, geometry information such as abnormal length and width, GPS location data, and the like.

When maintenance-related information is input through the user input unit 230 in the displayed pop-up window or a separate maintenance history management window, the processing unit 250 may update detailed information using the corresponding input information.

The processing unit 250 may use the marine bridge information acquired by the drone 100 for 3D marine bridge modeling. The processing unit 250 may convert an image of a sea bridge acquired while the drone 100 flies along a predetermined path into an orthographic image and pre-process the image to have a certain standard. The processing unit 250 may construct a 3D marine bridge model after converting the preprocessed image into 3D point cloud data.

The 3D offshore bridge model can be used as a surface abnormality mark in the management platform. The 3D marine bridge model supports 360-degree rotation, zoom-in and zoom-out on the management platform, which can be configured to help users intuitively understand the detailed shape. A separate option in the management platform can be set to display surface anomalies inferred by the artificial intelligence model. Surface anomalies are stored in sets according to the inspection period, and the surface anomaly inspection history can be checked by changing the set for each inspection time.

As described above, the present invention acquires the surface information of the pylons of marine bridges using various sensors mounted on drones, and then uses an artificial intelligence model to check in advance the risk factors of the pylons of marine bridges, such as concrete cracks, peeling, and whitening. can In addition, the present invention automatically records the occurrence location and type (type) of surface anomalies detected by artificial intelligence models, so that even non-experts in related organizations who want to manage marine bridges can obtain professional safety inspection evaluation results for target bridges at any time. can be provided and used for maintenance decision-making.

Hereinafter, with reference to FIG. 2 , a marine bridge information analysis process will be described in more detail.

2 is a diagram illustrating an AI-based maritime bridge information processing process according to an embodiment of the present invention.

The processing unit 250 of the server 200 may check the surface condition of the bridge by using a two-step deep learning artificial intelligence model. First, in step 1, the processing unit 250 may primarily select all marine bridge information obtained by the drone 100 using the first artificial intelligence model capable of fast processing due to a small amount of calculation.

The processing unit 250 may distinguish only marine bridge information determined to have an abnormality on the surface of the marine bridge regardless of the type (type) of the surface abnormality. That is, the processing unit 250 may select surface abnormality information of a sea bridge from among the acquired sea bridge information. As the first artificial intelligence model in the first step, a classification model based on a convolutional neural network (CNN) or a vision transformer (ViT) may be applied.

The first artificial intelligence model is an inference that processes the sea bridge information acquired by the drone 100 by transferring pre-learned weights to a previously developed network or newly learning by a developer building a separate deep learning network. It is a model. The first artificial intelligence model may analyze the marine bridge information and classify it as having surface abnormality or not having surface abnormality.

In step 2, the processing unit 250 analyzes the surface anomaly information of the marine bridge selected in step 1 with a second artificial intelligence model to infer detailed information such as the type and location of the surface anomaly.

As the second artificial intelligence model, a fully convolutional based segmentation model, a 1-stage object detection model, or a 2-stage object detection model may be applied. The second artificial intelligence model may use only data classified as surface abnormality in the first artificial intelligence model, that is, surface abnormality information of the sea bridge as input data.

The second artificial intelligence model may determine the location and type of surface abnormality by detecting local abnormal conditions among the selected individual data. For example, when an image is input, the second artificial intelligence model classifies the image pixel by pixel, distinguishes between normal and abnormal pixels, and expresses an abnormal area in the entire area. As another example, when LIDAR data is input, the second artificial intelligence model may infer that there is an abnormality in the coordinate plane where the space value is displayed in a 3D matrix on the 2D coordinate plane and the abnormality may be indicated.

The above two-step artificial intelligence model can be classified in detail based on the propensity of the learning data and used for inspection. In the AI model, weights are transferred or newly learned based on a pre-prepared learning dataset for general conditions, and it can be used as a basic model for daily inspection of marine bridges.

Afterwards, the learning dataset is augmented by reflecting changes in acquired data according to region, season, and shooting time, and can be used for transfer learning for the basic model. Several advanced models for various individual conditions can be established, and facility managers can select them according to the given situation and provide them as modular models so that they can be installed on the server. If necessary, by arranging several models for each condition in parallel, it is possible to take a method of obtaining results by fine-tuning the desired inspection situation, and it can be used for the purpose of precise inspection of structures.

Finally, the artificial intelligence model may be an adaptive artificial intelligence model that obtains optimal surface anomaly detection performance by learning and deriving optimization parameters in advance for various unlearned inspection situations of new offshore bridges. When applying the basic model and the advanced model to a new offshore bridge, it can be used as a technique for tuning the model.

3 is a flowchart illustrating a marine bridge inspection method according to an embodiment of the present invention.

The server 200 may acquire marine bridge information through the drone 100 (S100). The marine bridge information may include surface state information of the pylons of the marine bridge.

The server 200 may select surface abnormality information of the sea bridge from among the sea bridge information obtained by using the first artificial intelligence model (S110). The server 200 may analyze the marine bridge information using the first artificial intelligence model to determine whether there is a surface abnormality in the marine bridge information. The server 200 may classify marine bridge information having a surface abnormality as surface abnormality information of a marine bridge according to the determination result. A classification model based on a convolutional neural network (CNN) or a vision transformer (ViT) may be used as the first artificial intelligence model.

The server 200 may determine the type and location of the surface abnormality of the marine bridge by analyzing the selected surface abnormality information of the marine bridge with the second artificial intelligence model (S120). The server 200 may infer the type and occurrence location of the surface anomaly that has occurred on the offshore bridge. As the second artificial intelligence model, a fully convolutional based segmentation model, a 1-stage object detection model, or a 2-stage object detection model may be used.

The server 200 may output the surface abnormality type and location of the marine bridge based on the 3D marine bridge model (S130). The processing unit 250 may differently define colors and/or icons according to types of surface anomalies. The processing unit 250 may position and display a color or icon matching the surface anomaly type at the surface anomaly position of the 3D marine bridge model. At this time, the processing unit 250 may position a color mark or icon at any one point or central position in the surface abnormality area.

In the above, even though all the components constituting the embodiment of the present invention have been described as being combined or operated as one, the present invention is not necessarily limited to these embodiments. That is, within the scope of the object of the present invention, all of the components may be selectively combined with one or more to operate. In addition, although all of the components may be implemented as a single independent piece of hardware, some or all of the components are selectively combined to perform some or all of the combined functions in one or a plurality of pieces of hardware. It may be implemented as a computer program having. Codes and code segments constituting the computer program may be easily inferred by a person skilled in the art. Such a computer program may implement an embodiment of the present invention by being stored in a computer readable storage medium, read and executed by a computer.

Claims (13)

A communication unit for receiving marine bridge information obtained through a sensor mounted on a drone; and
A processing unit that analyzes the marine bridge information with an artificial intelligence model and checks surface abnormalities of the marine bridge;
The processing unit,
Using a first artificial intelligence model implemented as a CNN (convolutional neural network) or ViT (vision transformer)-based classification model, selecting marine bridge surface abnormality information among marine bridge information,
Analyzing the selected marine bridge surface anomaly information with a second artificial intelligence model implemented as a fully convolutional based segmentation model, a 1-stage object detection model, or a 2-stage object detection model to determine the type of surface anomaly and the location of the surface anomaly of the marine bridge,
Conversion of the sea bridge image acquired by the drone flying along a predetermined path into an orthographic image and preprocessing to a certain standard, converting the preprocessed image into 3D point cloud data to build a 3D marine bridge model,
Based on the built 3D marine bridge model, the surface abnormality type and surface abnormality location of the marine bridge, detection date and time, geometry information such as abnormal length, and GPS location data are displayed. Marine bridge inspection system
delete delete delete delete delete The method of claim 1,
The drone,
The marine bridge inspection system, characterized in that for obtaining the marine bridge information using at least one of an image sensor and lidar.
Receiving marine bridge information obtained through a sensor mounted on a drone;
Analyzing the marine bridge information with an artificial intelligence model to check surface abnormalities of the marine bridge; and
Displaying the surface abnormality type and surface abnormality location of the marine bridge based on the 3D marine bridge model;
The checking step is
Selecting surface anomaly information of a marine bridge from among marine bridge information using a first artificial intelligence model implemented as a classification model based on a convolutional neural network (CNN) or a vision transformer (ViT); and
Analyzing the selected marine bridge surface anomaly information with a second artificial intelligence model implemented as a fully convolutional based segmentation model, a 1-stage object detection model, or a 2-stage object detection model Including the step of determining the type of surface anomaly and the location of the surface anomaly of the marine bridge by
The 3D marine bridge model,
The image of the marine bridge obtained by flying along the predetermined path of the drone is converted into an orthographic image and preprocessed to have a certain standard, and the preprocessed image is converted into 3D point cloud data to be constructed, and the type and location of the surface anomaly of the marine bridge A marine bridge inspection method characterized by defining and displaying colors and icons differently according to.
delete delete delete delete The method of claim 8,
Receiving the marine bridge information,
Acquiring, by the drone, information on the sea bridge using at least one of an image sensor and a lidar; and
The marine bridge inspection method comprising the step of transmitting the acquired marine bridge information by the drone.

Images