KR102458771B1 - Method and apparatus for monitoring of bridge structure damage using graph neural network - Google Patents

Abstract

Bridge structure monitoring apparatus according to an aspect of the present application includes a communication module for performing communication with one or more sensors; a memory in which the bridge structure monitoring program is stored; and a processor executing the bridge structure monitoring program, wherein the bridge structure monitoring program converts a graph modeling the bridge structure to be evaluated based on the information collected from the sensor into an embedding vector, and a graph neural network (GNN) Input the embedding vector of the evaluation target bridge structure to the base bridge structure learning model, and based on the output of the GNN-based bridge structure learning model, the identification number of the cable in which the damage has occurred in the evaluation target bridge structure and the damage of the cable output the degree.

Description

Method and device for monitoring bridge structure using GNN

The present invention relates to a method and apparatus for monitoring a bridge structure using GNN (GRAPH NEURAL NETWORK).

A cable-stayed bridge, one of the major facilities of transportation infrastructure, is damaged and corroded due to external environment and complex factors such as natural disasters, climate, ambient vibration, and vehicle load. In particular, the cable, which is a component of cable-stayed bridges, is a very important element in maintaining the bridge, but also a component vulnerable to damage. When a cable is damaged, its stiffness and cross-sectional area decrease. Because the cable has a small cross-sectional area, it can be lost due to low resistance to accidental side loads, and the loss of the cable can overload the bridge and adversely affect adjacent cables .

Accordingly, in order to maintain the safe state of the bridge, it is necessary to continuously monitor the state of the cable. However, the data collected from the sensor of the bridge is about the tension of the cable, and this alone cannot directly determine what kind of damaged cable is and what the cross-sectional area of the damaged cable is. It is very inefficient and increases the maintenance cost to manually check all cables to check this. Therefore, the development of a new bridge monitoring method is required to ensure the safety and durability of the bridge.

Republic of Korea Patent Publication No. 10-1509305 (Title of the invention: cable damage signal processing apparatus and method)

The present invention is to solve the above problems, and it is a technical task to provide a method and apparatus for monitoring a bridge structure capable of outputting a damaged cable and a degree of damage among cables constituting a bridge structure using GNN. do.

However, the technical task to be achieved by the present embodiment is not limited to the above-described technical task, and other technical tasks may exist.

As a technical means for solving the above-described technical problem, a graph neural network (GNN)-based monitoring method of a bridge structure using the bridge structure monitoring apparatus according to the first aspect of the present disclosure is a graph modeling a bridge structure to be evaluated. converting to an embedding vector; inputting an embedding vector of the evaluation target bridge structure into a GNN-based bridge structure learning model; And based on the output of the GNN-based bridge structure learning model, comprising the step of outputting the identification number of the damaged cable in the evaluation target bridge structure and the degree of damage to the cable. The GNN-based bridge structure learning model learns training data of a plurality of bridge structures expressed in a graph through a Message Passing Neural Network (MPNN), and the graph is a plurality of vertices and a plurality of edges ( egde), wherein the vertex represents the position coordinates of the nodes of the bridge structure, and the edge represents the distance between each node, the type of connection element connecting each node, and the tension value of the connection element .

In addition, the bridge structure monitoring apparatus according to the second aspect of the present disclosure, a communication module for performing communication with one or more sensors; a memory in which the bridge structure monitoring program is stored; and a processor executing the bridge structure monitoring program, wherein the bridge structure monitoring program converts a graph modeling the bridge structure to be evaluated based on the information collected from the sensor into an embedding vector, and a graph neural network (GNN) Input the embedding vector of the evaluation target bridge structure to the base bridge structure learning model, and based on the output of the GNN-based bridge structure learning model, the identification number of the cable in which the damage has occurred in the evaluation target bridge structure and the damage of the cable output the degree. The GNN-based bridge structure learning model learns training data of a plurality of bridge structures expressed in graphs through a Message Passing Neural Network (MPNN), and the graph has a plurality of vertices and a plurality of edges ( egde), wherein the vertex represents the position coordinates of the nodes of the bridge structure, and the edge represents the distance between each node, the type of connection element connecting each node, and the tension value of the connection element .

According to any one of the above-described problem solving means of the present application, since the cable-stayed bridge is expressed in the form of a graph and a GNN learning model is built based on this, the state of the cables constituting the cable-stayed bridge can be more accurately monitored. In particular, it is possible to effectively determine which cable is damaged and to what extent the cable is damaged only with information collected from some of the sensors rather than all of the sensors disposed on the cable-stayed bridge.

1 is a block diagram showing the configuration of a bridge structure monitoring system according to an embodiment of the present invention.
Figure 2 is a block diagram showing the configuration of the bridge structure monitoring apparatus according to an embodiment of the present invention.
Figure 3 shows a bridge structure monitoring method used in an embodiment of the present invention.
4 shows an exemplary configuration of a bridge structure model used in an embodiment of the present invention.
5 is a view for explaining a process of building a learning model according to an embodiment of the present invention.
6 is a view showing the experimental results of the bridge structure monitoring method according to an embodiment of the present invention.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily implement them. However, the present application may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is said to be "connected" with another part, it includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. do.

Throughout this specification, when a member is said to be located “on” another member, this includes not only a case in which a member is in contact with another member but also a case in which another member is present between the two members.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing the configuration of a bridge structure monitoring system according to an embodiment of the present invention.

The bridge structure monitoring system 10 includes a bridge structure monitoring apparatus 100 and a plurality of sensors 200 .

The bridge structure monitoring apparatus 100 collects information on the tension of each cable transmitted from a plurality of sensors 200 attached to the cable-stayed bridge type bridge structure, and uses it to learn the bridge structure based on a graph neural network (GNN). By input to the model, the damaged cables and the degree of damage are output in the bridge structure to be evaluated.

The sensor 200 is coupled to each cable, measures the tension of each cable, and transmits the measured sensing data to the bridge structure monitoring apparatus 100 through data communication. In this case, each sensor 200 transmits sensing information in a form including its own identification information, and the bridge structure monitoring apparatus 100 may identify the sensing data for which cable through identification information of each sensing. A general wired or wireless sensor device may be used as the sensor 200 , and the detailed configuration of the sensor 200 is the same as that of the prior art, and thus a detailed description thereof will be omitted.

Figure 2 is a block diagram showing the configuration of the bridge structure monitoring apparatus according to an embodiment of the present invention.

As shown, the bridge structure monitoring apparatus 100 may include a communication module 110 , a memory 120 , a processor 130 , a database 140 , and an input module 150 .

The communication module 110 may transmit/receive data to and from the plurality of sensors 200 . The communication module 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired/wireless connection with other network devices.

The memory 120 stores the bridge structure monitoring program. The bridge structure monitoring program accumulates information on the tension data of each cable from the sensor 200, inputs it to the GNN-based bridge structure learning model, and based on the inference result of the GNN-based bridge structure learning model, the bridge Outputs whether damage has occurred and the degree of damage to the cables placed in the structure.

Various types of data generated during the execution of an operating system for driving the bridge structure monitoring apparatus 100 or a bridge structure monitoring program are stored in the memory 120 .

In this case, the memory 120 collectively refers to a non-volatile storage device that continuously maintains stored information even when power is not supplied, and a volatile storage device that requires power to maintain the stored information.

In addition, the memory 120 may perform a function of temporarily or permanently storing data processed by the processor 130 . Here, the memory 120 may include magnetic storage media or flash storage media in addition to the volatile storage device requiring power to maintain stored information, but the scope of the present invention is limited thereto. it's not going to be

The processor 130 executes the program stored in the memory 120, but controls the entire process according to the execution of the bridge structure monitoring program. Each operation performed by the processor 130 will be described in more detail later.

The processor 130 may include any type of device capable of processing data. For example, it may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed as a code or an instruction included in a program. As an example of the data processing device embedded in the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific (ASIC) An integrated circuit) and a processing device such as a field programmable gate array (FPGA) may be included, but the scope of the present invention is not limited thereto.

The database 140 stores or provides data necessary for the bridge structure monitoring system under the control of the processor 130 . The database 140 may be included as a component separate from the memory 120 , or may be built in a part of the memory 120 .

A configuration of a GNN-based bridge structure learning model used in the present invention and a method for monitoring damage to a bridge structure using the same will be described.

Figure 3 shows a bridge structure monitoring method used in an embodiment of the present invention.

First, a graph modeling the bridge structure to be evaluated is converted into an embedding vector (S310).

In the present invention, in the process of constructing a learning model or inferring a result based on it, a graph modeled by a bridge structure is generated, and each is converted into an embedding vector and used.

4 shows an exemplary configuration of a bridge structure model used in an embodiment of the present invention.

What is shown is a semi-half-type cable-stayed bridge model, and consists of a pylon, a girder, a cross beam, a plurality of cables, and the like. As shown in the figure, when viewed from the front, cables numbered 1 to 10 are connected to the first pylon, cables 11 to 20 are connected to the second pylon, and when viewed from the rear, cables numbered 21 to the first pylon No. 30 cables may be connected, and cables 31 to 40 may be connected to the second pylon. In such a structure, if one cable is damaged, the tensile force of the other cable is also affected. Therefore, by using the structural characteristics of the cable-stayed bridge, it is possible to check which cable is damaged through the machine learning model. In addition, it is possible to specify which cable is damaged even when the sensor is coupled to some cables rather than the entire cable.

Prior to building a learning model, through a model capable of setting up a bridge structure and performing a simulation on it, tension information of other cables according to damage to a specific cable can be generated, respectively. In the experimental stage of the present invention, a conventionally known structural analysis simulation model called PAAP (Practical Advanced Analysis Program) was used, but the present invention is not limited thereto. The structural analysis simulation program used in the present invention can calculate the tension value of the remaining cables constituting the cable-stayed bridge when the degree of damage (cross-sectional area) of each cable constituting the cable-stayed bridge is changed.

As in the cable-stayed bridge of FIG. 4 , for a bridge with 40 cables, the following sample set (C _i ) can be considered.

In this case, A _j represents the cross-sectional area of the j-th cable, and more specifically, it can be expressed through the following equation.

A _i represents the cross-sectional area of the cable in the undamaged state, A _d represents the area of the cable in the damaged state, and α represents the parameter. In this case, α has a value between 0 and 1.

At this time, for example, if the parameter α is changed from 0 to 1 in 0.01 steps, 100 changed values can be obtained per one cable, and in the case of a bridge with 40 cables, a total of 4000 data samples are obtained. will secure

The structural analysis simulation program can acquire, for example, the tension in 10 cables for 4000 cable cross-sectional area change data as follows.

In this way, through the structural analysis simulation program of the cable-stayed bridge, input data for changing the degree of damage to each cable and the tension data for each cable as an output can be obtained as an output. Then, a learning model is constructed by using the obtained data as training data.

In the present invention, to model the structural topology of the cable-stayed bridge, a graph neural network (GNN) is used. GNN is a powerful deep learning model that can process graph-structured data. It can effectively analyze and infer graphs by updating the hidden state of vertices with information from neighbors to capture hidden patterns in the graph. have.

In addition, in the present invention, the damage is evaluated by using both the tension and spatial information of the cable-stayed cable by using a Message Passing Neural Network (MPNN) as a framework of the GNN.

First, the graph modeling the structural topology of the cable-stayed bridge includes a plurality of vertices and a plurality of edges connecting each vertex. In this case, each vertex (x _v ) represents a three-dimensional position coordinate of a node representing an individual element of the bridge structure. In addition, the edge (e _uv ) is a 6-dimensional vector including the distance between each node, the type of connection element (cable, girder, cross beam, pylon) connecting each node, and the tension value of the corresponding connection element. As such, the graph includes a plurality of vertices and a plurality of edges, and by setting the identification number of each cable and the damage area of each cable differently, a graph for each distinct bridge structure model can be generated. .

Referring back to FIG. 3 , an embedding vector of a bridge structure to be evaluated is input to the GNN-based bridge structure learning model ( S320 ). Let's take a look at the specific configuration of the GNN-based bridge structure learning model.

5 is a view for explaining a process of building a learning model according to an embodiment of the present invention.

First, each GNN-based bridge structure model is generated using the training data obtained from the structural analysis simulation program described above. Training data is data expressed as a graph, and in particular, graph data including the tension data of each cable recorded at the edge (e _uv ) is used as input data, and the identification number of the damaged cable and the degree of damage (cross-sectional area) of the corresponding cable are used as input data. area) is used as output data. And, when the learning model is built, the tension data sensed through the sensor is input to the learning model, and based on this, an identification number of the damaged cable and a value indicating the degree of damage are output from the learning model.

Referring to the drawings, for each graph generated for each bridge structure model, vertices and edges constituting each graph are converted into embedding vectors, and this is learned through a Message Passing Neural Network (MPNN). In this case, the vertex (x _v ) indicating the position of the node can be embedded as a single hidden layer fully connected with 64 neurons and a ReLU activation function.

In the message delivery step, embedded vertices representing each vertex are delivered to adjacent vertices along adjacent edges, and such message delivery is repeatedly performed. In this way, a learning model is built by repeatedly performing the message passing step, and the hidden state (h _v ) of the repeatedly learned vertex (x _v ) is a message function (M _t ) and an update function (U) defined in the following equations: _t ).

), t 번째 은닉 상태(

) 등으로 표시될 수 있다.This message delivery step can be repeated a total of T times, and the initial hidden state (

), the t-th hidden state (

) can be expressed as

In this case, σ represents the ReLU activation function, and A( ) is a two-layer neural network that outputs a matrix, including a first layer with 128 neurons and a ReLU activation function, and a second layer with 64*64 neurons. have.

The update function (U _t ) may be defined as a gated recurrent unit (GRU). As with vertex embeddings, the dimension of the hidden state is still 64. The GRU integrates the state of the vertex itself and the message (M _t ) received from neighboring vertices. This message passing step is repeated 4 times, for example, and the hidden state of the last vertices is passed to the output function (R).

In this case, the output function R may be defined as a set2set model, which is a Long Short Term Memory (LSTM) network that is permutation invariant with respect to input data. Because Set2set is immutable to graph isomorphism, it can effectively unify the vertices of the graph and create graph-level embeddings.

Also, the set2set model with two LSTM stacks can be used for global pooling. In the two stacked LSTMs, the second LSTM maps the output of the first LSTM to a 128-dimensional vector, and sets the number of calculations, another hyperparameter of the set2set model, to 2. Then, 64 neurons and a hidden layer fully connected with the ReLU activation function were added. A prediction for the input data may be generated in two output layers, and may be generated in 40 linear units and 1 linear unit, respectively, as shown.

An embedding vector representing a graph of a bridge structure to be evaluated is input to the learning model constructed in this way, and a result is output. In this case, the graph of the bridge structure to be evaluated includes tension data collected from one or more sensors attached to the bridge structure. According to the configuration of the learning model of the present invention, even if the tension data of some cables among all cables is input, it is possible to output the damaged cables among all cables and the degree of damage to the cables.

Referring back to FIG. 3 , based on the output of the GNN-based bridge structure learning model, the identification number of the damaged cable in the evaluation target bridge structure and the degree of damage to the corresponding cable are output ( S330 ).

As such, in the present invention, the damaged cable identification number and the degree of damage of the corresponding cable are output through the learning model, and multi-task learning is used so that the MPNN can effectively learn two tasks.

The advantage of multi-task learning is that it can predict multiple tasks at the same time to learn related tasks more efficiently. Therefore, learning how to predict the degree of damage (cross-sectional area) of damaged cables and how to classify damaged cables at the same time improves learning efficiency. As shown in FIG. 3, the proposed MPNN provides outputs for a first task (task 1) of classifying identification numbers of damaged cables and a second task (task2) of estimating the cross-sectional area of the damaged cables, respectively.

The first task is classification, and the second task is prediction on continuous data. Therefore, the cross-entropy loss function for the first task and the mean absolute error loss function for the second task are used as follows.

는 출력 데이터를 의미한다. 또한, A_d는 손상된 케이블의 손상 정도를 나타내는 타겟 데이터이고,

는 출력 데이터를 의미한다. 이와 같이, 손실 함수를 통해 학습에 사용된 타겟 데이터와 출력 데이터 간의 차이가 최소화되도록 한다.D _i is target data indicating when the i-th cable among all cables is damaged,

is the output data. In addition, A _d is target data indicating the degree of damage to the damaged cable,

is the output data. In this way, the difference between the target data used for learning and the output data is minimized through the loss function.

A weight may be applied to each of the first loss function for the first task and the second loss function for the second task, and weighted summation obtained by summing them may be used as an integrated loss function.

6 is a view showing the experimental results of the bridge structure monitoring method according to an embodiment of the present invention.

First, to explain the specific experimental conditions, a total of 4000 cases were generated by reflecting 100 parameters (α) for each of 40 cables. Each case contains data on the value of tension applied to each cable when the parameters of any one of the 40 cables are set.

In the process of building the learning model, the graph of the bridge structure was input, and the input data of the training data was 10 cables out of 40 cables (No. 1, 4, 9, 11, 16, 21, 24, 29, 31, 36). The tension value of was used, and as the output data, the identification numbers of 40 damaged cables and the cross-sectional area of the damaged cables were used. Of the 4000 data obtained in this way, 2400 pieces were used as training data, 800 pieces were used as validation data, and 800 pieces were used as test data. Based on these data, the previously set loss function was trained to be minimized. At this time, the batch size was set to 32 and the epoch was set to 10000.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Although the methods and systems of the present invention have been described with reference to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general purpose hardware architecture.

The foregoing description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present application.

100: bridge monitoring device
110: communication module
120: memory
130: processor
140: database
150: input module
200: sensor

Claims (7)

A method for monitoring a bridge structure based on a graph neural network (GNN) using a bridge structure monitoring device, the method comprising:
(a) converting a graph modeling the evaluation target bridge structure into an embedding vector;
(b) inputting an embedding vector of the evaluation target bridge structure into a GNN-based bridge structure learning model; and
(c) based on the output of the GNN-based bridge structure learning model, outputting the identification number of the cable damaged in the evaluation target bridge structure and the degree of damage to the cable,
The GNN-based bridge structure learning model is obtained by learning the training data of a plurality of bridge structures expressed in graphs through a Message Passing Neural Network (MPNN),
The graph includes a plurality of vertices and a plurality of edges connecting each vertex, wherein the vertices represent the position coordinates of the nodes of the bridge structure, and the edges represent the distance between each node, each node A method for monitoring damage to a bridge structure, which indicates the type of connecting element to be connected and the tension value of the connecting element. The method of claim 1,
The GNN-based bridge structure learning model is constructed by repeatedly performing a message passing step of transferring the embedded vertex (x _v ) of the graph to the neighboring vertices along the adjacent edge (e _uv ) through the MPNN, and iteratively The hidden state (h _v ) of the learned vertex is defined by the message function and the update function defined in Equations 1 and 2 below, and the output value for the graph modeling the evaluation target bridge structure is in Equation 3 below. A method for monitoring damage to a bridge structure, which is defined by a defined output function (R).
[Equation 1]

[Equation 2]

t: number of iterations, σ: activation function A( ): neural network that outputs a matrix
[Equation 3]

3. The method of claim 2,
The GNN-based bridge structure learning model is a first loss function for a first task of estimating the identification number of the damaged cable and a second loss function for a second task of estimating the degree of damage to the estimated damaged cable. A method for monitoring damage to a bridge structure, learned using weighted summation. In the bridge structure monitoring device,
a communication module for communicating with one or more sensors;
a memory in which the bridge structure monitoring program is stored; and
Including a processor for executing the bridge structure monitoring program,
The bridge structure monitoring program converts a graph modeling the bridge structure to be evaluated based on the information collected from the sensor into an embedding vector, and embeds the evaluation target bridge structure in the GNN (graph neural network)-based bridge structure learning model. input, and based on the output of the GNN-based bridge structure learning model, output the identification number of the cable that is damaged in the evaluation target bridge structure and the degree of damage to the cable,
The GNN-based bridge structure learning model is obtained by learning the training data of a plurality of bridge structures expressed in graphs through a Message Passing Neural Network (MPNN),
The graph includes a plurality of vertices and a plurality of edges connecting each vertex, wherein the vertices represent the position coordinates of the nodes of the bridge structure, and the edges represent the distance between each node, each node A bridge structure monitoring device that indicates the type of connection element to be connected and the tension value of the connection element. 5. The method of claim 4,
The GNN-based bridge structure learning model is constructed by repeatedly performing a message passing step of transferring the embedded vertex (x _v ) of the graph to the neighboring vertices along the adjacent edge (e _uv ) through the MPNN, and iteratively The hidden state (h _v ) of the learned vertex is defined by the message function and the update function defined in Equations 1 and 2 below, and the output value for the graph modeling the evaluation target bridge structure is in Equation 3 below. The bridge structure monitoring device, which is defined by the defined output function (R).
[Equation 1]

[Equation 2]

t: number of iterations, σ: activation function A( ): neural network that outputs a matrix
[Equation 3]

5. The method of claim 4,
The GNN-based bridge structure learning model is a first loss function for a first task of estimating the identification number of the damaged cable and a second loss function for a second task of estimating the degree of damage to the estimated damaged cable. A bridge structure monitoring device, learned using weighted summation. A non-transitory computer-readable recording medium in which a computer program for performing the method for monitoring damage to a bridge structure according to any one of claims 1 to 3 is recorded.

Images