KR20220170122A - System for monitoring of structural and method ithereof - Google Patents

Abstract

The present invention is intended to provide a structural health monitoring system and a monitoring method using the same. The present invention is configured to evaluate structural health according to external force using an imaging device that takes images so as to measure the structural health without installing a device that measures a displacement of the structure separately. Thus, the present invention can monitor structures in real time and predict structural health. The structural health monitoring method includes steps of: extracting a series of effective frames for predicting the health of the structure from an image of the structure for monitoring the health; estimating an optical flow for the series of effective frames; and determining the health of structures represented in the series of effective frames based on the optical flow.

Description

Structure integrity monitoring system and monitoring method using the same {SYSTEM FOR MONITORING OF STRUCTURAL AND METHOD　ITHEREOF}

The present invention relates to a structural health monitoring system and a monitoring method using the same, and more particularly, to a structural health monitoring system capable of monitoring structural health in real time and a monitoring method using the same.

The contents described below are only described for the purpose of providing background information related to an embodiment of the present invention, and the contents described do not naturally constitute prior art.

Large structures and facilities constructed are subject to structural damage due to defects in the design and construction process or various factors not considered at the time of design. there is. For example, in the case of a structure in which a serious degree of structural damage occurs, it frequently occurs that the service life is shortened to a degree that is far short of the designed service life planned at the time of design. Accordingly, efforts to secure long-term safety and operability of building structures are urgently required. In particular, since large structures such as buildings, bridges, and dams are continuously exposed to various operational loads, impacts by external objects, earthquakes, wind loads, wave loads, and corrosion, the problem of securing the safety of structures from them is an economic and social problem. has become a subject of great interest. In order to accurately diagnose the safety of these large structures, diagnosis techniques are required through monitoring of displacement through appropriate experimental measurements, techniques for mechanically analyzing structural damage, and analysis techniques for modeling structural damage.

Structural health monitoring (SHM) is a method of monitoring structural damage of a structure using non-destructive testing and has received much attention in the field of civil engineering. For example, when a structure is excited by a natural or man-made hazard, properties such as stiffness and damping of the structure change. Changes in the structure observed through the sensor are transmitted to the SHM system, and the SHM system specifies and provides the degree of damage and location in real time so as to cope with it in advance.

However, the disclosed structure integrity monitoring is mainly based on sensors, and has a limitation in that the sensor must directly contact the measurement structure in order to compare with a reference level previously stored in the sensor. In addition, as the size of the structure increases, the number of sensors to be installed must increase so that structural integrity can be evaluated. Accordingly, in order to more accurately evaluate structural integrity, there is a limitation in that the sensor installation cost increases.

Accordingly, a technology capable of evaluating the soundness of a large-sized structure and minimizing direct installation to monitor the soundness of the structure is required.

The foregoing background art is technical information that the inventor possessed for derivation of the present invention or acquired during the derivation process of the present invention, and cannot necessarily be said to be known art disclosed to the general public prior to filing the present invention.

One object of the present invention is to make it possible to monitor the soundness of a structure without directly installing a device for monitoring the soundness on the structure.

In addition, one object of the present invention is to evaluate the soundness of a structure using artificial intelligence.

In addition, one object of the present invention is to eliminate the need for structural analysis to determine whether damage to a structure has occurred due to an external load, and to minimize installation and maintenance costs.

The object of the present invention is not limited to the above-mentioned tasks, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the embodiments of the present invention. It will also be seen that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

A method for monitoring the health of a structure according to an embodiment of the present invention extracts a series of effective frames for predicting the health of a structure from an image of a structure for monitoring structural health, and extracts a series of effective frames for predicting the health of the structure. After estimating the optical flow for valid frames, a process of storing a code for determining the integrity of structures represented in the series of valid frames based on the optical flow may be performed.

Specifically, when extracting an effective frame, a 3D image obtained by capturing the structure may be corrected into a 2D image.

When correcting an image, correction may be performed by selecting an arbitrary coordinate in the 3D image and generating projection coordinates projected onto the 2D image based on the arbitrary coordinate.

After generating the projected coordinates, the actual coordinates may be extracted by sequentially multiplying the scale of the 2D image, the conversion matrix using the homography matrix, and the image coordinates extracted using the pixels of the 2D image.

When determining soundness, the real coordinates are applied to a Gaussian Kernel,

(여기서, w와 h는 이미지의 너비와 높이이며,

는 실수 x를 입력값(input)으로 취하고 x보다 작거나 같은 가장 큰 정수를 출력값(output)으로 제공하는 플로어 함수(floor function)인)로 정의할 수 있다. The Gaussian filter (Gaussian Kernel),

(Where w and h are the width and height of the image,

can be defined as a floor function that takes as input a real number x and provides as output the largest integer less than or equal to x.

로 계산될 수 있다(여기서, X_i와 X_o는 이미지 픽셀의 현재 위치 좌표 및 원본 위치좌표이고, *은 요소 별 곱셈 연산자인).When applying the projection coordinates to the Gaussian filter, the soundness (d) of the structure is

(where X _i and X _o are the current position coordinates and original position coordinates of image pixels, and * is an element-by-element multiplication operator).

When extracting an effective frame, extracting projected coordinates based on feature detection from a 2D image, and determining movement of the projected coordinates by comparing the projected coordinates with previously stored reference coordinates when determining structural integrity can be done through

In addition, when extracting an effective frame, a weight of an image positioned behind a focus of a 2D image may be lowered, and a weight of an image positioned in front of a focus of the 2D image may be increased.

When estimating the optical flow, shaking of the effective frames may be corrected, and the shake-corrected valid frames may be input into a pre-trained optical flow estimation prediction model.

In order to extract the soundness of the structure in the process of inputting it to the optical flow estimation prediction model, different effective frames are stacked and passed through a network with 9 convolution layers (Generic Network), and the network passes through optical flow estimation. It may be performed using at least one of FlowNet-S, FlowNet-C, and FlowNet2 among prediction models.

Meanwhile, a system for monitoring the health of a structure according to an embodiment of the present invention includes at least one processor and a memory operatively connected to the processor and storing at least one code executed by the processor. When the memory is executed through the processor, it causes the processor to extract a series of effective frames for predicting the health of the structure from a moving picture of the structure for monitoring the structural health, and the series of effective frames for predicting the structural health. After estimating the optical flow for a frame, a code for determining the integrity of structures represented in the series of valid frames based on the optical flow may be stored.

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to the present invention, a structure in which displacement may occur is photographed, an image of the photographed structure is generated as learning data, the generated learning data is learned, and the image of the structure is input to a structure displacement prediction model. to monitor the health of

That is, since it is possible to estimate the soundness of a structure without separately installing a device for measuring the displacement of the structure, it is possible to effectively reduce installation costs for structural soundness evaluation.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram illustrating an environment of a structure displacement monitoring system according to an embodiment of the present invention.
2 is a diagram illustrating a server configuration according to an embodiment of the present invention.
3 is a diagram illustrating a structure displacement monitoring process using a structure displacement monitoring system according to an embodiment of the present invention.
4 is a diagram illustrating an example of monitoring a structure according to an embodiment of the present invention.
5 and 6 are diagrams illustrating an example of checking a structure displacement through an image frame in which a structure is photographed according to an embodiment of the present invention.
7 is a diagram illustrating an example of an optical flow according to an embodiment of the present invention.
8 is a diagram illustrating a process of calculating the behavior of a structure according to an embodiment of the present invention.
9 is a diagram schematically illustrating a process of measuring and evaluating the soundness of a structure according to an embodiment of the present invention.

Hereinafter, the present invention will be described in more detail with reference to the drawings. The invention may be embodied in many different forms and is not limited to the embodiments set forth herein. In the following embodiments, parts not directly related to the description are omitted in order to clearly describe the present invention, but this does not mean that the omitted configuration is unnecessary in implementing a device or system to which the spirit of the present invention is applied. . In addition, the same reference numbers are used for the same or similar elements throughout the specification.

In the following description, terms such as first and second may be used to describe various components, but the components should not be limited by the above terms, and the terms refer to one component from another. Used only for distinguishing purposes. Also, in the following description, singular expressions include plural expressions unless the context clearly indicates otherwise.

In the following description, terms such as "comprise" or "having" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that it does not preclude the possibility of addition or existence of features, numbers, steps, operations, components, parts, or combinations thereof.

1 is a diagram illustrating an environment of a structure displacement monitoring system according to an embodiment of the present invention.

Referring to FIG. 1 , a system for monitoring structure displacement according to an embodiment of the present invention photographs a structure such as a bridge or a large building, and measures the displacement of the structure based on the captured video or image so that the displacement of the structure occurs. In this case, the displacement is notified to a manager who manages the structure or a terminal connected to a device such as a photographing device so that the state of the structure can be checked.

To this end, in the structure displacement monitoring system 10, the photographing device 100, the server 200, and the terminal (equipment) 300 are communicatively connected through the network 400.

Specifically, the photographing device 100 may be one of devices capable of photographing structures such as bridges and large buildings in real time, such as cameras and CCTVs. In an embodiment of the present invention, a CCTV will be described as an example.

The photographing device 100 may be installed at a predetermined distance from the structure, and at least two or more photographing devices 100 may be installed at a location where the structure may be photographed differently with respect to the structure so that various displacement changes may be confirmed. do.

The photographing apparatus 100 obtains an image frame such as a still image or a moving image in a photographing mode. The acquired image frame may be stored in a memory (not shown) included in the photographing device 100 or may be transmitted to the server 200 and stored in the memory 240 of the server 200 .

The server 200 provides various services for monitoring the displacement of the structure described in the embodiment of the present invention. In detail, the server 200 may extract an effective image frame capable of determining whether the position of a structure has changed in a structure image or image captured by the photographing device 100 using artificial intelligence. In addition, the server 200 may calculate the degree of displacement (behavior change) of the structure based on the extracted effective frame, and may transmit the calculated result to the terminal 300 . Artificial intelligence will be described below.

The terminal 300 is a device capable of confirming a structure integrity result based on a structure image captured by the photographing device 100 . For example, the terminal 300 may be one of devices such as a mobile terminal, a laptop computer, a tablet PC, and the like, and the type of terminal 300 is not limited. Since the structure integrity result can be checked through the terminal 300, the structure manager can perform management corresponding to the structure integrity result more quickly and easily.

The terminal 100 may transmit and receive data to and from the server 200 through the network 400 . Specifically, the terminal 100 and the server 200 use at least one of Enhanced Mobile Broadband (eMBB), Ultra-reliable and low latency communications (URLC), and Massive Machine-type communications (mMTC) through the network 400. It can be configured to communicate data using the service of.

Here, eMBB (Enhanced Mobile Broadband) is a mobile broadband service, through which multimedia contents, wireless data access, etc. are provided. In addition, more advanced mobile services such as hot spots and broadband coverage to accommodate explosively increasing mobile traffic can be provided through eMBB. Hotspots allow high-capacity traffic to be accommodated in areas with low user mobility and high density. Wide and stable wireless environment and user mobility can be guaranteed through broadband coverage.

The URLLC (Ultra-reliable and low latency communications) service defines much stricter requirements than the existing LTE in terms of reliability and transmission delay of data transmission and reception, and is used in industrial field production process automation, remote medical treatment, remote surgery, transportation, safety, etc. 5G service for

Massive machine-type communications (mMTC) is a service that is not sensitive to transmission delays that require transmission of relatively small amounts of data. A much larger number of terminals than general mobile phones, such as sensors, can be simultaneously connected to the wireless access network by mMTC. In this case, the price of the communication module of the terminal should be low, and improved power efficiency and power saving technology are required so that the terminal can operate for several years without battery replacement or recharging.

Network 400 includes wired and wireless networks, such as local area networks (LANs), wide area networks (WANs), the Internet, intranets, and extranets, and mobile networks, such as It may be any suitable communication network including cellular, 3G, LTE, 5G, WiFi networks, ad hoc networks, and combinations thereof.

Network 400 may include connections of network elements such as hubs, bridges, routers, switches, and gateways. Network 400 may include one or more connected networks, such as a multiple network environment, including a public network such as the Internet and a private network such as a secure enterprise private network. Access to network 400 may be provided through one or more wired or wireless access networks.

Meanwhile, when extracting whether or not the structure is displaced using the structure displacement monitoring system 10 according to an embodiment of the present invention, a change in structure displacement may be extracted using artificial intelligence (AI).

To this end, the server 200 creates various artificial intelligence models in relation to the artificial intelligence model, trains them, evaluates them, completes them, and updates them using the user's personal data, in the process of updating them using the local area and Programs related to various cognitive intelligence algorithms stored in the server 200 may be used.

In addition, the server 200 may serve to collect learning data required for training of various artificial intelligence models and to train the artificial intelligence models using the collected data. When various artificial intelligence models trained through the server 200 are completed through evaluation, the terminal 300 uses the completed artificial intelligence model, or the artificial intelligence model itself becomes a subject and extracts the displacement of the structure. can be performed

For example, artificial intelligence (AI) is a field of computer science and information technology that studies ways to enable computers to do thinking, learning, and self-development that human intelligence can do. This means that behavior can be imitated.

Also, artificial intelligence does not exist by itself, but is directly or indirectly related to other fields of computer science. In particular, in modern times, attempts to introduce artificial intelligence elements in various fields of information technology and use them to solve problems in those fields are being actively made.

Machine learning is a branch of artificial intelligence, a field of study that gives computers the ability to learn without being explicitly programmed.

Specifically, machine learning can be said to be a technology that studies and builds a system that learns based on empirical data, makes predictions, and improves its own performance, as well as algorithms for it. Machine learning algorithms build specific models to make predictions or decisions based on input data, rather than executing rigidly defined, static program instructions.

The term 'machine learning' may be used interchangeably with the term 'machine learning'.

In machine learning, many machine learning algorithms have been developed regarding how to classify data. Representative examples include decision trees, Bayesian networks, support vector machines (SVMs), and artificial neural networks (ANNs).

A decision tree is an analysis method that performs classification and prediction by charting decision rules in a tree structure.

A Bayesian network is a model that expresses a stochastic relationship (conditional independence) among multiple variables in a graph structure. Bayesian networks are suitable for data mining through unsupervised learning.

A support vector machine is a supervised learning model for pattern recognition and data analysis, and is mainly used for classification and regression analysis.

An artificial neural network is an information processing system in which a plurality of neurons called nodes or processing elements are connected in the form of a layer structure by modeling the operating principle of biological neurons and the connection relationship between neurons.

An artificial neural network is a model used in machine learning, a statistical learning algorithm inspired by neural networks in biology (particularly the brain in the central nervous system of animals) in machine learning and cognitive science.

Specifically, an artificial neural network may refer to an overall model that has problem-solving ability by changing synapse coupling strength through learning of artificial neurons (nodes) that form a network by synapse coupling.

The term artificial neural network may be used interchangeably with the term neural network.

An artificial neural network may include a plurality of layers, and each of the layers may include a plurality of neurons. In addition, the artificial neural network may include neurons and synapses connecting neurons.

Artificial neural networks generally use the following three factors: (1) connection patterns between neurons in different layers, (2) a learning process that updates the weights of connections, and (3) an output value from the weighted sum of the inputs received from the previous layer. It can be defined by the activation function you create.

Artificial neural networks may include network models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Bidirectional Recurrent Deep Neural Network (BRDNN), Multilayer Perceptron (MLP), and Convolutional Neural Network (CNN). , but not limited thereto.

In this specification, the term 'layer' may be used interchangeably with the term 'layer'.

Artificial neural networks are classified into single-layer neural networks and multi-layer neural networks according to the number of layers.

A typical single-layer neural network consists of an input layer and an output layer.

In addition, a general multilayer neural network is composed of an input layer, one or more hidden layers, and an output layer.

The input layer is a layer that accepts external data. The number of neurons in the input layer is the same as the number of input variables. The hidden layer is located between the input layer and the output layer. do. The output layer receives a signal from the hidden layer and outputs an output value based on the received signal. The input signal between neurons is multiplied by each connection strength (weight) and then summed. If this sum is greater than the neuron's threshold, the neuron is activated and outputs the output value obtained through the activation function.

Meanwhile, a deep neural network including a plurality of hidden layers between an input layer and an output layer may be a representative artificial neural network implementing deep learning, which is a type of machine learning technology.

Meanwhile, the term 'deep learning' may be used interchangeably with the term 'deep learning'.

The artificial neural network may be trained using training data. Here, learning may refer to a process of determining parameters of an artificial neural network using learning data in order to achieve a purpose such as classification, regression analysis, or clustering of input data. can As representative examples of parameters of an artificial neural network, a weight assigned to a synapse or a bias applied to a neuron may be cited.

An artificial neural network learned from training data may classify or cluster input data according to a pattern of the input data.

Meanwhile, an artificial neural network trained using training data may be referred to as a trained model in this specification.

Next, the learning method of the artificial neural network will be described.

Learning methods of artificial neural networks can be largely classified into supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

Supervised learning is a method of machine learning to infer a function from training data.

Among the inferred functions, outputting a continuous value is called regression analysis, and predicting and outputting the class of an input vector is called classification.

In supervised learning, an artificial neural network is trained under a given label for training data.

Here, the label may mean a correct answer (or a result value) to be inferred by the artificial neural network when training data is input to the artificial neural network.

In this specification, when training data is input, an answer (or a result value) to be inferred by an artificial neural network is referred to as a label or labeling data.

Also, in this specification, setting labels on training data for learning of an artificial neural network is referred to as labeling labeling data on training data.

In this case, training data and labels corresponding to the training data constitute one training set, and may be input to the artificial neural network in the form of a training set.

Meanwhile, the training data represents a plurality of features, and labeling the training data with a label may mean that a label is attached to a feature represented by the training data. In this case, the training data may represent the characteristics of the input object in the form of a vector.

The artificial neural network may use the training data and the labeling data to infer a function for a correlation between the training data and the labeling data. In addition, parameters of the artificial neural network may be determined (optimized) through evaluation of the function inferred from the artificial neural network.

Unsupervised learning is a type of machine learning in which labels are not given to the training data.

Specifically, unsupervised learning may be a learning method for learning an artificial neural network to find and classify a pattern in training data itself rather than an association between training data and a label corresponding to the training data.

Examples of unsupervised learning include clustering or independent component analysis.

In this specification, the term 'clustering' may be used interchangeably with the term 'clustering'.

Examples of artificial neural networks using unsupervised learning include a Generative Adversarial Network (GAN) and an Autoencoder (AE).

A generative adversarial network is a machine learning method in which two different artificial intelligences, a generator and a discriminator, compete to improve performance.

In this case, the generator is a model that creates new data and can generate new data based on original data.

In addition, the discriminator is a model that recognizes data patterns and can play a role in discriminating whether input data is original data or new data generated by a generator.

In addition, the generator learns by receiving data that has not deceived the discriminator, and the discriminator can learn by receiving deceived data from the generator. Accordingly, the generator can evolve to deceive the discriminator as best as possible, and the discriminator can evolve to distinguish well between the original data and the data generated by the generator.

An autoencoder is a neural network that aims to reproduce the input itself as an output.

An auto-encoder includes an input layer, at least one hidden layer, and an output layer.

In this case, since the number of nodes in the hidden layer is smaller than the number of nodes in the input layer, the dimensionality of data is reduced, and compression or encoding is performed accordingly.

Also, the data output from the hidden layer goes into the output layer. In this case, since the number of nodes in the output layer is greater than the number of nodes in the hidden layer, the dimensionality of data increases, and accordingly, decompression or decoding is performed.

On the other hand, the autoencoder adjusts the connection strength of neurons through learning, so that input data is expressed as hidden layer data. In the hidden layer, information is expressed with fewer neurons than in the input layer, and being able to reproduce input data as an output may mean that the hidden layer discovered and expressed a hidden pattern from the input data.

Quasi-supervised learning is a type of machine learning and may refer to a learning method using both labeled training data and unlabeled training data.

As one of the techniques of semi-supervised learning, there is a technique of inferring the label of unlabeled training data and then performing learning using the inferred label. This technique is useful when the cost required for labeling is high. can

Reinforcement learning is a theory that if an agent is given an environment in which it can judge what action to take every moment, it can find the best way through experience without data.

Reinforcement learning may be performed mainly by a Markov Decision Process (MDP).

To explain the Markov decision process, first, an environment in which the information necessary for the agent to take the next action is given, second, how the agent will behave in that environment, and third, if the agent does well, a reward ( Fourth, the optimal policy is derived by repeatedly experiencing it until the future reward reaches the highest point.

The structure of an artificial neural network is specified by model configuration, activation function, loss function or cost function, learning algorithm, optimization algorithm, etc., and hyperparameters are set in advance before learning. After setting, the model parameter (Model Parameter) is set through learning, so that the content can be specified.

For example, factors determining the structure of an artificial neural network may include the number of hidden layers, the number of hidden nodes included in each hidden layer, an input feature vector, a target feature vector, and the like.

Hyperparameters include various parameters that must be initially set for learning, such as initial values of model parameters. And, the model parameters include several parameters to be determined through learning.

For example, the hyperparameters may include an initial value of weight between nodes, an initial value of bias between nodes, a mini-batch size, a number of training iterations, a learning rate, and the like. In addition, model parameters may include weights between nodes, biases between nodes, and the like.

The loss function may be used as an index (reference) for determining optimal model parameters in the learning process of an artificial neural network. In an artificial neural network, learning means a process of manipulating model parameters to reduce a loss function, and the purpose of learning can be seen as determining model parameters that minimize a loss function.

The loss function may mainly use mean squared error (MSE) or cross entropy error (CEE), but the present invention is not limited thereto.

Cross entropy error can be used when the correct answer label is one-hot encoded. One-hot encoding is an encoding method in which the correct answer label value is set to 1 only for neurons corresponding to the correct answer, and the correct answer label value is set to 0 for neurons with no correct answer.

In machine learning or deep learning, learning optimization algorithms can be used to minimize the loss function, and learning optimization algorithms include Gradient Descent (GD), Stochastic Gradient Descent (SGD), Momentum ), NAG (Nesterov Accelerate Gradient), Adagrad, AdaDelta, RMSProp, Adam, and Nadam.

Gradient descent is a technique that adjusts model parameters in the direction of reducing the value of the loss function by considering the slope of the loss function in the current state.

A direction for adjusting model parameters is called a step direction, and a size for adjusting the model parameters is called a step size.

In this case, the step size may mean a learning rate.

In the gradient descent method, a gradient may be obtained by partial differentiation of a loss function with respective model parameters, and the model parameters may be updated by changing the model parameters in the direction of the obtained gradient by a learning rate.

Stochastic gradient descent is a technique that increases the frequency of gradient descent by dividing training data into mini-batches and performing gradient descent for each mini-batch.

Adagrad, AdaDelta, and RMSProp are techniques that increase optimization accuracy by adjusting the step size in SGD. In SGD, momentum and NAG are techniques that increase optimization accuracy by adjusting the step direction. Adam is a technique that increases optimization accuracy by adjusting the step size and step direction by combining momentum and RMSProp. Nadam is a technique that increases optimization accuracy by adjusting the step size and step direction by combining NAG and RMSProp.

The learning speed and accuracy of an artificial neural network are characterized by being largely dependent on hyperparameters as well as the structure of the artificial neural network and the type of learning optimization algorithm. Therefore, in order to obtain a good learning model, it is important to set appropriate hyperparameters as well as to determine an appropriate artificial neural network structure and learning algorithm.

Typically, hyperparameters are experimentally set to various values to train the artificial neural network, and as a result of learning, the optimal values are set to provide stable learning speed and accuracy.

According to an embodiment of the present invention, in relation to an artificial intelligence model that extracts the behavior of a structure through displacement change, various artificial intelligence models are created, trained, evaluated, completed, and using user's personal data. In the process of updating it, programs related to various artificial intelligence algorithms stored in the local area and the server 200 can be used.

2 is a diagram illustrating a server configuration according to an embodiment of the present invention.

Referring to FIG. 2 , the server 200 includes a communication unit 210, a data acquisition unit 220, a learning processor 230, a memory 240, and a processor 260.

The communication unit 210 is a component capable of being communicatively connected to the terminal 300 and the photographing device 100 . In detail, the communication unit 210 may receive an image of a structure photographed by the photographing device 100 and transmit a result of whether the behavior of the structure has changed through the received image to the terminal 300 .

The data acquisition unit 220 may acquire input data for model learning. The data acquisition unit 220 may acquire raw input data. In this case, the input data obtained from the processor 260 or the learning processor 230 is preprocessed to provide training data that can be input to model learning or preprocessed data. Allows you to generate input data.

Here, the preprocessing of the input data means extracting input features from the input data.

The learning processor 230 is a device that learns a model composed of an artificial neural network using trained data.

In detail, the learning processor 230 may determine optimized model parameters of the artificial neural network by iteratively training an artificial neural network capable of extracting an image capable of determining whether a structure is healthy using various learning techniques described above.

In the specification of the present invention, an artificial neural network whose parameters are determined by learning using training data may be referred to as a learning model or a trained model. In this case, the learning model may be used to infer a result value with respect to new input data other than training data.

Further, learning processor 230 may be configured to receive, classify, store, and output information to be used for data mining, data analysis, intelligent decision making, and machine learning algorithms and techniques.

And learning processor 230 generally processes one or more processes of data to identify, index, categorize, manipulate, store, retrieve, and output data for use in supervised or unsupervised learning, data mining, predictive analytics, or other machines. It can be configured to be stored in a database. Here, the database may be implemented using other accessible remote memories, such as the memory 240 and the memory of the terminal 300.

The information stored in this learning processor 230 may be used by the processor 260 or one or more other controllers using any of a variety of different types of data analysis algorithms and machine learning algorithms.

Examples of such algorithms include k-nearest neighbor systems, fuzzy logic (e.g. probability theory), neural networks, Boltzmann machines, vector quantization, pulsed neural networks, support vector machines, maximum margin classifiers, hill climbing, inductive logic systems, Bayesian networks , ferritnets (e.g. finite state machine, millie machine, Moore finite state machine), classifier trees (e.g. perceptron trees, support vector trees, markov trees, decision tree forests, random forests), reading models and systems, artificial It includes fusion, sensor fusion, image fusion, reinforcement learning, augmented reality, pattern recognition, automated planning, and more.

The memory 240 may store image data taken by the photographing device 100, and also store data supporting various functions performed by the server 200.

For example, the memory 240 determines the displacement of the structure based on a command for extracting an effective frame capable of confirming whether the location of the structure has changed through image data captured by the photographing device 100, and the extracted effective frame. It may store instructions, data for the running processor 230 to operate, and the like.

The memory 240 may store the learned model by dividing it into a plurality of versions according to the learning time or learning progress, if necessary.

In this case, the memory 240 may further store input data acquired by the data acquisition unit 220, learning data (or training data) for model learning, and a learning history of the model.

In this way, the input data stored in the memory 240 may be not only processed data suitable for model learning but also unprocessed input data itself.

This memory 240 may include a model storage unit 241 and a database 242 and the like.

The model storage unit 241 stores a model (or artificial neural network, 241a) that is being learned or learned through the learning processor 230, and can store the updated model when the model is updated through learning.

The model storage unit 241 may classify and store the learned model into a plurality of versions according to learning timing or learning progress, as needed.

The artificial neural network 241a shown in FIG. 2 is only one example of an artificial neural network including a plurality of hidden layers, and the artificial neural network of the present invention is not limited thereto.

In addition, the artificial neural network 241a may be implemented as hardware software or a combination of hardware and software. When part or all of the artificial neural network 241a is implemented as software, one or more instructions constituting the artificial neural network 241a may be stored in the memory 240 .

The database 242 stores image data acquired by the data acquisition unit 220, learning data (or training data) used for model learning, and a learning history of the model.

The input data stored in the database 242 may be not only processed data suitable for model learning, but also unprocessed input data itself.

The processor 260 may control general operations of the server 200 in addition to operations related to the application programs described above. The processor 260 may provide or process appropriate information or functions to a user by processing signals, data, information, etc. input or output through the components described above or by running an application program stored in the memory 170.

In addition, the processor 260 may determine or predict at least one executable operation of the server 200 based on information determined or generated using data analysis and machine learning algorithms. To this end, the processor 260 may request, retrieve, and receive data from the learning processor 230, and may control the server 200 to perform a predicted operation or an operation determined to be desirable among at least one executable operation. .

Processor 260 may perform various functions that implement intelligent emulation (ie, knowledge-based systems, reasoning systems, and knowledge acquisition systems). It can be applied to various types of systems (eg, fuzzy logic systems), including adaptive systems, machine learning systems, artificial neural networks, and the like.

In addition, the processor 260 may collect structure image information captured by the photographing device 100 and store it in the memory 240 . The best match for executing a specific function is determined using the structure image information stored in the processor 260, pre-stored usage history information, and predictive modeling.

When a specific operation is performed, the processor 260 analyzes history information indicating the execution of the specific operation through data analysis and machine learning algorithms and techniques, and updates previously learned information based on the analyzed information. can

Accordingly, processor 260, along with learning processor 230, may improve the accuracy of data analysis and future performance of machine learning algorithms and techniques based on the updated information.

3 is a view showing a structure displacement monitoring process using a structure displacement monitoring system according to an embodiment of the present invention, FIG. 4 is a view showing an example of monitoring a structure according to an embodiment of the present invention, and FIG. 5 and FIG. 6 is a diagram showing an example of checking structure displacement through an image frame in which a structure is photographed according to an embodiment of the present invention, and FIG. 7 is a diagram showing an example of an optical flow according to an embodiment of the present invention. , FIG. 8 is a diagram illustrating a process of calculating the behavior of a structure according to an embodiment of the present invention.

As shown in FIGS. 3 and 4 , in the structure displacement monitoring process, a structure such as a bridge or a large building is photographed, and an effective frame capable of determining whether the displacement of the structure has changed is extracted from the photographed image (step S110). ).

Specifically, a structure for determining whether displacement has occurred may be photographed using the photographing device 100 . The image captured by the photographing device 100 is corrected by the photographing device 100 or the server 200, and the soundness of the structure can be determined based on the corrected image. Hereinafter, an example in which image correction is performed through the server 200 in an embodiment of the present invention will be described.

As described, the imaging device 100 captures the structure as a 3D image. The 3D image taken in this way only represents the shape of the structure, but does not represent the displacement of the structure. Therefore, a change in displacement of a structure can be calculated only when the captured 3D image is corrected with a 2D image.

Looking at the image conversion process, the captured 3D image includes various images such as structures and surrounding environments. Among them, the image conversion process means a process of projecting points capable of displaying a structure into a two-dimensional image. That is, image conversion is to generate projected coordinates by projecting arbitrary coordinates among real coordinates included in a 3D captured image onto a 2D flat image. This coordinate projection can be converted through a homography matrix, and such image conversion is called camera calibration. In this case, the image conversion can be performed more accurately only when conditions for capturing an image by the photographing device 100 are removed, such as a lens, a distance from a structure, and an angle at which a structure is photographed.

The calibration process may use any one of methods such as, for example, a scale factor, a full projection matrix, and a planar homography matrix transforms (PHT). In the embodiment, an example of acquiring a 2D image from a 3D image using a homography matrix conversion (PHT) will be given.

Here, the homography matrix conversion method is expressed by the following equation.

Equation (1)

Here, X can be referred to as real coordinates, and x refers to projected coordinates that are coordinates extracted from a 2D image. In addition, H is one of homography matrices that are 3x3 matrices, and s means an image scale.

The homography matrix conversion method is characterized in that the original image can be re-projected and the image can be projected without distortion even if the axis of the camera lens and the axis of the target structure are not parallel. Specifically, referring to FIG. 5, the PHT selects four random coordinates on a 3D image (see a, b, c, and d in FIG. 5 (a)) and projects the projected coordinates (SC_ 6 (a)) can be generated (see a, b, c and d in FIG. 5 (b)).

At this time, the projection coordinate (SC) projected to the two-dimensional image in Equation (1) is x, and can be calculated as the product of the real coordinate (X), the inverse matrix of the homography matrix (H), and the inverse function of the image scale (S). there is.

Here, the projection coordinate (SC) x is expressed by the following equation.

Equation (2)

Here, I _ij means the gray scale value (gray value_contrast value) of the pixel (i, j) of the 3D image, and m and n mean the region of interest (RoI) in the structure to be photographed. .

In addition, the image scale may mean a ratio between a size of a real object and an image size, a ratio between a distance from a camera to a target to be measured and a focal length, and the like.

In this way, when the projected coordinates SC of the 2D image are extracted from arbitrary coordinates of the 3D image, it is possible to determine an effective frame capable of inferring whether displacement has occurred in the structure.

The determined effective frame may be determined based on deep learning design for feature detection. For example, referring to FIG. 6 , when designing a structure, reference coordinates (BC) that can refer to the exact position of the structure may be set. A 2D image in which the projected coordinates SC derived from the 2D image calibrated based on the reference coordinates BC deviate from the reference coordinates BC may be determined as an effective frame ((a) of FIG. 6 ).

That is, the motion of the structure is determined by measuring the motion of the structure based on the reference coordinates (BC), and may be defined as points that can distinguish the structure from the surrounding environment on the image, such as the edge of the structure. In addition, the effective frame may refer to an image when the projected coordinates SC deviate from the reference coordinates BC by comparing movement occurring over time from the reference coordinates BC.

As described above, the motion of the structure can be determined by comparing the projected coordinates (SC) projected into a 2D image based on the defined reference coordinates (BC) with the reference coordinates (BC). It can be recognized (Fig. 6 (b)). These recognized valid frames can be confirmed as effective coordinates in real time (FIG. 6(c)).

In this case, the effective coordinate means that the behavior of the structure is converted into a wavelet by connecting the coordinates where the movement occurs among the previously acquired projection coordinates SC.

Here, in order to determine an effective frame, an image positioned behind the focus of the 2D image may have a lower weight and an image positioned in front of the focus of the 2D image may have an increased weight. That is, image noise can be reduced by weighting data close to the focus.

In this case, deep learning algorithms such as Convolutional Neural Networks (CNN), Boltzmann Machines, Belief Networks, Recurrent Neural Networks, Auto-encoders, and Generative Adversarial Networks (GANs) may be used as the deep learning algorithm for determining the effective frame. Here, a drop-out layer with a different drop-out ratio and a Max Pooling layer after the CNN classify the recorded behavior and identify the behavior by utilizing the transfer learning system (FC). can do.

If valid frames are extracted from the photographed image as described above, optical flow for the effective frames may be estimated (step S120).

Optical flow means estimating motion at the edge of a structure in which a structure and a background in a photographed image can be distinguished. Referring to FIG. 7, when there is a moving object, the object located at the coordinates (x, y) at time t considers the change in contrast appearing in the adjacent image in the image to estimate the coordinates at which the object will be located at a later time t + 1. It is an algorithm that can estimate the motion of An effective frame for estimating the optical flow may be performed through a 2-dimensional 2-dimensional image extracted from a 3-dimensional 3-dimensional image, and may be estimated through a deep learning algorithm.

For example, FlowNet or FlowNet2 may be used as a deep learning algorithm for estimating the optical flow for an effective frame. FlowNet or FlowNet2 stacks different effective frames in which the behavior of the structure has been transformed into wavelets through effective coordinates and passes through a generic network with 9 convolution layers, so that the network becomes It is an algorithm that allows itself to decide how to process image pairs to extract structure behavior information.

Here, the different effective frames stacked may be successive images according to the lapse of time, or may be successive images corresponding to each other.

사이즈의 CNN 필터(Kernel)를 가지며, 두 번째, 세 번째 계층의 필터는

, 네 번째부터 아홉 번째 필터는

의 사이즈를 갖는다. 각각의 계층의 차원은 다음과 같이 conv1 (

), conv2 (

), conv3 (

), conv3_1 (

), conv4 (

), conv4_1 (

), conv5 (

), conv5_1 (

), conv6 (

)로 구성된다. 마지막으로, Coarse feature map의 개선작업을 추가하여 해상도를 원본 이미지에 가깝게 올려주어 광학 흐름 예측이 수행될 수 있도록 한다. In FlowNet-S, the first layer is

It has a CNN filter (Kernel) of the size, and the filters of the second and third layers are

, the fourth through ninth filters are

has the size of The dimension of each layer is conv1 (

), conv2 (

), conv3 (

), conv3_1 (

), conv4 (

), conv4_1 (

), conv5 (

), conv5_1 (

), conv6 (

) is composed of Finally, by adding refinement of the coarse feature map, the resolution is raised close to the original image so that optical flow prediction can be performed.

Specifically, FlowNet-S can improve the accuracy of projected coordinates (SC). Depending on the resolution of the image, the line forming the edge of the structure may be inaccurate. Through FlowNet-S, the background and the boundary of the structure can be clearly distinguished, and the projection coordinates (SC) can be projected more clearly so that optical Allows flow to be estimated more accurately.

Instead of directly stacking two images, FlowNet-C first feeds the two images individually into three convolution layers and then combines them together with a correlation layer. The optical flow of the effective frame can be calculated by adding improvement work after the remaining 6 convolution layers.

FlowNet2 has the effect of improving the accuracy and speed of the existing FlowNet by stacking the previously described FlowNet-C and FlowNet-S based on FlowNet and integrating a sub-network specialized for small behavior.

Specifically, when FlowNet (or FlowNet2, etc.) is applied to an effective frame obtained from a 3D image, reliability of projection coordinates (SC) results in the obtained effective frame can be improved.

If the optical flow for the effective frame is estimated in this way, the soundness of the structures displayed in the effective frame can be determined based on the estimated optical flow (step S130).

Before determining the soundness of the structure, shake can be corrected according to whether there is shake in the extracted effective frame. In the process of photographing a structure with the photographing apparatus 100, vibration of the photographing apparatus 100 may occur due to environmental factors in which the photographing apparatus 100 is installed, such as vibration of the ground or wind. The shaking of the photographing device 100 may be mixed with the behavior of the structure, which may cause an error in the process of calculating the integrity of the structure. Therefore, in order to eliminate the shaking error of the photographing apparatus 100, a behavior measurement sensor capable of measuring the behavior of the photographing apparatus 100 may be installed in the photographing apparatus 100.

In addition, the photographing device 100 includes a wavelet filter, which is one of programs for removing noise from an image (image) having noise and recognizing a pattern in the image, to improve the signal for the behavior of the structure. can

In conclusion, when taking pictures using the photographing apparatus 100, the photographing apparatus 100 may be shaken by a change in the surrounding environment (eg, earthquake, wind, etc.). Therefore, the image captured by the photographing device 100 may include not only the behavior of the structure but also the shaking of the photographing device 100 and be photographed. At this time, a wavelet filter is installed in the imaging device 100, and only motion of the imaging device 100 can be removed through the installed wavelet filter, so that an effective frame capable of determining the soundness of the structure can be obtained.

In this way, after correcting the shaking of the effective frame represented by the 2D image, the real coordinates may be converted using the homography matrix conversion method again to extract the real coordinates from the 2D image.

Here, the actual coordinates (X) may be derived by sequentially multiplying the scale of the 2D image, the transformation matrix using the homography matrix, and the image coordinates extracted using pixels of the 2D image.

See equation (1)

That is, in order to extract real coordinates, a 2D image is extracted by applying a homography matrix conversion method to a 3D image. It is to apply the homography matrix conversion method again to the extracted 2D image so that the actual coordinates can be extracted.

As shown in FIG. 8 , an image of a structure located in a region of interest where displacement of the structure occurs is captured. Since the captured image is 3D, the homography matrix conversion method during image calibration can be applied to the 3D image to obtain a change value with respect to time of projection coordinates (SC) that can represent motion (FIG. 8 (a ) Note: image conversion_calibration).

The acquired projected coordinates (SC) are substituted into the homography matrix again to be converted into real coordinates representing the real structure (see (b) of FIG. 8).

Thereafter, the distance between the projected coordinates (SC) extracted by image calibration and the actual coordinates is calculated to calculate the displacement of the actual structure. In this case, the behavioral displacement of the actual structure can be applied to a Gaussian Kernel (refer to (c) of FIG. 8).

A Gaussian filter is expressed by the following equation.

Equation (3)

는 실수 x를 입력값(input)으로 취하고 x보다 작거나 같은 가장 큰 정수를 출력값(output)으로 제공하는 플로어 함수(floor function)일 수 있다. where w and h are the width and height of the image,

may be a floor function that takes a real number x as an input and provides the largest integer less than or equal to x as an output.

In summary, effective coordinates where behavior has occurred can be extracted through projected coordinates (SC) extracted from a two-dimensional image, and only images from which effective coordinates have been extracted can be set as valid frames. Since this valid frame setting extracts only the necessary area from the entire image (image), the size (size) of the image to check the behavior of the structure can be minimized.

In particular, it is possible to more accurately track the behavior of a structure in an image having only a frame in which a structure extracted as an effective frame moves. As described above, the motion of a structure acquired through optical flow, which means estimating motion at the edge of a structure in which a structure and a background in a photographed image can be distinguished, may be a motion at the edge of a structure.

At this time, the soundness (d) of the structure can be more specifically calculated by the following equation.

Equation (4)

Here, X _i and X _o are the current location coordinates and original location coordinates of image pixels, and * may be an element-by-element multiplication operator.

Specifically, X _o : coordinates in a frame of an image capturing an actual structure, and X _i : coordinates in an i-th frame of an image capturing an actual structure.

Accordingly, X _o -X _i may mean a displacement vector.

If the soundness of the structure is calculated in this way, the soundness of the calculated structure can be evaluated. Referring to the drawings below, a process of evaluating the soundness of the calculated structure will be described.

9 is a diagram schematically illustrating a process of measuring and evaluating the soundness of a structure according to an embodiment of the present invention.

Referring to FIG. 9 , the behavior of a structure may be measured based on a photographed image using a deep learning algorithm. At this time, the behavior of the structure may be estimated in the time and frequency domains by applying wavelet analysis. A wavelet refers to a wave that rapidly increases and decelerates around zero. Similarly, wavelet analysis means analyzing a case where the projected coordinates SC are out of the reference coordinates BC based on the reference coordinates BC.

Specifically, image processing may be performed on an image captured by the photographing device 100 . Through image processing, it is possible to acquire data on the characteristics, defects, etc. of a structure for learning structural problems and behavior of a structure.

Thereafter, an effective frame is extracted from the processed image, and the soundness of the structure is calculated by estimating the optical flow in the extracted effective frame.

As described above, a structure in which soundness may occur is photographed, and the soundness of the structure can be monitored through the captured image using a deep learning algorithm. At this time, since the soundness of the structure can be measured without separately installing a device for measuring the displacement of the structure, the cost of installing a device for evaluating the soundness of the structure can be effectively reduced.

In addition, the structure displacement monitoring of the present invention can predict the structure displacement using artificial intelligence technology. As a result, the displacement of the structure can be measured more quickly, and problems that may occur in the structure can be prevented in advance.

Embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed on a computer through various components, and such a computer program may be recorded on a computer-readable medium. At this time, the medium is a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a ROM hardware devices specially configured to store and execute program instructions, such as RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software. An example of a computer program may include not only machine language code generated by a compiler but also high-level language code that can be executed by a computer using an interpreter or the like.

In the specification of the present invention (particularly in the claims), the use of the term "above" and similar indicating terms may correspond to both singular and plural. In addition, when a range is described in the present invention, it includes the invention to which individual values belonging to the range are applied (unless there is a description to the contrary), as if each individual value constituting the range was described in the detailed description of the invention. .

The steps may be performed in any suitable order, provided that the order of steps constituting the method according to the present invention is not explicitly stated or stated to the contrary. The present invention is not necessarily limited by the order of description of the steps. In addition, the steps included in the methods according to the present invention may be performed by a processor or modules for performing the functions of the steps. The use of all examples or exemplary terms (eg, etc.) in the present invention is simply to explain the present invention in detail, and the scope of the present invention is limited due to the above examples or exemplary terms that are not limited by the claims. It is not. In addition, those skilled in the art can appreciate that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or equivalents thereof.

Therefore, the spirit of the present invention should not be limited to the described embodiments and should not be determined, and not only the claims described below but also all ranges equivalent to or equivalent to these claims belong to the scope of the spirit of the present invention. will do it

Claims (10)

As a method of monitoring structural health,
extracting a series of valid frames for predicting the health of the structure from an image of the structure for monitoring the health;
estimating an optical flow for the series of valid frames; and
determining the integrity of structures represented in the series of valid frames based on the optical flow;
Structural health monitoring methods.
According to claim 1,
The extraction step is
Comprising the step of correcting the three-dimensional image taken of the structure into a two-dimensional image,
The correcting step is
selecting an arbitrary coordinate in the 3D image; and
Generating projection coordinates projected onto the two-dimensional image based on the arbitrary coordinates,
Structural health monitoring methods.
According to claim 2,
After generating the projected coordinates,
Extracting real coordinates by sequentially multiplying the scale of the two-dimensional image, a transformation matrix using a homography matrix, and the projection coordinates,
Structural health monitoring methods.
According to claim 3,
The determining step is
Applying the real coordinates to a Gaussian Kernel;
The Gaussian filter (Gaussian Kernel),

(Where w and h are the width and height of the image,

is a floor function that takes as input a real number x and gives as output the largest integer less than or equal to x)
Structural health monitoring methods.
According to claim 4,
The applying step is
The integrity (d) of the structure is

which is calculated as
(Where X _i and X _o are the current position coordinates and original position coordinates of image pixels, and * is an element-by-element multiplication operator)
Structural health monitoring methods.
According to claim 1,
The extraction step is
Extracting projection coordinates based on feature detection from the two-dimensional image;
The determining step is
Comprising the step of determining the movement of the projected coordinates by comparing the previously stored reference coordinates with the projected coordinates,
Structural health monitoring methods.
According to claim 6,
The extraction step is
Lowering the weight of an image located behind the focus of the two-dimensional image and increasing the weight of an image located in front of the focus of the two-dimensional image.
Structural health monitoring methods.
According to claim 1,
The estimating step is
correcting shaking of the effective frame; and
Including the step of inputting the shake-corrected effective frames into a pre-trained optical flow estimation prediction model,
Structural health monitoring methods.
According to claim 8,
The input step is
In order to extract the integrity of the structure, stacking different valid frames and passing through a network with 9 convolution layers (Generic Network),
The passing step is
Using at least one of FlowNet-S, FlowNet-C, and FlowNet2 among the optical flow estimation prediction models,
Structural health monitoring methods.
As a system for monitoring the integrity of a structure,
at least one processor; and
a memory operatively connected to the processor and storing at least one code executed by the processor;
When the memory is executed by the processor, it causes the processor to:
After extracting a series of effective frames for predicting the health of the structure from a video taken of a structure for monitoring structural health, and estimating the optical flow for the series of effective frames, storing a code for determining the integrity of a structure represented in the series of valid frames based on the optical flow;
Structural health monitoring system.

Images