KR102293547B1 - Method and apparatus for detecting change - Google Patents

Abstract

According to an embodiment of the present disclosure, as a change detection method performed by a computing device including at least one processor. The method comprises the steps of: obtaining an image pair which is a change detection target, - the image pair include a reference image and a target image-; calculating feature data for each of the reference image and the target image using a neural network-based feature extraction model; and calculating change region information appearing in the target image from the feature data of the reference image and the feature data of the target image by using a neural network-based change detection model.

Description

Change detection method and device

The present invention relates to the field of artificial intelligence, and more particularly, to an artificial intelligence-based change detection method.

With the development of image processing technology using artificial neural networks, various methods for detecting a change in an image and detecting a changed region using a neural network have been disclosed. Typically, in the art, a neural network having a U-net structure has been used for change detection.

However, in the neural network of the U-net structure, the submodel of the encoder role and the submodel of the decoder role are connected in a skip connection form, so there is a problem in that it is difficult to replace or change each submodel.

This problem causes the inconvenience of having to change the overall structure of the model even when the feature representation for the input image can be more precisely generated through the change of the encoder submodel. Therefore, the demand in the art for a neural network capable of effectively performing change detection while each submodel is separated and has an independent neural network structure has been continuously increasing.

Korean Patent No. KR2189926 discloses "a method and system for detecting a change in a region of interest".

The present disclosure has been devised in response to the above-mentioned background art, and aims to provide information on the result of detecting changes based on an image using an artificial neural network.

A change detection method performed by a computing device including at least one processor according to an embodiment of the present disclosure for realizing the above-described problem, the method comprising: an image pair to be changed detection obtaining - the image pair includes a reference image and a target image; calculating feature data for each of the reference image and the target image using a neural network-based feature extraction model; and calculating change region information appearing in the target image from the feature data of the reference image and the feature data of the target image by using a neural network-based change detection model.

In an alternative embodiment, the neural network-based feature extraction model comprises at least one downsampling layer for data compression, wherein the downsampling layer comprises at least one convolutional filter and at least one One pooling filter may be included.

In an alternative embodiment, the neural network-based change detection model may include two or more pipelines, and the two or more pipelines may each include at least one upsampling layer.

In an alternative embodiment, the calculating of the change region information may include new intermediate data different from the plurality of intermediate data based on a plurality of intermediate data separately calculated from different layers respectively included in the two or more pipelines. calculating ; and calculating the change region information based on the new intermediate data.

In an alternative embodiment, the calculating of the change region information may include: first intermediate data calculated from a layer included in a first pipeline among the two or more pipelines; second intermediate data calculated from a layer included in a second pipeline among the two or more pipelines; and calculating the change region information based on a plurality of accumulated weights used for an operation related to the second pipeline.

In an alternative embodiment, each of the plurality of accumulated weights may correspond to each of a plurality of predetermined data related to the second pipeline.

In an alternative embodiment, the plurality of cumulative weights includes a first cumulative weight and a second cumulative weight, and the first cumulative weight corresponding to data calculated in a preceding calculation step is data calculated in a subsequent calculation step. may have a value smaller than the second cumulative weight corresponding to .

In an alternative embodiment, the change detection method may further include determining an image pair to be changed detection target from an image set including a plurality of images.

In an alternative embodiment, the determining of the image pair to be the target of the change detection comprises: generating a plurality of candidate image pairs consisting of two consecutive images in chronological order from the image set; calculating change occurrence information from the generated plurality of pair of candidate images; and determining an image pair that is a target of change detection based on the change occurrence information.

In an alternative embodiment, the calculating of the change occurrence information may include calculating a change score for each of the plurality of candidate image pairs based on a neural network-based change classification model.

In an alternative embodiment, the determining of an image pair to be subjected to change detection based on the change occurrence information includes: determining a priority according to a size of a change score included in the change occurrence information; and determining an image pair to be subjected to change detection based on the priority.

In an alternative embodiment, the neural network-based change classification model may include: at least one normal image pair including two images in which the degree of change between the two images is lower than a predetermined reference value; and at least one abnormal image pair including two images in which a degree of change between the two images is greater than a predetermined reference value.

Disclosed is a computer program stored in a computer-readable storage medium according to an embodiment of the present disclosure for realizing the above-described problems. The computer program, when executed on one or more processors, causes the following operations to be performed for detecting a change, the operations comprising: an image pair subject to change detection, the image pair being a reference image and a target image comprising - obtaining ; calculating feature data for each of the reference image and the target image by using a neural network-based feature extraction model; and calculating change region information appearing in the target image from the feature data of the reference image and the feature data of the target image by using a neural network-based change detection model.

A change detection apparatus is disclosed according to an embodiment of the present disclosure for realizing the above-described problems. The apparatus may include one or more processors; one or more memories; and a network unit, wherein the one or more processors are configured to: obtain an image pair to be subjected to change detection, wherein the image pair includes a reference image and a target image, and use a neural network-based feature extraction model to calculate feature data for each of the reference image and the target image, and a change region appearing in the target image from the feature data for the reference image and the feature data for the target image using a neural network-based change detection model information can be calculated.

The present disclosure may provide a change detection method.

1 is a block diagram of a computing device for change detection according to an embodiment of the present disclosure.
2 is a schematic diagram illustrating a network function according to an embodiment of the present disclosure.
3 is a conceptual diagram exemplarily illustrating two or more pipelines and a data flow according to each pipeline according to an embodiment of the present disclosure.
4 is a conceptual diagram illustrating a neural network-based feature extraction model and a neural network-based change detection model according to an embodiment of the present disclosure.
5 is a flowchart illustrating a process in which a computing device detects a change according to an embodiment of the present disclosure.
6 is a flowchart illustrating a process of determining, by a computing device, an image pair to be changed detection target according to an embodiment of the present disclosure.
7 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is apparent that these embodiments may be practiced without these specific descriptions.

The terms “component,” “module,” “system,” and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or execution of software. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and/or thread of execution. A component may be localized within one computer. A component may be distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored therein. Components may communicate via a network such as the Internet with another system, for example via a signal having one or more data packets (eg, data and/or signals from one component interacting with another component in a local system, distributed system, etc.) may communicate via local and/or remote processes depending on the data being transmitted).

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from context, "X employs A or B" is intended to mean one of the natural implicit substitutions. That is, X employs A; X employs B; or when X employs both A and B, "X employs A or B" may apply to either of these cases. It should also be understood that the term “and/or” as used herein refers to and includes all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and/or "comprising" should be understood to mean that the feature and/or element in question is present. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, elements and/or groups thereof. Also, unless otherwise specified or unless the context is clear as to designating a singular form, the singular in the specification and claims should generally be construed to mean "one or more."

In addition, the term “at least one of A or B” should be interpreted as meaning “includes only A”, “includes only B”, and “in the case of a combination of A and B”.

Those skilled in the art will further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or combinations of both. It should be recognized that they can be implemented with To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

Descriptions of the presented embodiments are provided to enable those skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art of the present disclosure. The generic principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments presented herein. This invention is to be construed in its widest scope consistent with the principles and novel features presented herein.

1 is a block diagram of a computing device for change detection according to an embodiment of the present disclosure. The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In an embodiment of the present disclosure, the computing device 100 may include other components for performing the computing environment of the computing device 100 , and only some of the disclosed components may configure the computing device 100 .

The computing device 100 may include a processor 110 , a memory 130 , and a network unit 150 .

The processor 110 may include one or more cores, and a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU). unit) and the like, and may include a processor for data analysis and deep learning. The processor 110 may read a computer program stored in the memory 130 and perform data processing for learning of a neural network-based model or inference of a neural network-based model according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the processor 110 may perform an operation for learning the neural network. The processor 110 for learning of the neural network, such as processing input data for learning in deep learning (DL), extracting features from the input data, calculating an error, updating the weight of the neural network using backpropagation calculations can be performed. At least one of a CPU, a GPGPU, and a TPU of the processor 110 may process learning of the neural network. For example, the CPU and the GPGPU can process neural network training and data classification using the neural network together. Also, in an embodiment of the present disclosure, learning of a neural network and data classification using a neural network may be processed by using the processors of a plurality of computing devices together. In addition, the computer program executed in the computing device according to an embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

According to an embodiment of the present disclosure, the memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150 . The memory 130 may store at least one parameter included in models based on the neural network. The memory 130 may store at least some of the parameter values of the neural network-based model received by the network unit 150 .

According to an embodiment of the present disclosure, the memory 130 is a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, SD or XD memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read (PROM) -Only Memory), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The computing device 100 may operate in relation to a web storage that performs a storage function of the memory 130 on the Internet. The description of the above-described memory is only an example, and the present disclosure is not limited thereto.

In the present disclosure, the network unit 150 may use various communication systems regardless of its communication mode, such as wired and wireless. The network unit 150 may receive an input image from an external device or an external server using a communication system such as wired or wireless. The network unit 150 may receive at least a portion of parameter values of the model learned from an external device or an external server using a communication system such as a wired or wireless communication system.

In an embodiment according to the present disclosure, the processor 110 may acquire an image pair that is a target of change detection. The processor 110 may acquire an image pair that is a target of change detection by loading it from a memory. The processor 110 may acquire an image pair by receiving image data transmitted from an external server through the network unit 150 . The processor 110 may acquire an image pair that is a target of change detection through a series of operations for selecting two images from among a plurality of images and determining the image pair. The image pair may include a reference image and a target image. The processor 110 may calculate change information of the target image based on the reference image.

In an embodiment according to the present disclosure, the processor 110 may calculate feature data for each of a reference image and a target image included in an image pair by using a neural network-based feature extraction model.

2 is a schematic diagram illustrating a network function according to an embodiment of the present disclosure.

In the present disclosure, terms such as “neural network-based X model” and “neural network-based Y model” refer to an operation performed by at least some of the nodes included in the neural network on input data, and the result of the operation is It can be used to refer to a neural network-based model that generates output data based on it. In terms of "neural network-based X model" and "neural network-based Y model", "X" and "Y" may be used to distinguish a structure of a neural network from each other. Throughout this specification, "neural network based X model" and "neural network based Y model" may be used interchangeably as "X model" and "Y model" respectively, for short.

In the present disclosure, the terms 'neural network', 'artificial neural network', 'network function', and 'neural network' may be used interchangeably. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network is configured to include at least one node. Nodes (or neurons) constituting the neural networks may be interconnected by one or more links.

In the neural network, one or more nodes connected through a link may relatively form a relationship between an input node and an output node. The concepts of an input node and an output node are relative, and any node in an output node relationship with respect to one node may be in an input node relationship in a relationship with another node, and vice versa. As described above, an input node-to-output node relationship may be created around a link. One or more output nodes may be connected to one input node through a link, and vice versa.

In the relationship between the input node and the output node connected through one link, the value of the data of the output node may be determined based on data input to the input node. Here, a link interconnecting the input node and the output node may have a weight. The weight may be variable, and may be changed by the user or algorithm in order for the neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node sets values input to input nodes connected to the output node and links corresponding to the respective input nodes. An output node value may be determined based on the weight.

As described above, in a neural network, one or more nodes are interconnected through one or more links to form an input node and an output node relationship within the neural network. The characteristics of the neural network may be determined according to the number of nodes and links in the neural network, the correlation between the nodes and the links, and the value of a weight assigned to each of the links. For example, when the same number of nodes and links exist and there are two neural networks having different weight values of the links, the two neural networks may be recognized as different from each other.

A neural network may include one or more layers. A layer may contain one or more nodes. Some of the nodes constituting the neural network may configure one layer based on distances from the initial input node. For example, a set of nodes having a distance of n from the first input node may constitute an n-th layer. The distance from the initial input node may be defined by the minimum number of links that must be traversed to reach the corresponding node from the initial input node. However, the definition of such a layer is arbitrary for description, and the order of the layer in the neural network may be defined in a different way from the above. For example, a layer of nodes may be defined by a distance from the final output node.

The initial input node may refer to one or more nodes to which data is directly input without going through a link in a relationship with other nodes among nodes in the neural network. Alternatively, in a relationship between nodes based on a link in a neural network, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network. In addition, the hidden node may mean nodes constituting the neural network other than the first input node and the last output node.

The neural network according to an embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as progresses from the input layer to the hidden layer. can In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be less than the number of nodes in the output layer, and the number of nodes decreases as the number of nodes progresses from the input layer to the hidden layer. have. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as the input layer progresses to the hidden layer. can The neural network according to another embodiment of the present disclosure may be a neural network in a combined form of the aforementioned neural networks.

A neural network according to an embodiment of the present disclosure may include a plurality of neural network layers. The plurality of neural network layers may constitute a sequence having a predetermined order in the neural network according to their functions and roles. The plurality of neural network layers may include, for example, a convolutional layer, a pooling layer, a fully connected layer, and the like. The initial input to the neural network may be received by the first layer among the layers included in the sequence. A neural network may sequentially input an initial input to layers in a sequence to generate a final output from the initial input. The initial input may be, for example, an image and the final output may be, for example, a score for each category in a set of categories comprising one or more categories.

The neural network layer according to an embodiment of the present disclosure may include at least one node. A weight or bias may be assigned to each node included in the neural network layer. The memory 130 of the computing device 100 according to the present disclosure may store a weight or bias value assigned to at least one node included in the neural network layer. Each neural network layer may receive as an input the first input to the convolutional neural network or the output of the previous neural network layer. For example, in a sequence including a plurality of neural network layers, the N-th neural network layer may receive the output of the N-1 th neural network layer as an input. Each neural network layer can generate an output from its input. When the neural network layer is the highest and last neural network layer in the sequence, the output of this neural network layer can be treated as the output of the entire neural network.

A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to identify the latent structures of data. In other words, it can identify the potential structure of photos, texts, videos, voices, and music (for example, what objects are in the photos, what the text and emotions are, what the texts and emotions are, etc.) . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, generative adversarial networks (GANs), and restricted boltzmann machines (RBMs). machine), a deep belief network (DBN), a Q network, a U network, a Siamese network, a Generative Adversarial Network (GAN), and the like. The description of the deep neural network described above is only an example, and the present disclosure is not limited thereto.

In an embodiment of the present disclosure, the network function may include an autoencoder. The auto-encoder may be a kind of artificial neural network for outputting output data similar to input data. The auto encoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between the input/output layers. The number of nodes in each layer may be reduced from the number of nodes of the input layer to an intermediate layer called the bottleneck layer (encoding), and then expanded symmetrically with reduction from the bottleneck layer to the output layer (symmetrically with the input layer). The auto-encoder can perform non-linear dimensionality reduction. The number of input layers and output layers may correspond to a dimension after preprocessing the input data. In the auto-encoder structure, the number of nodes of the hidden layer included in the encoder may have a structure that decreases as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes between the encoder and the decoder) is too small, a sufficient amount of information may not be conveyed, so a certain number or more (e.g., more than half of the input layer, etc.) ) may be maintained.

The neural network may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Learning of the neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

A neural network can be trained in a way that minimizes output errors. In the training of a neural network, iteratively inputs the training data to the neural network, calculates the output of the neural network and the target error for the training data, and calculates the error of the neural network from the output layer of the neural network to the input layer in the direction to reduce the error. It is a process of updating the weight of each node in the neural network by backpropagation in the direction. In the case of teacher learning, learning data in which the correct answer is labeled in each learning data is used (ie, labeled learning data), and in the case of comparative learning, the correct answer may not be labeled in each learning data. That is, for example, the learning data in the case of teacher learning regarding data classification may be data in which categories are labeled in each of the learning data. Labeled training data is input to the neural network, and an error can be calculated by comparing the output of the neural network with the label of the training data. In the present disclosure, neural network-based models may calculate an error by, for example, comparing a prediction label for a query sample with a training correct label for a query sample. As another example, in the case of comparison learning about data classification, an error may be calculated by comparing the input training data with the neural network output. The calculated error is back propagated in the reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back propagation. A change amount of the connection weight of each node to be updated may be determined according to a learning rate. The computation of the neural network on the input data and the backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of repetitions of the learning cycle of the neural network. For example, in the early stage of training of a neural network, a high learning rate can be used to enable the neural network to quickly acquire a certain level of performance, thereby increasing efficiency, and using a low learning rate at the end of learning can increase accuracy.

In the training of neural networks, in general, the training data may be a subset of real data (that is, data to be processed using the trained neural network), and thus the error on the training data is reduced, but the error on the real data is reduced. There may be increasing learning cycles. Overfitting is a phenomenon in which errors on actual data increase by over-learning on training data as described above. For example, a phenomenon in which a neural network that has learned a cat by showing a yellow cat does not recognize that it is a cat when it sees a cat other than yellow may be a type of overfitting. Overfitting can act as a cause of increasing errors in machine learning algorithms. In order to prevent such overfitting, various optimization methods can be used. In order to prevent overfitting, methods such as increasing the training data, regularization, dropout that deactivate some nodes of the network in the process of learning, and the use of a batch normalization layer can be applied.

In an embodiment of the present disclosure, the feature data that the processor 110 calculates for each image by using the neural network-based feature extraction model may be tensor data. Tensor data is a data structure having a predetermined n dimension, and n may be an integer greater than or equal to 0. For example, 0-dimensional tensor data may be expressed as a scalar, 1-dimensional tensor data may be expressed as a vector, and 2D tensor data may be expressed as a data structure in the form of a matrix. When the feature data is 2D tensor data, the feature data may be referred to as a 'feature map'. When the feature data consists of a plurality of feature maps, the feature data may be 3D tensor data having a channel, a height, and a width as axes, respectively.

In the present disclosure, the feature data that the processor 110 calculates for each image is data for expressing features inherent in the image. The processor 110 may calculate feature data for the image by mapping the image to the latent space using the feature extraction model.

The neural network-based feature extraction model according to an embodiment of the present disclosure may include at least one downsampling layer for data compression. The initial input data of the feature extraction model may be a reference image or a target image. The input data of each layer included in the feature extraction model may be output data of one of the preceding layers in the order of operation. The first layer among the layers included in the feature extraction model may use a reference image or a target image as input data.

, 출력되는 데이터는

로 호칭될 수 있다. 이 때

의 크기가 512*512*C(C는 자연수)인 경우, k번째 다운샘플링 레이어는 입력 데이터의 높이와 너비를 반으로 축소시키되, 입력 데이터에 N(N은 자연수)개의 서로 다른 필터를 적용함으로써 256*256*N의 크기를 갖는

를 출력할 수 있다. 전술한 예시는 다운샘플링의 데이터 연산 방법을 설명하기 위한 일 예시일 뿐, 본 개시를 제한하지 않는다.The downsampling layer included in the neural network-based feature extraction model according to an embodiment of the present disclosure may reduce the height and width of input data of the layer to generate output data of the layer. The downsampling layer may include a plurality of filters for extracting features from the input data. Output data of the downsampling layer may have the same number of channels as the number of filters included in the downsampling layer. In an embodiment, data input to the k-th downsampling layer among at least one downsampling layer included in the neural network-based feature extraction model is

, the output data is

may be called At this time

If the size of is 512*512*C (C is a natural number), the k-th downsampling layer reduces the height and width of the input data in half, but by applying N (N is a natural number) different filters to the input data. It has a size of 256*256*N.

can be printed out. The above-described example is only an example for describing a data operation method of downsampling, and does not limit the present disclosure.

The downsampling layer according to an embodiment of the present disclosure may include at least one convolutional filter and at least one pooling filter. The processor 110 may generate output data of the downsampling layer based on output data generated by the convolutional filter included in the downsampling layer and output data generated by the pooling filter included in the downsampling layer. . For example, the convolutional filter included in the downsampling layer may have a window size of 3×3, and a size of a stride may be 2. Also for example, the pulling filter may be a max pulling filter or an average pulling filter. When the stride size of the convolutional filter is 2 or more, the output data can be generated by reducing the height and width of the input data.

According to an embodiment of the present disclosure, the processor 110 may generate output data based on an operation of concatenating the output data generated by the convolutional filter and the output data generated by the pooling filter. When the output data of each filter is a 'feature map', the processor 110 may concatenate each feature map by aligning it in the channel axis direction. Specifically, when the size of the feature map generated by the convolutional filter is 16*16*1 and the size of the feature map generated by the pooling filter is also 16*16*1, the processor 110 generates two feature maps can be concatenated to generate output data having a size of 16*16*2. The above-described example is only an example in which the processor 110 generates output data based on the downsampling layer, and the processor 110 generates a feature map generated by the convolutional filter and a feature map generated by the pooling filter. Output data may be generated based on an operation of summing or averaging or weighted average.

The processor 110 may calculate feature data for each of the reference image and the target image based on the neural network-based feature extraction model including at least one downsampling layer as described above. The feature data may be three-dimensional tensor data. When the pooling filter and the convolutional filter are used together in the downsampling layer according to the present disclosure, some data loss problems caused in the process of data compression can be prevented, and feature data for an image can be effectively generated. That is, the method using the downsampling layer according to the present disclosure has an advantage in that it can generate more sophisticated feature data than the method of calculating feature data for an image using only the pooling filter.

The processor 110 may generate initial input data for the change detection model by concatenating the feature data of the reference image and the feature data of the target image to calculate change region information appearing in the target image. The processor 110 may generate initial input data for the change detection model by summing or averaging the feature data of the reference image and the feature data of the target image.

The processor 110 may calculate change region information appearing in the target image from the feature data of the reference image and the feature data of the target image by using the neural network-based change detection model.

In the present disclosure, the change region information may include information related to a change region appearing in the target image compared to the reference image. The change region information may include coordinate values related to the change region appearing in the target image. For example, the coordinate values for the changed region may include coordinate values of each vertex of a polygon including the changed region or coordinate values of pixels included in the changed region. The change region information may be a classification value for at least some pixels included in the target image. The processor 110 may allocate a classification value to each pixel. For example, the processor 110 assigns a value of 1 to the corresponding pixel when it is determined that a pixel included in the target image has changed based on respective pixel values corresponding to the same position in the reference image and the target image. can Conversely, when it is determined that a pixel included in the target image is not changed, a value of 0 may be assigned to the corresponding pixel. When there are a plurality of change states to be expressed through the calculated change region information, the processor 110 may assign different values to each change state in order to distinguish the plurality of change states from each other. A state of change may include a degree of change or an aspect of the change. The degree of change may be classified based on, for example, a confidence score calculated by the change detection model during a calculation process. Aspects of the change may include, for example, 'when a building disappears', 'when a building is created', or 'when replaced with a specific object'.

In an embodiment of the present disclosure, the neural network-based detection model may include two or more pipelines. In the present disclosure, 'pipeline' may be used as a term for expressing 'at least one layer set'. The pipeline may include at least one layer for data operation. A plurality of pipelines may be distinguished from each other by a layer included in the corresponding pipeline. Also, the plurality of pipelines may be classified based on the number and type of layers included in the corresponding pipeline. For example, when at least one of the layers included in each of the two pipelines is different, the two pipelines may be different pipelines. Also, for example, in a calculation process performed when the processor 110 calculates output data from input data based on a neural network-based change detection model, the calculation process may include a plurality of sub-operation processes. In this case, when layers used in the plurality of sub-operations are different, the plurality of sub-operations may be operation processes that follow different pipelines.

In the present disclosure, when the neural network-based detection model includes a plurality of pipelines different from each other, the processor 110 may independently perform a data operation along each pipeline. When the processor 110 independently performs data operations along each pipeline, data operations along different pipelines are respectively performed by different processors or computing devices, respectively. In addition, when the processor 110 independently performs data operations along each pipeline, data operations along different pipelines are respectively processed in parallel.

Two or more pipelines according to an embodiment of the present disclosure may each include at least one upsampling layer. The upsampling layer may increase the size of the height and width of the input data. The upsampling layer may increase the height and width of the input data by performing a deconvolution operation. Since the deconvolution operation itself is an algorithm generally known as an operation opposite to the convolution operation, a detailed description thereof will be omitted. The parameters included in the filter used for the deconvolution operation may be updated when the neural network-based change detection model is trained. A specific learning method of the change detection model will be described later in detail.

According to an embodiment of the present disclosure, the step of calculating, by the processor 110, change region information using the change detection model is performed on a plurality of intermediate data individually calculated from different layers included in two or more pipelines. It may include calculating new intermediate data different from the plurality of intermediate data based on the plurality of intermediate data. The new intermediate data may mean intermediate data calculated based on the existing intermediate data. Hereinafter, it will be described in detail with reference to FIG. 3 .

In the present disclosure before referring to FIG. 3 , when a plurality of layers are included in the neural network-based change detection model, 'output data of each layer' may be used interchangeably with 'intermediate data'. When a plurality of layers included in the change detection model have a series order (ie sequence) according to the operation processing order, in the present disclosure, in order to eliminate ambiguity between the output data of each layer and the final output data, the 'last layer' among the plurality of layers 'output data of' may be used interchangeably with 'output data of the feature extraction model' or 'final output data of the feature extraction model'.

등과 매칭되는 블록(block)들은 서로 다른 레이어로부터 각각 산출된 중간 데이터들을 도시한 것이다.3 is a conceptual diagram exemplarily illustrating two or more pipelines and a data flow according to each pipeline according to an embodiment of the present disclosure. in Figure 3

Blocks matching the etc. show intermediate data respectively calculated from different layers.

' 형태로 표현될 수 있다.

에 있어서, 'A'는 해당 중간 데이터를 생성하기 위해 사용된 레이어의 종류가 'A'임을 나타낸다.

에 있어서, 'x'는 'A' 유형의 레이어에 의해 생성된 중간 데이터가 포함된 파이프라인의 번호를 나타낸다. 기호

에 있어서, 'y'는 'x'번째 파이프라인에 포함되되, 'A' 유형의 레이어에 의해 생성된 중간 데이터의 번호를 나타낸다. In the present disclosure, the intermediate data each calculated from a plurality of layers is denoted as '

' can be expressed in the form

In , 'A' indicates that the type of layer used to generate the corresponding intermediate data is 'A'.

In , 'x' represents the number of the pipeline including the intermediate data generated by the 'A' type layer. sign

In , 'y' represents the number of intermediate data included in the 'x'-th pipeline and generated by the 'A' type layer.

In FIG. 3 , intermediate data in which A is U may indicate that a layer used to generate the corresponding intermediate data is an upsampling layer. Also, when A is D, it may indicate that a layer used to generate the corresponding intermediate data is a non-upsampling layer (ie, a layer other than the upsampling layer). The non-upsampling layer includes, without limitation, various types of layers for generating output data from input data without increasing the height and width of the input data. For example, the non-upsampling layer may be a residual layer. Unlike a general layer, the residual layer may generate secondary output data by additionally performing an operation of adding up input data input to the residual layer to primary output data of the residual layer. The primary output data means an operation result based on at least one node included in the residual layer before summing the input data. The secondary output data refers to a result of adding the input data input to the residual layer to the primary output data of the residual layer.

은 제 1 파이프라인에 포함된 첫번째 업샘플링 레이어에 의해 생성된 중간 데이터를 나타낼 수 있다.

은 제 1 파이프라인에 포함된 첫번째 비-업샘플링 레이어에 의해, 중간 데이터

에 기초하여 생성된 상이한 중간 데이터를 나타낼 수 있다. 또한

은 제 2 파이프라인에 포함된 첫번째 업샘플링 레이어에 의해, 중간 데이터

및

에 기초하여 생성된 중간 데이터를 나타낼 수 있다.3 of

may represent intermediate data generated by the first upsampling layer included in the first pipeline.

by the first non-upsampling layer included in the first pipeline,

may represent different intermediate data generated based on . In addition

is the intermediate data by the first upsampling layer included in the second pipeline

and

Intermediate data generated based on .

를 연산하기 위해 중간 데이터

및

에 기초할 수 있다. 또한, 프로세서(110)는

으로 표현되는 중간 데이터를 연산하기 위해

,

및

로 표현되는 중간 데이터에 기초할 수 있다. 도 3에 도시된 중간 데이터

은 변화 탐지 모델의 최종 출력 데이터를 나타낼 수 있다. 최종 출력 데이터는 목표 이미지와 높이 및 너비의 크기가 동일할 수 있다. 최종 출력 데이터는 변화 여부를 표현하기 위한 바이너리 클래스를 포함할 수 있다. 최종 출력 데이터는 변화의 정도 및 종류를 표현하기 위해 복수의 채널을 포함할 수도 있다.The solid line shown in FIG. 3 represents the flow of the calculation process related to the first pipeline. The broken line shown in FIG. 3 represents the flow of the calculation process related to the second pipeline. In one embodiment, the processor 110 provides intermediate data

intermediate data to compute

and

can be based on In addition, the processor 110

To compute the intermediate data expressed as

,

and

It may be based on intermediate data expressed as . Intermediate data shown in FIG. 3

may represent final output data of the change detection model. The final output data may have the same height and width as the target image. The final output data may include a binary class for expressing whether or not there is a change. The final output data may include a plurality of channels to represent the degree and type of change.

In general, when output data is calculated from input data using a single pipeline, most of the information included in the input data disappears due to repeated operations on the input data. However, the change detection method according to the present disclosure discloses a method based on two or more pipelines to detect a change appearing in a target image by utilizing information included in the characteristic data of the reference image and the characteristic data of the target image. That is, as described above, the processor 110, based on the first intermediate data generated from the first pipeline among the two or more pipelines included in the change detection model, and the second intermediate data generated from the second pipeline, Intermediate data different from the first and second intermediate data may be calculated, and change region information may be calculated based thereon. This method calculates a final output based on additional intermediate data - output by a layer located earlier in the sequence of the pipeline - in which relatively less important information contained in the feature data generated for the reference image and the target image is lost. has one characteristic. As in the present disclosure, when intermediate data respectively calculated by a plurality of pipelines are preserved and utilized in a change detection process, change detection performance is improved. Hereinafter, a specific embodiment of calculating different intermediate data based on a plurality of intermediate data will be further described.

According to an embodiment of the present disclosure, the processor 110 may include first intermediate data calculated from a layer included in a first pipeline among two or more pipelines included in the change detection model, a second one of the two or more pipelines. Change region information may be calculated based on the second intermediate data calculated from the layers included in the second pipeline and a plurality of accumulated weights used for an operation related to the second pipeline.

In the present disclosure, terms such as “first” and “second” are used to distinguish one component from other components, and are used only to maintain consistency of referenced objects throughout the specification. The scope of rights should not be limited. Accordingly, the entire specification may be renamed as “first” as “second” and “second” as “first” as necessary.

은 제 1 중간 데이터들을 나타낼 수 있다. 본 개시내용에 있어서 '제 N(N은 자연수) 중간 데이터'라는 용어는 '제 N 파이프라인에 포함된 레이어로부터 산출된 중간 데이터'와 상호 교환적으로 사용될 수 있다. 제 N 중간 데이터가 도 3에 도시된 바와 같이 복수 개인 경우 각각의 제 N 중간 데이터는 동일 파이프라인 내에서 산출되는 시간순서에 따라 다시 제 N-1 중간 데이터(e.g.

), 제 N-2 중간 데이터(e.g.

), 제 N-3 중간 데이터(e.g.

)와 같이 구분될 수 있다. 도 3의 참조번호 310은 둘 이상의 데이터에 대한 합산 연산을 나타내는 연산 기호를 나타낸다. 도 3의

는 각각 제 2 파이프라인과 관련된 연산을 위해 사용되는 복수의 누적 가중치를 나타낸다. Referring back to Figure 3,

may represent first intermediate data. In the present disclosure, the term 'Nth (N is a natural number) intermediate data' may be used interchangeably with 'intermediate data calculated from a layer included in the Nth pipeline'. When there are a plurality of N-th intermediate data as shown in FIG. 3 , each N-th intermediate data is again the N-1th intermediate data (eg, according to the time sequence calculated in the same pipeline).

), the N-2th intermediate data (eg

), the N-3 intermediate data (eg

) can be distinguished as Reference numeral 310 of FIG. 3 denotes an operation symbol indicating a summation operation for two or more data. 3 of

denotes a plurality of accumulated weights each used for an operation related to the second pipeline.

데이터를 산출할 수 있다.In one embodiment of the present disclosure, the processor 110 as in Equation 1

data can be calculated.

[Equation 1]

은

에 곱해지기 위한 누적 가중치이다.

과

은 서로 같은 크기의 데이터로서 합산될 수 있다.

은

데이터를 출력하는 업샘플링 레이어에 의한 연산 과정을 나타내는 함수 기호이다.here

silver

is the cumulative weight to be multiplied by .

class

may be summed as data having the same size as each other.

silver

It is a function symbol indicating the operation process by the upsampling layer that outputs data.

데이터를 산출할 수 있다.In one embodiment of the present disclosure, the processor 110 as in Equation 2

data can be calculated.

[Equation 2]

은 수학식 1에 기초하여 다른 중간 데이터(e.g.

,

)로부터 산출된 중간 데이터이다.

및

는 각각 도 3에 도시된 바와 같이, 사전 결정된 데이터들 각각에 대응하는 누적 가중치이다.

은

데이터를 출력하는 업샘플링 레이어에 의한 연산 과정을 나타내는 함수 기호이다.

,

, 및

는 서로 같은 크기의 데이터로서 합산될 수 있다. 수학식 2에 기초한

데이터는 다른 레이어에 입력되기 위한 중간 데이터일 수 있다. 이 때, 도 3의 실시예와 같이

데이터가 변화 탐지 모델에 의한 연산 과정의 마지막 중간 데이터인 경우, 상기

데이터는 변화 탐지 모델의 출력 데이터일 수 있다. 프로세서(110)는 변화 탐지 모델의 출력 데이터에 기초하여 변화 영역 정보를 산출할 수 있다.

is another intermediate data (eg, based on Equation 1)

,

) is the intermediate data calculated from

and

As shown in FIG. 3 , respectively, are cumulative weights corresponding to each of the predetermined pieces of data.

silver

It is a function symbol indicating the operation process by the upsampling layer that outputs data.

,

, and

may be summed as data of the same size as each other. based on Equation 2

The data may be intermediate data to be input to another layer. At this time, as in the embodiment of FIG.

When the data is the last intermediate data of the calculation process by the change detection model, the

The data may be output data of the change detection model. The processor 110 may calculate change region information based on output data of the change detection model.

The above-described example has been exemplarily described with reference to Equations 1 and 2, and describes in detail a data calculation process using two pipelines, that is, a first pipeline and a second pipeline. An embodiment in which there are three pipelines will be described later with reference to FIG. 4 . However, the present disclosure includes, without limitation, embodiments in which an additional pipeline is added to two pipelines, and those skilled in the art based on N (N is a natural number) pipelines from the contents disclosed above and the contents to be described later. It will be easy to understand how to perform the change detection method according to the present disclosure.

은 제 1 파이프라인의 첫번째 업샘플링 레이어에 의해 출력되는 제 1-1 중간 데이터(i.e.

)에 대응되는 누적 가중치일 수 있다. 수학식 2에서

은 제 2 파이프라인의 첫번째 업샘플링 레이어에 의해 출력되는 제 2-1 중간 데이터(i.e.

)에 대응되는 누적 가중치일 수 있다. 수학식 2에서

은 제 1 파이프라인의 두번째 업샘플링 레이어에 의해 출력되는 제 1-2 중간 데이터(i.e.

)와 제 2 파이프라인의 첫번째 업샘플링 레이어에 의해 출력되는 제 2-1 중간 데이터(i.e.

)를 합산한 데이터에 대응되는 누적 가중치일 수 있다.In an embodiment of the present disclosure, each of the plurality of accumulated weights may correspond to each of a plurality of predetermined data related to the second pipeline included in the change detection model. The plurality of predetermined data related to the second pipeline may include first intermediate data or second intermediate data. Referring back to the above example, in Equation 1

is the 1-1 intermediate data output by the first upsampling layer of the first pipeline (ie

) may be a cumulative weight corresponding to . in Equation 2

is the 2-1 th intermediate data output by the first upsampling layer of the second pipeline (ie

) may be a cumulative weight corresponding to . in Equation 2

is the first 1-2 intermediate data output by the second upsampling layer of the first pipeline (ie

) and the 2-1 intermediate data output by the first upsampling layer of the second pipeline (ie

) may be an accumulated weight corresponding to the summed data.

The plurality of accumulated weights according to the present disclosure may be updated during the training process of the neural network-based change detection model. For example, the processor 110 may change the values in the backpropagation process by setting each of the accumulated weights as an updateable and differentiable variable rather than a fixed constant. The cumulative weights according to the present disclosure may be set to a predetermined value.

을 연산하기 위해서는

이 필요하므로,

을 연산하는 단계는

을 연산하는 단계보다 후행 단계이다. 마찬가지로 수학식 2에 있어서

을 연산하기 위해서는

이 필요하므로,

을 연산하는 단계는

을 연산하는 단계보다 후행 단계이다. 위의 관계에 따라, 계속된 실시예에서 도 3에 도시된 복수의 누적 가중치들은

의 관계를 가질 수 있다. 즉, 본 개시내용은 후행하는 연산 단계에서 산출된 데이터에 대응되는 누적 가중치를 선행하는 연산 단계에서 산출된 데이터에 대응되는 누적 가중치보다 크게 설정할 수 있다. 이러한 본 개시내용에 따를 경우, 하위 레벨 정보를 갖는 중간 데이터를 반영하여 최종 출력 데이터를 생성하되, 최종 출력 데이터에 가까운 중간 데이터일수록 높은 누적 가중치를 부여하여 구체화 수준이 높은 데이터의 영향력을 높일 수 있다. 이는 복수의 중간 데이터에 포함된 정보를 활용하되, 중간 데이터들의 추상화 정도(또는 구체화 정도)에 따라 차등을 두어 보다 효과적으로 변화를 탐지하는 효과를 갖는다.In an embodiment of the present disclosure, the processor 110 applies a first cumulative weight corresponding to data calculated in a preceding operation step among the plurality of cumulative weights to data calculated in a subsequent calculation step among the plurality of cumulative weights. It may be set to be smaller than the corresponding second cumulative weight. In the present disclosure, the determination of the preceding and following of the calculation step is relative according to the calculation step as a reference. corresponds to the stage. For example, in Equation 1

In order to compute

Since this is needed,

The steps to calculate

It is a later step than the step of calculating . Similarly, in Equation 2

In order to compute

Since this is needed,

The steps to calculate

It is a later step than the step of calculating . According to the above relationship, in the continued embodiment, the plurality of accumulated weights shown in Fig. 3 is

can have a relationship with That is, according to the present disclosure, the cumulative weight corresponding to the data calculated in the subsequent calculation step may be set to be greater than the cumulative weight corresponding to the data calculated in the preceding calculation step. According to the present disclosure, the final output data is generated by reflecting the intermediate data having low-level information, but the closer the intermediate data to the final output data, the higher the cumulative weight is given to increase the influence of data having a high level of refinement. . This has the effect of more effectively detecting a change by utilizing information included in a plurality of intermediate data, but making a difference according to the degree of abstraction (or degree of refinement) of the intermediate data.

4 is a conceptual diagram illustrating a neural network-based feature extraction model and a neural network-based change detection model according to an embodiment of the present disclosure. The processor 110 may input the reference image 411 and the target image 413 to the neural network-based feature extraction model 430 , respectively. The processor 110 may acquire the feature data 431 of the reference image and the feature data 433 of the target image based on the operation result of the feature extraction model 430 . The processor 110 may input the combined data 435 combining the feature data 431 for the reference image and the feature data 433 for the target image into the neural network-based change detection model 450 . The processor 110 may calculate change region information 451 for the target image from the output data of the change detection model 450 .

는 제 2 파이프라인과 관련된 연산을 위해 사용되는 복수의 누적 가중치를 나타낸다.

는 제 3 파이프라인과 관련된 연산을 위해 사용되는 복수의 누적 가중치를 나타낸다. Hereinafter, the neural network-based change detection model 450 of FIG. 4 will be described in more detail. The neural network-based change detection model 450 of FIG. 4 is an extended embodiment of the example of FIG. 3 , and content overlapping with those described with reference to FIG. 3 will be omitted, and differences will be mainly described below. The solid line shown inside the neural network-based change detection model 450 of FIG. 4 represents the flow of data along the first pipeline. The broken line shown inside the neural network-based change detection model 450 represents the flow of data along the second pipeline. The dashed-dotted line shown inside the neural network-based change detection model 450 represents the flow of data along the third pipeline.

denotes a plurality of accumulated weights used for an operation related to the second pipeline.

denotes a plurality of accumulated weights used for an operation related to the third pipeline.

은 수학식 3과 같이 계산될 수 있다.In the embodiment of Fig. 4, intermediate data

can be calculated as in Equation (3).

[Equation 3]

은

데이터를 출력하는 업샘플링 레이어에 의한 연산 과정을 나타내는 함수 기호이다. 중간 데이터인

,

,

, 및

는 서로 같은 크기를 갖는 데이터로서 합산될 수 있다.

silver

It is a function symbol indicating the operation process by the upsampling layer that outputs data. intermediate data.

,

,

, and

may be summed as data having the same size as each other.

은 수학식 4와 같이 계산될 수 있다.In the embodiment of Fig. 4, intermediate data

can be calculated as in Equation (4).

[Equation 4]

은

데이터를 출력하는 업샘플링 레이어에 의한 연산 과정을 나타내는 함수 기호이다. 도 4의 실시예에 있어서, 중간 데이터

은 수학식 5와 같이 계산될 수 있다.

silver

It is a function symbol indicating the operation process by the upsampling layer that outputs data. In the embodiment of Fig. 4, intermediate data

can be calculated as in Equation 5.

[Equation 5]

은

데이터를 출력하는 업샘플링 레이어에 의한 연산 과정을 나타내는 함수 기호이다. 수학식 5에 있어서 중간 데이터인

,

,

,

, 및

는 서로 같은 크기를 갖는 데이터로서 합산될 수 있다.

silver

It is a function symbol indicating the operation process by the upsampling layer that outputs data. In Equation 5, the intermediate data

,

,

,

, and

may be summed as data having the same size as each other.

The neural network-based change detection model 450 according to the present disclosure may generate final output data based on Equations 3 to 5 described above. The processor 110 may calculate change region information displayed in the target image based on the final output data.

In general, when a change region in an image is to be detected using a neural network-based model, a skip connection structure in which at least a portion of a submodel for extracting a feature and a submodel for detecting a change region is connected. However, this structure has the disadvantage of making it difficult to replace or replace only the submodel for extracting features. On the other hand, the change detection method according to the present disclosure outputs information about a changed region in the target image without a structure connected between the feature extraction model and the change detection model. Furthermore, the method according to the present disclosure discloses a technique in which a change detection model detects a change by using a plurality of pipelines in order to effectively perform the change detection method without a conventional connection structure between submodels.

The neural network-based feature extraction model and the neural network-based change detection model according to the present disclosure may be simultaneously trained end-to-end. Referring back to FIG. 4 , in an embodiment, the processor 110 inputs the reference image 411 and the target image 413 , and then adds the final output change region information 451 and the target image 413 to the It is possible to train the feature extraction model and the change detection model end-to-end by comparing the training labels for each other and backpropagating the error. In another embodiment, the processor 110 fixes the parameters of the separately learned neural network-based feature extraction model 430 and updates only the parameters included in the neural network-based change detection model 450 to learn neural network-based models. may proceed.

The change detection method of the present disclosure may include determining an image pair to be changed detection target from an image set including a plurality of images. The processor 110 according to the present disclosure may perform change detection on the determined image pair after performing a separate step of determining an image pair to be changed detection.

The step of the processor 110 determining the image pair to be the target of change detection according to an embodiment of the present disclosure may include generating a plurality of candidate image pairs consisting of two consecutive images in chronological order from the image set. can The image set may include a plurality of image subsets. Each of the plurality of image subsets included in the image set may be classified based on a photographing target. The image subset may include a plurality of images of the same subject. For example, the image set may include a plurality of images of the same area, building, or object.

The processor 110 may align a plurality of images included in the image set based on time information corresponding to the images. The processor 110 may configure a pair of candidate images with two consecutive images in time order. Two images included in the pair of candidate images may be distinguished as a candidate reference image and a candidate target image. In this case, the candidate reference image may precede the candidate target image in chronological order. The processor 110 may generate a plurality of pair of candidate images.

According to an embodiment of the present disclosure, the processor 110 may include calculating change occurrence information from a plurality of generated pair of candidate images. The processor 110 may calculate change occurrence information for each pair of candidate images. The change occurrence information may include a data value related to whether a change has occurred in the candidate target image compared to the candidate reference image included in the pair of candidate images. The change occurrence information may include a classification result value indicating whether a change has occurred in the target image. The change occurrence information may include a change score indicating a degree to which a change has occurred in the target image. For example, the classification result value included in the change occurrence information may be 0 or 1, 'T' or 'F'. When the classification result value included in the calculated change occurrence information is 0 or 'F', the processor 110 may determine that no change has occurred in the corresponding candidate image pair. Similarly, when the classification result value included in the calculated change occurrence information is 1 or 'T', the processor 110 may determine that a change has occurred in the corresponding candidate image pair. Also, for example, the change score may be a value indicating the degree of change and may be a probability value according to the degree. As an example, assume that the processor 110 calculates a change score between 0 and 1. In this case, as the change score calculated by the processor 110 for the pair of candidate images is closer to 1, it may mean that the degree of change between the two images is greater.

The processor 110 according to an embodiment of the present disclosure may be based on a reference point-based change detection technique to calculate change occurrence information. The processor 110 may select at least one corresponding pixel pair each having the same coordinates in two images included in the candidate image pair. The processor 110 may measure a change amount between at least one corresponding pixel pair and calculate change occurrence information based thereon. The processor 110 may use at least one piece of predetermined coordinate data to select at least one corresponding pixel pair from the candidate image pair.

The processor 110 according to an embodiment of the present disclosure may be based on a neural network-based change classification model to calculate change occurrence information. The processor 110 may calculate a change score and/or a classification result value for each of the plurality of candidate image pairs by using the change classification model. The neural network-based change classification model may include at least one node. The description related to the neural network has been described above with reference to FIG. 2 , and differences will be mainly described below. The processor 110 may input a pair of candidate images to the change classification model, and may calculate a change score and/or a classification result indicating a degree of change between a candidate reference image and a candidate target image included in the candidate image pair.

The change classification model according to an embodiment of the present disclosure may be learned by training data including a plurality of training image pairs and a classification label for each training image pair. In this case, the change classification model may be finally learned based on the calculation of the value distribution and the error of the actual classification label after calculating the probability distribution for the plurality of classification labels. The plurality of training image pairs for training the change classification model include at least one normal image pair consisting of two images in which the degree of change between the two images is lower than a predetermined reference value and two images in which the degree of change between the two images is greater than the predetermined reference value. It may include at least one abnormal image pair consisting of images. A change degree value between two images included in the training image pair, which should be calculated to generate a plurality of training image pairs, may be calculated by the above-described reference point-based change detection technique. The predetermined reference value may be set as a value for distinguishing a normal image pair from an abnormal image pair. The predetermined reference value may be determined as a value for distinguishing the two groups by comparing the distribution of change degree values between the two images regarding the normal image pair and the distribution of change degree values between the two images regarding the abnormal image pair.

According to an embodiment of the present disclosure, the processor 110 may include determining, by the processor 110, an image pair as a target of change detection based on the calculated change occurrence information. According to an embodiment, the processor 110 may determine an image pair to be changed detection based on a classification result value included in the change occurrence information. For example, when the classification result calculated for [the first candidate image pair, the second candidate image pair, and the third candidate image pair] is [1, 0, 1], the processor 110 performs the first candidate image pair and the third candidate image pair may be determined as image pairs to be changed detection targets, respectively.

In an embodiment of the present disclosure, the processor 110 may determine a priority according to a size of a change score included in the change occurrence information, and then determine an image pair to be changed detection based on the priority. For example, when the calculated change score for [the first candidate image pair, the second candidate image pair, and the third candidate image pair] is [0.7, 0.9, 0.3], the processor 110 determines that the change score is the highest. The second candidate image pair may be determined as an image pair to be subjected to change detection. Also, for example, since the processor 110 gives priority according to the size of the change score, when change detection for the second candidate image pair is finished, the first candidate image pair is set as an image pair to be changed detection. can decide

As described above, a configuration for determining an image pair to be a target of change detection from an image set including a plurality of images is particularly useful when there are too many images in the image set. When it is impossible in time to perform change detection on all image pairs using a neural network-based change detection model, the present disclosure can effectively determine an image pair to be subjected to change detection from an image set. For example, when there is an image set for satellite photos of the same area, a significant number of images of the same area at multiple times may be included. In addition, when a single image data capacity is large, such as a satellite image, performing precise detection of all image pairs using a neural network-based change detection model may cause inefficiency. On the other hand, the present disclosure discloses a method of firstly giving priority to change detection by quickly searching a plurality of candidate image pairs, and secondly determining an image pair to be subjected to change detection according to the priority of change detection. . According to the present disclosure, it is possible to efficiently search the entire database while selecting a target image with a high probability of occurrence of a change.

5 is a flowchart illustrating a process in which a computing device detects a change according to an embodiment of the present disclosure. In step S510 , the processor 110 may acquire an image pair that is a target of change detection. The image pair to be detected for change may include a reference image and a target image. In operation S530 , the processor 110 may calculate feature data for each of the reference image included in the image pair and the target image included in the image pair by using the neural network-based feature extraction model. The feature extraction model may calculate feature data for the input image in the form of a feature map or a plurality of feature maps (i.e. three-dimensional tensors) by embedding the input image in the latent space. In operation S550 , the processor 110 may calculate change region information appearing in the target image from the feature data of the reference image and the feature data of the target image using the neural network-based change detection model. The processor 110 may calculate change region information appearing in the target image by inputting the feature data of the reference image and the feature data of the target image to the change detection model. A change detection model may include at least two pipelines. The pipeline may include at least one upsampling layer. The change detection model may increase the height and width of the input feature data by using at least one upsampling layer included in each pipeline. The change detection model may reduce a size of a channel of the input feature data, and the height and width may be based on at least one upsampling layer to match the height and width of the target image. The processor 110 may detect a change in an image pair that is a target of change detection through change region information output by the change detection model.

6 is a flowchart illustrating a process of determining, by a computing device, an image pair to be changed detection target according to an embodiment of the present disclosure. Each step of FIG. 6 may be performed by the processor 110 before step S510 of FIG. 5 . In step S610 , the processor 110 may generate a plurality of candidate image pairs including two consecutive images in chronological order from an image set including a plurality of images. Each of the plurality of pair of candidate images may include a candidate reference image and a candidate target image. In step S630 , the processor 110 may calculate change occurrence information from a plurality of pair of candidate images. The change occurrence information may include, for example, a classification result value indicating whether there is a change, a change score indicating a degree of change, and the like. Change occurrence information may be calculated by a reference point-based change detection technique. Change occurrence information may be calculated by a learned neural network-based change classification model. In step S650 , the processor 110 may determine an image pair to be changed detection target based on the change occurrence information. The processor 110 may determine a priority according to the magnitude of a change score included in the calculated change occurrence information, and determine an image pair to be detected for change based on the determined priority.

7 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented. Although the present disclosure has been described above generally as being capable of being implemented by a computing device, those skilled in the art will appreciate that the present disclosure is a combination of hardware and software and/or in combination with computer-executable instructions and/or other program modules that may be executed on one or more computers. It will be appreciated that it can be implemented as a combination.

Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In addition, those skilled in the art will appreciate that the methods of the present disclosure are suitable for single-processor or multiprocessor computer systems, minicomputers, mainframe computers as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which is It will be appreciated that other computer system configurations may be implemented, including those that may operate in connection with one or more associated devices.

The described embodiments of the present disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Any medium accessible by a computer can be a computer-readable medium, and such computer-readable media includes volatile and nonvolatile media, transitory and non-transitory media, removable and non-transitory media. including removable media. By way of example, and not limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media includes volatile and non-volatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. includes media. A computer-readable storage medium may be RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device, or other magnetic storage device. device, or any other medium that can be accessed by a computer and used to store the desired information.

Computer readable transmission media typically embodies computer readable instructions, data structures, program modules or other data, etc. in a modulated data signal such as a carrier wave or other transport mechanism, and Includes any information delivery medium. The term modulated data signal means a signal in which one or more of the characteristics of the signal is set or changed so as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

An example environment 1100 implementing various aspects of the disclosure is shown including a computer 1102 , the computer 1102 including a processing unit 1104 , a system memory 1106 , and a system bus 1108 . do. The system bus 1108 couples system components, including but not limited to system memory 1106 , to the processing device 1104 . The processing device 1104 may be any of a variety of commercially available processors. Dual processor and other multiprocessor architectures may also be used as processing unit 1104 .

The system bus 1108 may be any of several types of bus structures that may further be interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 1106 includes read only memory (ROM) 1110 and random access memory (RAM) 1112 . A basic input/output system (BIOS) is stored in non-volatile memory 1110, such as ROM, EPROM, EEPROM, etc., which is the basic input/output system (BIOS) that helps transfer information between components within computer 1102, such as during startup. contains routines. RAM 1112 may also include high-speed RAM, such as static RAM, for caching data.

The computer 1102 may also be configured with an internal hard disk drive (HDD) 1114 (eg, EIDE, SATA) - this internal hard disk drive 1114 may also be configured for external use within a suitable chassis (not shown). Yes—a magnetic floppy disk drive (FDD) 1116 (eg, for reading from or writing to removable diskette 1118), and an optical disk drive 1120 (eg, a CD-ROM) for reading from, or writing to, disk 1122, or other high capacity optical media such as DVD. The hard disk drive 1114 , the magnetic disk drive 1116 , and the optical disk drive 1120 are connected to the system bus 1108 by the hard disk drive interface 1124 , the magnetic disk drive interface 1126 , and the optical drive interface 1128 , respectively. ) can be connected to The interface 1124 for external drive implementation includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the description of computer readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will use zip drives, magnetic cassettes, flash memory cards, cartridges, etc. It will be appreciated that other tangible computer-readable media such as etc. may also be used in the exemplary operating environment and any such media may include computer-executable instructions for performing the methods of the present disclosure.

A number of program modules may be stored in the drive and RAM 1112 , including an operating system 1130 , one or more application programs 1132 , other program modules 1134 , and program data 1136 . All or portions of the operating system, applications, modules, and/or data may also be cached in RAM 1112 . It will be appreciated that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer 1102 via one or more wired/wireless input devices, for example, a pointing device such as a keyboard 1138 and a mouse 1140 . Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. Although these and other input devices are often connected to the processing unit 1104 through an input device interface 1142 that is connected to the system bus 1108, parallel ports, IEEE 1394 serial ports, game ports, USB ports, IR interfaces, It may be connected by other interfaces, etc.

A monitor 1144 or other type of display device is also coupled to the system bus 1108 via an interface, such as a video adapter 1146 . In addition to the monitor 1144, the computer typically includes other peripheral output devices (not shown), such as speakers, printers, and the like.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148 via wired and/or wireless communications. Remote computer(s) 1148 may be workstations, computing device computers, routers, personal computers, portable computers, microprocessor-based entertainment devices, peer devices, or other common network nodes, and are typically connected to computer 1102 . Although it includes many or all of the components described for it, only memory storage device 1150 is shown for simplicity. The logical connections shown include wired/wireless connections to a local area network (LAN) 1152 and/or a larger network, eg, a wide area network (WAN) 1154 . Such LAN and WAN networking environments are common in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a worldwide computer network, for example, the Internet.

When used in a LAN networking environment, the computer 1102 is connected to the local network 1152 through a wired and/or wireless communication network interface or adapter 1156 . Adapter 1156 may facilitate wired or wireless communication to LAN 1152 , which also includes a wireless access point installed therein for communicating with wireless adapter 1156 . When used in a WAN networking environment, the computer 1102 may include a modem 1158, be connected to a communication computing device on the WAN 1154, or establish communications over the WAN 1154, such as over the Internet. have other means. A modem 1158 , which may be internal or external and a wired or wireless device, is coupled to the system bus 1108 via a serial port interface 1142 . In a networked environment, program modules described for computer 1102 , or portions thereof, may be stored in remote memory/storage device 1150 . It will be appreciated that the network connections shown are exemplary and other means of establishing a communication link between the computers may be used.

Computer 1102 may be associated with any wireless device or object that is deployed and operates in wireless communication, for example, printers, scanners, desktop and/or portable computers, portable data assistants (PDAs), communications satellites, wireless detectable tags. It operates to communicate with any device or place, and phone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network or may simply be an ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) makes it possible to connect to the Internet, etc. without a wire. Wi-Fi is a wireless technology such as cell phones that allows these devices, eg, computers, to transmit and receive data indoors and outdoors, ie anywhere within range of a base station. Wi-Fi networks use a radio technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz radio bands, for example at 11 Mbps (802.11a) or 54 Mbps (802.11b) data rates, or in products that include both bands (dual band). .

One of ordinary skill in the art of this disclosure will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, instructions, information, signals, bits, symbols and chips that may be referenced in the above description are voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields particles or particles, or any combination thereof.

Those of ordinary skill in the art of the present disclosure will recognize that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein include electronic hardware, (convenience For this purpose, it will be understood that it may be implemented by various forms of program or design code (referred to herein as software) or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. A person skilled in the art of the present disclosure may implement the described functionality in various ways for each specific application, but such implementation decisions should not be interpreted as a departure from the scope of the present disclosure.

The various embodiments presented herein may be implemented as methods, apparatus, or articles of manufacture using standard programming and/or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include magnetic storage devices (eg, hard disks, floppy disks, magnetic strips, etc.), optical disks (eg, CDs, DVDs, etc.), smart cards, and flash drives. memory devices (eg, EEPROMs, cards, sticks, key drives, etc.). Also, various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. Based on design priorities, it is understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure. The appended method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the embodiments presented herein, but is to be construed in the widest scope consistent with the principles and novel features presented herein.

Claims (14)

A method of detecting changes performed by a computing device comprising at least one processor, the method comprising:
acquiring an image pair to be subjected to change detection, the image pair including a reference image and a target image;
calculating feature data for each of the reference image and the target image using a neural network-based feature extraction model; and
calculating change region information appearing in the target image from the feature data of the reference image and the feature data of the target image using a neural network-based change detection model including two or more pipelines;
including,
Calculating the change region information includes:
calculating new intermediate data different from the plurality of intermediate data based on a plurality of intermediate data individually calculated from different layers included in the two or more pipelines; and
calculating the change region information based on the new intermediate data;
containing,
Change detection method.
The method of claim 1,
The neural network-based feature extraction model is
at least one downsampling layer for data compression;
The downsampling layer is
comprising at least one convolutional filter and at least one pooling filter,
Change detection method.
The method of claim 1,
The two or more pipelines,
each comprising at least one upsampling layer,
Change detection method.
delete 4. The method of claim 3,
Calculating the change region information includes:
first intermediate data calculated from a layer included in a first pipeline among the two or more pipelines;
second intermediate data calculated from a layer included in a second pipeline among the two or more pipelines; and
a plurality of accumulated weights used for operations related to the second pipeline;
Comprising the step of calculating the change region information based on
Change detection method.
6. The method of claim 5,
Each of the plurality of accumulated weights,
corresponding to each of a plurality of predetermined data related to the second pipeline;
Change detection method.
7. The method of claim 6,
The plurality of accumulated weights includes a first accumulated weight and a second accumulated weight, and the first accumulated weight corresponding to the data calculated in a preceding operation step is a second accumulated weight corresponding to data calculated in a subsequent operation step. having a value less than the cumulative weight,
Change detection method.
The method of claim 1,
The change detection method comprises:
determining an image pair to be changed detection target from an image set including a plurality of images;
further comprising,
Change detection method.
9. The method of claim 8,
The step of determining the image pair to be the target of the change detection comprises:
generating a plurality of candidate image pairs including two consecutive images in chronological order from the image set;
calculating change occurrence information from the generated plurality of pair of candidate images; and
determining an image pair to be subjected to change detection based on the change occurrence information;
containing,
Change detection method.
10. The method of claim 9,
Calculating the change occurrence information includes:
calculating a change score for each of the plurality of candidate image pairs based on a neural network-based change classification model;
containing,
Change detection method.
10. The method of claim 9,
Determining an image pair to be a target of change detection based on the change occurrence information includes:
determining a priority according to the magnitude of the change score included in the change occurrence information; and
determining an image pair to be subjected to change detection based on the priority;
containing,
Change detection method.
11. The method of claim 10,
The neural network-based change classification model is
at least one normal image pair consisting of two images in which the degree of change between the two images is lower than a predetermined reference value; and
at least one abnormal image pair consisting of two images in which the degree of change between the two images is greater than a predetermined reference value;
learning based on
Change detection method.
A computer program stored on a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs the following operations for detecting a change, the operations comprising:
obtaining an image pair to be subjected to change detection, the image pair including a reference image and a target image;
calculating feature data for each of the reference image and the target image by using a neural network-based feature extraction model; and
calculating change region information appearing in the target image from the feature data of the reference image and the feature data of the target image using a neural network-based change detection model including two or more pipelines;
including,
The operation of calculating the change region information includes:
calculating new intermediate data different from the plurality of intermediate data based on a plurality of intermediate data individually calculated from different layers included in the two or more pipelines; and
calculating the change region information based on the new intermediate data;
comprising,
A computer program stored on a computer-readable storage medium.
A change detection device comprising:
one or more processors;
one or more memories; and
network department;
includes, and
The one or more processors,
obtaining an image pair to be subjected to change detection, wherein the image pair includes a reference image and a target image;
Calculating feature data for each of the reference image and the target image using a neural network-based feature extraction model, and
calculating change region information appearing in the target image from the feature data of the reference image and the feature data of the target image using a neural network-based change detection model including two or more pipelines;
Calculating the change region information includes:
Calculate new intermediate data different from the plurality of intermediate data based on a plurality of intermediate data separately calculated from different layers respectively included in the two or more pipelines, and
Comprising calculating the change area information based on the new intermediate data,
change detection device.

Images