KR102053906B1 - Methods and system for real-time supervised learning using geo-spatial information - Google Patents

Abstract

The present invention provides a method and system for performing supervised learning while acquiring images in real time from a moving object, converting the acquired images into georeferenced images, generating class information from georeferenced images, and georeferenced images and classes. Generate all the training data consisting of pairs of information, and input all the georeferenced images into the generated training data to supervise and train the model that extracts the object classification results. It is done. According to the present invention, by quickly and automatically performing image acquisition, learning, and classification in real time using geospatial information, there is no need to store massive learning data in advance. In addition, since part of the data acquired at the present time is used as the learning data, the data can be further optimized for object classification or identification at the present time.

Description

METHODS AND SYSTEM FOR REAL-TIME SUPERVISED LEARNING USING GEO-SPATIAL INFORMATION}

The present invention relates to a real-time supervised learning method and system using geospatial information, and more particularly, to generate learning data from an image acquired from a moving object, to supervise the generated learning data, and to acquire the acquired image as a learned model. The present invention relates to a method and a system for performing a whole process of classifying an object from an image in real time.

In order to detect anomalies or illegal phenomena in the observation area, images taken from moving objects such as manned aircraft, drones (UAVs), vehicles, and boats are frequently used. For example, to quickly identify disasters such as floods or landslides, detect illegal buildings in restricted areas, identify illegal ships in territorial waters, monitor construction sites, or monitor trees in national parks. There may be.

As an object classification method using an image, Korean Patent No. 10-1417498 discloses that a background and a foreground are separated from an input image, a feature is extracted from the separated foreground, and a descriptor is generated to search for object information by comparing with a database. Techniques are known. However, since this method can search only objects that are built in advance and stored in the database, it is difficult to classify or identify the objects that are not stored in advance.

Meanwhile, the supervised learning method using neural networks may generate a model capable of classifying or identifying a category of an object classified in the learning data using previously constructed learning data. However, since the supervised learning method calculates an optimal model using a plurality of learning data, a large amount of time and cost are required to secure a large amount of learning data. In addition, since a model that adaptively works well in various cases has a problem that a specific data at a specific point in time has a weakness that does not work relatively well.

KR 10-1417498.

The present invention was devised to solve the above-mentioned problems of the prior art, and it is possible to automatically and quickly perform real time all the processes of image acquisition, learning, and classification in real time while acquiring images without having to store huge learning data in advance. It is an object to provide a method and system.

According to an embodiment of the present invention, a supervised learning method is performed while acquiring an image in real time from a moving object, the method comprising: converting the acquired image into a georeferenced image; Generating class information based on the georeferenced image during a first time period, and generating learning data comprising the pair of the georeferenced image and the class information; And supervising and learning a model for extracting an object classification result for the georeferenced image using the generated learning data, wherein the first time interval is included in a time interval for acquiring the image in real time. The class information is generated based on geo-spatial information stored in a database.

The real-time supervised learning method may further include extracting an object classification result by inputting the georeferenced image to the learned model during a second time period, wherein the second time period is a time for acquiring the image in real time. Included in the section, the second time section may be a time section after the first time section.

The first time period may be determined based on one or more indicators indicating the completeness of the supervised learning.

The class information may be in a raster format generated based on the geospatial information from the georeferenced image.

The class information may be semantic segmentation data indicating the type of an object in pixels.

The training may include an optimization step of modifying the model to minimize the difference between the model-based object classification result of the georeferenced image and the object classification result identified from the class information.

In addition, according to another preferred embodiment, a computer program for executing the method according to each of the above-described methods is provided.

According to yet another preferred embodiment, there is provided a computer readable recording medium having the program recorded thereon.

According to still another preferred embodiment of the present invention, there is provided a system for performing supervised learning while acquiring an image in real time from a moving object, wherein the data processor converts the acquired image into a georeferenced image. ; A learning data generation unit generating class information based on the georeferenced image during a first time period and generating learning data consisting of the pair of georeferenced image and the class information; And a supervising learning unit configured to supervise a model for extracting an object classification result for the georeferenced image using the generated learning data, wherein the first time period is included in a time period for acquiring the image in real time. The class information is generated based on geospatial information stored in a database.

The real-time supervised learning system further includes a classification unit for extracting an object classification result by inputting the georeferenced image to the learned model during a second time period, wherein the second time period is a time for acquiring the image in real time. Included in the section, the second time section may be a time section after the first time section.

The supervised learning unit may determine the first time interval based on one or more indicators indicating the completeness of the supervised learning.

The class information may be in a raster format generated based on the geospatial information from the georeferenced image.

The class information may be semantic segmentation data indicating a type of an object in pixel units.

The supervising learner may perform optimization to modify the model to minimize the difference between the model-based object classification result of the georeferenced image and the object classification result identified from the class information.

As described above, according to the present invention, an image is acquired in real time in the entire process of generating learning data from an image acquired by a moving object, supervising the generated learning data, and classifying an object from the acquired image by the learned model. By performing all the while, there is an effect that can quickly and automatically perform image acquisition, learning and classification in real time.

In addition, since learning data is created in real time based on geospatial information, there is no need to store a large amount of learning data in advance, and since a part of data acquired at the present time is used as learning data, object classification or The effect is that it can be further optimized for identification.

1 is a diagram illustrating an environment in which a real-time supervised learning system operates according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a real-time supervised learning system according to an embodiment of the present invention.
3 is a flowchart illustrating a specific example of a real-time supervised learning method according to an embodiment of the present invention.
4 is a flowchart illustrating a specific example of a real-time supervised learning method according to another embodiment of the present invention.
5 is a flowchart illustrating a specific example of a real-time supervised learning method according to another embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. In the following description, only parts necessary for understanding the operation according to the embodiment of the present invention are shown and described, and illustrations and descriptions of other parts are omitted so as not to obscure the subject matter of the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention.

In addition, the terms or words used in the specification and claims described below are not to be construed as being limited to the ordinary or dictionary meanings, meaning that corresponds to the technical spirit of the present invention so that the present invention can be most appropriately expressed To be interpreted as

In addition, when a part is said to "include" a certain component, this means that it may further include other components, except to exclude other components unless otherwise stated.

For simplicity of explanation, although one or more methods are shown and described herein as a series of steps, for example in the form of a flowchart or flow chart, the present invention is not limited by the order of the steps, for the present invention. It will be appreciated that the present invention may be performed in a different order or simultaneously with other steps than those shown and described herein. Moreover, not all illustrated steps may have to implement a methodology in accordance with the present invention.

SUMMARY OF THE INVENTION Since the present invention aims to provide a method and system for quickly and automatically performing in real time all the processes of image acquisition, learning, and classification during image acquisition, without the necessity of storing huge learning data in advance. For convenience of explanation throughout the specification, the image acquisition is started from the moving object 10 and the initial predetermined time is referred to as a 'first time interval', and the predetermined time after the first time interval is referred to as a 'second time interval'. Let's do it. That is, the first time interval and the second time interval are included in the entire time interval for acquiring the image in real time, and the second time interval is the time interval after the first time interval.

The first time interval is preferably defined as a time interval until it is determined that the learning has been sufficiently performed through indices indicating the completeness of the learning, such as precision and recall. However, the present invention is not limited thereto and may be defined as a specific specific time interval. For example, a time interval of an initial 10 minutes or 10% in a time at which the moving object 10 acquires an image may be referred to as a first time interval, and then a remaining time interval as a second time interval.

1 is a diagram illustrating an environment in which the real-time supervised learning system 50 operates according to an embodiment of the present invention.

Referring to FIG. 1, the moving object 10, the server 20, and the observation area 30 are shown. The real-time supervised learning system 50 according to the present invention may be implemented in the moving object 10 and / or the server 20.

The moving object 10 may be a variety of moving objects or mobile platforms, such as a manned aircraft such as a helicopter, an unmanned aerial vehicle such as a drone (UAV), a vehicle, a boat, and the like. The moving object 10 may be equipped with equipment such as an optical camera, an infrared camera, GPS / INS, and various sensors capable of acquiring information and images of the observation area 30. The moving object 10 may acquire various data such as an optical image, a thermal image, and position / posture information of a mobile platform by using such equipment.

In addition, the moving object 10 may include a computing device such as a server or a computer and a storage device for storing a database, and perform various functions such as object recognition, classification, extraction, and learning.

The server 20 may be various devices such as a conventional server device, a mobile phone, a smartphone, a computer, a tablet, a PC, and the like. In addition, the server 20 may include a storage device capable of storing a database.

The mobile unit 10 and the server 20 may communicate with each other by various wireless communication means, and may transmit data with each other.

2 is a block diagram showing the configuration of a real-time supervised learning system 50 according to an embodiment of the present invention.

Referring to FIG. 2, the real-time supervised learning system 50 according to the present invention may include a data processor 52, a learning data generator 54, and a supervised learner 56. In addition, the real-time supervised learning system 50 may further comprise a classification unit 58.

The data processor 52 receives in real time an image acquired by the moving object 10 in real time, and converts the received image into a “georeferenced image” in real time. A georeferenced image is an image generated, corrected or converted with reference to geographic information. An example is an ortho image. This is not necessarily limited.

Orthophoto is an image which removes the displacement of the object caused by the camera posture and terrain ups and downs at the time of image shooting. The orthophoto is produced through ortho correction, such as numerical differential deviation correction. The mutual positional relationship becomes the same as the map. Therefore, in the general picture, the phenomenon in which the side surface of the object having the height appears not at the center of the picture is represented as the image which is vertically looked down in the orthodontic image.

The georeferenced image converted in real time by the data processing unit 52 is used by the learning data generation unit 54 in the first time interval (for an initial predetermined time period) when image acquisition is started from the moving object 10. The two-hour period (after the initial predetermined time) can be used in the classification unit 58.

The learning data generation unit 54 generates class information based on the georeferenced image during the first time period, and generates the learning data composed of the pair of the georeferenced image and the class information. That is, the acquisition of the image from the moving object 10 starts and generates the learning data from the georeferenced image generated at the beginning.

The class information may be generated from georeferenced images based on geo-spatial information of the observation area stored in the database. That is, class information may be generated based on geospatial information from georeferenced images.

Geospatial information can be secured and stored in advance as a database. Representative geospatial information includes a digital map of a target area, and information such as buildings, roads, rice fields, and fields that explicitly define the class of objects such as the terrain and features of the target area.

In one embodiment, the class information may be in the form of a raster of the same size as the georeferenced image, and the classification information may be determined by identifying which objects are defined on the digital map for the location of every pixel of the georeferenced image. Can be created by labeling the class. That is, the class information may be said to add geospatial information (absolute coordinate numerical information and object class information) on the georeferenced image.

Thus, the generated class information is semantic segmentation data representing the type of object such as a building or a road in pixel units.

As described above, in the present invention, the reason for generating the class information from the georeferenced image based on the geospatial information is that the georeferenced image and the geospatial information have the same map coordinate system so that they can be quickly matched. Because there is an advantage. Therefore, this enables real-time supervised learning according to the present invention.

On the other hand, the supervising learning unit 56 supervises the classification model using the learning data when the learning data composed of the pair of the image and the class information geo-referenced by the learning data generating unit 54 is generated. That is, a model (or classifier) for generating (or extracting) a result of classifying (or identifying) an object when a georeferenced image is input using learning data is generated.

Supervised learning is a method of machine learning for deriving a function from training data or training data, and neural networks such as deconvolutional networks are used, but not necessarily limited thereto. .

In addition, the supervised learning unit 56 may further perform an optimization process of correcting the learned model. That is, the learning model may be modified to minimize the difference between the object classification result calculated by inputting the georeferenced image into the learning model and the object classification result obtained from the class information.

For example, the georeferenced image contains some geometric distortions, so that some errors may exist in the training data itself. Even if these slight errors exist, the optimization is performed so that they work well (classification). The optimized coefficients of the classifier model can be determined to be robust.

In addition, the supervised learning unit 56 may determine the first time interval and the second time interval based on the completeness of the supervised learning. For example, based on the indicators (accuracy, recall, etc.) calculated in the process of optimizing the model with the generated training data, it may be determined whether the training has been sufficiently performed.

If it is determined that further learning is still needed, the learning may be further performed in the first time period. If it is determined that the completeness of the learning exceeds a certain threshold, that is, if the learning is determined to be sufficient, the second time period may be determined to terminate the learning and to determine the classifier model.

The various functions of the supervised learning unit 56 described above are performed in real time in parallel with the acquisition of the image from the moving object 10 during the first time period.

Finally, the classification unit 58 inputs a georeferenced image to the model generated by the supervising and learning unit 56 to extract a classification result.

In this case, the input georeferenced image is a georeferenced image converted in real time by the data processor 52 during the second time interval. That is, in the first time section, the learning data is generated and the learning model is generated, and in the second time section after that, the object classification is performed.

In other words, among the data acquired by the moving object 10 in real time, the first part of the data is used for learning, and the classification is performed on the remaining data (obtained in real time).

On the other hand, each of the functional blocks of the real-time supervised learning system 50 may be implemented in all or part of the moving object 10 and the server 20 in various ways.

For example, when the moving object 10 is responsible for image acquisition and transmits the image acquired by the moving object 10 to the server 20 in real time, the data processor 52, the learning data generator 54, and supervised learning. The unit 56 and the classification unit 58 may be implemented in the server 20.

In addition, the data processing unit 52 may be implemented in the moving object 10, and the learning data generation unit 54, the supervised learning unit 56, and the classification unit 58 may be implemented in the server 20.

In addition, the moving object 10 may include a data processor 52 and a learning data generator 54, and the server 20 may include a supervised learner 56 and a classifier 58.

In addition, all functions except for the classification unit 58 may be implemented in the moving object 10, and only the classification unit 58 may be implemented in the server 20.

In addition, all functional blocks are provided in the moving object 10, and the server 20 may receive only the classification result from the moving object 10.

As described above, the real-time supervised learning system 50 according to the present invention generates the learning data from the image acquired by the moving object 10, supervises the generated learning data, and classifies the acquired image into the learned model. By performing the entire process during the acquisition of the image in real time, it is possible to quickly and automatically perform image acquisition, learning and classification in real time.

In addition, since learning data is created in real time based on geospatial information, there is no need to store a large amount of learning data in advance, and since a part of data acquired at the present time is used as learning data, object classification or It can be further optimized for identification.

3 is a flowchart illustrating a specific example of a real-time supervised learning method according to an embodiment of the present invention (S100).

In step S110, the image acquired by the moving object 10 is received in real time and converted into a georeferenced image in real time.

In step S130, based on the geospatial information stored in the database, class information is generated from the image converted in step S110, and learning data composed of pairs of georeferenced images and class information is generated. This step may be continuously performed in the first time interval while the acquired image is received in real time.

The class information may be generated from the georeferenced image based on geospatial information of the observation area stored in the database. The geospatial information may include a digital map, and is preferably secured and stored in advance as a database. The generated class information may be in the form of a raster having the same size as the georeferenced image, and may include the absolute coordinate numerical information and the object class information of the observation area on the georeferenced image. Since georeferenced images and geospatial information have the same map coordinate system, it is possible to quickly match them.

In operation S140, the model for extracting an object classification result for the georeferenced image is supervised using the training data generated in operation S130. That is, a model (or classifier) is generated to calculate (or extract) a result of classifying (or identifying) an object when a georeferenced image is input using the training data. This step may be continuously performed in the first time interval while the acquired image is received in real time.

In addition, step S140 may include an optimization step of modifying the model to minimize the difference between the model-based classification result of the georeferenced image input and the classification result given to the training data. For example, the georeferenced image contains some geometric distortions, so there may be some errors in the training data itself. Optimized coefficients of the classifier model can be determined.

In step S145, it is determined whether the model has been sufficiently trained based on the indicators (accuracy, recall, etc.) calculated in the process of supervising the model with the generated training data.

If it is determined that the model is not sufficiently trained, that is, the first time interval, the transformed georeferenced image is used to generate training data in step S130.

As a result of the determination, if the model has been sufficiently learned, that is, the second time interval, the learning is terminated, and the converted georeferenced image is used for object classification in step S150.

When the model trained in the first time interval is generated, in step S150, the classification result is extracted by inputting a georeferenced image to the model generated in the training. This step may be continuously performed in the second time interval while the acquired image is received in real time.

Meanwhile, when the moving object 10 is responsible for image acquisition and transmits the image acquired by the moving object 10 to the server 20 in real time, steps S110 to S150 may be performed by the server 20.

In addition, the moving object 10 may be executed up to any specific step from step S110 to step S150, and the step after any particular step may be executed in the server 20.

As described above, the real-time supervised learning method 100 according to the present invention generates the learning data from the image acquired by the moving object 10, supervises the generated learning data, and classifies the acquired image into the learned model. By performing the entire process during the acquisition of the image in real time, it is possible to quickly and automatically perform image acquisition, learning and classification in real time.

4 is a flowchart illustrating a specific example of a real-time supervised learning method according to another embodiment of the present invention, in which an illegal building in a restricted development area is detected (S200).

In order to investigate illegal buildings in the restricted area, for example, the Greenbelt area, municipalities conduct aerial surveys, and people use illegal methods to investigate illegal buildings by comparing the results of last year's surveys with the relevant ledgers. Therefore, since it takes a lot of time and money, there is a problem that is often difficult to investigate.

According to the present invention, since it is possible to quickly and automatically perform learning and classification while acquiring an image in real time, it can be frequently and quickly investigated.

When the moving object 10 starts to acquire an image in the green belt region (S210), the moving object 10 receives the image and converts it into an orthoimage (geographically referenced image) in real time (S220).

Learning consists of pairs of orthoimages and class information by generating class information (raster) to classify illegal buildings and areas that are not illegal buildings on ortho images using geospatial information of the region stored in the database Generate data (S240).

The learning data is continuously increased during the first time period, and at the same time, the model is supervised and trained to produce a classifier that separates the illegal building and the non-illegal building area using the generated learning data (S250).

It is determined whether the model has been sufficiently trained based on the indicators (accuracy, recall, etc.) calculated in the process of supervising the model with the generated training data (S255).

If it is determined that the model has not been sufficiently trained, that is, the first time interval, step S240 is executed again.

If the determination result is that the model has been sufficiently learned, that is, the second time interval, the learning is terminated and the step S260 is executed.

When the classifier model is generated by the generation of the training data and the supervised learning in the first time interval, the classification result is extracted by inputting the orthoimage into the learned model during the second time interval. That is, the orthoimage is input to the calculated classifier to check the illegal building in the target area in real time (S260).

As described above, according to the present embodiment, it is possible to classify illegal buildings quickly and automatically because image acquisition, learning, and classification are performed in real time. In addition, since learning data is created in real time based on geospatial information, there is no need to store a large amount of learning data in advance, and since a part of data acquired at the present time is used as learning data, object classification or It can be further optimized for identification.

FIG. 5 is a flowchart illustrating a specific example of a real-time supervisory learning method according to another embodiment of the present invention, in which an illegal vessel monitoring and identification is performed in an ocean area (S300).

Similar to the detection of illegal buildings in the greenbelt of FIG. 4, the present invention can be applied to marine areas, particularly ship monitoring. Current ship surveillance is vulnerable to illegal ship surveillance because it relies on signals from ships, for example, Automatic Identification System (AIS) signals.

Applying the present invention, it is possible to quickly and automatically perform image acquisition, learning and classification (monitoring or identification) both in real time.

The aerial camera system can be used as a means of actively monitoring the marine area and performing vessel surveillance.

The manned or unmanned aerial vehicle 10 acquires an image in real time from the marine patrol area (S310), and converts the acquired image into a georeferenced image (orthoimage) in real time (S320).

In the first time interval, the georeferenced image and geospatial information of the region are used to classify the georeferenced image into water, islands, and other terrain / features defined on the map and non-regions. Class information is generated and learning data composed of a pair of georeferenced image and class information is generated (S340).

The classifier model is supervised using the generated training data (S350).

It is determined whether the model has been sufficiently trained based on the indicators (accuracy, recall, etc.) calculated in the process of supervising the model with the generated training data (S355).

If it is determined that the model has not been sufficiently trained, that is, the first time interval, step S340 is executed again.

If it is determined that the model has been sufficiently learned, that is, the second time interval, the learning is terminated and the step S360 is executed.

The georeferenced image is input to the generated classifier to check in real time the water, islands, and other terrain / features defined on the map and the non-regions in the target area in real time. At this time, water, islands, and other non-terrain / feature regions defined on the map may be referred to as ship candidates.

Since the ship candidates are classified in step S360, the actual ships may be detected using a rule-based method or a deep learning method using the characteristics of the ships having sizes or shapes among the candidates (S370).

Then, by checking whether the AIS is transmitted from the detected ship, it is possible to finally identify the illegal ship (S380).

As described above, according to various embodiments of the present disclosure, the entire process of generating the learning data from the image acquired by the moving object, supervising the generated learning data, and classifying the acquired image into the learned model in real time is obtained. By doing all this while it is possible to quickly and automatically perform image acquisition, learning and classification in real time.

In addition, since learning data is created in real time based on geospatial information, there is no need to store a large amount of learning data in advance, and since a part of data acquired at the present time is used as learning data, object classification or It is possible to be more optimized for identification.

In addition, the embodiment of the real-time supervised learning method described above may be implemented in the form of computer program instructions that can be executed through various computer components. In addition, the implemented computer program may be recorded in a computer readable recording medium. The mentioned recording medium may be a ROM, a magnetic disk or a compact disk, an optical disk, or the like, but is not necessarily limited thereto.

As described above, the present invention has been described with reference to the embodiments shown in the drawings, but this is merely exemplary, and those skilled in the art may various modifications and equivalent other embodiments from this. Will understand. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

10: moving object
20: server
30: observation area
50: real-time supervised learning system
52: data processing unit
54: learning data generation unit
56: supervised learning
58: classification

Claims (14)

As a supervised learning method performed while acquiring images in real time from a moving object,
Converting the acquired image into a georeferenced image;
Generating class information based on the georeferenced image during a first time interval, thereby generating learning data comprising the pair of georeferenced image and the class information; And
Supervising and learning a model for extracting an object classification result for the georeferenced image using the generated learning data;
Extracting an object classification result by inputting the georeferenced image to the learned model during a second time period;
Including,
The first time interval and the second time interval are included in the time interval for acquiring the image in real time, the second time interval is a time interval after the first time interval,
The class information is generated based on geo-spatial information stored in a database, real-time supervised learning method.
delete The method of claim 1,
And wherein the first time period is determined based on one or more indicators indicating the completeness of the supervised learning.
The method of claim 1,
The class information may be a raster format generated based on the geospatial information from the georeferenced image.
The method of claim 1,
The class information is semantic segmentation data representing the type of object in units of pixels, real-time supervised learning method.
The method of claim 1,
The learning may include an optimization step of modifying the model such that the difference between the model-based object classification result of the georeferenced image and the object classification result grasped from the class information is minimized. Supervised learning method.
A computer-readable recording medium having recorded thereon a computer program for executing the method according to claim 1.
delete A system for performing supervised learning while acquiring images in real time from a moving object,
A data processor converting the acquired image into a georeferenced image;
A learning data generation unit generating class information based on the georeferenced image during a first time period and generating learning data consisting of the pair of georeferenced image and the class information; And
A supervising learning unit for supervising and learning a model for extracting an object classification result for the georeferenced image by using the generated learning data;
A classification unit extracting an object classification result by inputting the georeferenced image to the learned model during a second time period;
Including,
The first time interval and the second time interval are included in the time interval for acquiring the image in real time, the second time interval is a time interval after the first time interval,
The class information is generated based on geospatial information stored in a database, real-time supervised learning system.
delete The method of claim 9,
Wherein the supervised learning unit determines the first time interval based on one or more indicators indicating the completeness of the supervised learning.
The method of claim 9,
And the class information is in a raster format generated based on the geospatial information from the georeferenced image.
The method of claim 9,
The class information is semantic segmentation data indicating the type of the object in units of pixels, real-time supervised learning system.
The method of claim 9,
The supervising learning unit may perform optimization to modify the model to minimize a difference between the model-based object classification result of the georeferenced image and the object classification result identified from the class information. system.

Images