KR102578456B1 - System and method for change management of facilities through recognition of structural objects and integration of multiple images - Google Patents

Abstract

The present invention relates to a system and method for recognizing changes in facilities through facility object recognition and multi-image matching based on images captured through drones or aircraft.

Description

System and method for change management of facilities through recognition of structural objects and integration of multiple images}

The present invention relates to a system and method for recognizing changes in facilities through facility object recognition and multi-image matching based on images captured through drones or aircraft.

In general, the Ministry of Land, Infrastructure and Transport, the Korea Forest Service, or the Ministry of Environment produced orthoimages based on high-resolution aerial photos or satellite photos to analyze the status of changes in facilities, and then analyzed changes in facilities based on these.

Here, orthophoto refers to an image that has removed the displacement of the object caused by the camera position and terrain undulations at the time of photography.

It is a high-resolution aerial photograph or satellite photograph viewed vertically from above that has undergone an orthocorrection process called “numerical differential deviation correction,” and has the same effect as a map, especially in terms of the mutual positional relationships of geographical features.

Therefore, conventionally, these orthoimages were secured and changes in facilities were detected through differences between the images.

However, in order to acquire orthoimagery, it is necessary to install a ground control point (GCP: Global Positioning System) and go through an orthocorrection process to reconstruct the same environment geometrically as at the time of shooting through ground surveying or aerial triangulation based on this, resulting in many There was a problem in that multiple changes could not be reflected within a short period of time due to time and cost incurred.

Therefore, as shown in Figure 1, a facility change detection system using conventional drones has been discussed, but in order to estimate differences between images, the positional accuracy between images must accurately match, but this has been difficult to achieve with a low-cost filming drone.

Even if the drone was equipped with a precision positioning device (RTK: Real Time Konematic) that can secure real-time positioning information, it was used only for limited purposes such as military purposes, and its price was too expensive to be utilized.

KR 10-2017754 B1

Therefore, the object to be solved by the present invention is to provide a method of recognizing the same facility and determining changes and alterations in orthoimages obtained and processed from a photographing device that does not have a built-in precision positioning device.

In other words, we aim to solve the technical challenge of recognizing the same facility and detecting changes even when the location mapping between orthoimages is inaccurate.

In order to solve the above problems, the present invention includes an image collection unit that acquires images taken from the air while equipped with GPS and a camera; an orthoimage generator that extracts an orthoimage from the image collected through the image collection unit; Facility object extraction unit for extracting specific facilities from orthophotos; a facility learning unit that learns the facilities extracted through the facility check extraction unit; A detection unit that detects changes in extracted facilities; A display unit that displays changes in facilities; It also includes a real-time mapping unit to detect rapid facility changes and changes in real time through real-time mapping.

Therefore, according to the present invention, according to the change estimation method based on object recognition and image matching method, it is possible to recognize the facility and infer the change using orthoimages taken from a low-cost image capture device, so it can be used in various types of small and medium-sized facility management industries. It can contribute to development.

Figure 1 is a system configuration diagram for facility change and management through conventional aerial photography.
Figure 2 is a system configuration diagram according to an embodiment of the present invention
Figure 3 is a system flow chart according to an embodiment of the present invention.
Figures 4 and 4a are exemplary diagrams and flowcharts showing a method for extracting the same point between orthophotographs according to the present invention.
5 and 5A are exemplary diagrams and flowcharts showing a method of mapping to the same point between orthogonal projections according to the present invention.
Figures 6 and 6a are exemplary diagrams and flowcharts showing a method for recognizing the same facility and detecting changes according to the present invention.

First, the terms used in the specification and claims of the present invention should not be construed as limited in their dictionary meaning, and are based on the principle that the inventor can appropriately define the concept of the term in order to explain his or her invention in the best way. Therefore, it should be interpreted with meaning and concept consistent with the technical idea of the present invention.

Therefore, since the embodiments described in the specification and the configurations shown in the drawings are only preferred embodiments of the present invention and do not express the entire technical idea of the present invention, these configurations can be replaced at the time of filing. It should be understood that various implementations may be made through equivalents and modifications.

Hereinafter, specific details for carrying out the present invention will be described with reference to the attached drawings.

Figure 2 briefly shows the configuration of a system for detecting location matching and changes in facilities according to an embodiment of the present invention. Figure 3 is a flowchart showing the operation process of the system, Figures 4 and 4a show the process of identifying the same point in multiple orthoimages through object recognition, and Figures 5 and 5a show whether or not the same facility is present in multiple orthoimages. This shows the process of determining, and Figures 6 and 6a show the process of estimating changes in the same facility in multiple orthoimages.

As shown, the present invention includes an image collection unit 10 equipped with a camera and GPS, such as a drone or aircraft, to photograph facilities vertically from the air, and an orthoimage generation unit to extract an orthoimage from the image collected through the image collection unit. A unit 20 and an orthoimage storage unit 70 for storing the same, a facility object extraction unit 30 for extracting a specific facility from the orthoimagery, and an artificial intelligence facility that learns the facilities extracted through the facility object extraction unit. A learning unit 40, a change detection unit 50 that detects changes in facilities, a display unit 60 that displays and displays changes in facilities, and real-time mapping of changes and changes in facilities. It is configured to include a real-time mapping unit 80 for detection.

Hereinafter, the operation process of the facility change management system is as follows.

First, the image collection unit 10, such as a drone or aircraft, is equipped with GPS and vertically photographs the facility. At this time, various imaging methods can be used, such as grid scan and circular scan, depending on the size and area characteristics of the diagnostic facility.

In addition, the image collection unit 10 records the initial departure and flight shooting times to maintain the same shooting angle in order to estimate changes in facilities, and then uses an automatic flight recorder to enable movement to the same flight path during future flights. It is desirable to have it available.

Then, the image collection unit 10 transmits the image captured as described above to the orthoimage generation unit. At this time, the image collection unit can collect data using a USB disk or through a network in real time or after shooting. In particular, the collected images contain information about shooting time, shooting location, shooting angle, focal length, altitude, etc.

And the orthoimage generator 20 generates an orthoimage of a wide area from the images based on the image transmitted from the image collection unit. At this time, the orthoimagery is based on GPS information and contains location errors per pixel of the image, but provides a form in which rough GPS location information can be determined.

And these orthoimages are converted into a database by shooting date through the orthoimage storage unit 70.

In addition, the artificial intelligence facility learning unit 40 learns objects to be extracted from the orthoimagery and transmits them to the facility object extraction unit 30. Typically, it is based on learning algorithms such as CNN, and may be equipped with various recognition learning models depending on the facility (e.g., vehicle object recognition learning model for recognizing abandoned vehicles in off-street parking lots, etc.).

And the facility object extraction unit 30 extracts the exact location of the facility object from the orthoimages using artificial intelligence facility recognition and image-to-image location matching techniques.

The change detection unit 50 determines whether the object is the same in the location-matched image and estimates the degree of change. At this time, the change points of facilities are created into a database by location and time zone, and when searched, they are displayed through the display unit 60.

At this time, the present invention can be provided with a real-time mapping unit 80 to detect changes in real time because facility management through orthophotos takes several hours to generate orthoimages.

The real-time mapping method is described below as “a method of identifying the same point in multiple orthoimages through object recognition.” It determines the mapping point in the reference orthoimage, performs rotation, size, and movement transformation, and displays it by overlaying it on the reference orthoimage. do. Detection of this change applies in the same way as other parts.

In addition, with reference to FIGS. 4 and 4A, the process of determining the same point in multiple orthoimages through object recognition is as follows.

First, a reference image is selected from among multiple orthoimages. The general form of selection would be to select an orthoimage corresponding to the start date of the estimated change point.

Select a facility object from the reference orthoimagery. The method of selection is through a bounding box by the user or through a recognition model learned from the facility object check learning unit of this system. The subsequent explanation is written based on selection from the learning model.

The GPS latitude and longitude location estimated from the pixel location of the facility object in the reference orthoimagery is estimated.

Based on the GPS latitude and longitude information estimated from the standard orthoimage, an area estimated to be the same point is selected from the orthoimage to be compared. The size of the area is a multiple of the facility size or an empirical value.

Even if it is the same GPS location, GPS errors cause facilities to be located nearby rather than at the same location. In other words, since the reference orthoimagery and the comparison target area do not have the same image coordinate system, the coordinate transformation parameters for estimating the same location are estimated using the image registration (homography) method.

Coordinate transformation parameters are selected by varying the size of the comparison area between the reference orthoimage and the comparison target orthoimage, and the weighted average is used to optimize the parameters. For example, the comparison area is set as the 2x, 4x, 6x, and 8x areas of the facility, and the estimated coordinate transformation parameters are used as a weighted average. Since this is a comparison between orthoimages, it is a parameter for position and size changes, so even if weighted average is performed, rotational deformation is not affected. Additionally, the comparison area can be set to include multiple facility locations within the reference orthoimagery.

Based on the estimated coordinate transformation parameter values, the orthoimage area to be compared is estimated from the portion corresponding to the object location in the reference orthoimage. As a result, the exact estimated location of the facility can be specified within the standard and comparative orthoimagery.

In addition, as shown in Figures 5 and 5A, the process of determining whether or not the facility is the same in multiple orthoimages is,

After determining the same point in multiple orthoimages through object recognition, the facility is detected in the orthoimage extraction area to be compared through a model learned from the facility object learning unit.

At this time, an image similarity determination algorithm is used to determine whether the detected facility is of the same species and has the same color and structural features.

If the species is the same and the similarity is high, it is recognized as the same facility.

In addition, as shown in Figures 6 and 6a, the process of estimating changes in the same facility in multiple orthoimages is,

After determining whether or not the same facility is present in the multiple orthoimages,

In order to extract whether there are any changed parts, the difference and structural differences between the reference image and the comparison image are checked.

At this time, the structural difference is that if the similarity between pixels is greater than a certain value through algorithms such as SSIM (Structural SIMilarity), the facility is recognized as having changed.

Therefore, the present invention makes it possible to check whether changes have been made to a facility through object recognition and image matching methods, and it is possible to recognize facilities and infer changes using orthoimages taken from low-cost image capture devices, thereby enabling various types of It will be possible to contribute to the development of the small and medium-sized facility management industry.

The present invention has been described so far with reference to the attached drawings, but these are merely illustrative, and those skilled in the art can easily implement various modifications and equivalent theories. There will be. Therefore, the true technical protection scope of the present invention should be determined by the scope of the patent claims.

10: Video collection unit 20: Orthophoto image generation unit
30: Facility object extraction unit 40: Artificial land facility learning unit
50: change detection unit 60: display unit
70: Orthoimage storage unit 80: Real-time mapping unit

Claims (7)

delete A step of an image collection unit acquiring an image taken from the air while equipped with a GPS and a camera, a step of an orthoimage generation unit generating an orthoimage based on the image acquired through the image collection unit, and a step of converting the orthoimage into a database; A step of learning the object to be extracted by the artificial intelligence facility learning unit based on the orthoimage generated through the orthoimage generation unit and transmitting it to the facility object extraction unit, and the facility object extraction unit extracts the facility through artificial intelligence facility recognition and location matching between images. An extraction step, a change detection step to determine whether the object is the same through location-matched images and a step to check whether there has been a change, and a display step to display the change point of the facility by location and time zone. Come true,
The real-time mapping unit further includes a step of determining the same point in multiple orthoimages through object recognition during real-time mapping to detect changes,
When the real-time mapping unit determines the same point in the multiple orthoimages, selecting a reference orthoimagery from the multiple orthoimages, selecting a facility object from the reference orthoimagery, and pixels of the facility object from the reference orthoimagery. It includes the step of confirming the GPS latitude and longitude position estimated from the location, and the step of selecting an area estimated to be the same point in the orthoimage to be compared based on the GPS latitude and longitude information estimated from the reference orthoimage,
The real-time mapping unit performs a coordinate transformation parameter to estimate the same location using an image matching (homography) method even if the same point is selected based on GPS latitude and longitude information, and compares the reference orthoimage and the comparison target orthoimage. A step of varying the size of the area to extract coordinate transformation parameters and optimizing the parameters by weighting the average, and a step of estimating the part corresponding to the object location in the reference orthoimagery as the orthoimage area to be compared based on the coordinate transformation parameter values. containing,
Management method of facility change management system through facility object recognition and multi-image registration. delete delete delete According to paragraph 2,
When the real-time mapping unit determines the same point in multiple orthoimages through object recognition, detecting the facility through a model learned from the facility object learning unit for the orthoimage extraction area to be compared, and determining whether the detected facility is the same as the same paper. , which includes a step of checking whether the color and structural features are the same, and a step of recognizing the same facility if it is the same as paper and has a high degree of similarity.
Management method of facility change management system through facility object recognition and multi-image registration. According to clause 6,
When the real-time mapping unit recognizes the same facility, it includes a step of checking whether there is a difference between the reference image and the comparison image to extract whether there is any change in the facility.
Management method of facility change management system through facility object recognition and multi-image registration.

Images