KR20220118865A - System and method for change management of facilities through recognition of structural objects and integration of multiple images - Google Patents

Abstract

The present invention relates to a system and method capable of confirming a change in a facility through facility object recognition and multi-image matching based on an image photographed by a drone or aircraft. A facility change management system includes: an image collection unit; an orthoimage generation unit; a facility object extraction unit; a facility learning unit; a change detection unit; a display unit; and a real-time mapping unit.

Description

{System and method for change management of facilities through recognition of structural objects and integration of multiple images}

The present invention relates to a system and method capable of confirming changes in facilities through facility object recognition and multi-image registration based on images captured by drones or aircraft.

In general, the Ministry of Land, Infrastructure and Transport, the Korea Forest Service, or the Ministry of Environment produced orthographic images based on high-resolution aerial or satellite images to analyze the status of changes in facilities, and then analyzed the changes in facilities based on this.

Here, an orthophoto refers to an image in which the displacement of an object caused by the camera posture and terrain undulations at the time of photographing has been removed.

It is a high-resolution aerial photograph or satellite photograph viewed vertically from top to bottom after orthogonal correction, which is usually called "numerical differential deviation correction".

Therefore, the conventional orthogonal images were secured and the change of the facility was sensed through the differences between the images.

However, in order to acquire an orthographic image, a global positioning system (GCP) is installed, and based on this, an orthographic correction process that geometrically reconstructs the same environment as at the time of shooting is required through ground survey or aerial triangulation. There was a problem that many changes and changes could not be reflected within a short period of time and cost.

Therefore, as shown in FIG. 1, a conventional facility change detection system using a drone has been discussed, but in order to estimate the difference between images, the positional accuracy between images must be accurately matched.

Even if the drone was equipped with a Real Time Konematic (RTK) that can secure real-time mobile positioning information, it was only used for limited purposes such as military use, and its price was too high to be utilized.

US 10-2017754 B1

Accordingly, an object of the present invention is to provide a method for recognizing the same facility in orthogonal images obtained and processed from a photographing device that does not have a built-in precision positioning device to determine changes and changes.

In other words, it is intended to solve the technical problem of recognizing the same facility and detecting change even in a state where the location mapping between orthographic images is not accurate.

In order to solve the above problems, the present invention is an image collecting unit for acquiring an image taken in the air in a state equipped with a GPS and a camera; an orthographic image generating unit for extracting an orthogonal image from the images collected through the image collecting unit; a facility object extraction unit for extracting a specific facility from an orthogonal image; a facility learning unit for learning the facility extracted through the facility check extraction unit; a sensing unit for detecting a change in the extracted facility; a display unit that displays the change and change of the facility; And it is configured to include a real-time mapping unit for detecting rapid facility change and change through real-time mapping in real time.

Therefore, according to the method of estimating change change by object recognition and image registration method, the present invention can recognize facilities and infer change of change with an orthographic image taken from a low-cost image capturing device, so that various types of small and medium-sized facility management industry can contribute to development.

1 is a system configuration diagram for changing and managing facilities through conventional aerial photography.
2 is a system configuration diagram according to an embodiment of the present invention;
3 is a system flowchart according to an embodiment of the present invention;
4 and 4A are exemplary diagrams and flowcharts showing a method for extracting the same point between orthogonal images according to the present invention.
5 and 5A are exemplary diagrams and flowcharts showing a method of mapping to the same point between orthographic projections according to the present invention.
6 and 6A are exemplary diagrams and flow charts showing a method for recognizing the same facility and detecting a change according to the present invention.

First, the terms used in the specification and claims of the present invention should not be construed as being limited to dictionary meanings, and based on the principle that the inventor can appropriately define the concept of the term to describe his invention in the best way. Therefore, it should be interpreted as meaning and concept consistent with the technical idea of the present invention.

Therefore, the configurations shown in the embodiments and drawings described in the specification are only preferred embodiments of the present invention, and do not express all the technical ideas of the present invention, so they can be substituted for these configurations at the time of filing. It should be understood that various implementations may be made by means of equivalents and modifications.

Hereinafter, specific details for carrying out the present invention will be described with reference to the accompanying drawings.

2 is a schematic diagram illustrating a system configuration for detecting a change in location and change of a facility according to an embodiment of the present invention. 3 is a flowchart showing the operation process of the system, FIGS. 4 and 4A show the process of dividing the same point in multiple normal images through object recognition, and FIGS. 5 and 5A are the same facilities in multiple orthogonal images It shows the process of determining, and FIGS. 6 and 6A show the process of estimating whether the same facility has changed or not in multiple orthogonal images.

As shown, the present invention is equipped with a camera and GPS such as a drone or an aircraft, and an image collection unit 10 for photographing a facility vertically in the air, and orthographic image generation for extracting an orthographic image from the image collected through the image collection unit The unit 20 and the ortho image storage unit 70 for storing it, the facility object extraction unit 30 for extracting a specific facility from the orthographic image, and an artificial intelligence facility for learning the facility extracted through the facility object extraction unit The learning unit 40, the change change detection unit 50 for detecting changes in the facility, the display unit 60 for displaying and expressing the change change of the facility, and real-time mapping to monitor the change and change of the facility in real time It is configured to include a real-time mapping unit 80 for detecting .

Hereinafter, an operation process of the facility change management system is as follows.

First, the image collection unit 10 vertically shoots a facility with a GPS, such as a drone or an aircraft. In this case, as a photographing method, various photographing methods such as grid scan, circular scan, etc. may be photographed according to the size and area characteristics of the diagnostic facility.

In addition, the image collection unit 10 records the initial flight and flight shooting time in order to maintain and maintain the same shooting angle in order to estimate the change of the facility, and then records an automatic flight recorder so that it can move to the same flight path during a later flight. It is preferable to have it.

And the image collecting unit 10 transmits the image captured as described above to the orthogonal image generating unit. In this case, the image collection unit may collect it with a USB disk or collect it through a network in real time or after shooting. In particular, the collected images contain information about shooting time, shooting location, shooting angle, focal length, altitude, etc.

In addition, the orthographic image generating unit 20 generates an orthogonal image based on the image transmitted from the image collecting unit for an image image of a wide area from the images. At this time, the orthographic image is based on GPS information and contains a position error per pixel of the image, but provides a form that can grasp the approximate GPS position information.

And these ortho images are converted into a database for each shooting date through the ortho image storage unit 70 .

In addition, the artificial intelligence facility learning unit 40 learns the object to be extracted from the orthographic image and transmits it to the facility object extraction unit 30 . Representatively, it is based on a learning algorithm such as CNN, and various recognition learning models may be provided depending on the facility (eg, a vehicle object recognition learning model for recognizing an abandoned vehicle in an off-street parking lot, etc.).

And the facility object extraction unit 30 extracts the exact location of the facility object from the orthoimages by using the artificial intelligence facility recognition and the location matching technique between the images.

The change change detection unit 50 determines whether or not the same object is in the position-matched image and estimates the degree of change. At this time, the change point of the facility change by place and time is converted into a database, and when it is inquired, it is expressed through the display unit 60 .

In this case, the present invention may include a real-time mapping unit 80 to detect changes in real time because facility management through orthographic images takes several hours to generate an orthographic image.

The real-time mapping method is a "method for determining the same point in multiple orthogonal images through object recognition" described below. do. Its change change detection is applied in the same way as other divisions.

In addition, referring to FIGS. 4 and 4A , a process of determining the same point in multiple orthogonal images through object recognition will be described as follows.

First, a reference image is selected from among multiple orthogonal images. The selection will be generally in the form of selecting an orthoimage corresponding to the change point estimation start date.

Select a facility object from the reference orthographic image. The selection method is selected by the user using the bounding box or through the recognition model learned from the facility guest check learning unit of the system. Subsequent explanations are made based on selections from the learning model.

The GPS latitude and longitude position estimated from the pixel position of the facility object in the reference orthogonal image is estimated.

Based on the GPS latitude and longitude information estimated from the reference orthogonal image, an area estimated to be the same point in the comparison target orthogonal image is selected. The size of the area is used as a multiple of the size of the facility or as an empirical value.

Even at the same GPS location, the facilities are located in the vicinity of the same point rather than at the same point due to a GPS error. That is, since the reference orthogonal image and the area to be compared do not have the same image coordinate system, coordinate transformation parameters for estimating the same position are estimated by image matching (homography) method.

The coordinate transformation parameters are selected by varying the size of the comparison area between the reference orthogonal image and the comparison target orthogonal image, and the parameters are optimized and used by weighted average. For example, set the comparison area as the 2x, 4x, 6x, 8x area of the facility and use the weighted average of each estimated coordinate transformation parameter. Since it is a comparison between orthogonal images, it is a parameter for position and size change, so even if a weighted average is performed, rotational deformation is not affected. In addition, the comparison area may be set to include the locations of a plurality of facilities in the reference orthogonal image.

Based on the estimated coordinate transformation parameter values, the area corresponding to the position of the object in the reference orthogonal image is estimated to be the comparison target ortho image area. In this way, it is possible to designate the exact estimated location of the facility in the reference and comparative orthogonal images.

In addition, as shown in Figures 5 and 5a, the process of determining whether the same facility in the multi-orthogonal image is,

After determining the same point in the multiple orthogonal images through the object recognition, the facility is detected through the model learned from the facility object learning unit in the ortho image extraction area to be compared.

At this time, it is determined whether the species of the detected facilities are the same and whether the color and structural features are the same using the image similarity determination algorithm,

If the species is the same and the similarity is high, it is recognized as the same facility.

In addition, as shown in Figs. 6 and 6a, the process of estimating the change of change of the same facility in the multi-orthogonal image is,

After determining whether the same facility is in the multi-orthogonal image,

In order to extract whether there is no changed part, the difference between the reference image and the comparison image and the structural difference are checked.

At this time, the structural difference is that the facility is recognized as changed and changed when the similarity between pixels is greater than a predetermined value through an algorithm such as SSIM (Structural SIMilarity).

Therefore, according to the present invention, as it is possible to check whether the change of the facility is changed by the object recognition and image matching method, it is possible to recognize the facility and to infer the change of the change by using the orthographic image taken from the low-cost image capturing device, It will be able to contribute to the development of the small and medium-sized facility management industry of

Although the present invention has been described with reference to the accompanying drawings, this is merely exemplary, and those of ordinary skill in the art to which it pertains can easily implement various modifications and equivalents therefrom. There will be. Therefore, the true technical protection scope of the present invention should be defined by the claims.

10: image collection unit 20: orthographic image generation unit
30: facility object extraction unit 40: artificial paper facility learning unit
50: change change detection unit 60: expression unit
70: ortho image storage unit 80: real-time mapping unit

Claims (7)

an image collecting unit for acquiring an image taken in the air in a state equipped with a GPS and a camera;
an orthographic image generating unit for extracting an orthogonal image from the images collected through the image collecting unit;
a facility object extraction unit for extracting a specific facility from an orthogonal image;
The artificial paper for learning the facility extracted object is a facility learning unit;
a change change detection unit for detecting a change change in the extracted facility;
a display unit that displays the change and change of the facility;
and a real-time mapping unit for detecting facility changes and changes in real time through real-time mapping.
Facility change change management system through facility object recognition and multi-image registration. Acquiring an image taken in the air in a state in which the image collecting unit is equipped with a GPS and a camera;
generating an ortho image by an orthogonal image generating unit based on the image acquired through the image collecting unit, and databaseizing it;
Learning the object to be extracted by the artificial intelligence facility learning unit based on the ortho image generated by the orthographic image generating unit and transferring it to the facility object extraction unit;
The facility object extraction unit extracts facilities through artificial intelligence facility recognition and location matching between images;
The change detection unit includes the steps of determining whether the same object is the same through the location-matched image and confirming whether there is a change;
The display unit is composed of a step of expressing so that the change point of the facility can be confirmed by place and time flow.
Management method of facility change management system through facility object recognition and multi-image registration. 3. The method of claim 2,
The real-time mapping unit further comprises the step of determining the same point in multiple orthogonal images through object recognition when performing real-time mapping to detect change or not in real time.
Management method of facility change management system through facility object recognition and multi-image registration. 4. The method of claim 3,
The step of the real-time mapping unit determining the same point in the multiple orthogonal images,
selecting a standard orthographic image from among multiple orthogonal images;
selecting a facility object from the reference orthogonal image;
Checking the GPS latitude and longitude position estimated from the pixel position of the facility object in the reference orthogonal image;
Based on the GPS latitude and longitude information estimated from the reference orthographic image,
Management method of facility change management system through facility object recognition and multi-image registration. 5. The method of claim 4,
The real-time mapping unit executes a coordinate transformation parameter for estimating the same position by an image matching (homography) method even at a point selected as the same point based on GPS latitude and longitude information;
selecting a coordinate transformation parameter by varying the size of the comparison area between the reference orthogonal image and the comparison target orthographic image, and optimizing the parameter by weighted average thereof;
Constructed including the step of estimating the part corresponding to the position of the object in the reference orthogonal image as the comparison target ortho image area based on the coordinate transformation parameter value
Management method of facility change management system through facility object recognition and multi-image registration. 4. The method of claim 3,
When the real-time mapping unit determines the same point in multiple orthogonal images through object recognition, detecting a facility through a model learned from the facility object learning unit for a comparison target ortho image extraction area;
Checking whether the species of the detected facilities are the same, and whether the color and structural feature points are the same;
If the species is the same and has a high degree of similarity, the
Management method of facility change management system through facility object recognition and multi-image registration. 7. The method of claim 6,
When the real-time mapping unit recognizes the same article,
In order to extract whether there is no change or change in the facility, it is composed
Management method of facility change management system through facility object recognition and multi-image registration.

Images