KR102289550B1 - Augmented reality information providing method and system using image deep learning for matching aerial image data and Geographic Information System data - Google Patents

Abstract

An AR information providing method and system for providing GIS data matched with real-time aerial image data in an AR method using image learning for matching aerial image data and GIS data are provided. AR information providing method according to an embodiment of the present invention, the AR information providing system, using the aerial image data, learning a deep learning-based image learning model; a matching step of matching, by the AR information providing system, real-time aerial image data input based on the learned image learning model with pre-stored GIS data; and allowing the AR information providing system to output the matched GIS data to the screen in an AR method. Thereby, by accurately calculating the GIS data matching the image collected using the camera and the location information of the object recognized as heavy equipment, and providing it in an AR method, it is possible to provide a service that enables efficient management to safety and security related industries. there is.

Description

AR information providing method and system using image learning for matching aerial image data and GIS data TECHNICAL FIELD Augmented reality information providing method and system using image deep learning for matching aerial image data and Geographic Information System data

The present invention relates to a method and system for providing AR information, and more particularly, AR information that provides GIS data matched with real-time aerial image data in an AR method using image learning for matching aerial image data and GIS data It relates to a method and system for providing.

Recently, a technology for providing AR (augmented reality) information to images collected using a drone has appeared. However, in the case of an image collected using a drone, due to a structural problem of the drone, the coordinates ) and the target of the image do not match, so there is a problem that the AR information provided based on the location information of the drone has a large error with the actual data.

In addition, even when heavy equipment such as an excavator is recognized as an object in an image collected using a drone, in order to accurately calculate the location information of the object recognized as heavy equipment, a means for separately providing location information within the heavy equipment should be provided. Inconvenience exists in that an additional cost occurs.

Therefore, it is required to find a way to accurately calculate GIS data matching the collected images and location information of objects recognized as heavy equipment and provide them in an AR method.

The present invention has been devised to solve the above problems, and an object of the present invention is to accurately calculate GIS data matching an image collected using a camera and location information of an object recognized as heavy equipment in an AR method. An object of the present invention is to provide a method and system for providing AR information using image learning for matching between provided aerial image data and GIS data.

According to an embodiment of the present invention for achieving the above object, a method for providing AR information includes, by an AR information providing system, learning a deep learning-based image learning model using aerial image data; a matching step of matching, by the AR information providing system, real-time aerial image data input based on the learned image learning model with pre-stored GIS data; and allowing the AR information providing system to output the matched GIS data to the screen in an AR method.

In addition, the aerial image data may include photographing information including photographing position information, photographing altitude information, and photographing angle information of the aerial image data.

Also, in the matching step, an actual photographing area for the aerial image data may be calculated, and GIS data corresponding to the calculated actual photographing area may be matched with the aerial image data.

In the matching step, the GIS data may be processed, and the processed GIS data may be matched with the aerial image data by converting the units of the coordinate system, removing the overlapping interpolation points of the scale to be used, or merging the connected points.

In addition, in the matching step, among the facilities in the real-time aerial image data, the main facility information retrieved through the R-Tree algorithm may be matched with the GIS data.

In the output step, when the latitude and longitude of the actual photographing area are calculated using the photographing information and the main facility information in the real-time aerial image data, GIS data matching the calculation result may be output in an AR method.

In addition, according to an embodiment of the present invention, a method for providing AR information includes, when real-time aerial image data is input, the AR information providing system recognizing heavy equipment in real-time aerial image data based on the learned image learning model; Further comprising, in the output step, when the heavy equipment is recognized, the recognition result of the heavy equipment may be output together with the GIS data in an AR method.

And, in the AR information providing method according to an embodiment of the present invention, when heavy equipment is recognized, the AR information providing system uses GIS data matching the latitude and longitude calculation results of the actual shooting area to provide location information of the heavy equipment. Inferring; may further include.

In addition, the GIS data includes location information and depth information of a gas pipe buried in the actual shooting area. In the AR information providing method, the AR information providing system uses the location information of heavy equipment and the information of the gas pipe in the actual shooting area. , detecting a collision risk situation between a gas pipe buried in an actual photographing area and heavy equipment; may further include.

In the output step, when the locations of the buried gas pipe and heavy equipment in the actual shooting area are adjacent to each other within a preset range, a warning alarm indicating a collision risk situation is output, and in a collision risk situation, the warning alarm outputted is the buried gas pipe The output method or output display may be set differently according to the depth of the .

And in the recognition step, when the heavy equipment is recognized, it is determined whether the heavy equipment is in a working state or a non-working state using the learned image learning model, and the detection step is, only when the heavy equipment is determined to be in a working state, the actual shooting area It is possible to detect a collision risk situation between a buried gas pipe and heavy equipment.

On the other hand, according to another embodiment of the present invention, AR information providing system, a storage unit for storing a deep learning-based image learning model; and a processor that trains a deep learning-based image learning model using aerial image data, and outputs GIS data matching real-time aerial image data input based on the learned image learning model in an AR method within the screen; include

As described above, according to embodiments of the present invention, by accurately calculating GIS data matching an image collected using a camera and location information of an object recognized as heavy equipment, and providing it in an AR method, safety and security related It is possible to provide a service that enables efficient management to the industry.

1 is a view provided for explaining an AR information providing system using image learning for matching aerial image data and GIS data according to an embodiment of the present invention;
2 is a flowchart provided for explaining a method for providing AR information using image learning for matching aerial image data and GIS data according to an embodiment of the present invention;
3 to 5 are views provided for explaining a process of calculating a photographing area of a drone, and
6 to 7 are diagrams illustrating aerial image data matched with GIS data.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

1 is a diagram provided for explanation of an AR information providing system (hereinafter, collectively referred to as 'AR information providing system') using image learning for matching aerial image data and GIS data according to an embodiment of the present invention. .

The AR information providing system according to the present embodiment is provided to accurately calculate GIS data matching an image collected using a camera and location information of an object recognized as heavy equipment, and provide it in an AR method.

To this end, the present AR information providing system may include a communication unit 110 , a storage unit 120 , and a processor 130 .

The communication unit 110 is a communication module capable of receiving aerial image data collected through a camera.

Here, the camera may be a camera mounted on an unmanned aerial vehicle such as a drone or a surveillance camera such as a Closed Circuit Television (CCTV).

The storage unit 120 may store programs and data necessary for the processor 130 to operate.

Specifically, the storage unit 120 may store GIS data, a deep learning-based image learning model, and training data for learning the image learning model.

The processor 130 trains an image learning model based on deep learning by using the aerial image data.

Specifically, the processor 130 receives a plurality of aerial image data including heavy equipment as learning data and trains the image learning model using deep learning. Through this, the image learning model can recognize and extract heavy equipment when the input image includes heavy equipment.

In addition, when real-time aerial image data is input, the processor 130 determines whether or not heavy equipment is included in the real-time aerial image data and whether heavy equipment is included in the aerial image data using the learned image learning model. location can be extracted.

In addition, the processor 130 may match the input real-time aerial image data with pre-stored GIS data. That is, the processor 130 calculates the coordinates of the actual photographing area corresponding to the photographed actual position of the input real-time aerial image data, extracts and matches the GIS data of the position corresponding to the calculated coordinate information, thereby providing aerial image data GIS data can be matched to

Here, the actual photographing area indicates an actual geographic area of the area included in the corresponding aerial image data, and since the aerial image is photographed in the air, a generally rectangular area is photographed as shown in FIG. 3 .

Accordingly, the actual photographing area information includes actual coordinate information (latitude and longitude information) of four vertices of a rectangle corresponding to the actual photographing area.

Specifically, the processor 130 may calculate an actual photographing area for the aerial image data using photographing information including photographing location information, photographing altitude information, and photographing angle information of the aerial image data.

Here, the photographing information may be recorded together with the image data while the aerial image data is photographed, and includes photographing location information, photographing altitude information, and photographing angle information.

For example, when aerial image data is photographed by a drone, the processor 130 checks the photographing location information using the GPS information of the drone, checks the photographing altitude information using the altitude information of the drone, and You can check the shooting angle information using the nose direction and camera angle.

In addition, the processor 130 may extract GIS data corresponding to the calculated actual photographing area and match it with the aerial image data. That is, the processor 130 extracts GIS data (especially, gas pipe-related data and major facility data) included in the actual photographing area from among the GIS data to be matched with the aerial image data. At this time, the GIS data may have a different format or form from the aerial image data or the actual photographing area. In this case, the processor 130 may process the GIS data to match the format or form with the aerial image data or the actual photographing area. there is.

In addition, the processor 130 may verify or correct the calculation result of the actual photographing area by using the main facility information in the real-time aerial image data.

Specifically, the processor 130 may recognize the main facilities included in the aerial image data by inputting and learning the image learning model aerial image data including the main facilities.

In addition, when there is a major facility recognized by the image learning model in the real-time aerial image data, the processor 130 may calculate the location of the recognized major facility using the calculated actual photographing area.

Then, the processor 130 may verify or correct the calculation result of the actual photographing area by comparing the calculated location of the major facility with the location of the corresponding major facility included in the GIS data.

At this time, the main facility means a facility that is easy to recognize in an image and has a small change in address, such as a building corresponding to a landmark such as a fire station, a police station, a city hall, a school, a large shopping mall, a department store, a mega building, etc. do.

Here, the GIS data represents Geographic Information System (GIS) data. The geographic information system is comprehensive information designed to create and manage maps and geographic information, which were used in printed form in the past, using a computer, and to collect, analyze, and process data based on the geographic information obtained from it and apply it to all fields related to topography. As a system, the geographic information system is a system in which a national geographic information system is established that digitizes, shares and utilizes all information on land, resources, environment, and facilities.

In addition, when real-time aerial image data is input, the processor 130 may recognize heavy equipment in the real-time aerial image data based on the learned image learning model.

And, when the heavy equipment is recognized, the processor 130 may calculate location information of the heavy equipment by using the actual photographing area information.

At this time, the processor 130 uses the location information of the heavy equipment and the information of the gas pipe in the actual photographing area to detect a collision risk situation between the buried gas pipe and the heavy equipment in the actual photographing area, and a real-time aerial image as shown in FIG. 6 . Data and GIS data matching the data (particularly, gas pipe-related data and major facility data) may be output on the screen of the display device in an AR method.

Here, the display device may be a display device included in the AR information providing system, or a separate display device connected to communicate with the AR information providing system.

2 is a flowchart provided to explain a method for providing AR information using image learning for matching aerial image data and GIS data (hereinafter, collectively referred to as 'AR information providing method') according to an embodiment of the present invention. , FIGS. 3 to 5 are views provided to explain a process of calculating a photographing area of a drone, and FIGS. 6 to 7 are diagrams illustrating aerial image data matched with GIS data.

The AR information providing method performed by the AR information providing system according to this embodiment is a learning step (S210) in which the processor 130 learns a deep learning-based image learning model using aerial image data, real-time aviation When the image data is input, the processor 130 uses the image learning model to match the input real-time aerial image data with the pre-stored GIS data ( S220 ). When the real-time aerial image data is input, the processor 130 ), a recognition step of determining whether heavy equipment is recognized in real-time aerial image data based on the learned image learning model (S230), when the heavy equipment is recognized (S230-Y), the processor 130, the latitude of the actual shooting area and an inference step (S240) of inferring the location information of the heavy equipment by using the GIS data matching the actual photographing area information, which is the result of the hardness calculation, the processor 130, the location information of the heavy equipment and the information of the gas pipe in the actual photographing area GIS matched with real-time aerial image data input based on the image learning model learned by the detection step ( S250 ) and the processor 130 to detect a collision risk situation between a gas pipe buried in an actual shooting area and heavy equipment using It may be composed of an output step (S260) of outputting data to the screen in an AR method.

In the matching step ( S220 ), the processor 130 calculates an actual photographing area for the aerial image data by using the real-time aerial image data, and processes the GIS data corresponding to the calculated actual photographing area to obtain the aerial image data can be matched with

Here, the aerial image data may include photographing information including photographing location information, photographing altitude information, and photographing angle information, and the GIS data may include location information and depth information of a gas pipe embedded in an actual photographing area. .

And in the matching step (S220), the processor 130, when processing the GIS data, removes the geometry structure information of the GIS data, converts the unit of the coordinate system, and as illustrated in FIGS. 3 to 4, the scale to be used Duplicate interpolation points can be removed or connected points can be merged.

In addition, in the matching step ( S220 ), the processor 130 may match the main facility information retrieved through the R-Tree algorithm among the facilities in the real-time aerial image data with the GIS data.

That is, in the matching step ( S220 ), the processor 130 may calculate the latitude and longitude of the actual photographing area by using the photographing information and the main facility information in the real-time aerial image data, and match the calculated result.

Here, the R-Tree algorithm is an algorithm that generates a spatial data structure that organizes data using the characteristics of a minimum range rectangle surrounding a specific area.

In the recognition step (S230), the processor 130 may recognize heavy equipment in real-time aerial image data based on the learned image learning model, and further, when the heavy equipment is recognized, using the learned image learning model, It can be determined whether the heavy equipment is in a working state or in a non-working state.

In this case, in the detection step ( S250 ), the processor 130 detects a collision risk situation between the gas pipe and the heavy equipment buried in the actual photographing area only when the heavy equipment is determined to be in a working state, so that non-work such as movement of heavy equipment It is possible to prevent the occurrence of an error in determining that the buried gas pipe and heavy equipment collide with each other.

In the inference step ( S240 ), the processor 130 may infer the location information of the heavy equipment by using the location information of major facilities in the screen included in the GIS data matched with the real-time aerial image data.

For example, the processor 130 obtains location information of major facilities on the screen using GIS data matched with real-time aerial image data, and considers the distance of heavy equipment from the obtained major facilities, and separates Without a sensor for measuring location information, location information of heavy equipment can be inferred.

At this time, for the accuracy of the inference result of the location information of the heavy equipment, the distance between each major facility and the heavy equipment is calculated using the location information of at least three or more main facilities, and the location information of the heavy equipment is calculated using the distance calculation result. It is preferable to infer.

Specifically, for example, when there are three or more of the main facilities in the screen, the processor 130 selects three major facilities having a relative distance to heavy equipment closer than other major facilities in a way that preferentially selects three, The distance between major facilities and heavy equipment can be calculated.

In the detection step (S250), the processor 130 may detect a collision risk situation between the buried gas pipe and the heavy equipment in the actual photographing area by using the location information of the heavy equipment and the information of the gas pipe in the actual photographing area, and additionally the processor (130), through the communication unit 110, when the work plan information including the location information of the area to perform the work using the heavy equipment is obtained, through the reasoning step, through the inferred location information or the communication unit 110 Based on the acquired location information of the heavy equipment, it is possible to verify whether the heavy equipment performs work in an area suitable for the work plan information.

On the other hand, in the output step (S260), when the heavy equipment is recognized, the processor 130 may output the recognition result of the heavy equipment together with the GIS data on the screen of the display device in an AR method.

Also, the processor 130 may output the calculation result of the distance between the gas pipe and the heavy equipment together with the recognition result of the heavy equipment and the GIS data on the screen of the display device in an AR method.

In addition, the processor 130 may display the location information of the heavy equipment on the screen in an AR method. In this case, the processor 130 may display the location information of the heavy equipment in a manner that displays the distance from the surrounding major facilities. For example, when the major facility close to the heavy equipment is the Bundang Police Station, the processor 130 may display location information of the heavy equipment such as “a location within a radius of 50 meters from the Bundang Police Station”.

Specifically, when the location of the buried gas pipe and the heavy equipment in the actual shooting area is adjacent to within a preset range, in the output step S260 , the processor 130 provides a warning alarm informing of a collision risk situation, and in a collision risk situation , the provided warning alarm is set to display text, color, shape, etc. differently depending on the depth of the buried gas pipe. can make you aware

At this time, the distance between the gas pipe and the heavy equipment means the horizontal distance between the point where the gas pipe is buried and the point where the heavy equipment is located, and the distance is calculated based on the GPS coordinates of the point where the gas pipe is buried and the GPS coordinates of the point where the heavy equipment is located similar results can be obtained.

However, the processor 130, in the detection step (S250), when calculating the distance between the gas pipe and the heavy equipment, if it is determined that the distance between the point where the gas pipe is buried and the point where the heavy equipment is located is adjacent within a preset range, the output step (S260) ), a warning alarm is output, and the distance between the buried gas pipe and heavy equipment is calculated using the depth information in which the gas pipe is buried, and the distance calculation result is displayed on the screen or different warning alarms according to the distance calculation result. You can have this output.

For example, the processor 130 sets a plurality of grades in which the boundary level of the collision risk is sequentially increased from the lowest grade according to the distance between the buried gas pipe and the heavy equipment, and the distance calculation result corresponding to each grade When is calculated, by outputting a warning alarm according to the grade, the administrator can quickly recognize the distance between the gas pipe and heavy equipment or the degree of a collision risk situation according to the distance calculation result.

Through this, after the real-time aerial image data is input, it is possible to smoothly manage resources for each task, and it is possible to provide AR information immediately and effectively for the real-time aerial image data.

Incidentally, in this embodiment, although the matching step (S220), the recognition step (S230), the inference step (S240), and the detection step (S250) are described as being sequentially performed, the image performing the matching step (S220) The learning model and the image learning model for performing the recognition step S230, the inference step S240, and the detection step S250 are separately provided, so that each task can be implemented in parallel.

In addition, it goes without saying that the technical idea of the present invention can also be applied to a computer-readable recording medium containing a computer program for performing the functions of the apparatus and method according to the present embodiment. In addition, the technical ideas according to various embodiments of the present invention may be implemented in the form of computer-readable codes recorded on a computer-readable recording medium. The computer-readable recording medium may be any data storage device readable by the computer and capable of storing data. For example, the computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, or the like. In addition, the computer-readable code or program stored in the computer-readable recording medium may be transmitted through a network connected between computers.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims Various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

110: communication department
120: storage
130: processor

Claims (12)

The AR information providing system, using the aerial image data, training a deep learning-based image learning model;
a matching step of matching, by the AR information providing system, real-time aerial image data input based on the learned image learning model with pre-stored GIS data; and
Including, by the AR information providing system, to output the matched GIS data to the screen in an AR method;
When real-time aerial image data is input, the AR information providing system recognizing heavy equipment in real-time aerial image data based on the learned image learning model; further comprising,
The output stage is
When heavy equipment is recognized, the recognition result of the heavy equipment is output in AR method along with GIS data.
How to provide AR information,
When the heavy equipment is recognized, the AR information providing system infers the location information of the heavy equipment by using the GIS data matching the latitude and longitude calculation results of the actual shooting area;
For GIS data,
The location information and depth information of the gas pipe buried in the actual shooting area are included.
How to provide AR information,
Detecting, by the AR information providing system, a collision risk situation between a gas pipe buried in the actual imaging area and the heavy equipment by using the location information of the heavy equipment and the information of the gas pipe in the actual photographing area; AR information characterized by further comprising: How to provide.
The method according to claim 1,
aerial image data,
AR information providing method, characterized in that it includes photographing information including photographing location information, photographing altitude information, and photographing angle information of aerial image data.
The method according to claim 1,
The matching step is
A method for providing AR information, comprising calculating an actual photographing area for the aerial image data, and matching GIS data corresponding to the calculated actual photographing area with the aerial image data.
delete 4. The method according to claim 3,
The matching step is
AR information providing method, characterized in that the main facility information retrieved through R-Tree algorithm among facilities in real-time aerial image data is matched with GIS data.
6. The method of claim 5,
The output stage is
AR information providing method, characterized in that when the latitude and longitude of the actual shooting area are calculated using the shooting information and main facility information in the real-time aerial image data, GIS data matching the calculation result is output in an AR method.
delete delete delete The method according to claim 1,
The output stage is
When the location of the buried gas pipe and heavy equipment within the actual shooting area is adjacent to within a preset range, a warning alarm indicating a collision risk situation is output;
In the event of a risk of collision, the warning alarm that is output is,
AR information providing method, characterized in that the output method or output display is set differently according to the depth of the buried gas pipe.
The AR information providing system, using the aerial image data, training a deep learning-based image learning model;
a matching step of matching, by the AR information providing system, real-time aerial image data input based on the learned image learning model with pre-stored GIS data; and
Including, by the AR information providing system, to output the matched GIS data to the screen in an AR method;
When real-time aerial image data is input, the AR information providing system recognizing heavy equipment in real-time aerial image data based on the learned image learning model; further comprising,
The output stage is
When heavy equipment is recognized, the recognition result of the heavy equipment is output in AR method along with GIS data,
The recognition stage is
When heavy equipment is recognized, it is determined whether the heavy equipment is in a working state or a non-working state using the learned image learning model,
The detection step is
AR information providing method, characterized in that only when it is determined that the heavy equipment is in a working state, a collision risk situation between the gas pipe buried in the actual shooting area and the heavy equipment is detected.
a storage unit for storing an image learning model based on deep learning; and
A processor that trains a deep learning-based image learning model using aerial image data, and outputs GIS data matching real-time aerial image data input based on the learned image learning model in an AR method within the screen; includes; do,
The processor is
When heavy equipment is recognized, the recognition result of heavy equipment is output in AR method along with GIS data.
The processor is
When heavy equipment is recognized, the location information of the heavy equipment is inferred using GIS data that matches the latitude and longitude calculation results of the actual shooting area,
For GIS data,
The location information and depth information of the gas pipe buried in the actual shooting area are included.
The processor is
AR information providing system, characterized in that by using the location information of the heavy equipment and the information of the gas pipe in the actual photographing area, a collision risk situation between the buried gas pipe and the heavy equipment in the actual photographing area is detected.

Images