KR102033075B1 - A providing location information systme using deep-learning and method it - Google Patents

Abstract

The present invention relates to a location information system using deep learning and a providing method thereof which provide location information and services by deep learning based on a SLAM technique. The location information system using deep learning comprises: a main server (10) including an image processing unit (11) to generate a virtual map after processing an image inputted from the outside, a communication unit (12) to communicate with a local server (20) and a terminal (30), and a data storage unit (13) to store data; the local server (20) connected to the main server (10) to transmit information inputted for each area to the main server (10) and distribute and store information for each area; and the terminal (30). The method comprises an image information acquisition step (S10), a key frame extraction step (S20), a SLAM processing step (S30), a data processing step (S40), a processing image information correction step (S50), a landmark extraction step (S60), a virtual map generation step (S70), and a location information providing step (S80).

Description

Location information system using deep learning and its providing method {A PROVIDING LOCATION INFORMATION SYSTME USING DEEP-LEARNING AND METHOD IT}

The present invention relates to a location information system using deep learning and a method of providing the same, and to provide more accurate location information and service by deep learning based on SLAM technique.

In general, the location information is serviced to the user in a variety of ways, the most representative of which is the GPS system applying the satellite navigation system.

However, such a GPS system is based on the communication with the satellites orbiting the earth orbit, the user is not possible if the communication with the satellite is not carried out, and the location information only if the communication with a certain number of satellites is performed This increases the accuracy for.

However, even in the case of such a GPS, the accuracy of several meters or less exist, so that only the approximate position can be grasped, and there is a shortage in providing the position information according to the pedestrian path or autonomous driving.

Accordingly, various methods have been proposed, and one of them is proposed by SLAM (Simultaneous Localization And Map-Building), a technology that generates a map of the environment without the help of the outside by using a sensor attached to the robot as the robot moves around the unknown environment. Among them, techniques for grasping location information by analyzing them through image photographing of terrain and features have been proposed.

As a representative technology applying the SLAM technology, Korean Patent Publication No. 10-2016-0003731 (Wide Location Recognition from SLAM Maps, published on January 11, 2016) is proposed, and SLAM (simultaneous localization and mapping) maps are proposed. DETAILED DESCRIPTION Exemplary methods, apparatuses, and systems for performing wide area localization are disclosed. The mobile device may select the first keyframe based SLAM map of the local environment using the one or more received images. Individual location of the mobile device within the local environment can be determined, and the individual location can be based on a keyframe based SLAM map. The mobile device may send the first keyframe to the server and receive a first global location response indicating a correction to the local map on the mobile device. The first global positioning response may include rotation, translation, and scale information. The server proposes a technique that can receive keyframes from the mobile device and locate keyframes in the server map by matching the keyframe features received from the mobile device to the server map feature.

In addition, the Republic of Korea Patent Publication No. 10-2013-0134986 (SLAM system and method of the mobile robot to receive a picture input for the environment from the user, published December 10, 2013) is a mobile robot that receives a picture input for the environment from the user SLAM system of claim 1, comprising: a user terminal for receiving information or commands about the environment from a user and transmitting the information to a mobile robot; And

It provides a SLAM system of a mobile robot, characterized in that it comprises a mobile robot for performing map making by grasping information and commands transmitted from the user terminal, surrounding environment information and mobile information obtained from a data acquisition device. A data acquisition device mounted to the robot, the data acquisition device being a means for acquiring information about the surrounding environment or for measuring movement of the mobile robot; A second processing device performing a function of determining a location of a mobile robot and determining a path based on a real-time mapping, managing and modifying a created map, and using the information transmitted from the data acquisition device and the user terminal; ; And it provides a technology for controlling the running of the mobile robot based on the information processed in the second processing device.

However, in the case of the prior art, only the image information is SLAM and applied based on this, and thus, it is vulnerable to changes in the external environment, so that the external environment is not applicable and thus there is a limit in providing a service.

Republic of Korea Patent Publication No. 10-2013-0023433 (apparatus and method for managing the map of the mobile robot based on SLAM technology, published March 08, 2013) Republic of Korea Patent Application Publication No. 10-2013-0134986 (SLAM system and method of a mobile robot receiving a picture of the environment from the user, published December 10, 2013) Republic of Korea Patent Publication No. 10-2014-0003987 (SLAM system for mobile robots fusion of vision sensor information and motion sensor information, published January 10, 2014) Republic of Korea Patent Publication No. 10-2016-0003731 (wide area recognition from SLAM maps, published January 11, 2016)

The present invention provides a location information system and a method for providing the information to provide a location information by using deep learning, more precise, and to adapt to changes in the external environment, in order to compensate for the shortcomings of the SLAM technology as described above. The purpose.

In the location information system using deep learning to solve the problems of the present invention, the image processing unit 11 and the local server 20 and the terminal 30 to generate a virtual map after processing and processing the image input from the outside Main server 10 comprising a communication unit 12 for communicating with the data storage unit 13 and a data storage unit 13 for storing data; and the main server 10 connected to the main server 10 for inputting information for each region. 10) local server 20 for transmitting and storing the regional information; And a terminal 30 to provide a location information system using deep learning.

In this case, the image processing unit 11 includes a key frame extracting unit 111 for extracting a key frame of the raw image input through the terminal 30 or an external video device, and a key frame in the key frame extracting unit 111. The SLAM processing unit 112 for SLAM processing the extracted image, the data processing unit 113 for deeply arranging the SLAM processed image on the virtual main frame by the SLAM processing unit 112, and the data processing unit 113. A data extractor 114 for correcting distorted information by matching the processed data with external information, and a landmark extractor 115 extracting a landmark based on the information corrected by the data corrector 114. And a virtual map generator 116 for generating a virtual map by applying the landmark extracted from the landmark extractor 115, and the data processor 113 displays the SLAM-processed image on the virtual mainframe. At the time of day After SLAM processing, this is arranged on the virtual mainframe by time slot to identify the overlapped parts, and the center point is found through deep learning that repeats these tasks.

In addition, the local server 20 stores a raw image input through a terminal located in a corresponding region, and uploads an image storage unit 21, a main server 10, and a terminal (uploading) to compare and analyze the processed image. And a communication unit 22 for communicating with the device 30, a data storage unit 23 for storing data, and an information sharing unit 24 for providing information to the terminal 30 and the main server 10. The image storage unit 21 compares the processed image and the stored backup data and the inputted raw image information by receiving the image input unit 211 for receiving and storing the raw image input from the terminal or a separate photographing equipment processed image of the region To provide a location information system using a deep learning, characterized in that consisting of a processed image uploading unit 212 for uploading.

According to another aspect of the present invention, there is provided a method for providing location information using deep learning, the method comprising: image information obtaining step (S10) of acquiring an image from a terminal of a user or a manager; A keyframe extraction step (S20) of extracting keyframes for each time zone from the obtained image information; A SLAM step of SLAM processing the image information for each keyframe and storing the information in chronological order (S30); A data processing step (S40) of deep-learning the stored information after the SLAM processing for each keyframe and placing it on the virtual mainframe (S40); Processing image information correction step (S50) of correcting the distorted information by matching the processing data and external information obtained through deep learning; A landmark extraction step (S60) of extracting a landmark based on the modified information; Generating a virtual map by applying the extracted landmark information (S70); And location information providing step (S80) of providing location information using a virtual map.

At this time, the data processing step (S40) is to locate the overlapped by placing the information obtained for each time zone on the virtual mainframe to the SLAM-processed information, and find the center point through the deep learning to repeat this operation The process of correcting the processed image information (S50) is performed on the virtual mainframe in the previous step (S40) and the accurate position information through matching of the centripetal point obtained through deep learning and external information. Acquire, but corrects the distorted information that may occur when shooting the image by matching the GPS information or location information of the object obtained through the center of gravity and the external information of the map, aerial photographs and cadastral information obtained By providing a location information providing method using deep learning to solve the problem of the present invention better.

By providing a location information system and a method of providing the same using the deep learning of the present invention, there is an effect that can provide highly reliable location information by being less affected by external environmental changes.

1 is a block diagram of a location information system using the present deep learning.
2 is a block diagram of the main server 10 according to the present invention.
3 is a block diagram of a local server 20 according to the present invention.
4 is a flowchart illustrating a method of providing location information using deep learning according to the present invention.

Hereinafter, a person skilled in the art will be described in detail with reference to the accompanying drawings so that the position information system using the deep learning of the present invention and a method of providing the same can be easily implemented.

1 is a configuration diagram of a location information system using the deep learning of the present invention, FIG. 2 is a configuration diagram of a main server 10 according to the present invention, and FIG. 3 is a configuration diagram of a local server 20 according to the present invention. .

1 to 3, the location information system using the deep learning of the present invention is connected to the main server 10 and the main server 10, the main server 10 is input information for each region; It is composed of a local server 20 and the terminal 30 to transmit to, and to store the regional information distributed.

The main server 10 is an image processing unit 11 for processing and processing the image input from the outside, and generates a virtual map and the communication unit 12 for communicating with the local server 20 and the terminal 30 And a data storage unit 13 for storing data.

Here, the image processing unit 11 has a key frame extracting unit 111 for extracting a key frame of the raw image input through the terminal 30 or an external video device and a key frame in the key frame extracting unit 111. The SLAM processor 112 and the SLAM processor 112 for SLAM processing the extracted image are processed by the data processing unit 113 and the data processing unit 113 to deeply arrange the SLAM processed image on a virtual mainframe. A landmark extractor 115 and a landmark extractor for extracting a landmark based on the information corrected by the data correction unit 114 and the data correction unit 114 to correct the distorted information by matching the corrected data with external information The virtual map generator 116 generates a virtual map by applying the landmark extracted by the unit 115.

The key frame extracting unit 111 extracts a key frame from the image information of the inputted raw image. The key frame extracting unit 111 extracts a key frame for each time zone from a continuously captured image and corrects the position of the terminal.

It extracts keyframes in order to reduce errors due to processing and processing of continuous images during post-production such as SLAM processing, data processing through deep learning, data modification through matching with external information, and landmark extraction. This is to proceed with the post-production.

The SLAM processing unit 112 performs SLAM processing on the input image information based on the key frame extracted by the key frame extracting unit 111.

Here, SLAM (Simultaneous Localization And Map-Building) is a technology for recognizing a terrain or a feature by binarizing or encoding a recognized image, and a detailed description thereof will be omitted.

The data processing unit 113 arranges the SLAM-processed image on the virtual mainframe. More specifically, the data processing unit 113 performs the SLAM processing on the consecutive photographed images for each time slot and then arranges it on the virtual mainframe according to the time slot. By finding the overlapping parts, deep learning that repeats these tasks is found.

This is to compensate for this, because in the case of continuous shooting images, the shooting position of each frame is different, or if there is a volume of an object, distortion may occur according to the shooting position of the image.

The data correction unit 114 is disposed on the virtual mainframe through the data processing unit 113 and the exact position of the terrain or feature as an object by matching the center information with the external information obtained through deep learning. To grasp the information.

At this time, the external information is one of the GPS information or the location information of the object obtained through the information such as map, aerial photographs and cadastral map.

The landmark extractor 115 extracts a landmark based on information corrected through the data correction unit 114, and more specifically, an element whose shape and position may change according to a temporal change. And remove the building with relatively little position variation or shape deformation and apply it as a landmark.

The virtual map generator 116 generates a virtual map by placing a landmark extracted through the landmark extractor 115 on a map.

The communication unit 12 is to communicate with the local server 20 provided for each region and the terminal 30 for using the service.

At this time, the communication method is to use an external Internet network or a separate dedicated internal computer network.

The data storage unit 13 temporarily stores the raw image input from the terminal 30 and the like, and temporarily stores the processed image through the time-based SLAM image and post-processing of the processed raw image.

At this time, in the case of the data completed work and service is transferred to the local server 20 provided in the area to save the data in each local server 20 to reduce the load that may occur in the main server 10 .

Here, when the user requests the service through the terminal 30, if there is backup data of the corresponding area, it is called and mounted on the data storage unit 13 of the main server 10 to store the service.

The local server 20 stores an original image input through a terminal located in a corresponding region, and uploads an image storage unit 21, a main server 10, and a terminal 30 to upload and analyze the processed image. Communication unit 22 for communicating with the data, a data storage unit 23 for storing data, and an information sharing unit 24 for providing information to the terminal 30 and the main server 10.

At this time, the image storage unit 21 compares the backup data and the inputted raw image information processed by the image input unit 211 for receiving and storing the raw image input from the terminal or a separate photographing equipment, It is composed of a processed image uploading unit 212 for uploading a processed image.

When the user inputs the raw image of the region through the terminal 30 and requests a service, the input image is compared with the processed image of the region and uploaded if the processed image is stored as backup data. To provide services.

In the case of the terminal 30, any means for communication capable of a photographing function possessed by an administrator or a user is possible.

The terminal 30 can communicate with the main server 10 or the local server 20 through the Internet, which is an external computer network, and can download and install a separate APP or a management program for providing a service. However, this is not limitative.

With the above configuration, the position information system using the deep learning of the present invention can be completed.

Another invention according to the present invention will be described in detail below with reference to FIG. 4, which is a flowchart of a method for providing location information using deep learning.

Referring to FIG. 4 in detail, the method for providing location information using deep learning according to the present invention includes an image information acquiring step (S10) of acquiring an image from a terminal of a user or a manager and time in the image information obtained as described above. Key frame extraction step (S20) for extracting keyframes per unit and SLAM step (S30) for storing chronological order by storing the image information for each keyframe (S30) and virtual mainframe by deep learning stored information after SLAM processing for each keyframe The processed image information correction step (S50) of matching the processed data obtained through deep learning with external data and correcting the distorted information by placing the data processing step (S40) disposed on the image, and landmarks based on the modified information. A landmark extraction step (S60) for extracting, a virtual map generation step (S70) for generating a virtual map by applying the extracted landmark information, and a location using the virtual map It is composed of a location information providing step of providing information (S80).

Here, the image information acquisition step (S10) refers to the image taken while moving in the field through the image taken by the terminal of the user or administrator or a separate recording equipment.

In this way, the obtained image is input to the local server 20 located in each region, and can be transmitted by accessing an APP or an open management program installed in the terminal.

This is because it is difficult for an administrator to obtain information of all regions in order to provide a service, so that a user who wants to provide a service can request a service after obtaining and transmitting image information of a desired region.

In the keyframe extraction step (S20) of extracting keyframes according to time slots from the obtained image information, keyframes are extracted according to time slots when the position of the terminal is changed when images are continuously photographed through the terminal. Done.

Here, a method of extracting keyframes for each time zone is applied at a predetermined time interval by the user, or triangulation method using GPS information of a terminal acquiring an image, a distance from a base station, or a change in the size of a specific object in the image. This can be applied.

For example, when using the GPS information of the terminal, it is possible to obtain the movement status and the direction of movement information by using the time zone GPS location information recorded with the video when shooting the video, through which the movement speed of the terminal can be determined. The keyframe acquisition time is automatically set according to the moving speed.

In addition, when the distance from the base station is used, the installation location of the base station is determined according to the frequency of the terminal. When the communication is moved to another neighboring base station after communication with one base station, the terminal is moved through the direction and the time traveled through it. The direction and speed of movement can be grasped.

In addition, by using a change in the size and shape of the specific object in the image and the image recording time to determine whether the movement, if the movement is determined, it is possible to determine the movement speed and direction.

More specifically, for example, in the case of telegraph poles installed on the roadside, the installation interval is determined, and the moving speed of the terminal is measured by measuring the image time during the movement from one telegraph pole to another adjacent pole, The distance change is measured by using the image time and the like through the triangulation method according to the change in the height.

The SLAM step (S30) of SLAM processing the image information for each keyframe and storing them in chronological order is to extract keyframes for each time slot, and then SLAM process the respective image information for each time slot, and store the processed information in chronological order.

At this time, when storing the information, each piece of information is given an identification code, which is given as an identification code for the place and time.

For example, identification information is given in a manner such as "Seoul-Gangnam-Tehranro-2018.01.01 12:00 23 '".

After the step S30, the data processing step S40 of deep-learning and storing the stored information after the SLAM processing for each keyframe is performed on the virtual mainframe to determine the exact position of the objects in the image by SLAM processing the continuous image information. Finding a central point to do so.

In more detail, the SLAM-processed information is arranged on the virtual mainframe to obtain information overlapped by time slots, and the overlapped portions are identified, and the center point is found through deep learning that repeats these operations.

This is because buildings, terrain, or objects that are objects in the image may be distorted depending on the image photographing position when they have a volume.

After the step S40, the processed image information correcting step S50 of correcting the distorted information by matching the processed data obtained through deep learning with external information is disposed on the virtual mainframe in the previous step S40 and deep learning. Accurate location information is obtained by matching information of the centripetal point acquired through the external information.

More specifically, through the matching of the GPS information or location information of the target through the acquired information, such as the center of gravity and external information such as maps, aerial photographs and cadastral maps to correct the distorted information that may occur when the image is taken will be.

A landmark extraction step S60 is performed to extract the landmark based on the information modified in the step S50.

In more detail, the elements whose shape and position may change according to time change are removed, and buildings with relatively little position variation or shape deformation are extracted and applied as landmarks.

At this time, the SLAM information with a high exposure frequency is extracted from the hourly image, regardless of the location of image capturing, and matched with location information such as maps, aerial photographs, and GPS, which are external information, and then set as landmarks.

In addition to the landmark setting, the classification for the identification of ground fixtures such as post boxes and trees and moving objects such as cars is also identified through the difference in exposure frequency in the same way as the landmark setting. The same applies to.

After the step S60, the virtual map generation step S70 is performed to generate the virtual map by applying the extracted landmark information.

This is to create a virtual map by placing a landmark set on the virtual two-dimensional or three-dimensional map.

In this case, in the landmark of the virtual map, the SLAM information of keyframes is stored and provided as DATA in the image processed in the step S30 of SLAM processing and storing the image information for each key frame in chronological order.

In the location information providing step (S80) of providing location information using the virtual map generated in step S70, when a user or a service is provided, an image of a corresponding region is input through a terminal, and the input image is input. After SLAM processing in real time, the landmark is extracted, and accurate location information is provided through comparison with the stored data base SLAM information for the extracted landmark.

Here, in the information providing step of providing the location information to the user (S80), the image of the corresponding region is input through the user's terminal, etc., and when the landmark is extracted after the SLAM processing in real time, a plurality of SLAM processing information is deep Run to improve accuracy.

More specifically, as in the previous process (S40), after repeating overlapping a plurality of SLAM processing information in one frame to find the centripetal point and then confirm the landmark and position information provided in the preceding steps through this And information such as a virtual map.

Through such location information is to be applied to a service for the blind or location information needs.

For example, in the case of wearable vision, a visual aid for the visually impaired, the visually impaired may prevent walking accidents by guiding the location of a desired area or a location of a terrain or ground that hinders walking.

10: main server 11: image processing unit
12: Communication unit 13: Data storage unit
20: Local server 21: Video storage unit
22: communication unit 23: data storage unit
24: Information Sharing Knit 30: Terminal
111: key frame extraction unit 112: SLAM processing unit
113: Department of Data Processing 114: Data Correction
115: landmark extraction unit 116: virtual map generation unit
211: video input unit 212: processed image uploading unit

Claims (8)

In the location information system using deep learning,
After processing and processing the image input from the outside, the image processing unit 11 for generating a virtual map, the communication unit 12 for communicating with the local server 20 and the terminal 30 and data storage for storing data A main server 10 composed of units 13; and
A local server 20 connected to the main server 10 to transmit information input for each region to the main server 10 and to store and store information for each region; And
The image processing unit 11 is composed of a terminal 30, the key frame extraction unit 111 for extracting a key frame of the raw image input through the terminal 30 or an external video device, and the key frame extraction unit ( A SLAM processing unit 112 for SLAM processing the image extracted from the key frame at 111, a data processing unit 113 for deeply arranging the SLAM processing image at the virtual main frame and placing the SLAM processing at the SLAM processing unit 112; A landmark that extracts landmarks based on the data correction unit 114 for correcting the distorted information by matching the data processed by the data processing unit 113 with external information, and a landmark based on the information corrected by the data correction unit 114. Location information system using a deep learning, characterized in that consisting of the extraction unit 115 and the virtual map generation unit 116 for generating a virtual map by applying the landmark extracted from the landmark extraction unit (115).
delete The method of claim 1,
The data processing unit 113 arranges the SLAM-processed image on the virtual mainframe, SLAM-processes the continuously photographed image by time zone, and then arranges it on the virtual mainframe by time zone to identify the overlapped portions. And, the location information system using deep learning, characterized in that to find the center point through the deep learning to repeat this operation.
The method of claim 1,
The local server 20 stores an original image input through a terminal located in a corresponding region, and uploads an image storage unit 21, a main server 10, and a terminal 30 to upload and analyze the processed image. Communication unit 22 for performing communication with, a data storage unit 23 for storing data, and an information sharing unit 24 for providing information to the terminal 30 and the main server 10. Location information system using deep learning.
The method of claim 4, wherein
The image storage unit 21 compares the processed image and the stored backup data and the input raw image information by receiving the image input unit 211 for receiving and storing the raw image input from the terminal or a separate photographing equipment processed image of the region Position information system using deep learning, characterized in that consisting of the processed image uploading unit 212 for uploading.
In the method of providing location information using deep learning,
Image information obtaining step of obtaining an image from a terminal of a user or an administrator (S10);
A keyframe extraction step (S20) of extracting keyframes for each time zone from the obtained image information;
A SLAM step (S30) of SLAM processing the keyframe image information and storing the information in chronological order;
A data processing step (S40) of deep-learning the stored information after the SLAM processing for each keyframe and placing it on the virtual mainframe (S40);
Processing image information correction step of correcting the distorted information by matching the processing data and external information obtained through deep learning (S50);
A landmark extraction step (S60) of extracting a landmark based on the modified information;
Generating a virtual map by applying the extracted landmark information (S70); And
Location information providing method using deep learning comprising a location information providing step (S80) for providing location information using a virtual map.
The method of claim 6,
In the data processing step (S40), the SLAM-processed information is arranged on the virtual mainframe to obtain information overlapped by time slots, and the overlapped portions are identified, and the center point is found through deep learning that repeats these operations. Location information providing method using deep learning.
The method of claim 7, wherein
The process of correcting the processed image information (S50) is obtained on the virtual mainframe in the previous step (S40) to obtain accurate position information by matching the information of the centripetal point obtained through deep learning and external information, Deep learning is characterized by correcting the distorted information that may occur when shooting an image by matching the GPS information or location information of the object obtained through the center point of the object and the map, aerial photograph, and cadastral information. Location information providing method.

Images