WO2020071573A1

WO2020071573A1 - Location information system using deep learning and method for providing same

Info

Publication number: WO2020071573A1
Application number: PCT/KR2018/012092
Authority: WO
Inventors: 이준혁
Original assignee: (주)한국플랫폼서비스기술
Priority date: 2018-10-05
Filing date: 2018-10-15
Publication date: 2020-04-09
Also published as: KR102033075B1

Abstract

The present invention relates to a location information system using deep learning and a method for providing same, so as to provide location information and services by deep learning on the basis of a SLAM technique. To this end, the present invention provides a location information system using deep learning, comprising: a main server (10) consisting of an image treating unit (11) for generating a virtual map after processing and treating an externally-inputted image, a communication unit (12) for communicating with a local server (20) and a terminal (30), and a data storage unit (13) for storing data; the local server (20), connected to the main server (10), for transmitting information input for each region to the main server (10) and distributing and storing regional information; and the terminal (30). In addition, the present invention provides a method comprising: an image information acquisition step (S10); a keyframe extraction step (S20); a SLAM treatment step (S30); a data processing step (S40); a processed image information modification step (S50); a landmark extraction step (S60); a virtual map generation step (S70); and a location information provision step (S80).

Description

Location information system using deep learning and its provision method

The present invention relates to a location information system using deep learning and a method for providing the same, and to provide a more accurate location information and service by deep learning based on a SLAM technique.

In general, location information is provided to users in various ways. Among them, a GPS system using a satellite navigation system is a representative one.

However, in the case of such a GPS system, based on the communication with the satellites orbiting the earth, if the communication with the satellites is not performed, the user itself is impossible, and communication with a certain number of satellites or more must be performed. The accuracy of is increased.

However, even in the case of such a GPS, the precision of the error exists within a few meters, so it is possible to grasp only the approximate location, and it is insufficient to provide location information according to pedestrians' path finding or autonomous driving.

Accordingly, various methods have been proposed, and one of them is proposed by SLAM (Simultaneous Localization And Map-Building), a technology that generates a map of the environment without external assistance with only sensors attached to the robot while the robot moves around an unknown environment. Among them, techniques for obtaining location information by analyzing this through image photographing of terrain and features have been proposed.

As a representative technology to which such SLAM technology is applied, Korean Patent Publication No. 10-2016-0003731 (wide-area recognition from SLAM maps, published on January 11, 2016) has been proposed, and SLAM (simultaneous localization and mapping) maps. Exemplary methods, apparatuses, and systems for performing wide area localization are disclosed. The mobile device may select the first keyframe based SLAM map of the local environment using one or more received images. The individual location recognition of the mobile device within the local environment may be determined, and the individual location recognition may be based on a keyframe based SLAM map. The mobile device can transmit the first keyframe to the server and receive a first global location-awareness response indicating a correction to the local map on the mobile device. The first global position recognition response may include rotation, translation, and scale information. The server proposes a technique for receiving keyframes from a mobile device and recognizing keyframes in the server map by matching keyframe features received from the mobile device to the server map features.

In addition, Republic of Korea Patent Publication No. 10-2013-0134986 (SLAM system and method of a mobile robot that receives a photo input from the user from the user, published on December 10, 2013) is a mobile robot that receives a photo input to the environment from the user A SLAM system comprising: a user terminal that receives information or commands about an environment from a user and transmits it to a mobile robot; And

It provides a mobile robot SLAM system, characterized in that it comprises a mobile robot that performs map creation by grasping the environment information and movement information obtained from the user terminal, information and commands, and a data acquisition device, and moving A data acquisition device mounted on the robot to obtain information about the surrounding environment or a means for measuring movement of the mobile robot; A second processing device that performs a function of determining the location of a mobile robot and determining a route based on real-time map creation, management and modification of the created map, and created map using the data obtained from the data acquisition device and the user terminal ; And a technology for controlling the driving of the mobile robot based on the information processed by the second processing device.

However, in the case of the prior arts, since only the video information is SLAM and applied based on it, it is vulnerable to changes in the external environment, and when the external environment changes, it cannot be applied and thus there is a limit in providing services.

The present invention is to provide a location information system and a method for providing the information to provide a location information using deep learning, to be more precise and adaptable to changes in the external environment, in order to compensate for the shortcomings of the SLAM technology as described above. The purpose.

In a location information system using deep learning for solving the problems of the present invention, an image processing unit 11 and a local server 20 and a terminal 30 that generate a virtual map after processing and processing an image input from the outside ) And a main server (10) consisting of a communication unit (12) for performing communication and a data storage unit (13) for storing data; and a main server (10) that is connected to the main server (10) and inputs information for each region. 10) a local server (20) that transmits and distributes regional information; And a terminal 30, to provide a location information system using deep learning.

At this time, the image processing unit 11 is a key frame extraction unit 111 for extracting a key frame of a raw image input through the terminal 30 or an external image device, and the key frame in the key frame extraction unit 111 SLAM processing unit 112 for SLAM processing the extracted image, a data processing unit 113 for deep-learning the SLAM-processed image by the SLAM processing unit 112 on a virtual mainframe, and a data processing unit 113 ) The data correction unit 114 that corrects distorted information by matching the processed data with external information, and the landmark extraction unit 115 that extracts landmarks based on the information corrected by the data correction unit 114. And a virtual map generation unit 116 that generates a virtual map by applying the landmark extracted from the landmark extraction unit 115, and the data processing unit 113 displays the SLAM-processed image on a virtual mainframe. Placed on the screen, but continuously photographed images by time After SLAM processing, it is arranged on a virtual main frame for each time zone to grasp the overlapped part, and a deep learning that repeatedly executes such a task finds a centripetal point.

In addition, the local server 20 stores the raw image input through the terminal located in the area, and uploads the image storage unit 21 and the main server 10 and the terminal (uploading) for comparative analysis of the processed image. 30) is composed of a communication unit 22 for communicating with, a data magnetic field unit 23 for storing data, and an information sharing unit 24 for providing information to the terminal 30 and the main server 10, The image storage unit 21 compares the image input unit 211 for receiving and storing the raw image input from the terminal or a separate photographing device, the processed and stored backup data and the input raw image information, and then processes the processed image in the corresponding region. It provides a location information system using deep learning, characterized in that it consists of a processed image upload section 212 for uploading.

A method for providing location information using deep learning, which is another means for solving the problems of the present invention, comprising: an image information acquisition step (S10) of acquiring an image from a terminal of a user or an administrator; A key frame extraction step (S20) of extracting key frames for each time zone from the obtained image information; SLAM step of storing the video information for each key frame in a chronological order (S30); A data processing step of deep-learning the stored information after SLAM processing for each key frame and placing it on a virtual main frame (S40); A processing image information correcting step (S50) of correcting distorted information by matching processing data obtained through deep learning with external information; A landmark extraction step of extracting a landmark based on the modified information (S60); A virtual map generation step of generating a virtual map by applying the extracted landmark information (S70); And a location information providing step (S80) for providing location information using a virtual map.

At this time, in the data processing step (S40), the SLAM-processed information is placed on the virtual mainframe for each time zone to identify the overlapped portion, and the deep learning that repeatedly executes these operations finds the center of gravity. Characterized in that, the processing image information modification step (S50) is arranged on the virtual mainframe in the previous step (S40) and accurate location information through matching with the information of the centripetal point obtained through deep learning and external information Acquiring, but correcting distorted information that may occur during video shooting by matching the GPS information or the location information of the object obtained through the information of the center of the obtained object and the information of the map, aerial photograph, and cadastral map that is external information By providing a method for providing location information using deep learning to be able to better solve the problem of the present invention.

By providing the location information system using the deep learning of the present invention and a method for providing the location information, there is an effect capable of providing highly reliable location information that is less affected by external environmental changes.

1 is a configuration diagram of a location information system using the deep learning of the present invention.

2 is a configuration diagram of the main server 10 according to the present invention.

3 is a block diagram of a local server 20 according to the present invention.

4 is a flowchart of a method for providing location information using deep learning according to the present invention.

Hereinafter, with reference to the accompanying drawings, it will be described in detail so that those skilled in the art can easily implement the location information system using the deep learning of the present invention and a method for providing the same.

1 is a configuration diagram of a location information system using deep learning of the present invention, FIG. 2 is a configuration diagram of a main server 10 according to the present invention, and FIG. 3 is a configuration diagram of a local server 20 according to the present invention. .

When described in detail with reference to FIGS. 1 to 3, the location information system using deep learning of the present invention is connected to the main server 10 and the main server 10, and inputs information for each region to the main server 10 It is composed of a local server 20 and a terminal 30 that transmits to and distributes regional information.

The main server 10 processes and processes the image input from the outside, and then generates a virtual map, and then an image processing unit 11 and a communication unit 12 that communicates with the local server 20 and the terminal 30. And a data storage unit 13 for storing data.

Here, the image processing unit 11 has a key frame extracted from the key frame extraction unit 111 and the key frame extraction unit 111 to extract the key frame of the raw image input through the terminal 30 or an external image device The SLAM processing unit 112 for SLAM processing the extracted image and the data processing unit 113 and data processing unit 113 for deep-learning the SLAM-processed image on the virtual mainframe by the SLAM processing unit 112. The data correction unit 114 that corrects distorted information by matching the old data with the external information, and the landmark extraction unit 115 and the landmark extraction unit that extract landmarks based on the information corrected by the data correction unit 114 It is composed of a virtual map generating unit 116 that generates a virtual map by applying the landmark extracted from the unit 115.

The key frame extraction unit 111 extracts a key frame from the input image information of the original image, and extracts a key frame for each time zone from the continuously photographed image to correct it when the position of the terminal changes.

This extracts key frames to reduce errors due to processing and processing of continuous images during post-production such as SLAM processing, data processing through deep learning, data modification through matching with external information, and landmark extraction. It is intended to proceed with the latter work.

The SLAM processing unit 112 SLAM-processes the inputted image information based on the key frame extracted by the key frame extraction unit 111.

Here, SLAM (Simultaneous Localization And Map-Building) is a technique for recognizing terrain or features by binarizing or coding the recognized image, and detailed description thereof will be omitted as a known technique.

The data processing unit 113 arranges a SLAM-processed image on a virtual mainframe. In more detail, SLAM is processed on a continuous-shot image by time slot, and then it is arranged on a virtual main frame by time slot. In this way, the overlapping part is grasped, and the center of gravity is found through deep learning that repeatedly executes these tasks.

This is to correct this because, in the case of a continuously photographed image, a photographing position for each frame is different or a volume of an object exists, distortion may occur according to the photographing location.

The data correction unit 114 is disposed on the virtual mainframe through the data processing unit 113 and the exact location of the target terrain or feature through matching with the information of the centripetal point obtained through deep learning and external information. Information will be grasped.

At this time, as the external information, information of one of GPS information or location information of an object obtained through information such as a map, an aerial photograph, and a cadastral map is used.

The landmark extracting unit 115 extracts a landmark based on the modified information through the data correction unit 114, and more specifically, an element whose shape and location may change according to a change in time. And removes buildings with relatively little change in position or shape, and applies them as landmarks.

The virtual map generation unit 116 generates a virtual map by placing the landmark extracted through the landmark extraction unit 115 on the map.

The communication unit 12 is to perform communication with the local server 20 provided for each region and the terminal 30 for service use.

At this time, an external Internet network is used as a communication method, or a separate dedicated inlet is used.

The data storage unit 13 temporarily stores the raw image input from the terminal 30 or the like, and temporarily stores the SLAM image for each time zone in which the input raw image is processed and the processed image through post-production.

At this time, in the case of data that has been completed for work and service, it is transferred to the local server 20 provided in the corresponding area to store data in each local server 20 to reduce the load that may occur on the main server 10. .

Here, when a user requests a service through the terminal 30, if there is backup data in the corresponding area, it is called and mounted on the data storage unit 13 of the main server 10 to store and store the service.

The local server 20 stores a raw image input through a terminal located in a corresponding region, and uploads an image storage unit 21 and a main server 10 and a terminal 30 to upload processed images for comparative analysis. It consists of a communication unit 22 for communicating with, a data magnetic field unit 23 for storing data, and an information sharing unit 24 for providing information to the terminal 30 and the main server 10.

At this time, the image storage unit 21 compares the image input unit 211 for receiving and storing the raw image input from the terminal or a separate photographing device, the processed and stored backup data and the input raw image information, and It is composed of a processed image uploading unit 212 for uploading a processed image.

This compares the inputted raw image with the processed image in the corresponding region when the user inputs the raw image in the corresponding region through the terminal 30 and requests service, and uploads it if the processed image is stored as backup data. To provide services.

In the case of the terminal 30, any communication means capable of a photographing function possessed by an administrator or a user can be used.

The terminal 30 can perform communication with the main server 10 or the local server 20 through the Internet, which is an external computer network, and download and install and use a separate APP or management program for service provision. However, this is not a limitation.

With the above configuration, it is possible to complete the location information system using the deep learning of the present invention.

Another invention according to the present invention will be described in detail below with reference to FIG. 4 which is a flowchart of a method for providing location information using deep learning.

Referring to FIG. 4 in detail, the method for providing location information using deep learning according to the present invention includes an image information acquisition step (S10) of acquiring an image from a terminal of a user or an administrator and time from the image information acquired as described above. A keyframe extraction step (S20) of extracting keyframes for each unit, and a SLAM step (S30) of processing the SLAM processing of the video information for each keyframe in chronological order, and deep learning of the stored information after SLAM processing for each keyframe to simulate a virtual mainframe Data processing step (S40) placed on the image and processing image information correction step (S50) to correct distorted information by matching the processing data obtained through deep learning with external information, and landmarks based on the modified information Landmark extraction step (S60) to extract, virtual map generation step (S70) to generate a virtual map by applying the extracted landmark information, and location using the virtual map It consists of a location information providing step (S80) for providing information.

Here, the image information acquiring step (S10) refers to an image captured by a user or an administrator's terminal or an image captured while moving in the field through a separate imaging device.

The obtained image is input to the local server 20 located in each region, so that it can be transmitted by accessing an APP installed in the terminal or an open management program.

This is to allow a user who wants to provide a service to request a service after acquiring and transmitting video information in a desired area because a problem may arise in which an administrator may have difficulty obtaining information in all areas in order to provide a service.

In the key frame extraction step (S20) of extracting the key frame for each time zone from the obtained image information, the key frame may be changed for each time zone because the key frame may fluctuate when the position of the terminal changes when the image is continuously photographed through the terminal. Will be extracted.

Here, as a method of extracting key frames for each time zone, a triangulation method using a GPS information of a terminal that acquires an image at a predetermined time interval, a distance from a base station or a size change of a specific object in an image, etc. This can be applied.

For example, when using the GPS information of the terminal, it is possible to obtain the movement status and the movement direction information by using the GPS location information for each time zone recorded with the image when shooting a video, through which the movement speed of the terminal can be grasped. The keyframe acquisition time according to the moving speed is automatically set.

In addition, when using a distance from the base station, the installation location of the base station is determined according to the frequency of the terminal. It is possible to grasp the moving direction and moving speed.

In addition, it is possible to grasp whether the movement is performed by using a change in the size and shape of a specific object in the image and the image shooting time. If the movement is determined, the moving speed and direction can be grasped.

More specifically, for example, in the case of a telegraph pole installed on the roadside, an installation interval is determined, and the moving speed of the terminal is measured by measuring the video time during movement from one telephone pole to another adjacent telephone pole, or a telegraph pole. Through the triangulation method according to the change in height, the distance change is measured using image time.

In the SLAM step (S30) in which the video information for each keyframe is SLAM-processed and stored in chronological order, after extracting keyframes for each time-slice, the video information for each time-slice is SLAM-processed and the processed information is stored in chronological order.

At this time, when storing information, an identification code is assigned to each information, and an identification code is given to a place and time.

For example, "Seoul-Gangnam-Tehran-ro-2018.01.01 12:00 23 '" is provided with identification information.

After the step (S30), the data processing step (S40) of performing SLAM processing for each key frame and deep-learning the stored information to be disposed on the virtual main frame is performed by SLAM processing the continuous image information to grasp the exact position of objects in the image. It is to find a centripetal point to do.

In more detail, the SLAM-processed information is placed on the virtual mainframe for each time zone to identify the overlapped portion, and the center of gravity is found through deep learning that repeatedly executes these tasks.

This is because if the object and the building or terrain in the image have a volume, it may be distorted depending on the location of the image.

After the step (S40), the processing image information correction step (S50) of correcting distorted information by matching the external data with the processed data obtained through deep learning is arranged on the virtual mainframe in the previous step (S40) and deep learning It is to obtain accurate location information by matching the information of the centripetal point obtained through and the external information.

In more detail, the distorted information that may occur during video shooting is corrected by matching the GPS information or location information of the object through information such as the center of the obtained object and information such as external information such as map, aerial photograph, and cadastral map. will be.

The landmark extraction step (S60) of extracting the landmark based on the information modified in the step (S50) is performed.

In more detail, elements that may change in shape and position according to changes in time are removed, and buildings with relatively little change in position or shape are extracted and applied as landmarks.

At this time, SLAM information with a high exposure frequency is extracted from the hourly image regardless of the location of the video shooting, and it is matched with location information such as external information such as map, aerial photograph and GPS, and then set as a landmark through this.

In addition, with the landmark setting, the distinction for identification between ground fixtures such as mailboxes and trees and moving objects such as automobiles is also identified through the difference in exposure frequency in the same way as the landmark setting. It also applies to.

After the step S60, a virtual map generation step S70 of generating a virtual map by applying the extracted landmark information is performed.

This is to create a virtual map by arranging landmarks set on a virtual 2D or 3D map.

At this time, in the landmark of the virtual map, SLAM information for each timeframe of the target object of the landmark is stored and provided as DATA in the image processed in step S30 in which the image information for each keyframe is SLAM processed and stored in chronological order.

In the step S80 of providing location information using the virtual map generated in the step S70, when a user or a service is provided, an image of a corresponding region is input through a terminal or the like, and the input image is received. The landmark is extracted after SLAM processing in real time, and accurate location information is provided through comparison with the database in which SLAM information for each time zone for the extracted landmark is stored.

Here, in the information providing step of providing location information to the user (S80), the image of the corresponding region is input through the user's terminal, etc., and when SLAM processing is performed in real time to extract the mark, a plurality of SLAM processing information is deeply diced. Make it possible to improve accuracy by running.

More specifically, as in the previous step (S40), after repeatedly overlapping a plurality of SLAM processing information in one frame, finding a centripetal point, confirming a landmark, and then providing location information through the tacking steps. And, accordingly, information such as a virtual map.

Through this location information, it is intended to be applied to services that require visually impaired or location information.

For example, in the case of the wearable vision, which is an auxiliary tool for the visually impaired, the visually impaired person can prevent a walking accident by guiding the route of a desired area or the location of a terrain or ground that interferes with walking when moving simply.

10: Main server 11: Image processing unit

12: Communication unit 13: Data storage unit

20: local server 21: video storage unit

22: communication unit 23: data storage unit

24: information sharing share 30: terminal

111: key frame extraction unit 112: SLAM processing unit

113: data processing unit 114: data correction

115: landmark extraction unit 116: virtual map generation unit

211: image input unit 212: processed image uploading unit

Claims

In the location information system using deep learning,

After processing and processing the image input from the outside, the image processing unit 11 that generates a virtual map and the communication unit 12 that communicates with the local server 20 and the terminal 30 and data storage that stores data Main server (10) consisting of a unit (13); And

A local server 20 that is connected to the main server 10 and transmits information input by region to the main server 10, and distributes and stores regional information; And

Location information system using deep learning, characterized in that consisting of a terminal (30).
According to claim 1,

The image processing unit 11 includes a key frame extraction unit 111 for extracting a key frame of a raw image input through the terminal 30 or an external image device, and a key frame extracted by the key frame extraction unit 111 The SLAM processing unit 112 for SLAM processing the image, and the data processing unit 113 and the data processing unit 113 for deep-learning the SLAM-processed image on the virtual mainframe. A data correction unit 114 for correcting distorted information by matching the processed data with external information, and a landmark extraction unit 115 and a land for extracting landmarks based on the information corrected by the data correction unit 114 A location information system using deep learning, characterized by comprising a virtual map generator (116) that generates a virtual map by applying the landmark extracted from the mark extractor (115).
According to claim 2,

The data processing unit 113 arranges the SLAM-processed image on a virtual mainframe, but after performing SLAM processing of continuously photographed images by time zone, it is arranged on a virtual main frame for each time zone to identify the overlapped portion. The location information system using deep learning is characterized by finding a centripetal point through deep learning that repeatedly performs such operations.
According to claim 1,

The local server 20 stores a raw image input through a terminal located in a corresponding region, and uploads an image storage unit 21 and a main server 10 and a terminal 30 to upload processed images for comparative analysis. Characterized in that it consists of a communication unit (22) for performing communication with, a data magnetic field unit (23) for storing data, and an information sharing unit (24) for providing information to the terminal (30) and the main server (10). Location information system using deep learning.
The method of claim 4,

The image storage unit 21 compares the image input unit 211 for receiving and storing the raw image input from the terminal or a separate photographing device, the processed and stored backup data and the input raw image information, and then processes the processed image in the corresponding region. A location information system using deep learning, characterized in that it consists of a processed image uploading unit 212 for uploading.
In the method of providing location information using deep learning,

An image information acquisition step (S10) of acquiring an image from a terminal of a user or an administrator;

A key frame extraction step (S20) of extracting key frames for each time zone from the obtained image information;

SLAM step of storing the video information for each key frame in a chronological order (S30);

A data processing step of deep-learning the stored information after SLAM processing for each key frame and placing it on a virtual main frame (S40);

A processing image information correcting step (S50) of correcting distorted information by matching processing data obtained through deep learning with external information;

A landmark extraction step of extracting a landmark based on the modified information (S60);

A virtual map generation step of generating a virtual map by applying the extracted landmark information (S70); And

A method of providing location information using deep learning, characterized by comprising a location information providing step (S80) of providing location information using a virtual map.
The method of claim 6,

In the data processing step (S40), the SLAM-processed information is placed on the virtual mainframe for each time zone to identify the overlapped portion, and it is characterized by finding a centripetal point through deep learning that repeatedly performs such operations. Location information providing method using deep learning.
The method of claim 7,

The processing image information modification step (S50) is arranged on the virtual mainframe in the previous step (S40) to obtain accurate location information through matching with the information of the centripetal point obtained through deep learning and external information. Deep learning, characterized by correcting distorted information that may occur during video shooting, by matching the GPS information or location information of an object obtained through the information of the centripetal point of the object and external information such as map, aerial photograph, and cadastral map Method of providing location information.