KR20210075619A - Autonomous robot, location estimation server of autonomous robot and location estimation or autonomous robot using the same - Google Patents

Abstract

When a location estimation server receives a plurality of received key frames generated based on an image of an autonomous driving robot, a plurality of first image feature points and a first feature descriptor for the first image feature points are extracted from each of the received key frames. The location estimation server calculates a vector of storage key frames based on visual vocabulary for each of the plurality of storage key frames stored in advance, arranges a storage key frame in which the second image feature points of the storage key frames constituting the calculated vector coincides with the first image feature points, and create it as a key frame index. In addition, the location estimation server calculates weights for the second feature descriptors of the storage key frames included in the key frame index, and selects a comparison candidate key frame from among the plurality of storage key frames. The present invention can estimate the location of a place where a robot is driving without prior information.

Description

Autonomous robot, location estimation server of autonomous driving robot, and method for estimating location of autonomous driving robot using same {Autonomous robot, location estimation server of autonomous robot and location estimation or autonomous robot using the same}

The present invention relates to an autonomous driving robot, a position estimation server for an autonomous driving robot, and a method for estimating a position of an autonomous driving robot using the same.

Visual place recognition in autonomous driving robots (hereinafter referred to as 'robots') is a technology that uses images collected through a camera equipped with a robot to determine whether the current place is a previously visited place.

In SLAM (Simulation Language for Alternative Modeling), to perform loop joint detection that reduces the accumulated error of robot position, visual place recognition provides important information. Visual place recognition is being applied and applied in various fields such as global location estimation and augmented reality. Global position estimation is a technology for estimating the position of the robot by recognizing the surrounding environment of the robot without initial position information of the robot.

When the robot moves through space, the server linked with the robot creates a map of the space visited by the robot using a visual place recognition technique. By providing the map generated by the server to the robot, the robot can check whether the current location is a previously visited space. However, when the robot travels in an infrequently visited space, the robot has difficulty in matching the space it is currently seeing with the space it has visited before.

In addition, strong spatial awareness is required even in a space, such as a hotel, where space relocation or lighting changes are frequent. And the robot should be able to determine its location without prior information about its location. And even if an accident occurs that the robot is lifted and moved to an arbitrary place by a physical shock or external force in a space used by an unspecified number of outsiders (for example, a hotel, etc.), the robot recognizes the space it is located in. You need skills to do it.

When a conventional robot performs global location estimation in the current frame to recognize space, it estimates its location by referring to a keyframe that is most similar to the scene the robot is currently viewing. To find the most similar keyframe, the robot requires a lot of computation because it compares all keyframes with the current frame. In addition, as the robot moves, the size of the map increases and the number of keyframes continues to increase, so a method is needed to solve this problem.

Accordingly, the present invention provides a technology capable of estimating the location of a place where a robot is traveling without prior information through an autonomous driving robot that runs on various floors of a building such as a hotel and a server that estimates the location of the robot.

As a server that provides a comparison target frame so that the robot can estimate a position in conjunction with an autonomous driving robot, which is one feature of the present invention for achieving the technical problem of the present invention,

receiving a plurality of keyframes selected from a specific image from the autonomous driving robot, and extracting a plurality of first image feature points and first feature descriptors for the plurality of first image feature points from the plurality of keyframes A keyframe processing module, a vector calculation module for calculating a keyframe vector of each of the plurality of keyframes, and generating a keyframe index vector including second image feature points of a plurality of stored keyframes stored in advance, the second image A keyframe index vector generation module that aligns keyframes including first feature descriptors corresponding to feature points to the keyframe index vector to generate a keyframe index, and calculates a weight of the keyframe index, and a comparison candidate keyframe selection module configured to select a plurality of comparison candidate keyframes for recognizing a position by comparing the autonomous driving robot with the plurality of keyframes from among a plurality of stored keyframes.

A keyframe storage module for storing a plurality of keyframes, first image feature points, and first feature descriptors processed by the keyframe processing module, and second image feature points and second feature descriptors for the stored keyframes may further include.

The keyframe processing module is configured to designate a mask area of each of a plurality of image feature points extracted from a first keyframe among the plurality of keyframes, and other than the mask area in a second keyframe consecutive to the first keyframe. A plurality of additional image feature points may be extracted from the region of .

The vector calculation module may obtain a visual vocabulary from each of the plurality of keyframes, and calculate a vector of received keyframes using a Bag of Words (BoW) method based on the acquired visual vocabulary.

The keyframe index vector generation module may generate the keyframe index by applying a term frequency-inverse document frequency (tf-idf) to the first image feature points.

As an autonomous driving robot that interworks with a location estimation server that is another feature of the present invention for achieving the technical problem of the present invention,

A camera that captures an image, a keyframe generation module that selects a plurality of keyframes for estimating the position of the robot from among a plurality of frames generated from the captured image, and transmits the plurality of keyframes to the position estimation server and an interface for receiving a plurality of comparison candidate keyframes selected based on the plurality of keyframes from the location estimation server, and comparing the plurality of keyframes with the plurality of comparison candidate keyframes, so that It includes a keyframe position recognition module for recognizing the position.

The image processing unit may further include an image processing unit generating the image captured by the camera as a plurality of images, and generating the plurality of frames for each of the plurality of images.

The keyframe location recognition module may further include a keyframe map generator configured to generate a keyframe map by using the plurality of keyframes used for location recognition when the location of the autonomous driving robot is recognized by the keyframe location recognition module.

As another feature of the present invention for achieving the technical problem of the present invention, a method for estimating a position of an autonomous driving robot by a position estimating server,

receiving a plurality of received keyframes generated based on an image in the autonomous driving robot; extracting a plurality of first image feature points and a first feature descriptor for the first image feature points from each of the received keyframes calculating a vector of the storage keyframes based on visual vocabulary for each of the stored plurality of storage keyframes, wherein a second image feature point of the storage keyframes constituting the calculated vector is the first image Sorting the storage keyframes matching the keyframes to generate a keyframe index, and calculating weights for the second characteristic descriptors of the storage keyframes included in the keyframe index, from among the plurality of storage keyframes. and selecting a comparison candidate keyframe.

The extracting of the first feature descriptor may include generating a feature descriptor space based on the first image feature points and the first feature descriptor, and clustering the generated feature descriptor space to obtain the first image feature points. It may include generating the first visual vocabulary.

The extracting of the first feature descriptor may include extracting a first keyframe image feature point from a first keyframe among the received keyframes, and a second keyframe image from a second keyframe adjacent to the first keyframe. extracting feature points; if the number of extracted feature points of the second keyframe image is less than a preset threshold value, designating a mask area using the feature points of the second keyframe image as a reference point; and in the second keyframe The method may include extracting an additional second keyframe image feature point in a region other than the mask region.

The calculating of the vector of the stored keyframes may include calculating a Bow vector based on a second visual vocabulary corresponding to the stored keyframes.

Another feature of the present invention for achieving the technical problem of the present invention is a method for estimating the position of an autonomous driving robot linked with a position estimation server,

Generating a plurality of images from the captured image, generating frames for the plurality of generated images, selecting a plurality of keyframes for estimating the position of the robot from among the generated frames, the selected transmitting a plurality of keyframes to the location estimation server, receiving a comparison candidate keyframe selected based on the plurality of keyframes from the location estimation server, and the received comparison candidate keyframe and the selected plurality of keys and estimating a position using the frames.

After estimating the location, the method may include generating a keyframe map using the plurality of keyframes.

According to the present invention, by classifying keyframes using the Bag of Words (BoW) technique, it is possible to improve the speed and precision of recognizing the visual space of the robot using the keyframes.

1 is an exemplary diagram of an environment to which a location estimation server according to an embodiment of the present invention is applied.
2 is a structural diagram of a robot according to an embodiment of the present invention.
3 is a structural diagram of a location estimation server according to an embodiment of the present invention.
4 is a flowchart of a location estimation method according to an embodiment of the present invention.
5 is an exemplary diagram of visual vocabulary generation according to an embodiment of the present invention.
6 is an exemplary diagram of a Bow vector calculated according to an embodiment of the present invention.
7 is an exemplary diagram of a keyframe index vector according to an embodiment of the present invention.
8 is a flowchart of a method for extracting image feature points according to an embodiment of the present invention.
9 is an exemplary diagram in which image feature points are extracted from a keyframe according to an embodiment of the present invention.
10 is an exemplary diagram of a visual vocabulary according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

Hereinafter, an autonomous driving robot, a position estimation server for an autonomous driving robot, and a method for estimating a position of an autonomous driving robot using the same according to an embodiment of the present invention will be described in detail with reference to the drawings.

1 is an exemplary diagram of an environment to which a location estimation server according to an embodiment of the present invention is applied.

As shown in FIG. 1 , an autonomous driving robot (hereinafter, referred to as a 'robot') 100 installed in a space (eg, a hotel, etc.) used by a large number of unspecified outsiders works with the location estimation server 200 . . In addition, the robot 100 and the position estimation server 200 interwork with the elevator control server 300 .

The robot 100 collects a photographed image by photographing a place in a direction in which the robot 100 travels by using a camera mounted in an internal or external form. The robot 100 generates a plurality of images by image processing the captured image.

The robot 100 generates each keyframe from a plurality of images. Here, the key frame means a frame selected to be important for recognizing the position of the robot 100 in every frame of the image captured by the robot 100 .

For example, when the robot 100 collects images while walking around the hotel lobby as shown in FIG. 1 , the location of the robot 100 is recognized by the border of the room door, the passageway leading to the elevator, and the room number sign. can be an important factor for Therefore, if the frame or the passage of the room door, the room number sign, etc. appear in the frame collected by the robot 100, the corresponding frame is generated as a key frame.

The robot 100 may generate one or more keyframes from one captured image. Since a method of generating a keyframe from an image can be performed in various ways, the embodiment of the present invention is not limited to any one method. Also, the number of keyframes generated from one image is not limited.

In addition, in the embodiment of the present invention, for convenience of description, the generation of the keyframe in the robot 100 is described as an example, but the location estimation server 200 may generate the keyframe. In this case, the robot 100 only needs to transmit the captured image captured by the camera to the location estimation server 200 .

Also, the robot 100 generates a keyframe map using a plurality of keyframes. At this time, the robot 100 generates a keyframe map for each layer, and a method of generating a keyframe map using the keyframes will be described later. In the embodiment of the present invention, the robot 100 generates a keyframe map as an example, but the location estimation server 200 may generate it.

The robot 100 receives a plurality of comparison candidate keyframes from the position estimation server 200 , and recognizes a position of the robot 100 itself by using the received plurality of comparison candidate keyframes.

The location estimation server 200 receives the floor information of the building from which the robot 100 alights from the elevator server 300 . When the location estimation server 200 receives the building floor information, it reads a pre-stored keyframe map of the corresponding building floor. Here, the keyframe map is divided into a global map including only components that rarely change, such as building pillars, corridors, and doors, and a local map displaying obstacles such as flowerpots and movable structures.

The process of the robot 100 updating the map in real time and recognizing its location is all performed on the local map. Information on the local map is configured in the robot 100 through image features, and the robot 100 recognizes its own position through comparison between keyframes.

At this time, the changed obstacle position is written on the local map. When the obstacle position is repeatedly recognized at the same position, the robot 100 updates the position of the obstacle on the global map. That is, highly variable obstacles and precise location recognition are performed on the local map. If the location of the obstacles included in the local map is recognized repeatedly more than a preset number of times, the location of the obstacle is reflected on the global map and then the global map is constructed. do.

For example, when a driving command is received from the hotel lobby to the front door, the robot 100 recognizes its current location in the local map and then performs driving. When a move command is executed from the hotel lobby to a room on another floor (hereinafter referred to as a 'destination room'), the destination room does not exist on the local map.

Accordingly, the robot 100 searches for a destination cabin on the global map and drives. That is, if the global map has spatial information of the entire hotel, it can be said that the local map has detailed area information of the space.

The position estimation server 200 receives a plurality of keyframes from the robot 100 . Also, the location estimation server 200 may receive a keyframe map from the robot 100 . If a plurality of keyframes are received from the robot 1000, the location estimation server 200 generates a keyframe map using the plurality of keyframes.

The location estimation server 200 calculates a Bow vector using a Bow (Bag of Words) technique for each of the keyframes used to generate the keyframe map. The Bow technique is one of the methods for automatically classifying documents, and refers to a technique of determining what kind of document this document is by looking at the distribution of words included in the text.

In the field of image processing or computer vision, the Bow technique is mainly used for the purpose of classifying or searching images, and has recently been used for recognizing objects or scenes. The Bow method for image classification will be described later.

The localization server 200 uses a visual vocabulary to calculate a Bow vector in a keyframe. In this case, the visual vocabulary is generated using Oriented FAST and Rotated BRIEF (ORB) feature points, which will be described in detail later.

The position estimation server 200 generates a keyframe index vector by using the calculated Bow vector. The position estimation server 200 selects a plurality of key frame candidates using the key frame index, and provides the selected plurality of key frame candidates to the robot 100 .

When the robot 100 gets on or off the elevator, the elevator control server 300 transmits the floor information of the floor where the robot alights based on the open/close state of the elevator door to the location estimation server 200 and the robot 100, respectively. do.

The structure of each of the robot 100 and the position estimation server 200 described above will be described with reference to FIGS. 2 and 3 .

2 is a structural diagram of a robot according to an embodiment of the present invention, and FIG. 3 is a structural diagram of a location estimation server according to an embodiment of the present invention.

As shown in FIG. 2 , the robot 100 includes a camera 110 , an image processing module 120 , a keyframe generation module 130 , an interface 140 , a keyframe location recognition module 150 , and a keyframe map generation a module 160 , and a robot driving module 170 . The robot 100 may include components other than those shown in FIG. 2 .

The camera 110 is built into the robot 100 or is attached to the outside of the robot 100, and takes a place (background, object, etc.) located in the moving direction of the robot 100 as an image.

The image processing module 120 generates an image captured by the camera 110 as a plurality of images, and generates a frame for each image. Since the image processing module 120 generates an image as an image or a method for generating a frame for the image in various ways, the embodiment of the present invention is not limited to any one method.

The keyframe generation module 130 selects a plurality of frames from among the plurality of frames generated by the image processing module 120 as keyframes. The key frame means an important frame for estimating the position of the robot 100 among frames generated based on an image.

It is already known through various methods that the keyframe generation module 130 selects a plurality of keyframes from among a plurality of frames. Therefore, in the embodiment of the present invention, a detailed description of the keyframe selection method will be omitted.

The interface 140 transmits the plurality of keyframes generated by the keyframe generation module 130 to the location estimation server 200 . Also, the interface 140 may transmit the keyframe map generated by the keyframe map generation module 160 to the location estimation server 200 . In addition, the interface 140 receives at least one comparison candidate keyframe selected from a plurality of keyframes from the location estimation server 200 .

The keyframe location recognition module 150 compares at least one comparison candidate keyframes received from the location estimation server 200 with the keyframes generated by the keyframe generation module 130, Recognize the location.

Conventionally, since the position of the robot 100 was recognized by comparing all the keyframes generated by the robot 100 with the generated keyframes, the position recognition operation took a long time. However, in the embodiment of the present invention, since the keyframe location recognition module 150 compares the generated keyframes with a small number of comparison candidate keyframes received from the location estimation server 200, the location recognition operation time is reduced. It works.

The keyframe map generation module 160, when the keyframe location recognition module 150 recognizes the location of the robot 100, uses keyframes used for location recognition to provide a keyframe map of the building in which the robot 100 is located. create The keyframe map generation module 160 stores the generated keyframe map and transmits it to the location estimation server 200 through the interface 140 .

The keyframe map generation module 160 will be described as an example of calling the local map stored in the robot 100 in response to the floor information on which the robot is located immediately after the robot 100 gets off the elevator. At this time, the floor information from which the robot 100 got off is received from the elevator control server 300 .

That is, the robot 100 recognizes floor information for the destination room based on the location (lake) of the destination room for room service. When the robot 100 moves in front of the elevator to board the elevator, the position estimation server 200 recognizes the elevator boarding wait of the robot 100 and commands the elevator control server 300 to call the elevator.

The elevator control server 300 is connected to the location estimation server 200 of the robot 100 and transmits elevator call, arrival, elevator door opening and closing, and floor information to the robot 100 . When the elevator called by the robot 100 arrives and the door is opened, the robot 100 proceeds to board the elevator.

At this time, in order to prevent a collision between the robot 100 and the person, the camera 110 recognizes the person through the image captured and then boards the vehicle. When the elevator arrives at the floor of the destination room, the robot 100 receives the corresponding floor information from the location estimation server 200 connected to the elevator control server 300 .

The robot 100 converts the corresponding floor information received from the location estimation server 200 into map information of the corresponding floor. Since there are various methods for the robot 100 to convert the map information using the floor information, the embodiment of the present invention is not limited to any one method.

Since the method for generating the keyframe map by the keyframe map generation module 160 may be performed in various ways (eg, Simultaneous Localization and Mapping (SLAM), etc.), a detailed description will be omitted in the embodiment of the present invention. Also, in the embodiment of the present invention, the robot 100 generates a keyframe map as an example, but the location estimation server 200 may also generate a keyframe map.

The robot driving module 170 drives the robot 100 to run. The robot driving module 170 includes various components such as a control module for controlling the driving of the robot 100, a wheel installed under the robot 100, and a driving module for enabling the robot 100 to autonomously drive, such as an engine can do. Since the function of the robot driving module 170 can be set in various ways, a detailed description will be omitted in the embodiment of the present invention.

On the other hand, as shown in FIG. 3, the position estimation server 200 includes a keyframe processing module 210, a keyframe storage module 220, a vector calculation module 230, a keyframe index vector generation module 240, and a comparison candidate keyframe selection module 250 .

When the keyframe processing module 210 receives a plurality of keyframes from the robot 200, it processes the received keyframes (hereinafter, referred to as 'received keyframes' for convenience of description) to generate image feature points and feature descriptors. extract each.

Here, the image feature point refers to a point that can be easily identified in a frame even if the shape, size, or position of an object changes, and that the corresponding point can be easily found in the frame even if the camera's viewpoint or lighting changes. And the feature descriptor describes the regional characteristics of image feature points, and enables comparison between image feature points.

The keyframe processing module 210 extracts image feature points from the received keyframe using a Feature from Accelerated Segment Test (FAST) method. When the keyframe processing module 210 extracts image feature points, if more than a preset number of feature points are not extracted, the keyframe processing module 210 designates a mask area based on the already extracted image feature points. And the keyframe processing module 210 additionally extracts image feature points from areas other than the mask area.

And, when extracting the image feature point, the keyframe processing module 210 extracts the image feature point of the next keyframe by using an optical flow between the previous keyframe and the next keyframe. Light flow is an algorithm that predicts the movement of each pixel, and compares the previous frame with the current frame to determine the direction and size of the pixel movement. A method for the keyframe processing module 210 to extract image feature points by using a light flow is already known, so a detailed description will be omitted in the embodiment of the present invention.

The keyframe processing module 210 extracts a feature descriptor from a received keyframe, and the feature descriptor includes a Scale Invariant Feature Transform (SIFT), a Histogram of Oriented Gradient (HOG), and a binary descriptor (BRIEF, ORB, BRISK). . In the embodiment of the present invention, the use of the BRIEF descriptor representing the descriptor in a binary column for quick comparison will be described as an example.

Here, the FAST method used by the keyframe processing module 210 to extract image feature points and the BRIEF descriptor used to extract the feature descriptor are collectively referred to as Oriented FAST and Rotated BRIEF (ORB). The FAST method and the BRIEF descriptor are already known, and detailed description will be omitted in the embodiment of the present invention.

The keyframe processing module 210 receives the keyframe map managed for each layer in the keyframe storage module 220 . The keyframe map is used when the comparison candidate keyframe selection module 250 selects the comparison candidate keyframes to be compared in order to recognize the position of the robot 100 .

The keyframe storage module 220 maps and stores the received keyframe received by the keyframe processing module 210 with the image feature point and the feature descriptor extracted from the keyframe processing module 210 . In the exemplary embodiment of the present invention, storing all keyframes in the form of a binary file is described as an example, but the present invention is not limited thereto.

In addition, the keyframe storage module 220 generates a keyframe map of the building in which the robot 100 is located by using the keyframes received from the robot 100 for each building floor. The keyframe storage module 220 stores the generated keyframe map and provides it to the keyframe processing module 210 .

The keyframe storage module 220 will be described as an example of generating a keyframe map immediately after the robot 100 gets off the elevator. At this time, in order to store the floor information from which the robot 100 got off corresponding to the keyframe map, the floor information signal is received from the elevator control server 300 .

In addition, in the embodiment of the present invention, the robot 100 may also generate a keyframe map. In this case, the keyframe storage module 220 only receives the keyframe map generated by the robot 100 together with floor information, and stores and manages them for each floor.

The vector calculation module 230 calculates a keyframe vector of each of the received keyframes received by the keyframe processing module 210 . In the embodiment of the present invention, the Bow method is used to calculate a keyframe vector from received keyframes as an example.

The Bow method is one of the methods for automatically classifying documents. It is a method used to determine what kind of document a document is based on the distribution of words included in the text. In an embodiment of the present invention, the vector calculation module 230 uses the Bow method to classify received keyframes.

In order for the vector calculation module 230 to calculate the vector of the received keyframes, a visual vocabulary of each of the received keyframes is required. To obtain a visual vocabulary, the vector calculation module 230 checks the image feature points and feature descriptors extracted by the keyframe processing module 210 for the received keyframes.

Here, the visual vocabulary is a result of the keyframe processing module 210 extracting a feature point and a feature descriptor from the keyframe, and learning the feature point using the extracted feature descriptor. Examples of visual vocabulary will be described in detail later.

The vector calculation module 230 generates a feature descriptor group of the received keyframes based on the image feature point and feature descriptor of the stored keyframes and the image feature point and the feature descriptor of the received keyframes. Here, the feature descriptor group is generated by clustering feature descriptors with similar directions by the vector calculation module 230 .

The vector calculation module 230 generates a plurality of visual vocabulary by clustering the generated feature descriptor group. Here, although the vector calculation module 230 performs K-means clustering when clustering the feature descriptor group as an example, the description is not necessarily limited thereto.

The keyframe index vector generation module 240 aligns the received keyframes having the image feature points extracted by the keyframe processing module 210 to generate the keyframe index vector. The keyframe index vector is a weight for assigning scores to keyframes, and the same method as document classification, tf-idf (term frequency-inverse document frequency), is applied to the visual vocabulary.

tf-idf is a weight used for information retrieval and search engines, and refers to a number indicating how important a word is within a specific document when there is a document group consisting of several documents. Taking English documents as an example, articles and conjunctions such as a, an, and, of, but appear frequently in certain documents, so they have low weight and are not selected as important words. When this is applied to the visual vocabulary according to the embodiment of the present invention, the image feature point having a distinctiveness can be regarded as a word in the document.

The keyframe index vector generation module 240 aligns the image feature points identified in the received keyframe with each element constituting the vector to generate the keyframe index vector. An image feature point, which is each element of the keyframe index vector, corresponds to a feature point included in the visual vocabulary.

The keyframe index vector generation module 240 uses the keyframe for the keyframe index vector element corresponding to the keyframe index vector element and the keyframe index vector element corresponding to the keyframe index vector element of the Bow vector of all the keyframes calculated by the vector calculation module 230 . Doing this for all keyframes results in a list of keyframes with a Bow vector containing that word for each element corresponding to that word.

The comparison candidate keyframe selection module 250 compares the words constituting the current frame with the words of the comparison candidate keyframes. In addition, the comparison candidate keyframe selection module 250 calculates the Tf-idf weight score of the current frame when there is a keyframe having a word overlapping more than a preset threshold (eg, 80%, etc.) as a result of the comparison. increase

The comparison candidate keyframe selection module 250 selects, as a comparison candidate keyframe, a preset number or a keyframe indicating a preset weighting score among keyframes having a high Tf-idf weight score. The comparison candidate keyframe selection module 250 transmits the selected comparison candidate keyframe to the robot 100 . Based on this, the robot 100 compares the keyframe of the image currently being photographed with the comparison candidate keyframe, so that the robot 100 grasps the current position.

A method of estimating the position of the robot 100 using the robot 100 and the position estimation server 200 described above will be described with reference to FIG. 4 .

4 is a flowchart of a location estimation method according to an embodiment of the present invention.

As shown in FIG. 4 , when the robot 100 captures a location using the camera 110 while moving, the captured video is processed into an image and generated into a plurality of frames ( S100 ). Among the plurality of generated frames, at least one or more frames that the robot 100 determines to be important frames for location estimation are generated as key frames and transmitted to the location estimation server 200 ( S101 and S102 ).

Here, the method for the robot 100 to select key frames from among the plurality of frames may be performed in various ways. Also, the robot 100 may transmit the image generated in step S100 to the location estimation server 200 , and the location estimation server 200 may select a keyframe.

The location estimation server 200 that has received the plurality of keyframes extracts the image feature point and the feature descriptor from the received keyframe (S103). Then, a plurality of image feature points are generated as visual vocabulary by clustering the feature descriptor space generated using the extracted image feature points and feature descriptors (S104).

The position estimation server 200 calculates a Bow vector of all stored keyframes in advance (S105). A visual vocabulary is required to calculate the Bow vector. Since the visual vocabulary is extracted for all stored keyframes, the Bow vector of the stored keyframes is calculated using the extracted visual vocabulary.

In order for the position estimation server 200 to calculate a BoW vector for a keyframe, a visual vocabulary is required. Since the embodiment of the present invention uses the ORB feature, a visual vocabulary learned with the ORB feature is used. The visual vocabulary is used to transform an image into a BoW vector, a sparse numerical vector. It extracts feature points and feature descriptors from numerous training images, and uses the feature descriptor space to form W visual words through K-median clustering. is expressed as

The location estimation server 200 compares the image feature points, which are elements of the Bow vector calculated in step S105, with the image feature points of the received keyframe extracted in step S103. Then, the stored keyframes having identical image feature points are aligned and generated as a keyframe index vector (S106).

The location estimation server 200 calculates the weight of the keyframe index vector generated in step S106, and selects a stored keyframe representing a preset score or more as a comparison candidate keyframe (S107). The position estimation server 200 transmits the comparison candidate keyframes selected in step S107 to the robot 100 (S108).

Here, when the location estimation server 200 selects the comparison candidate keyframes from among the plurality of stored keyframes in step S107, the comparison candidate keyframe is selected by calculating a similarity score with the Bow vector of the current keyframe. The similarity score is calculated through Equation 1 below.

Here, v denotes the Bow vector of the received current keyframe and the Bow vector of the w stored keyframe. n is the number of words, i is the index, v _i is the weight of the image feature corresponding to the index in the Bow vector of the current keyframe, and w _i is the weight of the image feature that corresponds to the index in the Bow vector of the saved keyframe. it means.

After calculating the similarity scores for all image singularities commonly included in the keyframe using Equation 1, the location estimation server 200 calculates the final similarity score for the current keyframe using Equation 2 below. .

In an embodiment of the present invention, in order to give priority to the weight of the current keyframe in the process of selecting the comparison candidate keyframe, the weight of the current keyframe and the stored keyframe are multiplied and used in the process of calculating the similarity score. Based on this, the number of cases in which the weight of the current keyframe can be ignored due to the low weight of the keyframe is reduced as much as possible, and the accuracy is increased.

The robot 100 compares the comparison candidate keyframes received in step S108 with the keyframe generated in step S101, and recognizes the current position of the robot 100 itself (S110). And the keyframe used to recognize the position of the robot is generated as a keyframe map (S111).

When the robot 100 recognizes its position through this procedure, it does not compare with all keyframes, but compares the selected comparison candidate keyframes with the keyframes generated by it, so that the amount of computation for keyframe comparison is can be reduced and the location can be quickly identified.

Among the procedures described above, an example of generating a plurality of visual vocabulary in step S104 will be described with reference to FIG. 5 .

5 is an exemplary diagram of visual vocabulary generation according to an embodiment of the present invention.

As shown in FIG. 5 , in order to improve the visual spatial recognition of the robot 100 using the BoW method, the position estimation server 200 generates a keyframe index vector. First, in order to estimate the global position of the robot 100, a binary file pre-stored in the position estimation server 200 is read and stored keyframes are called. The location estimation server 200 calls image feature points and feature descriptors for all stored keyframes.

The localization server 200 uses the visual vocabulary learned by ORB features with ORB SLAM to calculate the BoW vector for the received keyframe. Here, ORB SLAM is one of the SLAM technologies, and when keyframe features are extracted from ORB SLAM, the extracted features become ORB features. As shown in FIG. 5 , an example of generating a visual location having a total of 6 groups is shown, which is referred to as having 6 visual vocabulary.

For example, it is assumed that the location estimation server 200 clusters the image having the feature point and the feature descriptor of the room door in the keyframe into keyframes having the same feature. The feature points and feature descriptors of the room door clustered by the location estimation server 200 are used as comparison keyframes when the robot 100 wants to perform location recognition as an image in front of the room door.

5 shows an example in which six representative visual vocabulary is selected, and the number of representative visual vocabulary is not necessarily limited in this way.

Next, an example of the vector calculated in step S105 of FIG. 4 will be described with reference to FIG. 6 .

6 is an exemplary diagram of a Bow vector calculated according to an embodiment of the present invention.

As shown in FIG. 6 , in order to calculate the BoW vector of each keyframe, the localization server 200 selects a node that minimizes the Hamming distance at each level of the visual vocabulary for all feature descriptors of the corresponding keyframe. do. Then, from the root node, which is the first starting point, to the leaf node corresponding to the word, the visual vocabulary of the tree structure is searched and the word that can best be represented is selected.

This does not search for nodes corresponding to all words, but selects one node with the closest Hamming distance in each level and does not search for other nodes, so that words can be selected efficiently.

And the position estimation server 200 calculates the weight of the word selected for the key frame using tf-idf. A BoW vector is generated by collecting all pairs of the selected word and weight as a pair.

That is, in order to compare or learn the image feature points or feature descriptors in a plurality of key frames, the location estimation server 200 digitizes the feature descriptors having a direction at the image feature points of the key frames. For example, the horizontal feature pointing to the present direction from the corner of the room door is digitized as 1 and the vertical feature is managed by the location estimation server 200 , which is a standard for grouping keyframes.

Next, an example of the keyframe index vector generated in step S106 of FIG. 4 will be described with reference to FIG. 7 .

7 is an exemplary diagram of a keyframe index vector according to an embodiment of the present invention.

As shown in Figure 7, each element of the keyframe index vector corresponds to a word in the visual vocabulary learned with the ORB feature. After calculating the Bow vector of all stored keyframes, the location estimation server 200 uses the corresponding keyframe as an element of the keyframe index vector corresponding to the word that is the element of each Bow vector.

Referring to FIG. 7 for example, when the inverted triangle word element of the keyframe index vector is accessed, it can be seen that the keyframes having the BoW vector including the inverted triangle word are 1, 2, and 3. When finding a keyframe containing a specific word, it is not necessary to compare all keyframes, and only the corresponding word element is accessed, reducing execution time.

That is, the location estimation server 200 rearranges the keyframes based on the criteria for grouping the selected keyframes. In one embodiment, the location estimation server 200 groups keyframes having a keyframe index vector having a characteristic of a lighting edge of a hallway of a guest room, and keyframe images having a similar index vector.

The keyframe images grouped in advance by the position estimation server 200 in this way are selected as comparison images when the robot 100 performs position re-recognition with a camera near the lighting in the hallway of the room, and the robot quickly determines the position of the image with the image. makes it recognizable If this operation is performed for all keyframes, a list of keyframes having a Bow vector including the corresponding image feature point for each element corresponding to each image feature point is completed as shown in FIG. 7 .

Meanwhile, a method in which the location estimation server 200 extracts image feature points from a keyframe in step S103 of FIG. 4 described above will be described with reference to FIG. 8 .

8 is a flowchart of a method for extracting image feature points according to an embodiment of the present invention.

As shown in FIG. 8 , when the location estimation server 200 receives a plurality of keyframes from the robot 100, a plurality of image feature points are extracted from the first keyframe (S200). After extracting the plurality of image feature points from the first keyframe, the location estimation server 200 extracts the plurality of image feature points from the second keyframe by using the light flow with the first keyframe (S201). Here, the light flow refers to a visible operation pattern of an image object between two consecutive video frames. Since the light flow is already known, a detailed description thereof will be omitted in the embodiment of the present invention.

The location estimation server 200 checks whether the number of image feature points of the second keyframe extracted in step S201 is greater than a preset threshold value (S202).

If the number of image feature points extracted from the second keyframe is not sufficient, the location estimation server 200 uses the image feature point extracted in step S201 as a center point and creates a circle having a predefined length and radius and designates it as a mask area (S203). ). Since the method for the location estimation server 200 to designate a mask area based on image feature points can be performed through various methods, a detailed description will be omitted in the exemplary embodiment of the present invention.

The location estimation server 200 additionally extracts a plurality of image feature points from the remaining area except for the mask area in the second keyframe designated in step S203 (S204). Then, it is checked whether the number of additionally extracted plurality of image feature points and the number of image feature points extracted in step S201 is greater than a preset threshold value (S205).

If the number of all image feature points extracted from the second keyframe is greater than the threshold value, the location estimation server 200 determines the image feature points using the light flow with the second keyframe in the third keyframe continuous to the second keyframe. is extracted (S206). However, if the number of all image feature points extracted from the second keyframe is less than the threshold, the procedure after step S203 is repeatedly performed.

An example of extracting image feature points described above will be described with reference to FIG. 9 .

9 is an exemplary diagram in which image feature points are extracted from a keyframe according to an embodiment of the present invention.

9A is an exemplary diagram illustrating seven image feature points extracted from an arbitrary keyframe at time t. And FIG. 9(b) is an exemplary diagram showing 15 image feature points extracted from an arbitrary key frame at time t+1.

As shown in (a) of FIG. 9, the location estimation server 200 extracts 7 image feature points at time t in an arbitrary keyframe (①), then creates a circle based on each image feature point to mask Designate as an area (②, ③). In this case, it is assumed that the 7 image feature points extracted from the keyframe are less than the threshold value.

Then, the location estimation server 200 additionally extracts 8 image feature points in a region other than the mask region formed based on the 7 image feature points in an arbitrary key frame, as shown in FIG. 9(b) (④) ). Then, a mask area is designated including the additionally extracted 8 image feature points (⑤, ⑥).

As described above, in the embodiment of the present invention, since the image feature points are extracted in the mask method, the image feature points are not extracted only in a specific area of the key frame, but the image feature points can be evenly extracted throughout the key frame. Accordingly, more image feature points can be extracted. In addition, since image feature points favorable to light flow are extracted, light flow performance between the previous keyframe and the next keyframe can be improved.

The following describes a visual vocabulary used by the vector calculation module 230 to calculate a vector of received keyframes with reference to FIG. 10 .

10 is an exemplary diagram of a visual vocabulary according to an embodiment of the present invention.

When an image photographed by the robot 100 is received as shown in FIG. 10A , the keyframe processing module 210 extracts a characteristic pixel from the image. The areas marked in red and blue are the characteristic pixels. A corresponding pixel value may be expressed as a vector having a direction, and a value having a direction may be converted into a feature descriptor through a known technique.

And as shown in (b) of FIG. 10 , the feature descriptor obtains a direction histogram for the image feature point, and determines the strongest direction as the feature point direction. If similar feature descriptors are clustered among the feature descriptors, the keyframe processing module 210 represents a small number of representative visual vocabulary for performing tf-idf. A few representative visual vocabulary corresponds to FIG. 10( c ).

For example, when the robot 100 collects keyframe images while walking around the hotel lobby, the feature descriptors such as the border of the room door, the passageway leading to the elevator, and the room number sign will be the feature descriptors with the strongest direction. can Therefore, the keyframe with the corresponding characteristic can be regarded as a visual vocabulary to which the characteristic descriptor of the distinctive characteristic point belongs.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improved forms of the present invention are also provided by those skilled in the art using the basic concept of the present invention as defined in the following claims. is within the scope of the right.

Claims (14)

A server that provides a comparison target frame so that the robot can estimate a location in conjunction with an autonomous driving robot,
receiving a plurality of keyframes selected from a specific image from the autonomous driving robot, and extracting a plurality of first image feature points and first feature descriptors for the plurality of first image feature points, respectively, from the plurality of keyframes keyframe processing module,
a vector calculation module for calculating a keyframe vector of each of the plurality of keyframes and generating a keyframe index vector including second image feature points of a plurality of stored keyframes;
a keyframe index vector generation module that aligns keyframes including first feature descriptors corresponding to the second image feature points to the keyframe index vector to generate a keyframe index; and
A comparison candidate key for calculating a weight of the keyframe index and selecting a plurality of comparison candidate keyframes for recognizing a position by comparing the autonomous driving robot with the plurality of keyframes from among the plurality of stored keyframes stored in advance Frame selection module
A location estimation server comprising a. According to claim 1,
A keyframe storage module for storing a plurality of keyframes, first image feature points, and first feature descriptors processed by the keyframe processing module, and second image feature points and second feature descriptors for the stored keyframes
A location estimation server further comprising a. According to claim 1,
The keyframe processing module,
designating a mask area for each of a plurality of image feature points extracted from a first keyframe among the plurality of keyframes,
A location estimation server for extracting a plurality of additional image feature points in a region other than the mask region in a second keyframe successive to the first keyframe. According to claim 1,
The vector calculation module,
A location estimation server for acquiring a visual vocabulary from each of the plurality of keyframes, and calculating a vector of received keyframes by a Bag of Words (BoW) method based on the acquired visual vocabulary. 5. The method of claim 4,
The keyframe index vector generation module,
A position estimation server for generating the keyframe index by applying a term frequency-inverse document frequency (tf-idf) to the first image feature points. As an autonomous driving robot that interworks with a location estimation server,
a camera that shoots video,
a keyframe generation module for selecting a plurality of keyframes for estimating the position of the robot from among the plurality of frames generated from the photographed image;
an interface for transmitting the plurality of keyframes to the location estimation server and receiving a plurality of comparison candidate keyframes selected based on the plurality of keyframes from the location estimation server; and
A keyframe position recognition module for recognizing the position of the autonomous driving robot by comparing the plurality of keyframes with the plurality of comparison candidate keyframes
self-driving robot including 7. The method of claim 6,
An image processing unit generating the image captured by the camera as a plurality of images, and generating the plurality of frames for each of the plurality of images
An autonomous driving robot further comprising a. 8. The method of claim 7,
When the location of the autonomous driving robot is recognized by the keyframe location recognition module, a keyframe map generator for generating a keyframe map using the plurality of keyframes used for location recognition
An autonomous driving robot further comprising a. A method for a position estimation server to estimate a position of an autonomous driving robot, comprising:
Receiving a plurality of received keyframes generated based on an image in the autonomous driving robot;
extracting a plurality of first image feature points and a first feature descriptor for the first image feature points from each of the received keyframes;
calculating a vector of stored keyframes based on visual vocabularies for each of a plurality of previously stored stored keyframes;
generating a keyframe index by arranging storage keyframes in which second image feature points of the storage keyframes constituting the calculated vector coincide with the first image feature points; and
selecting a comparison candidate keyframe from among a plurality of storage keyframes by calculating weights for second characteristic descriptors of the storage keyframes included in the keyframe index;
A robot position estimation method comprising a. 10. The method of claim 9,
The step of extracting the first feature descriptor,
generating a feature descriptor space based on the first image feature points and a first feature descriptor; and
generating the first image feature points as a first visual vocabulary by clustering the generated feature descriptor space;
A robot position estimation method comprising a. 11. The method of claim 10,
The step of extracting the first feature descriptor,
extracting a first keyframe image feature point from a first keyframe among the received keyframes;
extracting a second keyframe image feature point from a second keyframe adjacent to the first keyframe;
designating a mask area using the second keyframe image feature point as a reference point when the number of extracted second keyframe image feature points is less than a preset threshold value; and
extracting an additional second keyframe image feature point in an area other than the mask area in the second keyframe
A robot position estimation method comprising a. 12. The method of claim 11,
Calculating the vector of the stored keyframes comprises:
A robot position estimation method for calculating a Bow vector based on a second visual vocabulary corresponding to the stored keyframes. As a method for estimating a location by an autonomous driving robot linked with a location estimation server,
generating a plurality of images from the captured image, and generating frames for the generated plurality of images;
selecting a plurality of keyframes for estimating the position of the robot from among the generated frames;
transmitting the selected plurality of keyframes to the location estimation server;
Receiving a comparison candidate keyframe selected based on the plurality of keyframes from the location estimation server, and
estimating a position using the received comparison candidate keyframe and the selected plurality of keyframes;
A robot position estimation method comprising a. 14. The method of claim 13,
After estimating the location,
generating a keyframe map using the plurality of keyframes;
Robot position estimation method further comprising.

Images