KR20230108997A - Method for visual localization, control server and buiding using the same - Google Patents

Abstract

The present invention relates to a visual positioning method, and a control server and a building using the same. The visual positioning method according to an embodiment of the present invention includes the steps of receiving, from a robot, a surrounding image of a space where the robot is located; selecting a positioning module according to spatial characteristics set in the space; and generating positioning information for the robot from the surrounding image using the selected positioning module.

Description

Visual localization method, control server and building using it {Method for visual localization, control server and buiding using the same}

The present invention relates to a visual positioning method for locating the position and pose of a robot to control the driving of the robot, a control server and a building using the same.

VL (Visual Localization) is a technology that can estimate the global position of a robot based on images taken from a robot, etc. location can be estimated. However, such a VL requires a preliminary operation of generating an entire map of the space by scanning the entire space, and expensive equipment such as a lidar is required at this time. In addition, when the position of the robot is recognized using only the VL in a step where the initial position is not given, there is a high possibility of false positives because the entire search is performed using the query image. In addition, there is a problem in that open spaces such as outdoors are vulnerable to utilizing existing VLs due to changes in scale and environment.

As another example, there is a technology for estimating the global position of a robot using a 2D image marker. For example, Korean Patent Publication No. 10-2006-0129960 relates to a mobile robot and a method for calculating its position and attitude. The data, identification data of the marker and positional data of a boundary line adjacent to the marker in the movement area are stored, the marker detection unit detects the marker in the image based on the positional data and identification data of the marker, and the boundary line detection unit detects the marker from the image. When a boundary line adjacent to the marker is detected, and the parameter calculation unit calculates the parameters of the boundary line in the image, the position and attitude calculation unit calculates the position and posture of the mobile robot in the moving area based on the parameters of the boundary line and the position data of the boundary line. Calculation is starting.

The present invention is to provide a visual positioning method that can perform more accurate visual positioning by utilizing a plurality of visual positioning methods specialized according to spatial characteristics of each space together, in addition to a general visual positioning method, and a control server and a building using the same. do.

An object of the present invention is to provide a visual positioning method having robust characteristics in spatial characteristics by integrating a plurality of positioning modules that perform visual positioning, and a control server and a building using the same.

An object of the present invention is to provide a building including robots that run in the building and provide services.

A visual localization method according to an embodiment of the present invention relates to a visual localization method of a control server for determining the position of a robot in a target area based on a surrounding image obtained from a camera provided in the robot, receiving, from the robot, a surrounding image of a space in which the robot is located; applying a positioning module based on spatial characteristics set in the space; and generating positioning information for the robot from the surrounding images using the positioning module.

The control server according to an embodiment of the present invention performs visual localization for determining the position of the robot in a target area based on a surrounding image acquired from a camera provided in the robot, and from the robot, an image receiving unit that receives an input image of the surroundings of the space where the robot is located; a positioning module determiner applying a positioning module based on spatial characteristics set in the space; and a positioning unit generating positioning information for the robot from the surrounding images using the positioning module.

In a building according to an embodiment of the present invention, at least one robot that travels in the building and provides services is disposed, the robot is controlled through communication with a control server, and the control server is provided in the robot. At least one processor implemented to execute a computer-readable command to perform visual localization for determining the location of the robot in the building based on a surrounding image acquired from a camera, wherein the At least one processor receives an input of a surrounding image of a space in which the robot is located from the robot, applies a positioning module based on a spatial characteristic set in the space, and uses the positioning module to obtain an input of a surrounding image of the space where the robot is located. Positioning information for the robot may be generated.

In addition, the solution to the above problem does not enumerate all the features of the present invention. Various features of the present invention and the advantages and effects thereof will be understood in more detail with reference to specific embodiments below.

According to the visual positioning method according to an embodiment of the present invention, a control server and a building using the same, in addition to the general visual positioning method, a plurality of visual positioning methods specialized according to the spatial characteristics of each space are used together, so more accurate visual positioning It is possible to do

Since the visual positioning method according to an embodiment of the present invention, a control server and a building using the same, can perform visual positioning for a robot by integrating a plurality of positioning modules specialized according to spatial characteristics, robust visual positioning for various spatial characteristics location can be provided.

However, the visual positioning method according to the embodiments of the present invention, the effects that can be achieved by the control server and the building using the same are not limited to those mentioned above, and other effects not mentioned are the present invention from the description below. This will be clearly understood by those skilled in the art.

1 is a schematic diagram showing a control system for controlling the operation of a robot according to an embodiment of the present invention.
2 is a block diagram showing a control server according to an embodiment of the present invention.
3 is an exemplary diagram illustrating visual positioning using a main positioning module according to an embodiment of the present invention.
4 is an exemplary diagram illustrating visual positioning using a marker positioning module according to an embodiment of the present invention.
5 is an exemplary diagram illustrating visual positioning using a signage positioning module according to an embodiment of the present invention.
6 is an exemplary diagram illustrating visual positioning using an elevator positioning module according to an embodiment of the present invention.
7 is a schematic diagram showing a control server according to an embodiment of the present invention.
8 is a schematic diagram showing a building in which a control system using visual positioning according to an embodiment of the present invention is built.
9 is a flowchart illustrating a visual positioning method according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar elements are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. That is, the term 'unit' used in the present invention means a hardware component such as software, FPGA or ASIC, and 'unit' performs certain roles. However, 'part' is not limited to software or hardware. A 'unit' may be configured to reside in an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, 'unit' refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functionality provided within the components and 'parts' may be combined into a smaller number of elements and 'parts' or further separated into additional elements and 'parts'.

In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

1 is a schematic diagram showing a control system for controlling the operation of a robot according to an embodiment of the present invention.

Referring to FIG. 1 , a control system according to an embodiment of the present invention may include a robot 1 and a control server 100.

A control system according to an embodiment of the present invention will be described with reference to FIG. 1 below.

The robot 1 can perform various services while moving within the target area A. Here, the target area (A) may be an indoor space such as a building, and the robot 1 may move within the target area (A) in an autonomous driving method, but positioning of the position of the robot 1 or control of movement, etc. may be performed by the control server 100. That is, the robot 1 may perform operations such as movement through communication with the control server 100 .

The robot 1 may include a camera, and a surrounding image corresponding to the movement of the robot 1 may be captured using the camera. The robot 1 may transmit a surrounding image to the control server 100, and the control server 100 may perform visual localization for the robot 1 based on the received surrounding image.

Depending on the embodiment, the robot 1 may further include various sensors such as an Inertial Measurement Unit (IMU) sensor, a wheel encoder sensor, and a LiDAR sensor in addition to a camera, and the control server 100 uses a method other than visual positioning for the robot. It is also possible to perform additional localization for (1).

The control server 100 can communicate with the robots 1 located in the target area A through the network N, and can control various operations including movement of the robot 1. . Here, the communication method between the control server 100 and the robot 1 is not limited, and the communication network (eg, mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network N may include In addition to the communication method used, short-distance wireless communication between devices may be included. For example, the network N may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , one or more arbitrary networks such as the Internet. In addition, the network N may include any one or more of network topologies including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, and the like. Not limited.

The control server 100 may be implemented as a computer device or a plurality of computer devices that communicate with the robot 1 through a network N to provide commands, codes, files, contents, services, and the like. Depending on the embodiment, it is also possible to implement the control server 100 in the form of a cloud server.

In order for the robot 1 to move within the target area A, it is necessary to perform positioning to generate positioning information including the current position and pose of the robot 1. Here, specific map information for the target area A may be stored in advance, and the control server 100 matches the current position and pose of the robot 1 to the map information to match the current position of the robot 1. and poses can be mapped within map information. Depending on the embodiment, the position and pose may be expressed as 3-dimensional coordinate values and 3-axis rotation values (pitch, roll, yaw), respectively.

The control server 100 may receive a surrounding image from the robot 1 and perform visual positioning of the robot 1 based on the received surrounding image. That is, the control server 100 can check the exact position and pose of the robot 1 in the map information as a result of the visual positioning of the robot 1, and based on this, the moving direction and rotation direction of the robot 1 , It is possible to control the movement of the robot 1 by determining the moving speed.

Specifically, the control server 100 may search for a reference image corresponding to the query image in map information by using the received surrounding image as a query image. Here, since the position and pose of the measurement equipment corresponding to the reference image may be set, it is possible to obtain the position and pose of the current robot 1 by comparing the reference image and surrounding images. That is, the position and pose of the robot 1 corresponding to the query image can be obtained through local feature matching.

However, if the space in which the robot 1 is traveling is repetitive or lacks features in the space, problems such as deterioration in visual positioning performance using surrounding images may occur. On the other hand, since the control server 100 according to an embodiment of the present invention utilizes a plurality of visual positioning methods specialized according to spatial characteristics of each space in addition to the general visual positioning method, it is possible to perform more accurate visual positioning. do. Hereinafter, a control server according to an embodiment of the present invention will be described.

2 is a block diagram showing a control server 100 according to an embodiment of the present invention.

Referring to FIG. 2 , the control server 100 according to an embodiment of the present invention may include an image receiving unit 110 , a positioning module determining unit 120 , a positioning unit 130 and a map database 140 .

The image receiving unit 110 may receive an input of a surrounding image of a space where the robot 1 is located, from the robot 1 . The robot 1 may generate an image of the surroundings captured while moving using a camera installed therein, and may transmit the captured image to the image receiving unit 110 through a network. Depending on the embodiment, in order to minimize the obstruction of the surrounding image by objects or people close to the robot 1 and to reduce the scale change in the image, the camera may be installed to face vertically upward. That is, the surrounding image may be an image obtained by photographing the ceiling of the target area A by photographing the upper side of the robot 1 .

The positioning module determiner 120 may apply a positioning module based on spatial characteristics set in the space. That is, a plurality of positioning modules may be included in the control server 100, and the positioning module determining unit 120 is the positioning module most suitable for the spatial characteristics of the space where the robot 1 is currently located among the plurality of positioning modules. can be applied.

Specifically, the target area A may be divided into a plurality of spaces, and spatial characteristics for each space may be set in map information about the target area A stored in the map database 140 . For example, when the target area (A) is a building, a space such as a separate charging station to support charging of the robots 1 may be provided in the building, and within the charging station, the robot 1 Since accurate position control for charging of the cells is required, a marker or the like to assist this may be included. In addition, signage, such as signs, can be installed in a work space in a building with a set standard according to a certain rule so that workers can recognize the location in the space. may include

That is, since spatial characteristics for each space are set in the map information, the most suitable positioning modules can be set in advance based on each spatial characteristic. Accordingly, the positioning module determiner 120 may apply each of the positioning modules preset based on the spatial characteristics to each space. In this case, since the most appropriate positioning module can be selected according to the spatial characteristics of each space, it is possible to perform visual positioning within the space more accurately and quickly.

Here, the positioning module determination unit 120 first needs to determine in which space each robot 1 is currently located. To this end, the positioning module determining unit 120 may utilize map information corresponding to the target area A and positioning information immediately before the robot 1 . The map information may be generated and stored precisely in advance so that the robot 1 can travel within the target area A, and within the map information, each space included in the target area A and the spaces Space characteristics to be set may be stored in advance. Therefore, the positioning module determiner 120 can determine which space is on the map information where the robot 1 is currently located based on the previous positioning information of the robot 1, and extracts the spatial characteristics set in the corresponding space. can do. Depending on the embodiment, the positioning module determiner 120 may obtain the odometry of the robot 1 by utilizing a separate IMU sensor or wheel encoder sensor included in the robot 1, and It is also possible to specify the space where the robot 1 is currently located by applying the corresponding odometry to the positioning information.

Additionally, an embodiment in which the positioning module determiner 120 distinguishes each space from surrounding images is also possible. For example, when a marker or signage is included in a surrounding image, the positioning module determiner 120 determines each spatial characteristic as a marker space, a signage space, etc., and applies a positioning module corresponding to each. .

Here, the positioning module may include a main positioning module V1, a marker positioning module V2, a signage positioning module V3, an elevator positioning module V4, and the like. However, it is not limited thereto, and depending on embodiments, it is possible to further include various positioning modules and the like.

The main positioning module V1 may generate positioning information of the robot 1 by extracting feature points from surrounding images and comparing the feature points with feature points in map information. Here, the map database included in the control server 100 may include a feature point map corresponding to the corresponding target area (A). Accordingly, the main positioning module V1 may set surrounding images as query images and then extract global feature points from the query images through a deep learning model or the like. Thereafter, as shown in FIG. 3 , a reference image i2 corresponding to the query image i1 may be searched from the feature point map using the extracted feature points. Here, each reference image may be tagged with information about a position and pose of a measuring device used when generating a feature point map. Accordingly, the main positioning module V1 may generate positioning information by comparing the reference image and the query image to specify the current position and pose of the robot 1 .

Meanwhile, since the main positioning module V1 is applicable to the entire space of the target region A where the feature point map is generated, the positioning module determiner 120 performs positioning by applying the main positioning module V1 in a general space. Then, when entering a specific space, positioning modules corresponding to respective spatial characteristics may be applied. At this time, the positioning module determiner 120 may switch from the main positioning module to another positioning module, but depending on the embodiment, it is also possible to operate other positioning modules together with the main positioning module V1. That is, the positioning module determiner 120 may select not only one positioning module, but a plurality of positioning modules to perform visual positioning of the position of the robot 1.

The marker positioning module V2 may extract markers from surrounding images and generate positioning information of the robot 1 based on coordinate information recognized from the markers.

The marker may be produced in the form of a printed matter attachable to a space such as a poster, wallpaper, or sticker. Depending on the embodiment, as shown in FIG. can create Here, each marker M may include a different identification code, and each marker M may be distinguished using the identification code. As shown in FIG. 4(b), the marker positioning module V2 can extract three-dimensional coordinate information preset for each marker M1 and M2 using the identification code of the marker M. there is. In the map database 140, information on the marker space to which each marker in the target area A is attached, and a marker map stored by matching the identification tag of each marker with the three-dimensional coordinates to which the corresponding marker is attached are stored. may be included Therefore, the marker positioning module V2 may detect the marker M in the query image by using the surrounding image captured by the robot 1 as a query image, and match the identification tag included in the marker M Based on the coordinate information, positioning information including the position and pose of the query image can be obtained. That is, the marker positioning module V2 may generate positioning information about the position and pose of the robot 1 from the position and angle of the marker M included in the query image.

On the other hand, since the marker positioning module V2 can perform positioning only in the marker space, the positioning module determining unit 120 is a marker positioning module as a positioning module if the spatial characteristic of the space where the robot 1 is located is the marker space ( V2) can be applied. That is, the marker space may be a space requiring precise control of the position of the robot 1 or a space in which accurate positioning is difficult only with the existing main positioning module V1, and the positioning module determining unit 120 is a marker space within map information. If it corresponds to the space set to , the marker positioning module (V2) can be applied.

The signage positioning module V3 may extract signage from surrounding images, obtain coordinate information of the signage, and then generate positioning information of the robot 1 based on the coordinate information of the signage.

The signage may be installed at a predetermined location in the space according to a set standard, and may be a sign installed so that a worker or robot in the target area (A) can recognize the location of the internal space. Depending on the embodiment, as shown in Figure 5 (a), the signage (S1, S2) may be attached in the space. Here, since the signage is installed in a preset location according to a certain set standard, the signage positioning module (V3) can extract the signage included in the surrounding image, and the coordinate information of the signage from the set standard of the signage. can be obtained. Depending on the embodiment, after the signage positioning module (V3) recognizes a character through OCR (Optical Character Reader) for the signage, etc., the signage is distinguished through the recognized character, and the 3D It is also possible to extract coordinate information and the like. The map database 140 includes information about the signage space to which each signage is attached in the target area (A), and a signage map in which the three-dimensional coordinates of each signage are mapped and stored. There may be. Therefore, the signage positioning module (V3) can detect the signage (S) in the query image by using the surrounding image captured by the robot (1) as a query image, and the coordinate information obtained from the signage (S) Based on this, positioning information including the position and pose of the query image can be obtained.

On the other hand, since the signage positioning module V3 can perform positioning only in the signage space, the positioning module determination unit 120 determines the positioning module if the space characteristic of the space where the robot 1 is located is the signage space. Signage positioning module (V3) can be applied.

The elevator positioning module V4 may extract specific feature points of the elevator from surrounding images and generate positioning information of the robot 1 based on the specific feature points. Since the inside of the elevator generally has a rectangular ceiling shape, the rectangular ceiling shape can be extracted as an elevator feature, and based on the ceiling shape of the elevator shown in the surrounding image, the position and pose of the robot 1 in the elevator can be determined. Positioning information can be created. Depending on the embodiment, the corresponding elevator may be a dedicated elevator in which only the robot 1 boards.

Specifically, as shown in FIG. 6 (a), the ceiling of the elevator may appear as a rectangle (R), and as shown in FIG. (F1, F2, F3) can be extracted. For example, the location and pose of the robot 1 in the elevator are determined based on the slopes (F1, F2) of each side forming the rectangular shape and the coordinate information (F3) where the center point of the surrounding image is located within the rectangular shape. It is possible to generate positioning information for In addition, the map database 140 may include an elevator map including information about the elevator space where the elevator is installed in the target area A and the type of specific feature points extracted from each elevator. Therefore, the elevator positioning module V4 may extract a specific feature point of the elevator from the query image by using the surrounding image captured by the robot 1 as a query image, and based on the specific feature point, the location and location of the query image Positioning information including a pose may be obtained.

On the other hand, since the elevator positioning module V4 can perform positioning only in the elevator space, the positioning module determination unit 120 determines that the positioning module is the elevator positioning module if the space characteristic of the space where the robot 1 is located is the elevator space. (V3) can be applied.

Additionally, here, it is assumed that the positioning module determiner 120 determines the positioning module based on the spatial characteristics of each space, but according to the embodiment, when positioning using one positioning module fails, another positioning module is used. It is also possible to do For example, the positioning module determiner 120 may set the main positioning module V1 as a basis to perform positioning, and then, if the positioning fails, such as low accuracy of positioning by the main positioning module V1. The positioning module can be changed to apply the marker positioning module (V2) or the signage positioning module (V3).

Specifically, the main positioning module V1 may apply Perspective-N-Point (PNP) based on a Random Sample Consensus (RANSAC) algorithm after matching a reference image corresponding to a query image. That is, a pose such as a rotation angle of the robot 1 may be obtained by comparing matched feature points in the query image and the reference image using PNP. However, among the matching feature points in the query image and the reference image, there may be a case where the number of feature points classified as inlier is less than the set number, and in this case, spatial positioning using the main positioning module V1 is determined to have failed. can That is, since it corresponds to a case in which matching is not sufficiently performed to be able to compare the query image and the reference image, positioning by the main positioning module V1 appears inaccurately. Therefore, the positioning module determination unit 120 may apply the marker positioning module V2 or the signage positioning module V3 instead of the main positioning module V1, and depending on the embodiment, the marker positioning module ( V2) and the signage positioning module (V3) can be applied simultaneously.

Among the spaces included in the target area A, there may be spaces in which positioning by the main positioning module V1 is difficult due to their characteristics, and markers or signage may be attached in advance to these spaces. Therefore, the positioning module determiner 120 performs positioning using the marker positioning module V1 or the signage positioning module V2 instead of the message positioning module V1 when positioning by the main positioning module V1 fails. can be implemented to try.

The positioning unit 130 may generate positioning information for the robot from surrounding images using a positioning module. That is, according to the main positioning module V1, the marker positioning module V2, the signage positioning module V3, and the elevator positioning module V4, positioning information may be generated by performing positioning, respectively. However, since the detailed positioning method performed by each positioning module has been described above, detailed description thereof will be omitted.

Meanwhile, the positioning module determination unit 120 may simultaneously apply a plurality of positioning modules, and in this case, the positioning unit 130 may perform positioning using a plurality of positioning modules. Specifically, the positioning unit 130 may generate final positioning information by collecting positioning information measured by each positioning module, and at this time, set weights for each positioning module, and positioning information to which each weight is applied. By combining them, it is possible to generate final positioning information. For example, when the main positioning module V1 and the marker positioning module V2 are selected at the same time, the weight of the main positioning module V1 is set to 1 and the weight of the marker positioning module V2 is set to 9, and the final positioning is performed. information can be generated. In addition, positioning information measured by each positioning module may be combined using a pose graph optimization technique or a Kalman filter widely used in the field of sensor fusion. Depending on the embodiment, the dynamic covariance scaling technique can be used among pose graph optimization techniques, and through this, even if some inaccurate poses are input, it is possible to perform robust pose estimation by adjusting covariance in the optimization process.

7 is a block diagram showing the control server 100 according to an embodiment of the present invention.

Referring to FIG. 7 , the control server 100 may include a memory 10, a processor 20, a communication interface 30, an input/output interface 40, and the like. The memory 10 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the control server 100 as a separate permanent storage device distinct from the memory 10. Also, an operating system and at least one program code may be stored in the memory 10 . These software components may be loaded into the memory 10 from a computer-readable recording medium separate from the memory 10 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 10 through the communication interface 30 rather than a computer-readable recording medium. For example, software components may be loaded into the memory 10 of the control server 100 based on a computer program installed by files received through the network N.

The processor 20 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 20 by memory 10 or communication interface 30 . For example, processor 20 may be configured to execute received instructions according to program codes stored in a recording device such as memory 10 .

The communication interface 30 may provide a function for the control server 100 to communicate with other devices (eg, the storage devices described above) through the network N. For example, the processor 20 of the control server 100 generates a request, command, data, file, etc. according to a program code stored in a recording device such as the memory 10 under the control of the communication interface 30 to the network ( N) can be transmitted to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received through the communication interface 30 of the control server 100 via the network N. Signals, commands, data, etc. received through the communication interface 30 may be transmitted to the processor 20 or memory 10, and files may be stored as storage media that the control server 100 may further include (described above). permanent storage).

The input/output interface 40 may be a means for interface with the input/output device 50 . For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input/output interface 40 may be a means for interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 50 may be configured as one device with the control server 100.

8 is a schematic diagram showing a building B in which a control system using visual positioning according to an embodiment of the present invention is built. As shown in FIG. 8, a building B can be set as a target area, and a plurality of robots 1 providing services within the target area can be included in the building B. In addition, the building B may include an indoor area in which a robot travels, and the indoor area may be divided into a plurality of spaces according to spatial characteristics. Here, the building B may include a marker space with markers attached thereto, a signage space including signage, and an elevator space including elevators among a plurality of spaces. The robot 1 can communicate with the control server 100 through the network N, and can perform operations such as movement under the control of the control server 100. The control server 100 may not be built in the building (B), and may be implemented as a cloud server separate from the building (B).

The control server 100 may perform visual positioning to determine the position of each robot 1 in the building B based on the surrounding image obtained from the camera provided in the robot 1. Since the specific operation of the control server 100 has been described above, a detailed description thereof will be omitted.

9 is a flowchart illustrating a visual positioning method according to an embodiment of the present invention. Here, each step of FIG. 9 may be performed by a control server according to an embodiment of the present invention.

The control server may receive an input of a surrounding image of a space where the robot is located from the robot (S10). The robot can create a surrounding image that captures the surroundings while moving using a camera installed in itself, and can transmit the captured surrounding image to the control server through a network. Depending on the embodiment, in order to minimize obstruction of surrounding images by objects or people close to the robot, the camera may be installed to face vertically upward. That is, the surrounding image may be an image obtained by photographing the ceiling of the target area by photographing the upper side of the robot.

The control server may select a positioning module according to the spatial characteristics set in the space (S20). That is, a plurality of positioning modules may be included in the control server, and the control server may select a positioning module most suitable for the spatial characteristics of the space where the robot is currently located from among the plurality of positioning modules.

Specifically, the target area may be divided into a plurality of spaces, and spatial characteristics of each space included in the target area may be previously set in map information stored in a map database. Therefore, the control server can perform visual positioning within the space more accurately and quickly by selecting the most appropriate positioning module according to the spatial characteristics of each space after distinguishing each space where the robot is located.

Here, the control server first needs to determine in which space each robot is currently located, and for this, the control server can utilize map information corresponding to the target area and positioning information immediately before the robot. That is, the control server can determine which space is on the map information where the robot is currently located based on the previous positioning information of the robot, and can extract the spatial characteristics set in the corresponding space using this. In addition, depending on the embodiment, since the odometry of the robot can be obtained using sensor values received from a separate IMU sensor or wheel encoder sensor included in the robot, the control server corresponds to the previous positioning information. It is also possible to accurately specify the space where the robot is currently located by applying odometry.

Here, the positioning module may include a main positioning module, a marker positioning module, a signage positioning module, an elevator positioning module, and the like. The marker positioning module may extract markers from surrounding images and generate robot positioning information based on coordinate information recognized from the markers. The signage positioning module extracts the signage from the surrounding image, obtains the coordinate information of the signage, and then generates the positioning information of the robot based on the coordinate information of the signage. The elevator positioning module may extract specific feature points of the elevator from surrounding images and generate positioning information of the robot based on the specific feature points. The main positioning module may generate positioning information of the robot by extracting feature points from surrounding images and comparing the feature points with feature points in map information.

Depending on the embodiment, after the control server primarily selects the main positioning module as a positioning module, it can be switched to a marker positioning module, a signage positioning module, an elevator positioning module, etc. according to spatial characteristics. That is, since the main positioning module can perform visual positioning for the entire target area, the main positioning module can be used as a basic module and switched to the corresponding positioning module if there is a positioning module specialized according to spatial characteristics. Further, in another embodiment, it is also possible to simultaneously operate the positioning module added according to each spatial characteristic along with the main positioning module.

The control server may generate positioning information for the robot from surrounding images using the selected positioning module (S30). That is, positioning information may be generated by performing positioning according to a main positioning module, a marker positioning module, a signage positioning module, an elevator positioning module, and the like. However, since the detailed positioning method performed by each positioning module has been described above, detailed description thereof will be omitted.

Meanwhile, the control server may select a plurality of positioning modules at the same time, and may perform positioning using the selected plurality of positioning modules at the same time. Specifically, the control server may collect positioning information measured by each positioning module to generate final positioning information. It is possible to generate final positioning information. In addition, depending on the embodiment, positioning information measured by each positioning module is utilized by using a pose graph optimization technique or a Kalman filter widely used in the field of sensor fusion. Combining is also possible.

The above-described present invention can be implemented as computer readable code on a medium on which a program is recorded. The computer-readable medium may continuously store programs executable by the computer or temporarily store them for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or combined hardware, but is not limited to a medium directly connected to a certain computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc. configured to store program instructions. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, a site that supplies or distributes various other software, and a server. Accordingly, the above detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

The present invention is not limited by the foregoing embodiments and accompanying drawings. It will be clear to those skilled in the art that the components according to the present invention can be substituted, modified, and changed without departing from the technical spirit of the present invention.

1: robot 10: memory
20: processor 30: communication interface
40: I/O interface 50: I/O device
100: control server 110: video receiver
120: positioning module determining unit 130: positioning unit
140: map database

Claims (11)

In the visual localization method of a control server for determining the position of the robot in a target area based on a surrounding image acquired from a camera provided in the robot,
receiving, from the robot, a surrounding image of a space in which the robot is located;
applying a positioning module based on spatial characteristics set in the space; and
and generating positioning information for the robot from the surrounding image using the positioning module.
The method of claim 1, wherein the step of applying the positioning module
Distinguishing a space where the robot is located based on map information corresponding to the target area, positioning information of the robot, or odometry of the robot, and determining spatial characteristics set in the space A visual positioning method of a control server that does.
The method of claim 2, wherein the positioning module
a marker positioning module for extracting a marker from the surrounding image and generating positioning information of the robot based on coordinate information recognized from the marker;
a signage positioning module that extracts signage from the surrounding image, obtains coordinate information of the signage, and then generates positioning information of the robot based on the coordinate information of the signage;
an elevator positioning module for extracting specific feature points of the elevator from the surrounding images and generating positioning information of the robot based on the specific feature points; and
A visual positioning method of a control server comprising at least one of main positioning modules that extracts feature points from the surrounding image, compares the feature points with feature points in the map information, and generates positioning information of the robot. .
The method of claim 3, wherein the step of applying the positioning module
After applying the main positioning module as the positioning module, switching to any one of the marker positioning module, signage positioning module and elevator positioning module according to the spatial characteristics, or additionally applied to operate simultaneously with the positioning module Visual positioning method of control server.
The method of claim 3, wherein the step of applying the positioning module
If the spatial characteristic is a marker space to which a marker including coordinate information is attached, the marker positioning module is applied.
The method of claim 3, wherein the step of applying the positioning module
The visual positioning method of the control server, characterized in that the application of the signage positioning module if the space characteristic is a signage space including a signage installed to a set standard in a predetermined location inside.
The method of claim 3, wherein the step of applying the positioning module
If the space characteristic is an elevator space, the visual positioning method of the control server, characterized in that for applying the elevator positioning module.
A computer program stored in a medium to execute the visual positioning method of the control server according to any one of claims 1 to 8 in combination with hardware.
In a control server that performs visual localization for determining the position of the robot in a target area based on a surrounding image obtained from a camera provided in the robot,
an image receiving unit that receives an input of a surrounding image of a space in which the robot is located, from the robot;
a positioning module determiner applying a positioning module based on spatial characteristics set in the space; and
A control server comprising a positioning unit generating positioning information for the robot from the surrounding images using the positioning module.
in the building,
At least one robot that travels in the building and provides a service is disposed;
The robot is controlled through communication with a control server,
The control server performs visual localization for determining the location of the robot in the building based on a surrounding image acquired from a camera provided in the robot, and is implemented to execute a computer-readable command. contains one processor;
The at least one processor,
From the robot, receive an input of a surrounding image of the space where the robot is located,
Based on the spatial characteristics set in the space, a positioning module is applied,
A building that generates positioning information for the robot from the surrounding image using the positioning module.
11. The method of claim 10, wherein the building
A building comprising an indoor area in which the robot travels, dividing the indoor area into a plurality of spaces according to spatial characteristics, and including a marker or signage in at least one of the plurality of spaces.

Images