KR102194426B1 - Apparatus and method for environment recognition of indoor moving robot in a elevator and recording medium storing program for executing the same, and computer program stored in recording medium for executing the same - Google Patents

Abstract

The present invention relates to an apparatus and a method for environment recognition of an indoor moving robot in an elevator, a recording medium storing a program for executing the same, and a computer program stored in a medium for executing the same and, more specifically, to an apparatus and a method for environment recognition of an indoor moving robot in an elevator, a recording medium storing a program for executing the same, and a computer program stored in a medium for executing the same, wherein the method for environment recognition of an indoor moving robot in the elevator is necessary for the indoor moving robot to get in and off the elevator. To this end, the apparatus for environment recognition of an indoor moving robot in the elevator includes a marker, a map storing unit, a vision sensor, an image processing unit, a posture estimating unit, a path planning unit, and a robot control unit.

Description

A device and method for an indoor mobile robot to recognize the environment in an elevator, a recording medium storing a program to implement it, and a computer program stored in the medium to implement it {APPARATUS AND METHOD FOR ENVIRONMENT RECOGNITION OF INDOOR MOVING　ROBOT IN A ELEVATOR AND RECORDING MEDIUM STORING PROGRAM FOR EXECUTING THE SAME, AND COMPUTER PROGRAM STORED IN RECORDING MEDIUM FOR EXECUTING THE SAME}

The present invention relates to an apparatus and method for an indoor mobile robot to recognize the environment in an elevator, a recording medium storing a program for implementing the same, and a computer program stored in the medium to implement the same, and more specifically, the indoor mobile robot is an elevator. An apparatus and method for an indoor mobile robot that provides a method for recognizing an environment necessary for boarding and alighting a vehicle to recognize the environment in an elevator, a recording medium storing a program for implementing the same, and a computer program stored in the medium to implement the same. .

In order for an indoor mobile robot to have the ability to provide various services through an autonomous driving function in a high-rise building with an elevator, the function of getting on or off the elevator is essential.

In order for an indoor mobile robot to board or get off an elevator, it must first be able to recognize an indoor space such as an elevator, and must be able to estimate a pose and plan a route by itself.

In an indoor environment such as an elevator, environmental obstacles located at all times are mainly composed of simple obstacles such as walls, and the space is relatively narrow and closed compared to a general indoor environment.

In addition, in an indoor environment such as an elevator, distortion of spatial information frequently occurs due to non-environmental obstacles temporarily located such as people or luggage.

In general, in an indoor environment such as an elevator, a global positioning system (GPS) cannot be used, and it mainly relies on an image and a distance sensor mounted on a robot to recognize the environment locally and estimate a posture.

The grid map, which is a method mainly used by indoor mobile robots to recognize the indoor environment, divides the space into small grids, and uses a map that shows the degree to which each grid is occupied by obstacles. It's a way to recognize.

In an indoor space such as an elevator, since environmental obstacles are generally obscured from the robot's field of view by non-environmental obstacles, it is difficult for the robot to properly recognize the environment using only a simple space occupancy value by a grid map.

Korean Patent Registration [10-0877071] discloses a particle filter-based method and apparatus for estimating a posture of a mobile robot.

Korean Patent Registration [10-0877071] (Registration Date: December 26, 2008)

Accordingly, the present invention was conceived to solve the above-described problems, and an object of the present invention is to provide a method for recognizing an environment required for an indoor mobile robot to board and alight an elevator, and the indoor mobile robot It provides an apparatus and method for recognizing, a recording medium in which a program for implementing the same is stored, and a computer program stored in the medium to implement the same.

Objects of the embodiments of the present invention are not limited to the above-mentioned objects, and other objects not mentioned will be clearly understood by those of ordinary skill in the art from the following description. .

An apparatus for recognizing an environment in an elevator by an indoor mobile robot according to an embodiment of the present invention for achieving the above object includes: a marker 10 displayed on the inner floor of the elevator; A map storage unit 100 in which information related to the location of the inner wall of the elevator and a map of the elevator floor in which information related to the location of the markers 10 displayed on the inner floor of the elevator are stored are stored; A vision sensor 200 for obtaining image information of the front floor in a specific range from the robot; Based on the map of the elevator floor stored in the map storage unit 100, a marker 10 is determined among the image information obtained from the vision sensor 200, and the unoccupied floor including the determined marker 10 An image processing unit 300 for generating or updating a map of the elevator floor for; A posture estimating unit 400 for estimating the posture of the robot based on a map of the elevator floor generated or updated by the image processing unit 300; Based on the map of the floor of the elevator created or updated by the image processing unit 300 and the attitude of the robot estimated by the attitude estimation unit 400, after checking whether there is a target area that the robot can reach, A route planning unit 500 that determines a point and plans a moving route to the target point; And a robot control unit 600 for controlling the robot to move along the movement path generated by the path planning unit 500.

In addition, the marker 10 is characterized in that any one or more of an Arco marker type, a checker board type, and a color pattern type is used.

In addition, the image processing unit 300 acquires the areas of the markers 10 that are completely identified for the elevator floor, and then compares the previously stored information on the elevator floor to compare the previously stored unit blocks, that is, the robot It is characterized by distinguishing only the movable area.

In addition, the attitude estimating unit 400 calculates the coordinates of the vision sensor 200 in the elevator's internal map coordinate system.

는 엘리베이터 내부 지도 좌표계에서 비전센서의 좌표,

은 3차원 회전변환 행렬,

는 이동변환 벡터)과 같이 구하고, (here,

Is the coordinate of the vision sensor in the map coordinate system inside the elevator,

Is a three-dimensional rotation transformation matrix,

Is obtained as a movement transformation vector),

), 피치(pitch,

) 및 요(yaw,

)를 다음식The direction information of the vision sensor, roll,

), pitch,

) And yaw,

) To the following expression

는 비전센서 좌표계의 y축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이고,

는 비전센서 좌표계의 z축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이고,

,

,

는 각각 비전센서 좌표계의 x축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 x, y, z축 성분이고,

은 역사인(arcsine) 함수이다)과 같이 구하여, , 로봇의 자세를 계산하는 것을 특징으로 한다.(Where Atan2 is an arctangent function that takes two arguments,

Is the unit vector in the y-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

Is the unit vector in the z-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

,

,

Is the unit vector in the x-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the x, y, z axis component of,

Is an arcsine function) and calculates the robot's posture.

In addition, the robot control unit 600 is characterized in that when the target area and the movement path are not secured, the robot returns to the starting point or controls to wait in place.

The method for the indoor mobile robot to recognize the environment in an elevator according to an embodiment of the present invention is a method for the indoor mobile robot to recognize the environment in the elevator in the form of a program executed by an operation processing means including a computer. In the floor sensing step (S10) of obtaining image information of the front floor of a specific range from the robot with the vision sensor 200; Markers among the image information obtained from the floor sensing step (S10) are based on a map of the elevator floor in which the information related to the location of the interior wall of the elevator and the markers 10 displayed on the interior floor of the elevator are stored. 10) determining a floor map acquisition step (S20) of generating or updating a map of the elevator floor for an unoccupied floor including the determined marker 10; Based on the map of the elevator floor updated from the floor map acquisition step (S20), the attitude of the robot is estimated, and after checking whether there is a target area that can be reached by the robot, a target point that can be reached is determined, and the moving path to the target point It characterized in that it comprises a; moving route planning step (S30) to plan.

In addition, the floor map acquisition step (S20) acquires the areas of the completely identified markers 10 for the elevator floor, and then compares the previously stored information on the elevator floors, that is, the aggregation area of unoccupied unit blocks, that is, It is characterized in that only the area in which the robot can move is divided.

In addition, in the movement route planning step (S30), the coordinates of the vision sensor 200 in the elevator internal map coordinate system are calculated as follows:

는 엘리베이터 내부 지도 좌표계에서 비전센서의 좌표,

은 3차원 회전변환 행렬,

는 이동변환 벡터)과 같이 구하고, (here,

Is the coordinate of the vision sensor in the map coordinate system inside the elevator,

Is a three-dimensional rotation transformation matrix,

Is obtained as a movement transformation vector),

), 피치(pitch,

) 및 요(yaw,

)를 다음식The direction information of the vision sensor, roll,

), pitch,

) And yaw,

) To the following expression

는 비전센서 좌표계의 y축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이고,

는 비전센서 좌표계의 z축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이고,

,

,

는 각각 비전센서 좌표계의 x축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 x, y, z축 성분이고,

은 역사인(arcsine) 함수이다)과 같이 구하여, 로봇의 자세를 계산하는 것을 특징으로 한다.(Where Atan2 is an arctangent function that takes two arguments,

Is the unit vector in the y-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

Is the unit vector in the z-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

,

,

Is the unit vector in the x-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the x, y, z axis component of,

Is an arcsine function) and calculates the robot's posture.

In addition, according to an embodiment of the present invention, a computer-readable recording medium in which a program for implementing a method for the indoor mobile robot to recognize an environment in an elevator is provided is provided.

In addition, according to an embodiment of the present invention, in order to implement a method for the indoor mobile robot to recognize an environment in an elevator, a program stored in a computer-readable recording medium is provided.

According to an apparatus and method for recognizing an environment in an elevator by an indoor mobile robot according to an embodiment of the present invention, a recording medium storing a program for implementing the same, and a computer program stored in the medium to implement the same, There is an effect of estimating the attitude of the robot based on the markers of the image and planning the movement path of the robot, thereby providing the necessary environment for the robot to board and alight the elevator autonomously.

In addition, by using simple markers such as the shape of the Aruco marker, the shape of the checker board, and the shape of a color pattern, there is an effect of reducing the amount of computation required for posture estimation and path planning.

In addition, by dividing the collective area of unoccupied unit blocks into an area in which the robot can move, in addition to environmental obstacles such as walls displayed on the elevator interior map, non-environmental obstacles such as people or boxes not displayed on the elevator interior map are prevented. Even if it exists, it is possible to get on and off the elevator, and there is an effect of further reducing the amount of computation required for this.

In addition, by using a vision sensor, if the lidar sensor is used due to the characteristics of the elevator interior (a space surrounded by a metal material with high reflectivity), it is effective to solve the problem that many errors occur in detecting objects. .

In addition, by estimating the posture on a two-dimensional coordinate system, there is an effect of minimizing the amount of computation required for posture estimation.

In addition, even if the robot fails to board the elevator, it is possible to return to the boarding waiting point (departure point) and wait, thereby enabling more stable elevator boarding.

1 is a block diagram of an apparatus for an indoor mobile robot to recognize an environment in an elevator according to an embodiment of the present invention.
FIG. 2 is an exemplary view showing an example of an elevator door being opened in a state in which a robot is waiting at an elevator boarding waiting point (departure point), and the floor inside the elevator is visible.
3 is an exemplary view showing an example of comparing the coordinate system of the vision sensor of the robot and the coordinate system of the elevator interior map.
4 is an exemplary view showing an example of a case in which an Aruco marker is installed on the floor of an elevator.
5 is an exemplary view showing an example of a case where a checker board is installed on the floor of an elevator.
6 is an exemplary view showing an example of a case in which a color pattern is installed on the floor of an elevator.
7 is an exemplary view showing an example of a map of the elevator floor according to an embodiment of the present invention.
8 is an exemplary view showing an example in which a robot sets a primary target point and plans a route according to an embodiment of the present invention.
9 is an exemplary view showing an example in which a robot sets a second target point from a first target point and plans a route according to an embodiment of the present invention.
10 is an exemplary view showing an example in which a map of an elevator floor according to an embodiment of the present invention has different sizes of markers according to the importance of an interior space.
11 is a flowchart of a method for an indoor mobile robot to recognize an environment in an elevator according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. Should be.

On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle.

The terms used in the present specification are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, processes, operations, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the possibility of addition or presence of elements or numbers, processes, operations, components, parts, or combinations thereof is not preliminarily excluded.

Unless otherwise defined, all terms including technical or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings. Prior to this, terms or words used in the specification and claims should not be construed as being limited to their usual or dictionary meanings, and the inventors appropriately explain the concept of terms in order to describe their own invention in the best way Based on the principle that it can be defined, it should be interpreted as a meaning and concept consistent with the technical idea of the present invention. In addition, unless there are other definitions in the technical terms and scientific terms used, they have the meanings commonly understood by those of ordinary skill in the art to which this invention belongs, and the gist of the present invention in the following description and accompanying drawings Description of known functions and configurations that may be unnecessarily obscure will be omitted. The drawings introduced below are provided as examples in order to sufficiently convey the spirit of the present invention to those skilled in the art. Accordingly, the present invention is not limited to the drawings presented below and may be embodied in other forms. In addition, the same reference numbers throughout the specification indicate the same elements. It should be noted that the same elements in the drawings are indicated by the same reference numerals wherever possible.

1 is a block diagram of an apparatus for recognizing an environment in an elevator by an indoor mobile robot according to an embodiment of the present invention, and FIG. 2 is a block diagram of an elevator door while the robot waits at an elevator boarding waiting point (departure point). It is an example view showing an example where the floor inside the elevator is open, and FIG. 3 is an example view showing an example of comparing the coordinate system of the vision sensor of the robot and the coordinate system of the elevator interior map, and FIG. 4 is an Arco on the floor of the elevator. Aruco) is an exemplary diagram showing an example of a case where the shape of a marker is installed, FIG. 5 is an exemplary view showing an example of a case where the shape of a checker board is installed on the floor of an elevator, and FIG. 6 is a color pattern on the floor of the elevator. It is an exemplary view showing an example of a case in which the form is installed, FIG. 7 is an exemplary view showing an example of a map of an elevator floor according to an embodiment of the present invention, and FIG. 8 is an exemplary view showing an example of an elevator floor map according to an embodiment of the present invention. Accordingly, it is an exemplary diagram showing an example in which a robot sets a primary target point and plans a route, and FIG. 9 is a diagram in which the robot sets a secondary target point from a primary target point and plans a route according to an embodiment of the present invention. It is an exemplary view showing an example, and FIG. 10 is an exemplary diagram showing an example in which the size of a marker is changed according to the importance of an interior space in a map of an elevator floor according to an embodiment of the present invention, and FIG. 11 is an embodiment of the present invention. It is a flowchart of a method for an indoor mobile robot according to an example to recognize an environment in an elevator.

An apparatus and method for the indoor mobile robot according to an embodiment of the present invention to recognize the environment in an elevator, a recording medium storing a program for implementing the same, and a computer program stored in the medium to implement the same, the indoor mobile robot rides the elevator. And by providing a technology that recognizes the environment required to get off, the indoor mobile robot checks whether there is space to board the elevator by itself, estimates its location, and moves by planning a route, so that it can get on and off the elevator by itself. You can do it.

For convenience of calculation, the following can be assumed.

It can be assumed that the door of the elevator does not close while the indoor mobile robot is getting on or off the elevator.

That is, the elevator knows whether the indoor mobile robot is attempting to board or get off, and can know whether the attempt is successful.

Indoor mobile robots can detect elevator doors fully open.

That is, the indoor mobile robot can be detected using a sensor, but it is possible to accurately know the operation of the elevator in the same way as radio communication, and the elevator can also accurately know the operation of the robot.

When the indoor mobile robot boards the elevator, it can be assumed that the vision sensor 10 installed in the robot is waiting in a direction facing the front of the elevator.

The method of moving the mobile robot to the elevator boarding standby position in advance may be manually manipulated and moved, or autonomous driving may be moved by an algorithm without human manipulation.

The method of moving the mobile robot to the elevator boarding standby position is not covered in the present invention.

That is, it is assumed that the mobile robot moves to the elevator boarding standby position to achieve various services and purposes.

The boarding standby position may be a final target point when the mobile robot gets off the elevator.

As shown in Fig. 1, an apparatus for recognizing an environment in an elevator by an indoor mobile robot according to an embodiment of the present invention includes a marker 10, a map storage unit 100, a vision sensor 200, an image processing unit ( 300), a posture estimating unit 400, a path planning unit 500, and a robot control unit 600.

Markers 10 are displayed on the inner floor of the elevator (see Figs. 2 to 3).

On the floor of the elevator, there is a marker 10 that can be recognized by the vision sensor 200 of the mobile robot.

The marker 10 may be a specific mark or pattern that can be recognized.

In addition, the marker 10 may be disposed on the entire inner floor of the elevator, but may be disposed according to a predetermined rule.

Further, in the marker 10, a plurality of different types of markers may be arranged according to a predetermined rule.

For example, the shape of the checkerboard and the shape of a QR code (Quick Response code) may be arranged together.

In this case, the QR code may contain information related to the characteristics of the elevator, such as the size and shape of the elevator.

The map storage unit 100 has information (coordinates, features, sizes, etc.) related to the location of the interior walls of the elevator and information (coordinates, features, sizes, etc.) related to the locations of the markers 10 displayed on the interior floor of the elevator. A map of the elevator floor is saved.

The map storage unit 100 stores a map of the elevator floor, which is basic data used to estimate the attitude of the robot and plan a moving route, and the map of the elevator floor has information related to the location of the marker 10. .

The vision sensor 200 acquires image information of the front floor in a specific range from the robot.

The mobile robot may collect image data using the vision sensor 10 installed in the mobile robot.

In this case, it is preferable to set the front of the robot in a specific range from the bottom vertically below the robot as the sensing area. This is to be prepared in case there is an obstacle in front of the robot.

The image processing unit 300 determines the marker 10 among the image information obtained from the vision sensor 200 based on the map of the elevator floor stored in the map storage unit 100, and the determined marker 10 is Create or update a map of the elevator floor for the included unoccupied floor.

With respect to the marker 10 installed on the floor of the elevator, the mobile robot acquires a 2D image with the vision sensor 200, which can be expressed in a coordinate system in the form of FIG. 3.

Taking the process of generating or updating the elevator floor map as an example,

With respect to the image acquired from the vision sensor 200, the boundary line of the left/right wall is extracted through an image processing process (refer to Figs. 4 to 6(c)), and the boundary line of the floor for the unoccupied floor is extracted. (Refer to (d) of FIGS. 4 to 6), a value for the occupancy of the floor can be obtained.

The mobile robot may estimate its own position while boarding the elevator in the same manner as in FIG. 8.

By using this, it is possible to know the space occupancy status of non-environmental obstacles to the space inside the elevator.

Non-environmental obstacles are not obstacles (environmental obstacles) fixedly arranged in elevators, but are moving obstacles such as people.

FIG. 7 shows an example of initially generating a map of the elevator floor, and FIG. 8 shows a map of the elevator floor updated while the robot moves.

That is, the image processing unit 300 may obtain an image inside the elevator including the marker 10 and generate (convert) the elevator interior map as shown in FIG. 7, and generate (convert) the elevator interior map according to the movement of the mobile robot. The target area can be updated while updating as shown in FIG. 8.

The generated or updated elevator floor map can be stored and managed in a separate storage space.

The posture estimating unit 400 estimates the posture of the robot based on the map of the elevator floor generated or updated by the image processing unit 300.

The indoor mobile robot may estimate a posture including its own position and direction based on the coordinate system of the elevator interior map from the marker.

The route planning unit 500 is based on the map of the elevator floor created or updated by the image processing unit 300 and the attitude of the robot estimated by the attitude estimation unit 400, and the target area that the robot can reach is After confirming that there is, it decides the target point that can be reached and plans the route to the target point.

The target point means a point at which the robot can board or get off the elevator avoiding the obstacle, and a target area is required for this.

The robot control unit 600 controls the robot to move along the movement path generated by the path planning unit 500.

As shown in Fig. 1, the marker 10 of the device for recognizing the environment in the elevator by the indoor mobile robot according to an embodiment of the present invention is in the form of an Aruco marker, a form of a checker board, and a color pattern. It may be characterized in that any one or a plurality of forms are used.

The marker 10 is an artificial mark made in a certain format and can be used for tracking.

The Aruco marker is composed of a two-dimensional bit pattern of size n x n and a black border region surrounding it. The black border area is for quick recognition of the marker, and the inner two-dimensional bit pattern is used to identify the marker by expressing the unique ID of the marker by a combination of white cells and black cells (see Fig. 4).

The shape of the checker board refers to a shape in which two colors (white, black, etc.) are alternately arranged in a grid-like arrangement (see Fig. 5).

The shape of the color pattern refers to a shape in which various colors (red, blue, green, yellow, etc.) are alternately arranged in a grid-like arrangement (see Fig. 6).

Examples of the various markers 10 have been described above, but the present invention is not limited thereto, and the ChAruco board in which an Aruco marker is inserted in the bright color (white, blank, etc.) of the checker board It goes without saying that any markers 10 capable of tracking such as shape can be applied.

As shown in FIGS. 4 to 6, the image processing unit 300 of the apparatus for recognizing the environment in the elevator by the indoor mobile robot according to an embodiment of the present invention includes a marker 10 completely identified for the elevator floor. After acquiring the area of the elevator, it may be characterized in that only the aggregate area of unoccupied unit blocks, that is, the area in which the robot can move, is distinguished by comparing previously stored information on the elevator floor.

In the case of Aruco markers, the acquired image is first converted into a binary image using a threshold value, including Canny edge detection or local Various methods, such as a local adaptive thresholding approach, can be used, but are not limited thereto.

The contours of the converted binary images are extracted, and various methods such as the method of using the Sobel filter and the topological structural analysis of digitized binary images by border following are used. Can, but is not limited to this.

From the extracted contour image, a candidate group position of a marker having a rectangular shape can be found using the Douglas-Peucker algorithm.

In addition to the Douglas-Peker algorithm, various methods such as Harris corner detection can be used.

Here, it is desirable to exclude squares that are not of an appropriate size from the candidate group of markers that have a square shape that are shown at the position where the robot is waiting to take the elevator. In other words, it is desirable to exclude squares other than predefined markers.

Here, the appropriate size means the size of a rectangle that can be detected as a marker when looking at the marker on the floor inside the elevator from the position where the robot is waiting to take the elevator, and may be determined in advance.

In the original image, the perspective projection effect was removed using a homography matrix for the portion corresponding to the candidate group of markers composed of squares having appropriate sizes, and the floor marker of the elevator was vertically overlooked. It can be converted into a rectangular shape when viewed.

The internal marker image obtained in the above step is made into a binary image using Otsu's binarization method, and divided into grids at regular intervals, based on the inner color of each grid, black is 0, and white is Can be converted to 1.

In the converted data, by referring to the combination of numbers excluding the border part converted to 0, it is checked whether the marker belongs to the marker dictionary of the robot, and markers that do not belong to the marker dictionary of the robot are excluded. Area can be determined.

Here, if there is a non-zero part in the border information, it is preferable to regard the corresponding part as an obstacle, not a marker.

In the case of the checkerboard shape, it is similar to the case in the shape of an Aruco marker.

First, a candidate group location of a marker having a square shape is found in the same way.

Then, the areas where the interior color is filled with black or not filled with white are excluded.

In addition, when a white square does not appear at the top, bottom, left, and right positions of the black square, or a black square does not come at the top, bottom, left, and right positions of the white square, the corresponding area is regarded as an area other than a marker, and areas of complete markers are determined.

In the case of the shape of the color pattern, it proceeds similarly to the case of the shape of an Aruco marker.

First, a candidate group of markers having a square shape is found in the same way.

Then, it checks whether the color inside the robot is filled with the color registered in advance of the robot, and determines whether the designated color is found in the upper, lower, left, and right of the corresponding color area to determine the complete marker area.

Through the above method, the area of complete markers for the elevator floor is obtained, and then the previously stored information on the elevator floor is compared to identify only the unoccupied part, that is, the area in which the mobile robot can move.

In addition, even though it is an area that should be classified as a marker by comparing information on the previously stored elevator floor, the area determined as a non-marker area can be determined as an area where the marker is occupied or invisible by non-environmental obstacles. have.

The boundary line between the elevator floor and the wall can be found through comparison between the previously stored elevator floor information and the acquired image.

For example, among the markers of the elevator floor detected from the acquired image, information on the markers corresponding to the left and right ends may be derived, and then information on the previously stored elevator floor may be compared to determine the interior wall of the elevator. .

As an example, referring to FIG. 4, if a reference marker as shown in (a) is installed on the floor of an elevator at regular intervals, and a non-environmental obstacle is located as shown in (b), through the image processing process of the mobile robot ( The left and right walls of the elevator, such as the blue line in c), can be detected, and the unoccupied space can be detected as in (d).

In the above example, the unoccupied space means a set of parts in which the reference marker is completely recognized.

In the above example, a portion where the reference marker is not recognized may be regarded as an occupied space.

As another embodiment, referring to FIG. 5, if the checker board as shown in (a) is installed and non-environmental obstacles are located as shown in (b), the blue line and the blue line in (c) through the image processing process of the mobile robot The same elevator left and right walls can be detected, and unoccupied space can be detected as shown in (d).

In the above example, the unoccupied space means a set of parts in which the checker is recognized as a complete rectangle.

If the shape of the checker detected in the above example is not a rectangle, it can be regarded as an occupied space.

As another embodiment, referring to FIG. 6, if the color pattern as shown in (a) is installed and the non-environmental obstacle is located as shown in (b), the blue line and the blue line in (c) through the image processing process of the mobile robot The same elevator left and right walls can be detected, and unoccupied space can be detected as shown in (d).

In the above example, the unoccupied space refers to a set of spaces in which only one predefined color is detected at each position of the rectangular color pattern.

In the above example, if the space detected by one color is not a square, it can be regarded as an occupied space.

As shown in Figure 1, the posture estimation unit 400 of the device for recognizing the environment in the elevator indoor mobile robot according to an embodiment of the present invention

The coordinates of the vision sensor 200 in the elevator's internal map coordinate system are calculated as follows:

는 엘리베이터 내부 지도 좌표계에서 비전센서의 좌표,

은 3차원 회전변환 행렬,

는 이동변환 벡터)(here,

Is the coordinate of the vision sensor in the map coordinate system inside the elevator,

Is a three-dimensional rotation transformation matrix,

Is the shift transformation vector)

Obtain as,

), 피치(pitch,

) 및 요(yaw,

)를 다음식The direction information of the vision sensor, roll,

), pitch,

) And yaw,

) To the following expression

는 비전센서 좌표계의 y축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이고,

는 비전센서 좌표계의 z축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이고,

,

,

는 각각 비전센서 좌표계의 x축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 x, y, z축 성분이고,

은 역사인(arcsine) 함수이다)(Where Atan2 is an arctangent function that takes two arguments,

Is the unit vector in the y-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

Is the unit vector in the z-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

,

,

Is the unit vector in the x-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the x, y, z axis component of,

Is an arcsine function)

It can be obtained as described above and characterized in that the posture of the robot is calculated.

The map storage unit 100 may store an elevator interior map in the form of FIG. 7.

The elevator interior map may display the interior wall of the elevator and the coordinates where each marker is located.

It is possible to have the coordinates of the markers located in the space in which the robot can move based on the coordinate system of the elevator interior map that the robot has in advance.

This can be obtained by artificially removing non-environmental obstacles in the elevator in advance, placing the robot at the starting position for boarding the elevator, acquiring vision sensor data, and utilizing camera calibration.

As shown in Fig. 3, assume that a point A on the marker on the floor of the elevator is located at (x, y, z) based on the coordinate system inside the elevator.

It is assumed that a point A'on the image measured by the robot's vision sensor for the point A is in (a, b, c) based on the three-dimensional coordinate system, that is, the vision sensor coordinate system, with the position of the vision sensor as the origin. Suppose that, is located at (m, n) in the 2D image coordinate system of the acquired image.

The following relationship is established between the coordinates (m, n) and the coordinates (a, b, c).

는 비전센서의 초점 거리이고,

와

는 각각 비전센서의 한 픽셀의 x와 y방향 크기를 의미하며,

는 비대칭(skewness) 계수이며

로써 구해진다.here

Is the focal length of the vision sensor,

Wow

Is the size of one pixel of the vision sensor in the x and y directions, respectively,

Is the coefficient of skewness

It is obtained by

는 비전센서의 y축이 x축에 대해 기울어진 정도를 각도로 표현한 값이다.here

Is a value expressed as an angle of the degree of inclination of the y-axis of the vision sensor with respect to the x-axis.

와

는 각각 비전센서의 광축과 영상평면이 만나는 주점(principal point)의 x와 y방향 좌표를 의미한다.

Wow

Denotes the x- and y-direction coordinates of the principal point where the optical axis of the vision sensor and the image plane meet, respectively.

,

,

,

,

,

은 방사 왜곡에 대한 계수이고,

과

는 접선 왜곡에 대한 계수이고,

이다. Also,

,

,

,

,

,

Is the coefficient for radiation distortion,

and

Is the coefficient for tangential distortion,

to be.

Through the above relational equation, a point (a, b, c) in the vision sensor coordinate system is expressed as a point A'in the two-dimensional coordinate system of the image acquired by the vision sensor.

The following relationship is established between the coordinates (a, b, c) and the coordinates (x, y, z).

은 3차원 회전변환 행렬이고,

는 이동변환 벡터(vector)이다. here,

Is the three-dimensional rotation transformation matrix,

Is the movement transformation vector.

과

를 구할 수 있다.Assuming that the markers have a square shape as shown in Figs. 4 to 6, using the relational equations, information on four vertices of the marker outline, and the Levenberg-Marquardt algorithm

and

Can be obtained.

는 비전센서 좌표축 기준으로 (0, 0, 0)이기 때문에 아래와 같이 구해진다. The coordinates of the vision sensor in the map coordinate system inside the elevator

Is (0, 0, 0) based on the coordinate axis of the vision sensor, so it is calculated as follows.

), 피치(pitch,

)와 요(yaw,

)는 다음과 같이 구해질 수 있다.-Roll, which is the direction information of the vision sensor

), pitch,

) And yaw,

) Can be obtained as

은 역사인(arcsine) 함수이다.-Where Atan2 is an arctangent function that takes two arguments,

Is an arcsine function.

는 비전센서 좌표계의 y축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이다.- here

Is the unit vector in the y-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of

는 비전센서 좌표계의 z축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이고- here

Is the unit vector in the z-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of

,

,

는 각각 비전센서 좌표계의 x축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 x, y, z축 성분이다.- here

,

,

Is the unit vector in the x-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the x, y, z axis component of.

-Through the above process, the coordinates of the vision sensor can be obtained from the elevator's internal map coordinate system, and through this, the posture of the mobile robot can be calculated.

In this embodiment, pitch information and roll information of the vision sensor are not used, but if a three-dimensional map is used as an elevator interior map, pitch information and roll information may be required.

After acquiring an image with a vision sensor and processing the image for marker recognition, the area in which the intact marker is found may be determined as a space in which the robot can move.

After completing the recognition of the marker, the map is updated as shown in FIG. 8.

That is, the elevator interior map as shown in FIG. 8 is generated in real time.

In other words, the marker of the 2D image acquired by the vision sensor and the marker on the elevator interior map of the robot can be matched one-to-one and expressed as coordinates.

Here, in the case of an Aruco marker, in matching markers, unique identification information (ID) of each marker may be used.

If it is in the form of a checker board, it can be matched by using the actual size of the elevator, the actual size information of each marker, and scale information that is reduced or enlarged in the image.

If it is in the form of a color pattern, it may be matched by using unique color information and color patterns arranged up, down, left and right.

That is, when the robot identifies the marker 10 on the floor inside the elevator, the area where the robot can go and the area that cannot be reached can be distinguished in the map of FIG. 8 on the elevator map.

The robot control unit 600 of the apparatus for recognizing the environment in the elevator by the indoor mobile robot according to an embodiment of the present invention controls the robot to return to the starting point or to wait in place if the target area and the movement path are not secured. It can be characterized.

If there is no space to board the elevator inside, you have to wait for the next.

If the robot is boarding inside the elevator (if a part of the robot is located inside the elevator), it is desirable to return to the starting point if it is determined that the robot cannot board the elevator. When the elevator door is opened, the robot will board the elevator. If it is determined that it is not possible, it is desirable to wait in place.

As shown in Fig. 10, the marker 10 of the device for recognizing the environment in the elevator by the indoor mobile robot according to an embodiment of the present invention is arranged by varying its size (resolution) according to the importance of the interior floor of the elevator. It is also possible to become.

This is to reduce the amount of calculation while allowing precise control by increasing the resolution of only important parts.

In other words, the purpose is not to accurately calculate all areas, but to more accurately calculate areas of high importance such as boundary areas and edges.

For example, the size of the marker 10 may decrease as it goes toward the inner floor rim of the elevator.

In addition, it goes without saying that any marker 10 can be used as long as the posture of the robot can be estimated through the marker 10.

As shown in FIG. 11, in the method for the indoor mobile robot to recognize the environment in an elevator according to an embodiment of the present invention, an indoor mobile robot in the form of a program executed by an arithmetic processing means including a computer is used in the elevator. A method for recognizing an environment includes a floor sensing step (S10), a floor map acquisition step (S20), and a moving route planning step (S30).

In the floor sensing step S10, the vision sensor 200 acquires image information of the front floor of a specific range from the robot.

The vision sensor 200 may periodically or in real time acquire image information of the front floor in a specific range from the robot.

That is, the floor detection step (S10) is performed periodically or in real time.

First of all, the robot can start an algorithm to get on the elevator in the direction of initially looking at the elevator inside.

That is, when the marker 10 is detected among the image information acquired by the vision sensor 200, the floor map acquisition step (S20) described later may be performed.

In the floor map acquisition step (S20), the floor detection step (S10) is based on a map of the elevator floor in which information related to the location of the interior wall of the elevator and information related to the location of the markers 10 displayed on the interior floor of the elevator are stored. The marker 10 is determined among the image information obtained from and a map of the elevator floor for the unoccupied floor including the determined marker 10 is generated or updated.

In the floor map acquisition step (S20), the mobile robot acquires an image with the vision sensor 200 at the elevator boarding standby position, and searches for the marker 10 from the acquired image.

Information on the actual size and position of the marker 10 and the size and position of the marker 10 expressed as pixels within the image recognized at the initial starting point of the corresponding robot are stored in advance, and the operation processing means It is desirable to be able to use it.

The movement route planning step (S30) estimates the attitude of the robot based on the map of the elevator floor updated from the floor map acquisition step (S20), checks whether there is a target area that the robot can reach, and then determines the reachable target point. Determine and plan the route to the target point.

If the occupancy of the walls and the floor of the elevator is detected in the floor map acquisition step (S20), in the movement route planning step (S30), it searches whether there is a reachable target area and a target point in consideration of the size of the mobile robot. You can plan a route that is there.

If the target area to which the target point belongs is a space enough for the robot to go, a path can be planned to that area.

If the target area and the movement path are secured, the robot can be moved after estimating the pose of the current mobile robot and generating the movement path.

That is, it is possible to create a path to a target point of a corresponding target area and start moving.

If the target area and route of movement are not secured, it can be terminated and returned to the starting point or wait on the spot.

The moving path can create a path that can move to the end at once, or can be configured with a finite number of paths.

In the case of generating movement paths with a finite number of paths, it is possible to determine whether or not to arrive at the target point of the target area whenever the local target point is reached.

If it arrives at the target point of the target area, the process is terminated. If not, the process returns to the step of acquiring the image again and the above process may be repeated.

If the target point of the target area is not reached and ends, there may be a method of returning to the initial point.

Here, a general autonomous driving algorithm can be used to create a route.

For example, global route planning algorithms such as A* algorithm and Dijkstra algorithm, and local route planning algorithms such as Dynamic Window Approach (DWA) algorithm and Timed Elastic Band (TEB) algorithm can be used.

The present invention is to provide an environment recognition method and apparatus for autonomous driving, not an autonomous driving algorithm.

When moving and estimating the robot's posture (position and direction), there may be a method of using a particle filter.

There may be other methods for estimating the self- posture, but the present invention is not limited thereto, and the present invention provides a method for recognizing the environment.

The robot can recognize its current posture as follows using its driving information and previous posture.

는 시간

에서 로봇의 자세,

은 시간

에서 로봇의 자세,

는 시간

에서

까지의 로봇의 자세변화량,

는 시간

에서 자세변화를 감지했을 때의 노이즈이다. here

The time

In the robot's pose,

Silver time

In the robot's pose,

The time

in

The amount of change in the robot's posture up to,

The time

This is the noise when the posture change is detected.

에서 로봇의 자세 변화량

는 자이로 센서 및 모터의 엔코더 값을 통해 알 수 있다. Specific time

The amount of change in the robot's posture

Can be known through the gyro sensor and the encoder value of the motor.

로 표현한다. However, due to non-ideal environments such as slip on the floor or unevenness of the floor, it is difficult to provide accurate information on the amount of change in the robot's posture.

Expressed as

An example of a particle filter-based magnetic posture estimation method using the environment recognition method presented in the present invention is as follows.

번째 로봇의 자세 후보

를 다수의(

) 파티클로 생성하고,

, 각 파티클들의 모션(motion)을 측정된 자세 변화량을 통해 업데이트한다.Particle filter

Robot posture candidate

A number of (

) Create particles,

, Update the motion of each particle through the measured posture change amount.

Then, after updating the weight of each particle using the robot's pose measured through the marker, the current pose is determined.

In a method of resampling the particles again using the weight information of each particle, the current posture is continuously estimated whenever a certain time and posture change.

Here, resampling is a process of excluding particles with a low weight and dividing particles with a high weight into several and updating N particles.

When N particles are updated, the resampling process is repeated by updating the motion and weight again.

Here, through the resampling process, the robot can converge to a set of particles having a high weight among particles expressed as candidates for their posture, thus reducing uncertainty resulting from posture estimation.

In the process of updating the weight of the particle filter, the environment recognition method of the present invention may be used.

를 추정한다.First, by using the information of the marker acquired from the actual robot,

Estimate

The attitude information of the robot can be calculated from the attitude of the vision sensor of the coordinate system inside the elevator.

Next, a weight may be calculated as follows so as to be inversely proportional to an error value obtained by comparing the posture information including the position and direction of each particle with the estimated posture information of the robot.

,

,

는 비례 계수이며,

를 만족하도록 설정된다.here

Is the proportionality factor,

Is set to satisfy.

In addition, after the weight update, the candidate having the highest weight can be regarded as the current location.

The robot continuously updates its position and plans and moves the path toward the target point in the target area.

If the robot enters the elevator, it may not be able to see the marker through the vision sensor.

In this case, it estimates its position depending on the encoder value while maintaining the most recent weight value.

If the marker is checked inside, the weight is updated again, and the process of resampling is repeated.

At this time, when entering the elevator, the blind spot that was not initially seen can be recognized by the vision sensor.

In this case, as shown in FIG. 9, the target area may be updated to a new point.

The criterion for updating the target area can be that a high weight is placed on the inside of the elevator, that is, in the y-axis direction.

That is, if it can move in the inward direction, it starts to move, and if it cannot go in the inward direction, it does not move.

Even if a new place is found by putting equal weights in the x-axis direction, there may be a method of waiting in the current target area without moving if the value of the y-axis is the same.

If the marker is not recognized before boarding the elevator, you can wait at that location until it is recognized or seen as a failure.

Using the present invention, it is possible for a mobile robot to enter a narrow space such as an elevator at low cost.

The floor map acquisition step (S20) of the method for the indoor mobile robot to recognize the environment in the elevator according to an embodiment of the present invention acquires the areas of the completely identified markers 10 for the elevator floor, and then By comparing information on the elevator floor, it may be characterized in that only a set area of unoccupied unit blocks, that is, an area in which the robot can move, is distinguished.

In the case of Aruco markers, the acquired image is first converted into a binary image using a threshold value, including Canny edge detection or local Various methods, such as a local adaptive thresholding approach, can be used, but are not limited thereto.

The contours of the converted binary images are extracted, and various methods such as the method of using the Sobel filter and the topological structural analysis of digitized binary images by border following are used. Can, but is not limited to this.

From the extracted contour image, a candidate group position of a marker having a rectangular shape can be found using the Douglas-Peucker algorithm.

In addition to the Douglas-Peker algorithm, various methods such as Harris corner detection can be used.

Here, it is desirable to exclude squares that are not appropriately sized markers that are displayed at the position where the robot is waiting to take the elevator. In other words, it is desirable to exclude areas that cannot be passed due to the size of the robot.

In the original image, the perspective projection effect is removed using a homography matrix for the part corresponding to the candidate group of markers, and the elevator floor marker is transformed into a rectangular shape when viewed vertically. I can.

Using Otsu's binarization method, the image inside the marker is converted into a binary image, divided into grids at regular intervals, converted to a number of 0 or 1, and then a combination of numbers excluding the border that is converted to 0 With reference to, it is possible to determine whether the marker is a marker stored in a dictionary of the robot, and exclude a marker that is not stored in a dictionary to determine an area of complete markers.

Here, if there is a non-zero part in the border information, it is preferable to regard the corresponding part as an obstacle, not a marker.

In the case of the checkerboard shape, it is similar to the case in the shape of an Aruco marker.

First, a candidate group location of a marker having a square shape is found in the same way.

Then, the areas where the interior color is filled with black or not filled with white are excluded.

In addition, when a white square does not appear at the top, bottom, left, and right positions of the black square, or a black square does not come at the top, bottom, left, and right positions of the white square, the corresponding area is regarded as an area other than a marker, and areas of complete markers are determined.

In the case of the shape of the color pattern, it proceeds similarly to the case of the shape of an Aruco marker.

First, a candidate group of markers having a square shape is found in the same way.

Then, it checks whether the color inside the robot is filled with the color registered in advance of the robot, and determines whether the designated color is found in the upper, lower, left, and right of the corresponding color area to determine the complete marker area.

Through the above method, the area of complete markers for the elevator floor is obtained, and then the previously stored information on the elevator floor is compared to identify only the unoccupied part, that is, the area in which the mobile robot can move.

The boundary line of the elevator floor can be found through a comparison between the previously stored elevator floor information and the acquired image.

For example, among the markers of the elevator floor detected from the acquired image, the information of the marker corresponding to the left and right ends is derived, and then the information on the previously stored elevator floor is compared to determine whether the information is at the end. I can.

As an example, referring to FIG. 4, if a reference marker as shown in (a) is installed on the floor of an elevator at regular intervals, and a non-environmental obstacle is located as shown in (b), through the image processing process of the mobile robot ( The left and right walls of the elevator, such as the blue line in c), can be detected, and the unoccupied space can be detected as in (d).

In the above example, the unoccupied space means a set of parts in which the reference marker is completely recognized.

In the above example, a portion where the reference marker is not recognized may be regarded as an occupied space.

As another embodiment, referring to FIG. 5, if the checker board as shown in (a) is installed and non-environmental obstacles are located as shown in (b), the blue line and the blue line in (c) through the image processing process of the mobile robot The same elevator left and right walls can be detected, and unoccupied space can be detected as shown in (d).

In the above example, the unoccupied space means a set of parts in which the checker is recognized as a complete rectangle.

If the shape of the checker detected in the above example is not a rectangle, it can be regarded as an occupied space.

As another embodiment, referring to FIG. 6, if the color pattern as shown in (a) is installed and the non-environmental obstacle is located as shown in (b), the blue line and the blue line in (c) through the image processing process of the mobile robot The same elevator left and right walls can be detected, and unoccupied space can be detected as shown in (d).

In the above example, the unoccupied space refers to a set of spaces in which only one predefined color is detected at each position of the rectangular color pattern.

In the above example, if the space detected by one color is not a square, it can be regarded as an occupied space.

In the method for the indoor mobile robot according to an embodiment of the present invention to recognize the environment in an elevator, the movement route planning step (S30) is

The coordinates of the vision sensor 200 in the elevator's internal map coordinate system are calculated as follows:

는 엘리베이터 내부 지도 좌표계에서 비전센서의 좌표,

은 3차원 회전변환 행렬,

는 이동변환 벡터)(here,

Is the coordinate of the vision sensor in the map coordinate system inside the elevator,

Is a three-dimensional rotation transformation matrix,

Is the shift transformation vector)

Obtain as,

), 피치(pitch,

) 및 요(yaw,

)를 다음식The direction information of the vision sensor, roll,

), pitch,

) And yaw,

) To the following expression

는 비전센서 좌표계의 y축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이고,

는 비전센서 좌표계의 z축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이고,

,

,

는 각각 비전센서 좌표계의 x축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 x, y, z축 성분이고,

은 역사인(arcsine) 함수이다)(Where Atan2 is an arctangent function that takes two arguments,

Is the unit vector in the y-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

Is the unit vector in the z-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

,

,

Is the unit vector in the x-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the x, y, z axis component of,

Is an arcsine function)

It can be obtained as described above and characterized in that the posture of the robot is calculated.

The map storage unit 100 may store an elevator interior map in the form of FIG. 7.

The elevator interior map may display the interior wall of the elevator and the coordinates where each marker is located.

It is possible to have the coordinates of the markers located in the space in which the robot can move based on the coordinate system of the elevator interior map that the robot has in advance.

This can be obtained by artificially removing non-environmental obstacles in the elevator in advance, placing the robot at the starting position for boarding the elevator, acquiring vision sensor data, and utilizing camera calibration.

As shown in Fig. 3, assume that a point A on the marker on the floor of the elevator is located at (x, y, z) based on the coordinate system inside the elevator.

It is assumed that a point A'on the image measured by the robot's vision sensor for the point A is in (a, b, c) based on the three-dimensional coordinate system, that is, the vision sensor coordinate system, with the position of the vision sensor as the origin. Suppose that, is located at (m, n) in the 2D image coordinate system of the acquired image.

The following relationship is established between the coordinates (m, n) and the coordinates (a, b, c).

는 비전센서의 초점 거리이고,

와

는 각각 비전센서의 한 픽셀의 x와 y방향 크기를 의미하며,

는 비대칭(skewness) 계수이며

로써 구해진다.here

Is the focal length of the vision sensor,

Wow

Is the size of one pixel of the vision sensor in the x and y directions, respectively,

Is the coefficient of skewness

It is obtained by

는 비전센서의 y축이 x축에 대해 기울어진 정도를 각도로 표현한 값이다.here

Is a value expressed as an angle of the degree of inclination of the y-axis of the vision sensor with respect to the x-axis.

와

는 각각 비전센서의 광축과 영상평면이 만나는 주점(principal point)의 x와 y방향 좌표를 의미한다.

Wow

Denotes the x- and y-direction coordinates of the principal point where the optical axis of the vision sensor and the image plane meet, respectively.

,

,

,

,

,

은 방사 왜곡에 대한 계수이고,

과

는 접선 왜곡에 대한 계수이고,

이다. Also,

,

,

,

,

,

Is the coefficient for radiation distortion,

and

Is the coefficient for tangential distortion,

to be.

Through the above relational equation, a point (a, b, c) in the vision sensor coordinate system is expressed as a point A'in the two-dimensional coordinate system of the image acquired by the vision sensor.

The following relationship is established between the coordinates (a, b, c) and the coordinates (x, y, z).

은 3차원 회전변환 행렬이고,

는 이동변환 벡터(vector)이다. here,

Is the three-dimensional rotation transformation matrix,

Is the movement transformation vector.

과

를 구할 수 있다.Assuming that the markers have a square shape as shown in Figs. 4 to 6, using the relational equations, information on four vertices of the marker outline, and the Levenberg-Marquardt algorithm

and

Can be obtained.

는 비전센서 좌표축 기준으로 (0, 0, 0)이기 때문에 아래와 같이 구해진다. The coordinates of the vision sensor in the map coordinate system inside the elevator

Is (0, 0, 0) based on the coordinate axis of the vision sensor, so it is calculated as follows.

), 피치(pitch,

)와 요(yaw,

)는 다음과 같이 구해질 수 있다.-Roll, which is the direction information of the vision sensor

), pitch,

) And yaw,

) Can be obtained as

은 역사인(arcsine) 함수이다.-Where Atan2 is an arctangent function that takes two arguments,

Is an arcsine function.

는 비전센서 좌표계의 y축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이다.- here

Is the unit vector in the y-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of

는 비전센서 좌표계의 z축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 z축 성분이고- here

Is the unit vector in the z-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of

,

,

는 각각 비전센서 좌표계의 x축 방향 단위 벡터

를 엘리베이터 내부 지도 좌표계로 변환한 벡터

의 x, y, z축 성분이다.- here

,

,

Is the unit vector in the x-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the x, y, z axis component of.

-Through the above process, the coordinates of the vision sensor can be obtained from the elevator's internal map coordinate system, and through this, the posture of the mobile robot can be calculated.

In this embodiment, pitch information and roll information of the vision sensor are not used, but if a three-dimensional map is used as an elevator interior map, pitch information and roll information may be required.

After acquiring an image with a vision sensor and processing the image for marker recognition, the area in which the intact marker is found may be determined as a space in which the robot can move.

After completing the recognition of the marker, the map is updated as shown in FIG. 8.

That is, the elevator interior map as shown in FIG. 8 is generated in real time.

In other words, the marker of the 2D image acquired by the vision sensor and the marker on the elevator interior map of the robot can be matched one-to-one and expressed as coordinates.

Here, in the case of an Aruco marker, in matching markers, unique identification information (ID) of each marker may be used.

If it is in the form of a checker board, it can be matched by using the actual size of the elevator, the actual size information of each marker, and scale information that is reduced or enlarged in the image.

If it is in the form of a color pattern, it may be matched by using unique color information and color patterns arranged up, down, left and right.

That is, when the robot identifies the marker 10 on the floor inside the elevator, the area where the robot can go and the area that cannot be reached can be distinguished in the map of FIG. 8 on the elevator map.

In the above, a method for the indoor mobile robot according to an embodiment of the present invention to recognize the environment in an elevator has been described, but a computer-readable record storing a program for implementing a method for the indoor mobile robot to recognize the environment in an elevator Of course, a program stored in a computer-readable recording medium for implementing a method for the medium and indoor mobile robot to recognize the environment in an elevator can also be implemented.

That is, the above-described method for the indoor mobile robot to recognize the environment in an elevator is tangibly implemented by a program of instructions for implementing it, so that it is easy for those skilled in the art to be included in a recording medium that can be read through a computer You can understand. In other words, it is implemented in the form of program instructions that can be executed through various computer means, and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include a program command, a data file, a data structure, or the like alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and constructed for the present invention, or may be known and usable to those skilled in computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and floptical disks. Magneto-optical media and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, USB memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

The present invention is not limited to the above-described embodiments, and of course, various modifications can be made without departing from the gist of the present invention as claimed in the claims.

10: marker
100: map storage unit
200: vision sensor
300: image processing unit
400: posture estimation
500: route planning department
600: robot control unit
S10: Floor detection step
S20: Floor map acquisition stage
S30: Moving route planning stage

Claims (10)

A marker 10 displayed on the inner floor of the elevator;
A map storage unit 100 in which a map of the elevator floor in which information related to the position of the inner wall of the elevator and the information related to the position of the markers 10 displayed on the inner floor of the elevator is stored;
A vision sensor 200 for obtaining image information of the front floor in a specific range from the robot;
Based on the map of the elevator floor stored in the map storage unit 100, a marker 10 is determined among the image information obtained from the vision sensor 200, and the unoccupied floor including the determined marker 10 An image processing unit 300 for generating or updating a map of the elevator floor for;
A posture estimating unit 400 for estimating the posture of the robot based on a map of the elevator floor generated or updated by the image processing unit 300;
Based on the map of the floor of the elevator created or updated by the image processing unit 300 and the attitude of the robot estimated by the attitude estimation unit 400, after checking whether there is a target area that the robot can reach, A route planning unit 500 that determines a point and plans a moving route to the target point; And
A robot control unit 600 for controlling the robot to move along a movement path generated by the path planning unit 500;
Including,
The posture estimation unit 400
The coordinates of the vision sensor 200 in the elevator's internal map coordinate system are calculated as follows:

(here,

Is the coordinate of the vision sensor in the map coordinate system inside the elevator,

Is a three-dimensional rotation transformation matrix,

Is the shift transformation vector)
Obtain as,
The direction information of the vision sensor, roll,

), pitch,

) And yaw,

) To the following expression

(Where Atan2 is an arctangent function that takes two arguments,

Is the unit vector in the y-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

Is the unit vector in the z-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

,

,

Is the unit vector in the x-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the x, y, z axis component of,

Is the arcsine function)
Obtained as,
A device for an indoor mobile robot to recognize an environment in an elevator, characterized in that it calculates the robot's posture.
The method of claim 1,
The marker 10 is
A device for an indoor mobile robot to recognize an environment in an elevator, characterized in that one or more of a shape of an Arco marker, a shape of a checker board, and a color pattern is used.
The method of claim 1,
The image processing unit 300
An indoor mobile robot characterized in that the area of the completely identified markers 10 for the elevator floor is obtained, and then the information on the previously stored elevator floor is compared to distinguish only the aggregate area of unoccupied unit blocks. A device to be aware of the environment.
delete The method of claim 1,
The robot control unit 600 is
If the target area and the movement path are not secured, the device for indoor mobile robot to recognize the environment in the elevator, characterized in that controlling the robot to return to the starting point or wait in place.
In the method for an indoor mobile robot in the form of a program executed by an operation processing means including a computer to recognize an environment in an elevator,
A floor sensing step (S10) of obtaining image information of a front floor of a specific range from the robot by the vision sensor 200;
Markers among the image information obtained from the floor sensing step (S10) are based on a map of the elevator floor in which the information related to the location of the interior wall of the elevator and the markers 10 displayed on the interior floor of the elevator are stored. 10) determining a floor map acquisition step (S20) of generating or updating a map of an elevator floor for an unoccupied floor including the determined marker 10;
Based on the map of the elevator floor updated from the floor map acquisition step (S20), the attitude of the robot is estimated, and after checking whether there is a target area that can be reached by the robot, a target point that can be reached is determined, and the moving path to the target point A moving route planning step (S30) of planning a;
Including,
The movement route planning step (S30)
The coordinates of the vision sensor 200 in the elevator's internal map coordinate system are calculated as follows:

(here,

Is the coordinate of the vision sensor in the map coordinate system inside the elevator,

Is a three-dimensional rotation transformation matrix,

Is the shift transformation vector)
Obtain as,
The direction information of the vision sensor, roll,

), pitch,

) And yaw,

) To the following expression

(Where Atan2 is an arctangent function that takes two arguments,

Is the unit vector in the y-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

Is the unit vector in the z-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the z-axis component of,

,

,

Is the unit vector in the x-axis direction of the vision sensor coordinate system

Transformed into the elevator interior map coordinate system

Is the x, y, z axis component of,

Is an arcsine function)
Obtained as,
A method for an indoor mobile robot to recognize an environment in an elevator, characterized in that the robot's posture is calculated.
The method of claim 6,
The floor map acquisition step (S20)
After acquiring the areas of the completely identified markers 10 for the elevator floor, it is characterized in that only the aggregate area of unoccupied unit blocks, that is, the area in which the robot can move, is distinguished by comparing previously stored information on the elevator floor. A method for indoor mobile robots to recognize the environment in an elevator.
delete A computer-readable recording medium storing a program for implementing the method for the indoor mobile robot according to claim 6 or 7 to recognize the environment in an elevator.
A program stored in a computer-readable recording medium for implementing a method for the indoor mobile robot according to claim 6 or 7 to recognize the environment in an elevator.

Images