WO2021040214A1 - Mobile robot and method for controlling same - Google Patents

Abstract

A mobile robot and a method for controlling same are provided. The mobile robot comprises: a laser scanner installed in a predetermined position on the mobile robot so as to output laser scan data on the basis of a scan unit, the laser scan data being produced through a scan within a viewing angle; a memory configured to store a program for performing object recognition and object tracking by using at least one image production unit and the laser scanner; and at least one processor configured to execute the program. The program comprises instructions for: converting an image produced by the at least one image production unit into a skeleton, thereby producing a skeleton image, and storing same; detecting an object (tracking target) on the basis of the laser scan data; driving the at least one image production unit, if noise or interference occurs or if object tracking fails, so as to produce a skeleton image from an image captured at the current timepoint; and re-detecting an object on the basis of laser scan data that is output at the current timepoint if the skeleton image at the current timepoint is identical to a prestored skeleton image.

Description

Mobile robot and its control method

The present invention relates to a mobile robot and a control method thereof.

As the automation of the logistics market expands, there are increasing cases of applying an object tracking driving technology that automatically follows an object by recognizing an object on a manual cart or cart in use at the logistics site. Object tracking driving technology is used in various fields, and representatively, there is a guide service robot. The information service robot follows the customer and guides the location or carries the baggage of the customer or object. However, such object tracking driving technology has difficulty in spreading commercialization by operating based on expensive sensors or driving in a constrained environment.

Typically, many expensive sensors, such as LiDAR sensors, are used for object tracking, and there is an increase in cost and difficulty in maintenance because map information and positioning infrastructure are used for location positioning. have.

In addition, short-range wireless communication-based tracking technology, as in the case of personal travel bags, follows a one-to-one mapping when a user wears a wearable device, so there are many restrictions in using a large number of objects and customers.

In addition, the conventional object tracking method using a depth camera often misses an object due to the low resolution of the depth camera and a narrow recognition area.

The problem to be solved by the present invention is to provide a mobile robot having an efficient object recognition and tracking function, and a control method thereof, using a laser scanner and an image capturing unit without the aid of a positioning infrastructure.

According to one feature of the present invention, as a mobile robot, at least one image capturing unit installed at a predetermined position of the mobile robot, a laser scanner installed at a predetermined position of the mobile robot, the at least one image capturing unit, and the A memory in which a program for recognizing an object and tracking an object using a laser scanner is stored, and at least one processor executing the program, wherein the program is a tracking target based on the image generated by the at least one image capturing unit. Recognizes an object, detects and tracks an object to be tracked through the laser scan data, and if object tracking fails, the at least one image capture unit is driven to compare the image captured at the current time with the previously stored image , And commands for re-detecting the object based on the laser scan data output at the current time.

The program crosses a plurality of object areas detected from the laser scan data collected at a first time point and a plurality of object areas detected from the laser scan data collected at a second time point later than the first time point, and crosses the A command for recognizing an object having the largest area ratio as a tracking object may be included.

The program may include a command for removing noise by applying a morphology operation to laser scan data arranged in two dimensions.

The program labels the laser scan data from which noise has been removed, detects at least one area with the same label as each object, and tracks the matching object by comparing the area detected as each object with the image It may include a command to select an object.

The program includes a command for detecting an object based on laser sensor data collected in a predetermined scan area, and the predetermined scan area may have a range corresponding to a set height for each body part of a person from the ground.

The program divides the data group based on the shortest distance between the point groups constituting the grouped laser sensor data from the laser sensor data grouped by the predetermined scan area, and sets a characteristic variable for each scan area within the divided data group. It may contain instructions to calculate.

The characteristic variable may include a total width of group data including the point groups, distances between two adjacent point groups, a width of the group, and an average of the distances.

According to another feature of the present invention, as a control method of a mobile robot, collecting laser scan data, converting laser scan data composed of a polar coordinate system into a plurality of point cloud forms having position values in a Cartesian coordinate system, and converting into a point cloud form Comparing the generated laser scan data with image data captured by a camera and binarizing using pixel values of an image corresponding to each point group, labeling the laser scan data to detect a plurality of objects, and detecting a plurality of objects Comparing the image data to select a matching object as a tracking object, and driving toward the tracking object.

The collecting of the laser scan data further includes removing noise by applying a morphology mask, and the morphology mask may be set in advance to a size capable of covering a person's stride.

The driving may include comparing a plurality of objects detected from laser scan data at a first scan time point with a plurality of objects detected from laser scan data at a second scan time later than the first scan time, It may further include the step of recognizing the object having the largest ratio as a tracking object.

Recognizing the object having the largest ratio of the intersecting areas as the tracking object may be selectively driven according to a preset condition.

The collecting of the laser scan data may further include filtering scan data corresponding to a height set for each body part of a person from among the laser scan data.

The detecting may include applying the binarized laser scan data as a labeling target, dividing and grouping the data based on the distance between each point group, from the data applied as the labeling target, and grouping the grouped point groups It may include performing labeling for classifying them into the same object.

According to another feature of the present invention, as a method of operating a mobile robot, the step of performing authentication for object tracking in connection with a remote server, and if the authentication is successful, photographing the front side to generate an image, and the generated image Generating a skeleton image by making a skeleton, determining whether the generated skeleton image is a first user by comparing the skeleton image of the first users registered in advance in a user database with the generated skeleton image, as a first user When it is determined, performing a task according to the request of the first user while tracking the first user using a laser scanner, storing the work history, and if it is determined that it is not the first user, second user information and destination information And performing a route guidance to the received destination while requesting and receiving an input of, and tracking a second user using the laser scanner.

In the first user tracking or second user tracking using the laser scanner, laser scan data composed of a polar coordinate system is converted into a plurality of point cloud forms having position values in a Cartesian coordinate system, and the laser scan data converted into a point cloud form is photographed with a camera. A method of comparing one image data and binarizing using pixel values of an image corresponding to each point group, comparing a plurality of objects detected by labeling the binarized laser scan data with the image data, and selecting matching objects as tracking objects Can be made.

The storing may include extracting and registering feature point information from a captured image if it is determined as the first user, performing a task requested by the first user and storing a work history, and using the laser scanner Tracking the user, if the tracking of the first user fails, feature point information is extracted from the image acquired by photographing the front of the current point of view, and the extracted feature point information is compared with the registered feature point information to determine the object of the matching point. The step of determining as the first user, and moving to the determined first user's location, guiding the previous work history and inquiring if there is any intention to follow. It may include moving to a location.

After the step of performing the route guidance, if the tracking of the second user fails, the image generated by photographing the front of the robot is skeletonized to generate a skeleton image at the present time, and the previously generated skeleton image and the The method may further include the step of searching for a matching object by comparing the skeleton image of the current viewpoint, and tracking the searched object.

According to an embodiment, by using a relatively inexpensive laser scanner and an image capture unit instead of a LiDAR sensor of hundreds to tens of millions of won, an object recognition/tracking algorithm using an image capture unit and an object recognition/tracking algorithm using a laser scanner Combined to improve the object tracking rate. That is, the recognition range and processing speed of the image capturing unit can be supplemented through the laser scanner.

In addition, the problem of tracking failure caused by the low resolution of the camera and the narrow angle of view is compensated by using a laser scanner having a field of view capable of covering most of the front.

In addition, by efficiently processing the data collected by the laser scanner, continuous tracking performance is improved.

In addition, when tracking an object, it is possible to track only with software without the help of an infrastructure for locating indoor and outdoor locations, so when applied to an actual site, price competitiveness and maintenance efficiency can be improved.

1 is a block diagram showing the configuration of a mobile robot according to an embodiment of the present invention.

FIG. 2 is a view for explaining the angle of view of the mobile robot of FIG. 1.

3 is an example of scan data converted into two dimensions according to an embodiment of the present invention.

4 is an exemplary view of setting a region of a mobile robot according to an embodiment of the present invention.

5 is a flow chart showing the operation of the mobile robot according to an embodiment of the present invention.

6 is a flowchart illustrating an object recognition/tracking operation of a mobile robot according to an embodiment of the present invention.

7 is a flowchart illustrating an object recognition/tracking operation of a mobile robot according to another embodiment of the present invention.

8 is a flowchart illustrating a process of detecting an object through a labeling process according to an embodiment of the present invention.

9 is a diagram illustrating a laser scan area according to an embodiment of the present invention.

10 is a diagram illustrating a coordinate system for collecting laser scan data from a mobile robot according to an embodiment of the present invention.

11 is a diagram illustrating a process of extracting reliable data from labeled laser scan data according to an embodiment of the present invention.

12 is a diagram for explaining feature variables from laser scan data according to an embodiment of the present invention.

13 is a diagram illustrating a process of dividing an object based on a feature variable according to an embodiment of the present invention.

14 is a diagram illustrating a frame matching technique according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

In addition, terms such as "... unit", "... group", and "... module" described in the specification mean a unit that processes at least one function or operation, which can be implemented by hardware or software or a combination of hardware and software. I can.

The devices described in the present invention are composed of hardware including at least one processor, a memory device, a communication device, and the like, and a program that is combined with the hardware and executed is stored in a designated place. The hardware has the configuration and capability to implement the method of the present invention. The program includes instructions for implementing the operating method of the present invention described with reference to the drawings, and executes the present invention by combining it with hardware such as a processor and a memory device.

In the present specification, "transmission" or "providing" may include not only direct transmission or provision, but also transmission or provision indirectly through another device or using a bypass path.

Expressions described in the singular in this specification may be interpreted as the singular or plural unless an explicit expression such as "one" or "single" is used.

In the present specification, the mobile robot is configured to travel in a desired direction while avoiding obstacles by mounting various detection sensors and autonomous driving control algorithms. Mobile robots do not need to build indoor infrastructure or wear additional equipment such as wearable devices, so they can be applied not only to logistics, but also to hotel room services and shopping carts.

Now, a mobile robot and a control method thereof according to an embodiment of the present invention will be described with reference to the drawings.

1 is a block diagram showing the configuration of a mobile robot according to an embodiment of the present invention, FIG. 2 is a view for explaining the angle of view of the mobile robot of FIG. 1, and FIG. 3 is a two-dimensional transformation according to an embodiment of the present invention. Is an example of the scanned data, and FIG. 4 is an example of setting an area of a mobile robot according to an embodiment of the present invention.

Referring to FIG. 1, the mobile robot 100 includes an image capture unit 101, a laser scanner 103, a traveling device 105, an input device 107, an output device 109, a communication device 111, and a storage device. It includes a device 113, a memory 115 and a control unit 117.

The image capturing unit 101 is attached to the outside of the mobile robot 100 and generates an image by photographing the outside of the mobile robot 100. One image capturing unit 101 may be installed, but a plurality of image capturing units 101 may be installed depending on the embodiment.

According to an embodiment, the image capture unit 101 may use a depth camera that combines an RGB camera and a depth sensor.

The laser scanner 103 may be installed at an angle of view overlapping the image capturing unit 101. The laser scanner 103 detects the periphery of the laser scanner 103 using a laser measuring beam that is attached to the outside of the mobile robot 100 and rotates rapidly.

At this time, the laser scanner 103 obtains scan data by irradiating a laser in the scanning direction while being mounted on the mobile robot 100 and scanning the surface of the object positioned in the scanning direction while the mobile robot 100 is moving.

Referring to FIG. 2, the image capturing unit 101 is installed on the front side of the mobile robot 101 and photographs the front side at the current position of the mobile robot 101. One image capture unit 101 is described as being installed, but a plurality of image capturing units 101 may be installed according to embodiments.

The image capture unit 101 may be installed on the top of the mobile robot 100 to recognize the entire skeleton of the object.

A plurality of laser scanners 103 may be installed at the angle of view overlapping the image capturing unit 101. For example, two laser scanners 103 may be installed at the front and one at the rear of the mobile robot 101.

At this time, the angle of view of the image capturing unit 101 is approximately 120 degrees, and the laser scanner 103 has a wider angle of view than that of the image capturing unit 101, which may be approximately 360 degrees.

Accordingly, even if the object being tracked deviates from the angle of view of the image capturing unit 101, the object tracking can be maintained through the laser scanner 103.

Again, referring to FIG. 1, the traveling device 105 is a device that allows the mobile robot 101 to travel, and a driving means such as a plurality of wheels and a motor, and a driving force applied to the traveling means to drive the mobile robot 100. It may include a driving source for providing. This traveling device 105 belongs to a well-known technology and a detailed description thereof will be omitted here.

The input device 107 is a means for receiving a user command. The input device 107 may include devices such as a keyboard, a touch pad, an optical joystick, and a mouse.

The output device 109 may output a visual or an audible output according to a user command transmitted from the input device 107. The output device 109 may include an output device such as a display device or a speaker. The output device 109 may output a user interface (hereinafter, collectively referred to as'UI') screen. The output device 109 may output voice, sound, and the like.

The input device 107 and the output device 109 may be implemented as a touch screen. That is, a touch keypad, a touch button, etc. may be activated on the screen to receive a user command and output various information.

The communication device 111 is connected to a remote server (not shown) through a communication network (not shown) to transmit and receive data.

The storage device 113 stores data necessary for the operation of the mobile robot 100 and data transmitted and received through the communication device 111.

The memory 115 stores programs and data necessary for the operation of the mobile robot 100. These programs are composed of instructions for performing the operation and control of the mobile robot 100.

The controller 117 is at least one processor and executes a program stored in the memory 115.

The control unit 117 operates based on a robot operating system (ROS), and in detail, an object tracking unit 119, a driving control unit 121, a user interface (UI) control unit 123, a remote control unit 125, and a task management unit Includes (127).

The object tracking unit 119 interlocks with the image capturing unit 101 and the laser scanner 103 to recognize and track an object. The object tracking unit 119 requests the driving control unit 121 to detect and avoid an obstacle, reduce the speed, or make an emergency stop in the process of recognizing and tracking an object. The object tracking unit 119 generates a workplace map for navigation of the mobile robot 100 and processes a navigation function. Here, the workplace refers to the environment in which the mobile robot 100 travels. The object tracking unit 119 may operate in a tracking mode and a driving mode.

The object tracking unit 119 receives and processes raw data from the image capturing unit 101 and the laser scanner 103 so that it can be used for object recognition and tracking.

The object tracking unit 119 extracts a depth map and skeleton data from an image generated by the image capturing unit 101.

The object tracking unit 119 drives the driving control unit 121 to follow only the designated object by recognizing the object and tracking it continuously. Here, the object is a moving object, and in the present invention, the object means a person to be tracked. According to one embodiment, when a mobile robot is used in a hotel, the objects may be employees and guests.

The object tracking unit 119 identifies an object to be tracked based on the image received from the image capturing unit 101 and tracks the identified object using scan data received from the laser scanner 103. In this way, the object tracking unit 119 may improve the object tracking rate by combining the image capturing unit 101 and the laser scanner 103 to recognize and track the object.

The object tracking unit 119 recognizes the object by grasping the position of the skeleton of the object using the image received from the image capturing unit 101.

The object tracking unit 119 generates a skeleton image by skeletonizing the image received from the image capturing unit 101. Skeletonization is a method of expressing the location of relatively unmodified body parts and connection information between the body parts, and various known algorithms can be used.

In this case, the object tracking unit 119 assumes that the skeleton image is for one person. When generating an initial skeleton image, the object tracking unit 119 may process an error and request re-recognition when a plurality of skeletons are detected. In addition, an input for selecting one of a plurality of skeletons may be received from a user through the input device 107. At this time, the selected skeleton is determined as a tracking object.

The object tracking unit 119 collects laser scan data scanned in seconds from the laser scanner 103. And, from the collected laser scan data, primary filtering is performed to extract only the radar scan data of a preset area. In this case, the preset area may be set to a height corresponding to a person's legs, hips, and the like.

The object tracking unit 119 performs secondary filtering according to a filtering criterion among the primary filtered laser scan data. At this time, based on the point cloud data located at the center of the primary filtered laser scan data, secondary filtering is performed in which only a predetermined number of point cloud data are left for each area. Here, the predetermined number is set differently for each leg height, hip height, and back height.

The object tracking unit 119 performs a binarization operation on the secondary filtered laser scan data. Then, a morphology mask and labeling are applied to the binarized laser scan data to detect a plurality of objects.

The object tracking unit 119 compares the plurality of object areas with the skeleton image and selects an object identical to the skeleton image (shape) as the tracking object. Since the skeleton image can be regarded as a type of shape data, the object tracking unit 119 selects an object area that matches the skeleton image as a tracking object.

After recognizing the object through the above process, the object tracking unit 119 then compares the previous object area with the next object area and recognizes the object with the largest intersection ratio as the tracking object. Here, "previous" and "next" can be viewed as the collection point of laser scan data.

The object tracking unit 119 tracks the object by converting the scanned data scanned with the laser scanner into an object through pre/post processing processes.

The scan data obtained by the object tracking unit 119 from the laser scanner 103 is composed of a polar coordinate system consisting of a radius and an angle.

The object tracking unit 119 converts the scan data of the polar coordinate system into a rectangular coordinate system composed of X and Y. These scan data form a number of station points, that is, point clouds, each having position coordinates of X and Y.

As such, an example of a code for converting scan data in the form of polar coordinates into scan data in the form of rectangular coordinates may be as shown in Table 1.

The object tracking unit 119 generates scan data in units of at least 1 centimeter (cm) so that the mobile robot 100 can smoothly track the object. Here, the scan data generation unit refers to a moving distance of the mobile robot 100. That is, each time the mobile robot 100 moves in units of 1 centimeter, scan data may be generated. In addition, the scan distance for initial object recognition can be secured within 2 meters.

The object tracking unit 119 converts the scan data of the polar coordinate system into 2D data of the Cartesian coordinate system, and the converted 2D data is expressed as shown in FIG. 3.

Referring to FIG. 3, 2D data has a value of 0 or 1. That is, the object tracking unit 119 performs a job of binarizing the scan data of the polar coordinate system. At this time, the object tracking unit 119 maps a number of station points each having position coordinates of X and Y to the image acquired from the image capturing unit 101, and is 0 or 0 based on the value of the pixel corresponding to each station point in the image. It can be expressed as 1. The object tracking unit 119 applies the RGB function to the laser scan data, and assigns 1 as the value of the pixel of the corresponding two-dimensional array to the area composed of R=255, G=255, and B=255, and has other RGB values. The area can be set to 0.

For example, a reference pixel value (eg, 255) may be determined, and if the pixel value corresponding to each point is smaller than the reference pixel value, it may be displayed as 0, and if it is greater than the reference pixel value, it may be displayed as 1.

Each station having a position coordinate of X and Y and a value of 0 or 1 is defined as a pixel. In addition, by performing a morphology operation on the scan data shown in FIG. 3 in which the measurement points are arranged in a two-dimensional coordinate system, noise is removed and missing values are compensated. In general, morphology operation is a technique that is widely used in image processing. One of the morphological operations, dilation, is a method of combining two adjacent objects and filling a hole inside the object. The erotion operation removes the noise and pillars of the object boundary.

According to an embodiment, a morphology mask having a size of 21×21 may be used to cover the stride of an object. This mask can cover a person's normal stride of 40 centimeters. Of course, the morphology mask is not limited thereto, and morphology masks of various sizes may be used.

The morphology operation of the object tracking unit 119 may use code examples shown in Table 2 below.

The object tracking unit 119 classifies an object to be tracked from an object to be tracked using the scan data acquired from the laser scanner 103 to speed up the initial recognition speed and performs labeling. In addition, other objects are removed from the list of objects of interest, centering on the tracked object. Table 3 shows an example of a code for removing the remaining objects except for the tracking object.

The labeling technique is particularly widely used as a preprocessing process for object recognition. The object tracking unit 119 may define a set of adjacent pixels having a pixel value of 1 in the scan data as shown in FIG. 3 as an object. An object consists of one or more adjacent pixels. Multiple objects can exist in one scan data, and labeling is the task of assigning a unique number to all pixels belonging to the same object.

According to an embodiment, the object tracking unit 119 may perform labeling using a 4-neighbor connectivity mask. Therefore, pixels located in a diagonal relationship have different labels. Two adjacent neighboring pixels among the neighboring pixels adjacent to a pixel at an arbitrary position (X, Y) can be classified into four cases as follows.

The object tracking unit 119 creates an equivalent table by propagating a label in the first scan, and gives a unique label to a pixel corresponding to the object by referring to the label in the equivalent table in the second scan.

When labeling is completed, the object tracking unit 119 recognizes an object composed of pixels having the same label. Then, a plurality of recognized objects are matched with a skeleton image to select a tracking object from among the plurality of objects.

The object tracking unit 119 may fail to recognize the tracking object. To compensate for this, the object tracking unit 119 uses a frame-to-frame matching algorithm to classify an object to be tracked and another object. The object tracking unit 119 assumes that the previously detected object does not suddenly move to a long distance or disappear in order to efficiently match frames. When a part of the scan data where a person is detected is identified as a space and the person's stride is taken into account, a person can be recognized by spatial region matching because a person exists within a certain range of space in the next frame. Since scan data is generated in units of frames per second, a large overlap area occurs. The object tracking unit 119 may perform a quick correction operation when a tracking object is temporarily missed through a spatial region matching technique.

The object tracking unit 119 uses a method of tracking an object having the largest ratio of an intersecting area among the objects detected in the previous frame and the objects detected in the current frame.

When the object tracking unit 119 succeeds in recognizing the object, the object tracking unit 119 tracks the object recognized through the laser scanner 103 and cross-comparisons the laser scan data acquired in real time on a frame-by-frame basis. The object with the largest crossing area is tracked by cross-comparison of the area where the tracked object is recognized in the previous frame and the current frame. Another name for this spatial domain matching technique is the best matching method. At this time, after assuming that the detected object does not suddenly move to a long distance or disappear, an optimal matching algorithm is used. In this way, the path the object has moved is predicted and traced without the need for repetitive detection of the object. At this time, an algorithm called Kalman filter can be additionally used.

The object tracking unit 119 may deactivate the spatial area matching technique according to conditions during tracking driving. For example, when there is no movement of the mobile robot 100 and the object, that is, when a person waits in a space such as an elevator, other objects may be detected around the mobile robot 100. I can. At this time, if the tracking object is changed to another object, a problem occurs in the service. In this case, the spatial domain matching technique is deactivated.

The object tracking unit 119 may be deactivated according to conditions during tracking driving. For example, when there is a situation where a robot and a worker are waiting in a space such as an elevator, other workers or customers may be exposed around the camera and the laser sensor. At this time, if the tracking target is changed, the service may be affected, so it can be added to the deactivation condition.

When the object tracking unit 119 re-recognizes an object that has failed to be tracked, it is assumed that the distance change in the area of the object that has failed to be tracked will be minimal. If this is expressed by an equation, it is equal to Equation 1.

[Equation 1]

According to Equation 1, if the change value of the previous coordinate (previous.coordicate) and the current coordinate (current.coordinate) of the object is less than the threshold, it is recognized as a tracking user, otherwise, another object ( otherUser).

Further, the object tracking unit 119 recognizes the object as a tracking object when the current coordinates of the object are smaller than the matching area, and this is expressed as an equation, as shown in Equation 2.

[Equation 2]

According to Equation 2, the tracking object user(s) represents a case where the current coordinates (urrentUserPosition(x,y,z)) of the tracking object user(s) are smaller than the intersection area.

The object tracking unit 119 may recognize a region in which a person is based on the laser scan data, and use an image when a person is specifically identified. The object tracking unit 119 classifies a person, that is, an object other than a tracked object in an area recognized as an object, as a non-tracking object such as an obstacle. When a tracked object is selected, information on the non-tracked object is deleted in order to determine that objects other than the selected object are not the target of tracking.

In the case of object recognition and tracking using the laser scanner 103, it is good because the tracking speed is fast and the angle of view is wide, but it is difficult to increase the level of object identification and sometimes misses. In this case, the object tracking unit 119 may match the skeleton image with the object recognized by the laser scanner 103 to continuously track while minimizing the re-recognition process.

The object tracking unit 119 may acquire 3D information on an image of a target scene in real time using distance sensors.

The state of the object recognized by the object tracking unit 119 is defined as "New", "Visible", "Tracked", "NotVisible", and "Lost" states. The "New" state is a state in which an object is newly detected, and the start of object recognition can be known. The "Visible" state is a state in which an object is detected within the angle of view. In the "Tracked" state, the object is being tracked and skeleton information is available. The "NotVisible" state is a state in which an object cannot be detected and is invisible within the angle of view. The "Lost" state is a state in which the object has been completely lost.

The object tracking unit 119 updates the current object tracking state while tracking the object based on these object states.

The object tracking unit 119 may process the object in a “NotVisible” state when the object is out of the field of view of the image capturing unit 101.

In addition, when the object tracking unit 117 fails to track the object, it may re-recognize other objects and fail to track the object. Even though the object tracking unit 119 exists in the angle of view, the object tracking unit 119 may miss the object according to the posture or situation of the object. For this reason, if the object is not visible within about 10 seconds or more after the object tracking fails, it is determined that the object has been lost and can be processed in a "Lost" state.

When the object tracking unit 119 re-recognizes an object that has failed to be tracked, the object ID is assigned as before, and may be processed in a "Tracked" state.

When a situation in which an object can be assigned a new ID occurs, the object tracking unit 119 may process the object in a "New" state.

When an object is detected within the angle of view, the object tracking unit 119 may process the object in a "Visible" state.

In addition, the object tracking unit 119 recognizes a new object within a preset radius, for example, within 1m, based on the coordinate values (X, Y, Z) in the space of the missed object as the main object and proceeds with the tracking. If object recognition is not performed for a certain period of time, it is processed as a “Lost” state, and an emergency stop is requested to the driving control unit 121. In addition, an emergency stop alarm may be requested from the UI controller 123.

At this time, various alarms may occur according to the service provided by the mobile robot 100. For example, if the object tracking unit 119 requests an emergency stop due to an object tracking failure while tracking a hotel employee, it outputs a stop alarm to the outside through the UI control unit 123 so that the employee takes action to re-recognize the object. Do it. For example, a speaker alarm can be output to allow an employee to move within the object tracking radius.

In addition, if the object tracking unit 119 fails in the middle of tracking a customer who uses the hotel, it instructs the remote control unit 125 to request an alarm from a server (not shown) located in a remote location, and the driving control unit 121 is set in advance. Request to return to the designated location. In addition, when a destination is set by a customer, the object tracking unit 119 does not regard the object as a departure, even if an object is not detected within a certain radius around the destination, and if it reaches the destination, it is processed as a normal arrival and then a preset designated location. It can be requested to the driving control unit 121 to return to.

In addition, in order to maintain a constant distance when tracking an operator, the object tracking unit 119 performs dynamic distance calculation and speed control. To this end, three distance states, that is, "Far", "Near", and "InZone" are defined according to how far the distance between the object and the mobile robot 100 is.

In Table 3, according to an embodiment, the Max(d) and Min(d) values may be set to 1.1m and 1.0m, respectively. Since the laser scanner 103 scans in real time per second, the object tracking unit 119 divides the distance between the mobile robot 100 and the tracked object into one of "Far", "Near", and "InZone" in seconds. . If this is expressed by an equation, it is shown in Equation 3

[Equation 3]

According to Equation 3, "Far" is defined as a case where the user distance (the distance between the object and the mobile robot) is greater than Max(d). "Near" is defined as a case where the user distance is less than Min(d). "inZone" is defined as a case where the user distance is greater than Min(d) and less than Max(d).

The object tracking unit 119 determines the state of the user distance and provides it to the driving control unit 121. The driving control unit 121 pre-designates driving control information for each user distance state. Based on this, when the state of the user distance is "Far", the driving control unit 121 increases the driving speed of the mobile robot 100. In addition, the driving control unit 121 maintains the driving speed of the mobile robot 100 when the state of the user distance is “inZone”. In addition, when the state of the user distance is "Near", the driving control unit 121 may reduce the driving speed of the mobile robot 100 or stop the mobile robot 100 in an emergency.

Referring to FIG. 4, the mobile robot 100 is divided into an obstacle zone and a tracking zone. At this time, when an object is detected in the obstacle area, the object tracking unit 119 requests an emergency stop from the driving control unit 121.

According to an embodiment, the obstacle area may be a=0.8m, b=0.3m~0.4m, and the tracking area may be c=4.2m, d=1.1m. Here, the obstacle area and the tracking area may be changed according to embodiments, and may be set differently according to the tracking environment.

In addition, depending on the performance of the laser scanner 103, an object can be tracked up to 10 meters, but in the present invention, the object tracking unit 119 scans so as not to scan an object farther than the distance of the object selected as the target to be tracked. You can limit the area dynamically. By limiting the scan area, unnecessary scans and operations can be minimized and the accuracy of scan data can be improved. An example of a code limiting the scan area may be shown in Table 6.

The object tracking unit 119 may generate a map and recognize its own location through SLAM (Simultaneous localization and mapping) technology.

The object tracking unit 119 determines whether the location of the mobile robot 100 measured through the distance sensor matches the location information of the mobile robot 100 on the map.

The object tracking unit 119 determines the initial position of the mobile robot 100. This position indicates how far the mobile robot 100 is from the charging station (not shown) based on the distance from the charging station (not shown) where the mobile robot 100 charges the battery. Collected from) and the rotation angle collected from the gyro sensor. That is, the initial position information of the mobile robot 100 is composed of (x, y, radian), and radian represents the rotation angle.

If the initial position of the mobile robot 100 is not correct, the coordinate information of the tracking object is also not accurate, and thus the initial position of the mobile robot 100 needs to be matched.

When the initial position of the mobile robot 100 is matched, the object tracking unit 119 acquires distance information about how far the object is from the mobile robot 100 through the laser scanner 103, and the distance information and the initial position By using, the position of the object may be calculated as a position value relative to the mobile robot 100.

Referring again to FIG. 1, the driving control unit 121 is a robot adapter and performs linear/angular speed calculation, motor revolutions per minute (RPM) calculation, speed control, encoder travel distance calculation, robot state value management, and the like.

The driving control unit 121 includes a controller and an interface for driving and controlling hardware of the mobile robot 100. The driving control unit 121 performs acceleration/deceleration by calculating the linear speed according to the distance to the tracked object output from the object recognition unit 119. Further, the angular velocity is calculated according to the moving direction of the object to calculate RPM (revolutions per minute) for the differential motor, and based on this, the traveling device 105 is controlled.

The driving control unit 121 transmits the rotation direction, and the RPM of the left and right motors converted based on the linear and angular speeds to the driving device 105 in real time to safely and smoothly move the mobile robot 100 to the target position. I can. In addition, for safety, feedback control by transmitting the state values of the mobile robot 100 such as whether the front and rear bumpers of the mobile robot 100 collide, and the encoder value of the left and right motor rotation of the mobile robot 100 to the traveling device 105 , The moving distance and the rotation direction of the mobile robot 100 may be calculated and used as base information necessary for creating a map and tracking a moving path of the mobile robot 100.

Destination information recognized through a screen output through the UI control unit 123 or voice recognition is transmitted to the driving control unit 121. The driving control unit 121 outputs a driving command such as a rotation direction, a linear speed, and an angular speed to the driving device 105 based on the result of calculating the position of the object and its own position on the map.

The driving control unit 121 may divide the speed control of the driving unit into L1 to L7 (the speed at which a person walks quickly, for example, 6km/h).

The UI control unit 123 outputs the contents necessary for the operation using the mobile robot 100 to the output device 109. In addition, data input from the input device 107 may be output on the screen. For example, the UI controller 123 may provide a UI through which an operator can improve work efficiency by using a mobile robot, such as a logistics list, item inspection, and a transport route.

According to an embodiment, the UI controller 123 may provide a UI such as barcode recognition, product information, order information, and route search.

The remote control unit 125 is connected to the remote server 200 through the communication device 111 and interlocks with the remote server 200 to perform an operation for performing the state and control of the mobile robot 100. According to an embodiment, the remote control unit 125 provides the object recognition status (5 types) determined by the object recognition unit 119 to the remote server 200 to query the object recognition status of the mobile robot 100 at a remote location. To be able to do it. The remote control unit 125 may receive a control command from the remote server 200 and control the mobile robot 100 to operate based on the control command.

The task management unit 127 is in charge of efficient operation of the mobile robot 100 through real-time monitoring, task management, operation status management, and automation information system linkage.

The task management unit 127 may manage tracking authority. The tracking authority is used when a specific user, such as a hotel employee, not a number of unspecified users, but a specific user such as a hotel employee, is used to connect to other employees while working in tracking mode or when the job is terminated. At this time, it is set so that it cannot be used except for the changed tracking target.

Then, the operation of the mobile robot 100 will now be described.

First, FIG. 5 is a flow chart showing an operation of a mobile robot according to an embodiment of the present invention, and shows an operation according to a service scenario in a specific place where the mobile robot is installed. Here, the specific place may be a hotel, but is not limited thereto.

Referring to FIG. 5, a skeleton image of a first user (eg, employees, hereinafter, collectively referred to as "employees" for convenience of explanation) on the storage device 113 or the remote server 200 of the mobile robot 100 The DB is built (S101).

The control unit 117 of the mobile robot 100 performs authentication before starting object recognition/tracking (S103). For example, the controller 117 may perform authentication by requesting input of authentication information (eg, biometric authentication information such as a fingerprint, a face, etc., a password, etc.) stored in the storage device 113. Alternatively, the controller 117 may perform authentication by requesting input of authentication information stored in the remote server 200.

If authentication is successful, the image photographing unit 101 of the mobile robot 100 photographs the front side and generates an image (S105).

The object tracking unit 119 of the mobile robot 100 generates a skeleton image by skeletonizing the generated image (S105) (S107).

The object tracking unit 119 of the mobile robot 100 compares the skeleton image generated (S107) with the skeleton image registered in the employee DB (S109) to determine whether the skeleton image represents the employee (S111).

If it is determined as an employee, the object tracking unit 119 of the mobile robot 100 extracts and registers feature point information from the image generated in step S105 (S113). Here, the feature point information may be referred to as feature information representing a specific user. For example, it may include information such as the color and mark of an employee's uniform. The mark may include a hotel label, name tag, or the like written on the uniform. Also, the feature point information may be a skeleton image generated in step S107.

The job management unit 127 of the mobile robot 100 performs a job according to the employee's request and stores the job history (S115). Work history includes creating a map for driving or tracking people while carrying luggage.

The object tracking unit 119 of the mobile robot 100 tracks the employee recognized (S109) using the laser scanner 103 (S117). Here, the employee tracking using the laser scanner 103 corresponds to the operation of the object tracking unit 119, as described above.

The object tracking unit 119 of the mobile robot 100 determines whether the employee being tracked has left, that is, whether the object tracking has failed (S119).

If object tracking does not fail, it is performed again from step S117.

On the other hand, if it is determined that object tracking has failed, the object tracking unit 119 extracts a feature point from the image captured by the image capture unit 101 at the current time, compares it with the previously registered feature point (S113), and matches the object. That is, the employee is searched (S121). That is, a person with the same feature point information within a certain radius is searched through the camera. Alternatively, a skeleton image may be generated and an object matching the previously registered (S113) skeleton image may be searched.

The object tracking unit 119 requests the driving control unit 121 to move the mobile robot 100 to the position where the searched feature point is found (S123). In addition, the job management unit 127 informs the employee of the previously stored job history (S115) to the employee through the UI control unit 123, and queries whether or not to accept the tracking intention (S123). In other words, the history of previous work is notified by voice or screen, and the intention to follow is confirmed.

In this case, when the UI controller 123 receives a response to accept the tracking intention, the object tracking unit 119 starts again from step S115.

On the other hand, if the intention to track is rejected, the object tracking unit 119 requests the driving control unit 121 to end tracking the object and move it to the start position (S127). Alternatively, you can call an employee.

On the other hand, if it is determined that it is not an employee in step S111, if it is determined as a second user (eg, customer, hereinafter, collectively referred to as "customer" corresponding to "employee" for convenience of explanation), the task management unit 127 Additional information and destination information are queried to the customer through the UI control unit 123 (S129). Here, the additional guest information may be a room number or the like.

The object tracking unit 119 extracts and temporarily registers feature point information from the image generated in step S105 (S131). Here, the feature point information may be a skeleton image generated in step S107.

In addition, the object tracking unit 119 generates navigation information to the destination based on the content received by the UI control unit 123 and starts the route guidance (S133).

In addition, the object tracking unit 119 uses the laser scanner 103 to track the customer, that is, the object determined in step S109 (S135). Here, customer tracking using the laser scanner 103 corresponds to the operation of the object tracking unit 119, as described above.

The object tracking unit 119 determines whether the customer leaves while driving to the destination (S137), and if the customer does not leave, it starts again from step S135.

On the other hand, when the customer leaves, the image generated by photographing the front through the image photographing unit 101 is skeletonized, and an object matching the previously generated skeleton image is searched for (S139).

It is determined whether the search is successful (S141), and if the search is successful, that is, the area in which the skeleton is detected is recognized as an object, that is, a customer, and starts again from step S135.

On the other hand, if the search fails, the object tracking unit 119 calls an employee through the UI controller 123 or requests the driving controller 121 to end object tracking and move to the starting position (S143).

The object tracking unit 119 generates skeleton data as an image signal and re-recognizes it when noise or interference occurs while tracking an object with a laser signal.

For example, if a large number of people are recognized during tracking, tracking may fail. In this case, it may be difficult to identify objects only by laser scanning because a large number of people can continue to flow into the range of the raise scan in the elevator or in the hotel lobby. Alternatively, when laser scan data is distorted by an obstacle or noise during object tracking, object tracking may fail. For example, it could be due to highly reflective walls, glass, or an unexpected obstacle.

As described above, when object tracking fails by laser scanning, skeleton data is generated and registered skeleton data is re-recognized as an object, as described above.

At this time, if re-recognition is successful, object recognition and tracking are performed with a laser scanner to track to the destination. This will be described in FIG. 6.

6 is a flowchart illustrating an object recognition/tracking operation of a mobile robot according to an embodiment of the present invention.

Referring to FIG. 6, the image photographing unit 101 photographs the front of the mobile robot 100 and generates an image (S201). The object tracking unit 119 generates a skeleton image by skeletonizing the image generated in step S201 (S203).

The object tracking unit 119 determines whether the skeleton image generated in step S203 is a registered image (S205). Here, S205 is to determine whether or not registered as a tracking target. If not registered as a tracking target, the skeleton image generated in step S203 is registered as a tracking candidate object (S207), and then step S205 is restarted.

If it is determined that the skeleton image is registered in step S205, the object tracking unit 119 recognizes the object with the laser scanner 103 (S209) and tracks the object (S211).

At this time, it is determined whether noise or interference occurs in the process of tracking the object (S213). If noise or interference does not occur, the process returns to step S211.

On the other hand, if noise or interference occurs, it is determined whether the object being tracked by comparing whether the skeleton image generated at the current time (S215) is the skeleton image generated at the current time (S215) is the skeleton image generated at the S203 or registered at the S207 S219).

If it is determined that the object is being tracked, it starts again from step S209. On the other hand, if the object being tracked is not the object being tracked, object tracking is terminated and the object is moved to the start position (S221).

A process in which the object tracking unit 119 combines the image generator 101 and the laser scanner 103 to recognize/track an object will be described as follows.

7 is a flow chart showing an object recognition/tracking operation of a mobile robot according to another embodiment of the present invention, and FIG. 8 is a flow chart showing a process of detecting an object through a labeling process according to an embodiment of the present invention. Is a diagram illustrating a laser scan area according to an embodiment of the present invention, FIG. 10 is a diagram illustrating a coordinate system for collecting laser scan data from a mobile robot according to an embodiment of the present invention, and FIG. 11 is a view illustrating the present invention. Is a diagram for explaining a process of extracting reliable data from labeled laser scan data according to an embodiment of the present invention, FIG. 12 is a diagram for explaining feature variables from laser scan data according to an embodiment of the present invention, and FIG. A diagram illustrating a process of dividing an object based on a feature variable according to an embodiment of the present invention, and FIG. 14 is a diagram illustrating a frame matching technique according to an embodiment of the present invention.

First, referring to FIG. 7, steps S301 to S313 correspond to step S209 in FIG. 6, that is, an object recognition step. In addition, steps S315 to S329 correspond to step S211 in FIG. 6, that is, an object tracking step.

The object tracking unit 119 acquires the laser scan data at the time point t1 from the laser scanner 103 (S301).

The object tracking unit 119 filters (S303) the acquired laser scan data (S301), and performs a binarization operation on the filtered laser scan data (S305).

The object tracking unit 119 removes noise by applying a morphology operation to the laser scan data that has undergone the binarization operation (S307).

The object tracking unit 119 performs labeling processing on the laser scan data from which noise has been removed (S309). The object tracking unit 119 compares a plurality of labeled objects with the skeleton image data, and selects a matching object as a tracking object (S311). As an example, the distance and area of the labeled object may be calculated, and an object matching the skeleton shape on a plane of the same dimension may be detected.

The object tracking unit 119 deletes the remaining objects other than the tracking object selected in step S307 among the plurality of labeled objects (S313).

Thereafter, the object tracking unit 119 acquires laser scan data at a time point t2 (S315). Here, t1 and t2 denote a scan time unit, and may be, for example, a second unit.

The object tracking unit 119 performs filtering (S317) and binarization (S319) on the laser scan data at the point of time t2, and applies morphology calculation and labeling processing to the filtered and binarized laser scan data (S321, S323). do.

The object tracking unit 119 calculates the ratio of the area where the objects labeled at the time t1 and the objects labeled at the time t2 intersect (S325). In addition, the object having the largest ratio of the calculated intersection area is recognized as a tracking object (S327).

The object tracking unit 119 corrects the position of the tracked object calculated by the laser scanning method using a Kalman filter (S329). Here, as described above, the position of the tracking object is calculated as a relative position from the mobile robot 100 by using the initial position of the mobile robot 100 and distance information from the mobile robot 100 to the tracking object. Here, the distance information is generally calculated through a time when the beam irradiated by the laser scanner 103 arrives after being reflected from the object, but is not limited thereto, and various methods may be used.

The filtering process (S303, S317) of the object tracking unit 119 will be described in detail as shown in FIG. 8.

Referring to FIG. 8, the object tracking unit 119 performs primary filtering of only data having a height designated as a scan area from the laser scan data collected from the laser scanner 103 (S401).

The object tracking unit 119 may define a detailed position of a person as a laser scan area for identifying a tracked object in order to distinguish each person. The first area is the legs. It is a part that has the lowest interference with other body parts and is suitable for extracting features. However, if there are many people and are close together, it may be difficult to track the target with only the legs. To compensate for this, the buttocks and back are additionally analyzed to extract a tracking object corresponding to a person.

FIG. 9(a) shows the main height to each part for identification of a tracked object, that is, a leg, a hip, and a back. 9B is laser scan data collected from the laser scanner 103. (C) of FIG. 9 shows scan data obtained by filtering out all data except for the data corresponding to the scan area among the collected laser scan data. That is, compared to FIG. 9B, in FIG. 9C, all scan data except for the scan data of the knee height, the scan data of the pelvis, and the scan data of the back height are deleted. Therefore, unnecessary scan data is excluded from the operation.

The object tracking unit 119 compares the laser scan data with the heights (h1, h2, h3) of each scan area, and stores the data of the corresponding height if the _{error limit value and threshold are not exceeded.} The object tracking unit 119 refers to data of the same height among data classified into three heights (legs, hips, etc.) as group data.

Referring to FIG. 10, such laser scan data may be represented by a coordinate system (x, y) with the mobile robot 100 as an origin.

The object tracking unit 119 processes the labeling of group data collected from the laser scanner 103 and filtered into the scan area to group data having similar characteristics to distinguish objects having different characteristics. The criterion that the characteristics are similar can determine a human body part according to the form of data (point cluster) of laser scan data. In addition, people can be classified and grouped according to the clustered form.

The object tracking unit 119 defines variables that can be considered for labeling feature candidates from reliable laser scan data, as shown in Table 7. These variables are shown in Figure 12.

In Table 7, since an object can be identified by the group's width (d), width (h), and p value, the characteristic variables are d, h, and p. If this is derived, it is equal to Equation 4.

[Equation 4]

Again, referring to FIG. 8, the object tracking unit 119 divides group data into reference points based on the shortest distances Δd and D of two adjacent points at a specified height (S403).

Referring to FIG. 13, D denotes the distance between two adjacent points among consecutive points, and data division, i.e., detection as a different object, when exceeding _{the distance limits (D leg} , D _hip , D _{back) of two points at each height} Data partitioning is performed for this purpose.

That is, when the distance D between two adjacent points exceeds the threshold, it is recognized as belonging to another group. Here, the limit values (D _leg , D _hip , D _back ) are set as the distance at which an adjacent part of a person's body can fall to the maximum. Reliable data is extracted from the classified data group (S405). That is, secondary filtering is performed in which unnecessary data is removed by applying a filtering criterion within the divided data group (S405). This will be described as follows.

Referring to FIG. 11, a process of extracting reliable data from group data for each height is shown. Here, the triangle represents unnecessary data, and the sensor data in the center is A. Using the extracted laser scan data, the object tracking unit 119 derives a variable value to be considered as a feature for object identification.

The start of feature analysis and labeling is to extract the number of reliable data from the sensor data in the center of the labeled laser scan data.

In this case, the object tracking unit 119 applies differently to each body part when selecting the number of reliable data. That is, in the case of the leg and the back of the tracking target, unnecessary data exists in the group data. Accordingly, the object tracking unit 119 determines a reliable number of data from the center of the remaining data after removing a certain number of data from both ends of the group data. On the other hand, in the case of hips, arm and hip data are often extracted together. When the arms are classified into the same group, since the number of unnecessary data is large according to the movement of the arm, the object tracking unit 119 applies the maximum value of unnecessary data and subtracts it from the number of group data, and is reliable based on the number of remaining data. Select the number of data.

In the group data, both ends are in the horizontal direction. In FIG. 11, since the number of unnecessary data is 6 and the number of unnecessary data may increase as the scan data of the hip portion and the arm are scanned together, the number of data to be removed in advance can be set large. Equation 5 is an equation for defining data analysis of leg and back height.

[Equation 5]

Here, N _trust is the number of selected reliable data, N _group is the total number of group data, and N _outloer is the number of data excluded from group data. In other words, Equation 4 indicates that reliable data is selected by subtracting the number of unnecessary data that appears fixedly from the entire group data.

Using Equation 5, it is possible to analyze based on reliable data by subtracting the number of unnecessary data that appears fixedly from the entire group data. And because the number of reliable data varies, each person can reflect their characteristics.

Equation 6 is an equation defining data analysis of the height of a person's hips.

[Equation 6]

Here, N _trust is the number of selected reliable data, N _group is the total number of group data, and outlier _max is the maximum number of data that can be excluded.

That is, Equation 6 indicates that the maximum number of reliable data is fixedly used. In this case, consistent features can be derived from situations in which unnecessary data is generated.

Again, referring to FIG. 7, S325 and S327 of the object tracking unit 119 will be described as in FIG. 14.

Referring to FIG. 14, the object tracking unit 119 continuously tracks an object detected through a frame-to-frame matching technique. Examples of frame matching codes of the object tracking unit 119 are shown in Table 8.

The object tracking unit 119 calculates an overlapping area where the previous frame P10 and the current frame P20 intersect each other. At this time, delta_x(a) and delta_y(b) are measured through the coordinates of ①, ②, ③, and ④, and the intersection area is calculated by the product of delta_x(a) and delta_y(b). If this is formulated, it is as shown in Equation 7.

Here, the coordinates of ①, ②, ③, and ④ are calculated through a well-known mathematical theory, and are assumed to be values provided in advance.

[Equation 7]

Among the currently detected objects, the object whose intersection over union with the object detected in the previous frame has the largest match is selected as the object to be tracked, and this is expressed as Equation 8.

[Equation 8]

The "intersection over union" of Equation 8 is obtained by dividing the ratio of the intersection area by a value obtained by subtracting the intersection area from the two object areas.

The embodiments of the present invention described above are not implemented only through an apparatus and a method, but may be implemented through a program that realizes a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (17)

1. As a mobile robot,

   At least one image capture unit installed at a predetermined position of the mobile robot,

   A laser scanner installed at a predetermined position of the mobile robot,

   A memory in which a program for performing object recognition and object tracking using the at least one image capturing unit and the laser scanner is stored, and

   Including at least one processor to execute the program,

   The above program,

   Recognizes an object to be tracked based on the image generated by the at least one image capture unit, detects and tracks the object to be tracked through the laser scan data, and drives the at least one image capture unit when object tracking fails Thus, the mobile robot comprising commands for comparing the image captured at the current time and the pre-stored image, and re-detecting the object based on the laser scan data output at the current time.
2. In claim 1,

   The above program,

   The plurality of object areas detected from the laser scan data collected at the first time point and the plurality of object areas detected from the laser scan data collected at a second time point later than the first time point are intersected, and the ratio of the intersection area is Mobile robot containing a command for recognizing the largest object as a tracking object.
3. In paragraph 2,

   The above program,

   A mobile robot comprising a command for removing noise by applying a morphology operation to the two-dimensionally arranged laser scan data.
4. In paragraph 3,

   The above program,

   By labeling the laser scan data from which noise has been removed, at least one area with the same label is detected as each object,

   A mobile robot comprising a command for comparing an area detected as each object with the image and selecting a matching object as a tracking object.
5. In claim 1,

   The above program,

   Includes a command to detect an object based on the laser sensor data collected in a predetermined scan area,

   The predetermined scan area,

   A mobile robot that has a range corresponding to the height set for each body part of a person from the ground.
6. In clause 5,

   The above program,

   In the laser sensor data grouped by the predetermined scan area, commands for dividing a data group based on the shortest distance between point groups constituting the grouped laser sensor data, and calculating a feature variable for each scan area within the divided data group Containing mobile robot
7. In paragraph 6,

   The characteristic variable is,

   A mobile robot including the total width of the group data including the point groups, the distances between two adjacent point groups, the width of the group, and the average of the distances
8. As a control method of a mobile robot,

   Collecting laser scan data,

   Converting laser scan data composed of a polar coordinate system into a plurality of point groups having position values in a Cartesian coordinate system,

   Comparing the laser scan data converted in the form of a point cloud with image data captured by a camera, and binarizing using pixel values of the image corresponding to each point group,

   Labeling the laser scan data to detect a plurality of objects,

   Comparing a plurality of detected objects with the image data and selecting matching objects as tracking objects, and

   Driving toward the tracking object

   Containing, the control method.
9. In clause 8,

   Collecting the laser scan data,

   Further comprising the step of removing noise by applying a morphology mask,

   The morphology mask,

   A control method that is preset to a size that can cover a person's stride.
10. In clause 8,

    The driving step,

    The object with the largest ratio of the intersecting area is determined by comparing the plurality of objects detected from the laser scan data at the first scan time to the plurality of objects detected from the laser scan data at the second scan time later than the first scan time. Steps to recognize as tracking object

    Further comprising a, control method.
11. In claim 10,

    Recognizing the object having the largest ratio of the intersecting regions as a tracking object,

    A control method that is selectively driven according to a preset condition.
12. In clause 8,

    Collecting the laser scan data,

    Filtering the scan data corresponding to the height set for each body part of the person among the laser scan data

    Further comprising a, control method.
13. In clause 8,

    The detecting step,

    Applying the binarized laser scan data as a labeling target,

    In the data applied as the labeling target, dividing and grouping the data based on the distance between each point group, and

    Step of performing labeling to classify grouped point clouds into the same object

    Containing, control method
14. As a method of operating a mobile robot,

    The step of performing authentication for object tracking in connection with a remote server,

    If the authentication is successful, generating an image by photographing the front side, and generating a skeleton image by skeletonizing the generated image,

    Comparing the skeleton image of first users registered in advance in a user database with the generated skeleton image to determine whether the generated skeleton image is a first user,

    If it is determined as the first user, performing a task according to the request of the first user while tracking the first user using a laser scanner, and storing the work history, and

    If it is determined that it is not the first user, requesting and receiving the input of second user information and destination information, and performing a route guidance to the input destination while tracking the second user using the laser scanner

    Containing, operating method.
15. In clause 14,

    The first user tracking or the second user tracking using the laser scanner,

    Convert laser scan data composed of polar coordinates into a plurality of point groups having position values in a Cartesian coordinate system,

    The laser scan data converted in the form of a point cloud is compared with the image data captured by the camera and binarized using the pixel values of the image corresponding to each point group,

    An operation method comprising a method of comparing a plurality of objects detected by labeling the binarized laser scan data with the image data to select matching objects as tracking objects.
16. In paragraph 15,

    The storing step,

    If it is determined as the first user, extracting and registering feature point information from a captured image,

    Performing the operation requested by the first user and storing the operation history,

    Tracking a first user using the laser scanner,

    If the tracking of the first user fails, feature point information is extracted from the image acquired by photographing the front of the current point of view, and the extracted feature point information is compared with the registered feature point information to determine the object of the matching point as the first user. Step, and

    Step of moving to the determined first user's location, guiding the previous work history, inquiring if there is a willingness to follow, and continuing tracking if tracking is accepted, and terminating object tracking if tracking is rejected, and moving to the starting location

    Containing, operating method.
17. In paragraph 15,

    After the step of performing the route guidance, if the tracking of the second user fails, generating a skeleton image at the current time by skeletonizing an image generated by photographing the front of the robot, and

    Comparing a previously generated skeleton image with the skeleton image at the present time to search for a matching object, and tracking the searched object

    Further comprising, the operation method.

Images