KR102408327B1 - AI-based Autonomous Driving Robot System That Supports Gesture Recognition For Autonomous Driving Sales - Google Patents

Abstract

The AI-based autonomous driving robot system according to an embodiment of the present invention includes a mobile robot and a computing server unit that remotely controls the mobile robot, and the mobile robot acquires an image of the surroundings, and performs an object and a gesture of the object in the image. It is characterized in that it is recognized through the AI processing unit of the computing server, and can move the mobile robot or automatically sell goods according to the recognized gesture.

Description

AI-based Autonomous Driving Robot System That Supports Gesture Recognition For Autonomous Driving Sales

The present invention uses AI (Artificial Intelligence) rules to recognize a user's specific gesture as an expression of intention to purchase, and a mobile robot capable of autonomous driving and automatic selling that can sell goods to the user and a mobile robot that communicates with the mobile robot and remotely It is about an AI-based autonomous driving robot system including a computer server that can be controlled by

With the development of robot technology, artificial intelligence technology, and autonomous driving technology, robots are gradually developing into a form of providing services to users rather than simply manufacturing items of a specific shape in the manufacturing industry.

However, at present, the technical level of robots capable of commercial services is limited to delivering goods from stores or performing service actions at a predetermined level to users (for example, washing dishes or lifting and moving goods).

Therefore, in order to overcome the above-mentioned limitations of using the robot, the user's intention is confirmed, and according to the confirmed intention, the robot finds a path by itself and moves, and thereafter, the automatic vending machine where the user can purchase the necessary goods is autonomous within a specific area. There is a need for research on mobile robots or mobile robot systems that are possible through in-driving.

Republic of Korea Patent Publication No. 10-2019-0088854 (published on July 23, 2019) Republic of Korea Patent Publication No. 10-2019-0130214 (published on November 22, 2019)

An object of the present invention is to recognize a user and a specific gesture of the user, perform autonomous driving to a target location where the user is located, and provide a specific service (eg, selling a product) to the user, an AI-based autonomous driving robot system is to provide

Another object of the present invention is to utilize an AI processing unit when recognizing the user's gesture in the AI-based autonomous driving robot system, and the AI processing unit uses an artificial intelligence algorithm such as Mask R-CNN to increase processing speed and It is to make it suitable for environments with frequent instantaneous movement, such as mobile robots, by increasing the accuracy.

Another object of the present invention is to estimate its own location while simultaneously estimating its location by using information obtained from various sensor units rather than simply using a location-navigation-based method such as GPS when the mobile robot of the AI-based autonomous driving robot system is driven. It is to provide features that can create maps for

An AI-based autonomous driving robot system according to an embodiment of the present invention includes a mobile robot and a computing server for remotely controlling the mobile robot,

The mobile robot includes an image acquisition unit for acquiring an image of the surroundings; a sensor unit including a plurality of sensors to predict the position of the mobile robot; a control unit for processing and determining a plurality of sensor data obtained from the sensor unit; a communication unit for transmitting the sensor data processed from the control unit and the image data obtained from the image acquisition unit to the computing server unit at a remote location, or receiving robot control data related to control of the mobile robot from the computing server unit; a driving unit for controlling the operation of the mobile robot; and an automatic sales module that recognizes an object and automatically sells a product by using AI from the image data.

The computing server unit according to an embodiment of the present invention comprises: a gesture recognition module for recognizing a specific object and a specific gesture of the object from the image data received through the communication unit of the mobile robot; A destination coordinate (waypoint) is calculated to control the moving path of the mobile robot using a plurality of sensor data obtained from the mobile robot, and according to the calculated destination coordinate, the mobile robot traverses a predetermined path. Path traversal control module to control; a navigation module for controlling navigation of the mobile robot according to the destination coordinates; and a server communication unit controlling data transmission and reception with the mobile robot.

According to an embodiment of the present invention, the gesture recognition module includes a learning unit for learning the neural network from the image data and an AI processing unit for determining an object and a gesture of the object based on a result learned from the learning unit.

According to an embodiment of the present invention, the AI processing unit uses Mask R-CNN as a neural network for recognizing the object as an individual class, and uses binary classification consisting of a person and a background as the individual class, Mask processing is performed on the area of the person object of the image data only when the person class is recognized.

According to an embodiment of the present invention, the learning unit uses Resnet-based model learning based on the mask-processed image data in order to determine the gesture of the object, and allows learning of the Resnet-based model For this purpose, it is characterized by using a dataset generating program that generates images of people of various gestures.

According to an embodiment of the present invention, the navigation module is characterized in that when moving to the destination coordinates, a Simultaneous Localization and Mapping (SLAM) method using the acquired plurality of sensor data is used.

According to an embodiment of the present invention, the route traversal control module moves based on a single or a plurality of destination coordinates determined by a gesture of the user's object or designated by the user, but reaches the final destination coordinates among the plurality of destination coordinates Then, the mobile robot moves by setting the movement path in reverse (reverse), and when the specific gesture is recognized by the gesture recognition module, it is stopped.

According to an embodiment of the present invention, when the user specifies a destination coordinate (waypoint) in the route traversal control module, an Rviz method is used for visualization, and the form of data used in the Rviz method is a destination coordinate In addition, the posture information of the mobile robot is included.

According to an embodiment of the present invention, the navigation module includes a global planner for voyage to a relatively long distance and a local planner for voyage closer than the global planner.

According to an embodiment of the present invention, the global planner uses a Dijkstra algorithm or A* algorithm, and the local planner uses a Dynamic Window Approach method.

The AI-based autonomous driving robot system of the present invention recognizes a user and a specific gesture of the user, autonomously drives to a target location where the user is located, and provides a specific service such as selling goods to the user to improve the service capability of the robot is to make

The present invention utilizes an AI processing unit when recognizing a user's gesture, and the AI processing unit uses an artificial intelligence algorithm such as Mask R-CNN to increase both processing speed and accuracy to provide functions such as automatic sales It is suitable for environments where instantaneous movement is frequent, such as mobile robots.

The present invention uses information obtained from various sensor units rather than simply a location-navigation-based method such as GPS when driving a mobile robot, estimating its own location and creating a map for movement at the same time, and creating a map for movement when moving to a destination. By dualizing the operation method of moving a long distance and the operation method of moving a short distance with the

1 is a block diagram showing the overall configuration of a mobile robot according to an embodiment of the present invention.
2 is a diagram illustrating an AI-based autonomous driving robot system including a mobile robot and a computing server according to an embodiment of the present invention.
3 is a diagram illustrating the application of Mask R-CNN of the AI processing unit according to an embodiment of the present invention.
4 exemplifies a dataset program that generates human images of various gestures in order to perform Resnet-based learning in the AI processing unit.
5 is an exemplary design diagram of a mobile robot of the present invention, the left drawing is a front view, and the right drawing is a side view.
6 is a view for explaining the dynamic window approach, which is an operation method of the local planner of the mobile robot of the present invention.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the presented embodiments, and those skilled in the art who understand the spirit of the present invention may add, change, delete, etc. other elements within the scope of the same spirit, and may use other degenerative inventions or the present invention. Other embodiments included within the scope of the present invention may be easily proposed, but these will also be included within the scope of the present invention. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals.

1 and 5 are a block diagram showing an overall configuration of a mobile robot 1000 according to an embodiment of the present invention, respectively, and an exemplary design diagram of the mobile robot 1000 according to an embodiment of the present invention.

The mobile robot 1000 includes an image acquisition unit 1100 for acquiring an image of the surroundings, a sensor unit 1200 including a plurality of sensors to predict the position of the mobile robot, and a plurality of sensors obtained from the sensor unit 1200 . The control unit 1300 that processes and determines the sensor data of the control unit 1300, and transmits the sensor data processed from the control unit 1300 and the image data obtained from the image acquisition unit to the computing server unit 2000 to be described later in a remote location, and a communication unit 1400 for receiving robot control data related to the control of the mobile robot 1000 from the computing server unit 2000 .

In addition, the mobile robot 1000 includes a driving unit 1500 for controlling the operation of the mobile robot to move the lo-mobile robot; and an automatic vending module 1600 that recognizes an object and automatically sells a product by using AI from the image data.

Specifically, the mobile robot 1000 refers to a mobile robot capable of basic autonomous driving.

In addition, the sensor unit 1200 may include a plurality of sensors, and may include an encoder capable of measuring the plurality of sensors, an inertial measurement unit (IMU), the number of rotations of the wheel, etc. By outputting it, it may be utilized for calculation of destination coordinates in the computing server unit 2000, which will be described later.

In addition, the control unit 1300 controls the communication unit 1400 that communicates with the computing server unit 2000 to be described later, or through the destination coordinates or other robot control data received from the computing server unit 2000. It performs a function of controlling the driving unit 1500 .

In addition, the communication unit 1400 transmits the sensor data obtained from the sensor unit 1200 or the image data obtained from the image acquisition unit 1100 to the server communication unit 2400 of the computing server unit 2000, and , performs a function of receiving various robot control data (including data for controlling robot driving or destination coordinates to be driven) from the computing server unit 2000 .

In addition, the communication unit 1400 also transmits information related to various event data (user's product purchase, user's payment information, etc.) generated in the automatic vending module 1600 to be described later to the computing server unit 2000 or the computing unit 1400 . Data related to the automatic sales may be received from the server unit 2000 .

In addition, in the communication method between the communication unit 1400 and the server communication unit 2400, a wide variety of wireless communication methods such as Bluetooth, Wi-Fi, LTE-M, and 5G may be used, and for the security of data during such wireless communication, Several security schemes may be implemented. More specifically, the security scheme may use a one-way encryption security method that does not require high computing power, such as a lightweight hash function, because the mobile robot 1000 is implemented in the form of an embedded device, and the computing power is not high.

Finally, although not shown in detail in the drawings, in the case of the automatic vending module 1600, in order to provide a function of supplying a product or providing a certain service to a user and receiving a user's feedback therefor, a display unit is additionally (reference numeral not shown) may be included. In addition, the vending module 1600 additionally includes a user input unit (not shown), or provides an accommodating space for storing goods, and provides data from the computer server unit 2000 (eg, appropriate user payment). Since this has been completed, the product may further include a product storage unit that provides an appropriate locking means so that the product can be supplied only after receiving the product.

2 is a diagram illustrating an AI-based autonomous driving robot system 10000 including a mobile robot 1000 and a computing server 2000 according to an embodiment of the present invention.

Since the detailed configuration and operation of the mobile robot 1000 have already been described above, the computing server unit 2000 will be described herein.

The computing server unit 2000 may further include a gesture recognition module 21000 for recognizing a specific object and a specific gesture of the object from the image data received through the communication unit 1400 of the mobile robot 1000 . .

In addition, the computing server unit 2000 calculates a destination coordinate (waypoint) so as to control the movement path of the mobile robot 1000 using a plurality of sensor data obtained from the mobile robot 1000, A path traversal control module 2200 for controlling the mobile robot 1000 to traverse a predetermined path according to the destination coordinates may be further included.

More specifically, the destination coordinates calculated by the route traversal control module 2200 are transmitted to the mobile robot 1000 through the server communication unit 2400, which is transmitted through the communication unit 1400 of the mobile robot 1000 to the control unit ( 1300 ), and is used so that the control unit 1300 can control the driving unit 1500 .

In addition, the computing server unit 2000 according to the destination coordinates, the navigation module 2300 for controlling the navigation (navigation) of the mobile robot; and a server communication unit 2400 for controlling data transmission and reception with the mobile robot 1000 .

In addition, the gesture recognition module 2100 further includes the following detailed configuration to accurately recognize the object of the image data and the gesture of the object using artificial intelligence, that is, AI.

More specifically, the gesture recognition module 2100 further includes a learning unit for learning the image data from the neural network and an AI processing unit for determining an object and a gesture of the object based on a result learned from the learning unit.

In particular, the learning unit and the AI processing unit should be able to accurately determine only the fast recognition of an object and the presence or absence of a specific gesture among various neural network algorithms so that only a specific gesture of the object can be quickly and accurately recognized.

More specifically, the AI processing unit uses a Mask R-CNN (Regional Convolutional Neural Network) as a neural network for recognizing the object as an individual class, and as the individual class, a binary classification consisting of a person and a background (classification) is used to mask the area of the person object of the image data only when the person class is recognized.

That is, by classifying the class to be recognized into only two classes, human and background, the accuracy of neural network learning can be improved. In addition, the neural network receives RGB as an input, and outputs a mask image in which a human shape is detected as an output.

In particular, the region where the object is estimated to be located is extracted from the image data acquired through R-CNN, the CNN neural network is applied only to this region, and the region recognized as the desired object is masked to increase the learning processing speed. , it has the advantage that the processing speed can be performed much faster than other recognition methods.

In the case of FIG. 3, the mask R-CNN is applied to the above-described AI processing unit to recognize and mask a specific human gesture (the act of raising a hand to meet the mobile robot).

In addition, in order to determine the gesture of the object, the learning unit uses Resnet (Residual Neural Network)-based model learning based on the mask-processed image data, and to learn the Resnet-based model, A dataset creation program that generates images of people with various gestures is used.

In the case of Resnet, as described above, with respect to the previously masked objects, it is a neural network used to accurately recognize whether the gesture of the object we want to find is a gesture, and it is relatively easy to optimize neural network learning through residual learning, It also has an effective feature in lowering the error rate during learning.

In the case of FIG. 4, in order to perform Resnet-based learning in the learning unit, it is exemplified that a data set program for generating human images of various gestures is used.

In addition, the navigation module 2300 of the computing server unit 2000 uses a Simultaneous Localization and Mapping (SLAM) method that can move using the acquired plurality of sensor data when moving to the destination coordinates described above. characterized in that

Through the SLAM method, the navigation module 2300 uses various information obtained from the sensor unit (encoder and IMU, etc.) of the mobile robot 1000 to estimate its location and create a map at the same time have The SLAM method has a characteristic that is particularly suitable for indoor movement, compared to a method of generating and moving a map using only the GPS method.

In addition, the route traversal control module 2200 moves based on a single or a plurality of destination coordinates determined by the gesture of the user's object or designated by the user, and when reaching the final destination coordinates among the plurality of destination coordinates, the mobile It is possible to set the movement path of the robot 1000 in reverse (reverse) to move.

In addition, the path traversal control module 2200 recognizes the specific gesture in the gesture recognition module 2100, calculates the destination coordinates, and then arrives at the destination coordinates (that is, the user who has taken the specific gesture is close). If judged), stop. Thereafter, the user may use the automatic vending module 1600 of the mobile robot 1000 to purchase a desired product or the like.

Also, specifically, the route traversal module 2200 uses the Rviz method for visualization when the user designates a destination coordinate (waypoint).

Rviz is a 3D-based visualization tool used in ROS (Robot Operating System), and has a feature that can be visualized using sensor data of the mobile robot 1000, and the form of data used in the Rviz method is only the destination coordinates. Rather, the posture information of the mobile robot 1000 is included so that the user can more precisely control the mobile robot 1000 as a result.

In addition, the navigation module 2300 is characterized in that two approaches are used together when moving in order to use the destination coordinates accurately and quickly.

More specifically, the navigation module 2300 includes a global planner for navigation to a relatively long distance and a local planner for a navigation closer than the global planner.

In the case of the global planner, the Dijkstra algorithm or the A* algorithm may be used to lead the navigation to the destination coordinates up to a relatively long distance.

On the other hand, in the case of the local planner, it is characterized in that a dynamic window approach method is used.

More specifically, in the Dynamic Window Approach, the destination of a general navigation system is x and y coordinates, while avoiding obstacles that may collide with the robot and selecting a speed that can quickly reach the destination as an objective function. , the objective function G is characterized by selecting the translational speed v and the rotational speed w. That is, through this method, it is possible to present the effect of approaching the destination as quickly as possible while moving at a relatively short distance.

6 is a diagram for explaining the above-described dynamic window approach.

10000: AI-based autonomous driving robot system
1000 : mobile robot
1100: image acquisition unit
1200: sensor unit
1300: control unit
1400: communication department
1500: drive unit
1600: vending module
2000: Computing Server Department
2100: gesture recognition module
2200: path traversal control module
2300 : navigation module
2400: server communication unit
4000: mobile device

Claims (5)

In the autonomous driving robot system,
the system is
Comprising a mobile robot and a computing server for remotely controlling the mobile robot,
The mobile robot
an image acquisition unit for acquiring an image of the surroundings;
a sensor unit including a plurality of sensors to predict the position of the mobile robot;
a control unit for processing and determining a plurality of sensor data obtained from the sensor unit;
a communication unit that transmits the sensor data processed from the control unit and the image data obtained from the image acquisition unit to the computing server unit at a remote location, or receives robot control data related to control of the mobile robot from the computing server unit;
a driving unit for controlling the operation of the mobile robot; and
An automatic selling module that recognizes an object and automatically sells a product by using AI from the image data,
The computing server unit
a gesture recognition module for recognizing a specific object and a specific gesture of the object from the image data received through the communication unit of the mobile robot;
A destination coordinate (waypoint) is calculated to control the movement path of the mobile robot using a plurality of sensor data obtained from the mobile robot, and according to the calculated destination coordinate, the mobile robot traverses a predetermined path. a path traversal control module to control;
a navigation module for controlling navigation of the mobile robot according to the destination coordinates; and
Further comprising a server communication unit for controlling data transmission and reception with the mobile robot,
The gesture recognition module
a learning unit for learning the image data by a neural network;
Further comprising an AI processing unit for determining the object and the gesture of the object based on the result learned from the learning unit,
The AI processing unit
Mask R-CNN is used as a neural network for recognizing the object as an individual class, and binary classification consisting of a person and a background is used as the individual class, and only when the object is recognized as the person class, the image data is It is characterized by masking the area of the human object,
The route traversal module uses the Rviz method for visualization when the user specifies the destination coordinates (waypoint),
The navigation module includes a global planner for navigation to a relatively long distance when moving, and a local planner for navigation closer than the global planner, in order to use the destination coordinates accurately and quickly. AI-based autonomous robot system characterized by using two approaches. delete delete delete The method according to claim 1,
The learning unit is characterized in that it uses Resnet-based model learning based on the mask-processed image data to determine the gesture of the object, and generates human images of various gestures to learn the Resnet-based model AI-based autonomous driving robot system, characterized in that it uses a data set creation program that

Images