WO2022127439A1

WO2022127439A1 - Robot obstacle avoidance processing method and apparatus, device, and computer readable storage medium

Info

Publication number: WO2022127439A1
Application number: PCT/CN2021/129398
Authority: WO
Inventors: 李泽华; 张涛; 陈永昌; 申鑫瑞
Original assignee: 深圳市普渡科技有限公司
Priority date: 2020-12-17
Filing date: 2021-11-08
Publication date: 2022-06-23
Also published as: CN112506204B; CN112506204A

Abstract

A robot obstacle avoidance processing method and apparatus, a device, and a computer readable storage medium. The method comprises: when it is detected that there is an obstacle on a current walking route of a robot, controlling the robot to stop walking on the current walking route (step S101); detecting whether the obstacle on the current walking route of the robot is a person (step S102); if the obstacle on the current walking route of the robot is a person, enabling a first prompt mode, to prompt the person to avoid the robot (step S103); and if the obstacle on the current walking route of the robot is not a person, enabling a second prompt mode, to prompt a worker in the scene to assist in eliminating the obstacle which is not a person (step S104).

Description

Robot failure handling method, apparatus, device and computer-readable storage medium

technical field

This application requires the priority of a Chinese patent application filed with the State Intellectual Property Office of China on December 17, 2020, the application number is 202011497739.X, and the application name is "Robot obstacle handling method, device, equipment and computer-readable storage medium" rights, the entire contents of which are incorporated herein by reference.

The present application relates to the field of artificial intelligence, and in particular, to a method, apparatus, device and computer-readable storage medium for handling a robot in trouble.

Background technique

With the rapid development of artificial intelligence (AI), AI technology has gradually been applied to fields closely related to people's lives. For example, robot restaurants and smart warehouses are typical applications of AI technology in the field of people's livelihood. In these application scenarios, intelligent robots not only bring a sense of high-tech to users, but also bring users other experiences, such as dining pleasure, efficient handling, and so on.

In robot restaurants, smart warehouses and other occasions where robots undertake the main work and are crowded with people, the walking route of the robot may be blocked by real people or other objects. For the dilemma in this scenario, the existing solution is to try to make the robot have adaptive ability, that is, to make the robot have the function of automatic obstacle avoidance, and change the preset walking route by itself when encountering obstacles such as real people, so as to avoid obstacles. Open obstacles.

However, the above-mentioned existing solutions need to be built on the basis of extensive training of robots in advance. On the one hand, it will mean high costs; on the other hand, if the final training results of the robots are not intelligent enough, not only can the robots fail to Obstacle avoidance may also encounter obstacles and cause damage to the robot, resulting in higher costs.

technical solutions

According to various embodiments of the present application, on the one hand, a method for handling an obstacle of a robot is provided, including:

When detecting that there is an obstacle on the current walking route of the robot, controlling the robot to stop moving forward on the current walking route;

Detecting whether the obstacles on the current walking route of the robot are real people;

If the obstacle on the current walking route is a real person, enable a preset first prompt mode to prompt the real person to avoid the robot;

If the obstacle on the current walking route is not a real person, a preset second prompt mode is enabled to prompt the staff in the scene to assist in removing the obstacle that is not a real person.

On the other hand, the present application provides an obstacle handling device for a robot, comprising:

the robot stops moving on the current walking route;

a detection module for detecting whether the obstacles on the current walking route of the robot are real people;

a first prompting module, configured to enable a first prompting mode if the obstacle on the current walking route is a real person, so as to prompt the real person to avoid the robot;

The second prompting module is configured to enable a second prompting mode if the obstacle on the current walking route is not a real person, so as to prompt the staff in the scene to assist in removing the obstacle that is not a real person.

In a third aspect, the present application provides a device, the device includes a memory and a processor, the memory stores a computer program, the computer program can be executed on the processor, and the processor executes the The computer program implements the steps of the technical solution of the above-mentioned method for handling an obstacle in a robot.

In a fourth aspect, the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the technical solution of the above-mentioned method for handling an obstacle in a robot are realized. .

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features and advantages of the present application will become apparent from the description, drawings, and claims.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that are used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, the drawings of other embodiments can also be obtained according to these drawings without creative efforts.

FIG. 1 is a flowchart of a method for handling an obstacle in a robot provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of forming a syntax tree by splitting a SQL statement into corresponding statements, conditions, expressions, etc. by performing semantic analysis on SQL provided by an embodiment of the present application;

3a is a schematic diagram of a robot provided by an embodiment of the present application adopting a ">"-type walking mode to avoid obstacles on the current walking route of the robot;

3b is a schematic diagram of a robot provided by an embodiment of the present application adopting a "<" type walking mode to avoid obstacles on the current walking route of the robot;

4 is a schematic structural diagram of a robot obstacle handling device provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a device provided by an embodiment of the present application.

Embodiments of the present invention

In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the related drawings. The preferred embodiments of the present application are shown in the accompanying drawings. However, the application may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that a thorough and complete understanding of the disclosure of this application is provided.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field of the invention. The terms used herein in the description of the invention are for the purpose of describing particular embodiments only and are not intended to limit the application. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The present application proposes a method for handling obstacles of a robot, which can be applied to a robot. The robot can be a robot operating in a restaurant, such as a food delivery robot, or a medicine delivery robot operating in a medical place, such as a hospital, or It is a transfer robot that works in warehouses and other places, and so on. The places where these robots work may have both adults (eg, adult patients in hospitals and other places) and minors such as young children (eg, toddlers, children, etc., eating in restaurants). As shown in FIG. 1 , the method for handling an obstacle to a robot mainly includes steps S101 to S104, which are described in detail as follows:

Step S101 : when it is detected that there is an obstacle on the current walking route of the robot, the robot is controlled to stop moving forward on the current walking route.

The so-called obstacle refers to the person or thing on the predetermined route of the mobile device, such as a robot, that hinders the mobile device from moving forward. In this embodiment of the present application, an optoelectronic method, for example, a visual sensor, a laser radar, or an infrared thermal imaging device, etc., can be used to detect whether there is an obstacle on the current walking route of the robot. Once it is detected that there is an obstacle on the current walking route of the robot, the central processing unit in the robot sends a stop operation instruction to the driving unit, so that the robot stops moving forward on the current walking route.

Step S102: Detect whether the obstacles on the current walking route of the robot are real people.

As mentioned above, the obstacles on the walking route of the robot are mainly divided into people or objects. In the embodiment of this application, different obstacle removal schemes can be adopted for different obstacles. Therefore, when an obstacle is detected on the current walking route of the robot, it is also necessary to detect whether the obstacle on the current walking route of the robot is a real person. A real person”, in this application, refers to a real person, a natural person.

Since the heat signal emitted by the human body is different from the heat signal emitted by other objects for the infrared sensor, as an embodiment of the present application, to detect whether the obstacle on the current walking route of the robot is a real person, the heat signal installed on the robot can be used. The infrared sensor receives one or more heat signals from obstacles on the current walking route of the robot; if the wavelength of the heat signal is about 5~12um, it is determined that the obstacles on the current walking route of the robot are real people.

The above embodiment is based on the principle of thermal imaging to detect whether the obstacle on the current walking route of the robot is a real person. In another embodiment of the present application, whether the obstacle on the current walking route of the robot is a real person can be detected based on the principle of computer graphics. Specifically, In other words, it can be implemented through steps S201 to S205 as shown in FIG. 2, and the description is as follows:

Step S201: According to the image of the obstacle acquired by the image acquisition device, determine whether the obstacle on the current walking route of the robot is a humanoid.

In this application, the so-called "human-like" refers to a person or object with human physical characteristics. From the perspective of computer graphics, real people and objects similar to real people, such as mannequins, artificial robots, etc., all belong to human-like bodies. Therefore, when it is determined that the obstacle on the current walking route of the robot is not a humanoid, the obstacle cannot be a real person. The image of the obstacle can be obtained through the image acquisition device integrated on the robot, such as a camera, and then the image of the obstacle is matched with the pre-stored human-like images in the database on geometric parameters such as shape and size to determine the current walking of the robot. Whether the obstacles on the route are humanoid.

Step S202: if the obstacle on the current walking route of the robot is a human body, collect the key feature point information in the human body-like face image for the human body-like face, wherein the key feature point information in the human body-like face image includes the key feature point information. The plane position information of the feature point.

As mentioned earlier, humanoids include real people and real-life-like objects. Therefore, when it is determined that the obstacle on the robot's current walking route is not a humanoid, the obstacle cannot be a real person; when it is determined that the obstacle on the robot's current walking route is a humanoid, it is necessary to further confirm whether the humanoid is a real human. The fact is that the properties of real people and the properties of objects similar to real people still have essential differences in physiological and/or psychological sense. For example, real people can understand semantics, so they can actively avoid robots when they hear the prompt voice issued by robots, but Humanoids, such as mannequins, cannot understand the semantics and thus cannot avoid it.

In order to further improve the acquisition accuracy of the human-like face image, in the embodiment of the present application, when collecting the human-like face image, different image acquisition devices may be used to target the human-like face from different positions at the same time. At least two images are collected, and then the at least two human-like face images are corrected so that the at least two human-like face images are consistent in the horizontal direction. After acquiring at least two human-like face images in the same horizontal direction, perform key feature point detection on each human-like face image respectively, and determine several key feature point information in each human-like face image. In the embodiment of the present application, the key feature points include 65 to 106 key points including the tip of the nose, the lower eyelid point of the left eye, the lower eyelid point of the right eye, the corner of the left mouth, and the corner of the right mouth. For example, after determining these key feature points in the human-like face image, the plane position information and reliability of these key feature points in the human-like face image can be further determined, wherein the plane position information can be the key feature points in the The two-dimensional coordinates in the human-like face image, and the reliability is used to indicate the accuracy of the positioning of key feature points in the human-like face image.

Step S203: According to the plane position information of the key feature points in the human-like face image, obtain the three-dimensional position information of these key feature points.

As mentioned above, the credibility indicates the accuracy of the key feature points in the human-like face image. Therefore, according to the credibility of each key feature point, the key features whose credibility is greater than a certain threshold can be selected. point to determine the three-dimensional position information of these key feature points, for example, the three-dimensional coordinates of the key feature points in the three-dimensional coordinate system. Specifically, the three-dimensional position information of each key feature point can be calculated according to the plane position information of each key feature point based on the similar triangle principle.

Step S204 : obtaining a face fitting curved surface according to the three-dimensional position information of the key feature points.

After obtaining the three-dimensional position information of the key feature points, a surface fitting algorithm, such as the least squares method, can be used to fit the three-dimensional position information of these key feature points as a fitting factor to fit a surface, which is the face fitting surface .

Step S205: According to the distance between the key feature points in the human-like face image and the face fitting surface, determine whether the obstacle on the current walking route of the robot is a real person.

Generally, for non-real face images, the depth information of the key feature points is not obvious, and the face fitting surface obtained by fitting is generally a smooth face surface, and the key feature points are connected to such a face fitting surface. The distance is also small, approaching zero, and the depth information of the key feature points in the face image of a real person is obvious. The fitted face fitting surface is generally not a smooth face surface, and the key feature points are like this The distance of the face fitting surface is also larger. Based on the above imaging characteristics, in the embodiment of the present application, according to the distance between the key feature points in the human-like face image and the face fitting surface, judging whether the obstacle on the current walking route of the robot is a real person may be: calculating the human-like face image The sum of the distances between the key feature points in the face image and the face fitting surface. If the sum of the distances is greater than the dynamic depth information threshold, it is determined that the humanoid is a real person. If the sum of the distances is not greater than the dynamic depth information threshold, Then it is determined that such a human body is not a real person. In the above-mentioned embodiment, the a priori depth distance threshold preset in the dynamic depth information threshold is determined in real time, and the credibility of each key feature point used to calculate the sum of the distance between the key feature point and the face fitting surface is determined. degree related.

Step S103 : if the obstacle on the current walking route is a real person, the preset first prompt mode is activated to prompt the real person to avoid the robot.

Specifically, the first prompting mode may be a prompting manner of voice prompting, image prompting or a combination of voice and image.

Although the obstacles on the current walking route are all real people, there are still some differences between real people, for example, young children and adults, young children generally refer to children under a certain age, such as under 5 years old, may not be accurate or accurate enough Understand semantic information, but can understand images (including videos, animations, etc., videos and animations can be regarded as images that are played continuously) and other visual representation methods, but adults generally do not have semantic understanding under the premise of normal intelligence. obstacle. Therefore, for this difference, different prompting methods can be given. Specifically, step S103 can be implemented through the following steps S1031 to S1033:

Step S1031: Determine whether the real person on the current walking route of the robot is a child.

In the embodiment of the present application, the image of the real person on the current walking route of the robot can be collected through an image collection device integrated on the robot, such as a camera, etc., and then the collected image of the real person is compared with the image of the child preset in the database. Matching is carried out from the physical features to determine whether the real person on the current walking route of the robot is a young child. It is also possible to determine whether the real person on the robot's current walking route is a child by collecting the voice information of the real person on the robot's current walking route. For example, when the robot plays a voice to the real person, the real person may playfully say "I won't let go. "To respond to the robot, when the robot collects the voice information of "I will not let go", it can determine whether the real person on the robot's current walking route is a young child according to its voiceprint features.

Step S1032 : if the real person is a young child, form a terrain feature animation of the surrounding environment of the young child, and prompt the young child to avoid the robot through the terrain feature animation combined with voice.

As mentioned above, compared with adults, young children have limited understanding of semantic information, and the combination of semantic explanation based on images can make it easier for young children to understand semantic information. Therefore, in this embodiment of the present application, after determining that the real person on the current walking route of the robot is a young child, the environment around the young child can be formed into a terrain feature animation, and the young child can be prompted to avoid the robot through the terrain feature animation combined with voice. Specifically, forming a terrain feature animation of the surrounding environment of the child, and prompting the child to avoid the robot through the terrain feature animation combined with voice can be: by sensing the surrounding environment data of the child, determine the prompt information of the robot's current walking route; The animation processing model extracts the features of the prompt information of the robot's current walking route to form a terrain feature animation; projects the terrain feature animation on the target projection surface; uses the child's voice to explain the terrain feature animation projected on the target projection surface to Guide young children to avoid robots.

Step S1033: If the real person is not a child, prompt the real person who is not a child to avoid the robot through voice and the generated facial expression image.

Because adults have a strong ability to understand semantics, and at the same time, in order to make the request more humanized, as an embodiment of the present application, if the real person is not a child, the real person who is not a child will be prompted through voice and generated facial expressions. The avoidance robot may be: displaying the generated expression image to express embarrassment on a display device with a preset exaggeration method, and cyclically playing the expression image to express embarrassment to the real person who is not a child at a fixed interval. facial expression images and prompt voices with actual semantics, requesting the real person who is not a child to avoid the robot. When adults see these exaggerated embarrassing emoji images, plus looping these embarrassing emoji images and prompting voices with actual semantics (eg, please let me go, etc.), from the human Angle, under normal circumstances, these adults will actively avoid the robot and let it pass smoothly.

Step S104: If the obstacle on the current walking route of the robot is not a real person, the second prompt mode is enabled to prompt the staff in the scene to assist in removing the obstacle that is not a real person.

Specifically, the second prompt module may select an alert mode.

Obviously, when the obstacle on the robot's current walking route is not a real person, the aforementioned voice prompt mode and/or image prompt mode cannot take effect or the effect is not good (of course, when the voice prompt is played, the staff in the scene may also come to exclude the The obstacle of a real person, but there is still an effect difference from the active avoidance of a real person). Therefore, if the obstacles on the current walking route of the robot are not real people, the warning prompt mode is enabled to prompt the staff in the scene to assist in removing non-human obstacles. For example, after the warning prompt mode is enabled, the robot emits an audible and visual alarm to remind the staff in the scene that the robot encounters obstacles that are not real people, and needs him/her to assist in removing these non-human obstacles.

Further, after step S104, step S105 may also be included: after enabling the first prompt mode, or after enabling the second prompt mode, if the real person still does not avoid the obstacle of the robot or the non-human person, then upgrade the first prompt mode. The prompting mode or the second prompting mode continues to prompt.

In some scenarios, it is not excluded that after enabling the voice prompt mode and/or image prompt mode, or after enabling the warning prompt mode, the obstacle that the real person still does not avoid the robot or the non-human person is still not eliminated. In this scenario, the prompting mode can be upgraded. For example, after enabling the voice prompting mode and/or image prompting mode and repeating the voice prompting and/or image prompting to the real person for more than 3 times, the real person still does not avoid the robot and can enhance the voice Voice decibels and/or use more exaggerated methods when prompting, for example, playing angry facial expressions; or, after enabling the warning prompt mode, if the non-human obstacle has not been eliminated, increase the intensity of sound and light alarms, for example, Make a louder alarm sound, make the alarm light glow more intensely, or increase the flashing frequency of the alarm light, etc.

In some scenarios, it is not ruled out that after upgrading the prompt mode, the real person still does not avoid the robot or the obstacle of the non-real person is still not eliminated, for example, the young children on the current walking route of the robot still cannot understand the voice prompts and/or image prompts, or , in a specific place such as a hospital, although adult patients on the robot's current walking route can understand the voice prompts and/or image prompts, they still cannot actively avoid the robot due to disease reasons, or, despite the upgraded warning prompt mode, out of order For various other reasons, the staff could not arrive in time to eliminate the obstacles that are not real people. In this scenario, the detouring trajectory of the current walking route around the obstacle can be adjusted according to the size of the obstacle, and the robot can be controlled to travel on the adjusted walking route to avoid the obstacle. For example, the robot can be controlled Use the "<" or ">" walking method to avoid obstacles on the robot's current walking route. As shown in Figure 3a, the robot can first deviate a small angle to the right as shown by the dotted line, walk a distance from the right side of the obstacle, and then turn left (at this time, a ">"-type route is formed), and return to the preset Or, as shown in Figure 3b, the robot can first deviate a small angle to the left, walk a distance from the left of the obstacle, and then turn right (at this time, a "<" type route), return to the preset walking route. Because whether the "<"-type walking method or the ">"-type walking method is adopted, the robot deviates from the predetermined route by a small angle, so the algorithm is not complicated, and there is no need for complex training of the robot. accomplish.

In other embodiments, it may also be the first prompt mode or the second prompt mode, or after the upgrade of the first prompt mode or the second prompt mode reaches a preset time. The detouring trajectory of the current walking route around the obstacle can be adjusted according to the size of the obstacle, and the robot is controlled to travel on the adjusted walking route to avoid the obstacle.

In order to save the consumption of resources such as electricity, in the embodiment of the present application, the voice prompt mode and/or the image prompt mode may be turned off after the real person avoids the robot, or the warning prompt mode may be turned off after the non-human obstacle is removed.

It can be seen from the above-mentioned method for dealing with obstacles in the example of Fig. 1 that when the obstacle is confirmed to be a real person, the voice prompt mode and/or image prompt mode are enabled to prompt the real person to avoid the robot, and when it is confirmed that the obstacle is not a real person, the warning prompt mode is enabled to prompt work Personnel assists in removing obstacles, because no matter it is voice prompts, image prompts, warning prompts or the upgrade of the three, there is no need to train the robot. Therefore, compared with the scheme of carrying out a lot of training on the robot to remove obstacles, on the one hand, the application of the The cost of the technical solution is low; on the other hand, as long as the voice prompt mode, the image prompt mode, the warning prompt mode or the upgrade mode of the two are enabled, the purpose of removing obstacles can be achieved, and there is no need for robot training as in the prior art. The worry of not being able to achieve the expected results when it is in place.

Please refer to FIG. 4 , which is an apparatus for handling an obstacle of a robot provided by an embodiment of the present application, which may include a stop operation module 401 , a detection module 402 , a first prompt module 403 , and a second prompt module 404 , and may further include The third prompt module 405 is described in detail as follows:

The stop operation module 401 is used to control the robot to stop moving forward on the current walking route when it is detected that there is an obstacle on the current walking route of the robot;

The detection module 402 is used to detect whether the obstacle on the current walking route of the robot is a real person;

The first prompting module 403 is configured to enable the first prompting mode if the obstacle on the current walking route of the robot is a real person, so as to prompt the real person to avoid the robot;

The second prompting module 404 is configured to enable the second prompting mode if the obstacle on the current walking route of the robot is not a real person, so as to prompt the staff in the scene to assist in removing the obstacle that is not a real person;

The third prompting module 405 is used for, after the first prompting module 403 enables the first prompting mode, or after the second prompting module 404 enables the second prompting mode, if the real person still does not avoid the obstacle of the robot or the non-human person, Then the upgrade prompt mode continues to prompt.

Optionally, the detection module 402 in the example of FIG. 4 includes a first determination unit, a collection unit, a three-dimensional position information acquisition unit, a fitting unit and a judgment unit, wherein:

a first determining unit, configured to determine whether the obstacle on the current walking route of the robot is a humanoid according to the image of the obstacle acquired by the image acquisition device;

The acquisition unit is used to collect the key feature point information in the human-like face image for the human-like face if the obstacle on the current walking route of the robot is a human-like face, wherein the key feature point information in the human-like face image includes: The plane position information of the key feature point;

a stereoscopic position information acquisition unit, configured to obtain the stereoscopic position information of the key feature points according to the plane position information of the key feature points in the human-like face image;

The fitting unit is used to obtain the face fitting surface according to the three-dimensional position information of the key feature points;

The judgment unit is used for judging whether the obstacle on the current walking route of the robot is a real person according to the distance between the key feature points in the human-like face image and the face fitting surface.

Optionally, the above judgment unit may include a calculation unit and a second determination unit, wherein:

a calculation unit, used to calculate the sum of the distances between the key feature points in the human-like face image and the face fitting surface;

The second determining unit is configured to determine that the human-like face is a real person if the sum of the distances between the key feature points in the human-like face image and the face fitting surface is greater than the dynamic depth information threshold, and if the key feature in the human-like face image is a real person If the sum of the distances between the feature points and the face fitting surface is not greater than the dynamic depth information threshold, it is determined that this type of human body is not a real person.

Optionally, the first prompting module 403 in the example of FIG. 4 may include a third determining unit, an animation prompting unit and an audio and video prompting unit, wherein:

The third determination unit is used to determine whether the real person on the current walking route of the robot is a young child;

The animation prompting unit is used to form a terrain feature animation of the surrounding environment of the child if the real person on the robot's current walking route is a child, and remind the child to avoid the robot through the terrain feature animation combined with voice;

The audio and video prompting unit is used to prompt the real person who is not a young child to avoid the robot through voice and generated facial expressions if the real person on the current walking route of the robot is not a child.

Optionally, the above animation prompting unit includes a fourth determining unit, a feature extraction unit, a projection unit and a guiding unit, wherein:

The fourth determining unit is used to determine the prompt information of the current walking route of the robot by sensing the surrounding environment data of the child;

The feature extraction unit is used to extract features from the prompt information of the current walking route of the robot by the preset animation processing model, so as to form a terrain feature animation;

The projection unit is used to project the terrain feature animation onto the target projection surface;

The guidance unit uses a child's voice to explain the terrain feature animation projected onto the target projection surface, so as to guide the young child to avoid the robot.

Optionally, the above-mentioned audio and video prompting unit may include a display unit and a playback unit, wherein:

A display unit, used to display the generated embarrassed expression image in an exaggerated manner on the display device;

The playback unit is used to play the embarrassed expression images and the prompt voice with actual semantics to the real person who is not a child in a cycle at a fixed interval, so as to request the real person who is not a child to avoid the robot.

Optionally, the apparatus shown in FIG. 4 may also include a control module or a shutdown module, wherein:

The control module is used to control the robot to avoid obstacles by walking in the "<" or ">" shape if the real person still does not avoid the robot or the non-human obstacle after upgrading the prompt mode;

The closing module is used to turn off the voice prompt mode and/or image prompt mode after the real person avoids the robot, or close the warning prompt mode after the non-human obstacle is removed.

It can be seen from the description of the above technical solutions that when it is confirmed that the obstacle is a real person, the voice prompt mode and/or image prompt mode are enabled to prompt the real person to avoid the robot. Since it is not necessary to train the robot whether it is a voice prompt, an image prompt, a warning prompt or an upgrade of the three, therefore, compared with the solution of performing a large amount of training on the robot to remove obstacles, on the one hand, the technical solution of the present application is low in cost; On the other hand, as long as the voice prompt mode, the image prompt mode, the warning prompt mode or the upgrade mode of the two are enabled, the purpose of removing obstacles can be achieved, and there is no need for the robot to be poorly trained and unable to achieve expectations as in the prior art. Worry about the effect.

FIG. 5 is a schematic structural diagram of a device provided by an embodiment of the present application. As shown in FIG. 5 , the device 5 of this embodiment mainly includes: a processor 50 , a memory 51 , and a computer program 52 stored in the memory 51 and executable on the processor 50 , such as a program of a robot failure handling method. When the processor 50 executes the computer program 52, it implements the steps in the above embodiment of the method for handling an obstacle in a robot, for example, steps S101 to S105 shown in FIG. 1 . Alternatively, when the processor 50 executes the computer program 52, the functions of the modules/units in the above-mentioned device embodiments are implemented, for example, the stop operation module 401, the detection module 402, the first prompt module 403, the second prompt module 404 and The function of the third prompt module 405 .

Exemplarily, the computer program 52 of the method for dealing with a robot encountering an obstacle mainly includes: when it is detected that there is an obstacle on the current walking route of the robot, stop the robot from moving forward on the current walking route; and detect whether the obstacle on the current walking route of the robot is a real person. ;If the obstacle on the robot's current walking route is a real person, enable the voice prompt mode and/or image prompt mode to prompt the real person to avoid the robot; if the obstacle on the robot's current walking route is not a real person, enable the warning prompt mode to prompt The staff in the scene assists in removing non-human obstacles; after enabling the voice prompt mode and/or image prompt mode, or after enabling the warning prompt mode, if the real person still does not avoid the robot or the non-human obstacle is still not removed, upgrade prompt mode. The computer program 52 may be divided into one or more modules/units, which are stored in the memory 51 and executed by the processor 50 to complete the present application. One or more modules/units may be a series of computer program instruction segments capable of performing specific functions, the instruction segments being used to describe the execution of the computer program 52 in the device 5 . For example, the computer program 52 can be divided into the functions of the stop operation module 401, the detection module 402, the first prompt module 403, the second prompt module 404 and the third prompt module 405 (modules in the virtual device), and the specific functions of each module are as follows : stop operation module 401, used to stop the robot from moving forward on the current walking route when it is detected that there is an obstacle on the current walking route of the robot; detection module 402, used to detect whether the obstacle on the current walking route of the robot is a real person; A prompting module 403, for enabling a first prompting mode, such as a voice prompting mode and/or an image prompting mode, if the obstacle on the current walking route of the robot is a real person, to prompt the real person to avoid the robot; the second prompting module 404, for If the obstacle on the current walking route of the robot is not a real person, a second prompting mode, such as a warning prompting mode, is enabled to prompt the staff in the scene to assist in removing obstacles that are not real people; the third prompting module 405 is used to prompt the first prompting After the module 403 enables the voice prompt mode and/or the image prompt mode, or after the second prompt module 404 enables the warning prompt mode, if the real person still does not avoid the obstacle of the robot or the non-human person, the prompt mode is upgraded.

Device 5 may include, but is not limited to, processor 50 , memory 51 . Those skilled in the art can understand that FIG. 5 is only an example of the device 5, and does not constitute a limitation to the device 5. It may include more or less components than the one shown, or combine some components, or different components, such as Computing devices may also include input and output devices, network access devices, buses, and the like.

The so-called processor 50 may be a central processing unit (Central Processing Unit, CPU), other general-purpose processors, digital signal processors (Digital Signal Processors) Processor, DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 51 may be an internal storage unit of the device 5 , such as a hard disk or a memory of the device 5 . The memory 51 can also be an external storage device of the device 5, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a Secure Digital (SD) card, a flash memory card (Flash Card) equipped on the device 5 Wait. Further, the memory 51 may also include both an internal storage unit of the device 5 and an external storage device. The memory 51 is used to store computer programs and other programs and data required by the device. The memory 51 can also be used to temporarily store data that has been output or is to be output.

The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity, all possible combinations of the technical features in the above-described embodiments are not described. However, as long as there is no contradiction between the combinations of these technical features, All should be regarded as the scope described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but should not be construed as a limitation on the scope of the patent application. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims

A method for handling an obstacle in a robot, the method comprising:

When detecting that there is an obstacle on the current walking route of the robot, controlling the robot to stop moving forward on the current walking route;

Detecting whether the obstacles on the current walking route of the robot are real people;

If the obstacle on the current walking route is a real person, enable a preset first prompt mode to prompt the real person to avoid the robot;

If the obstacle on the current walking route is not a real person, a preset second prompt mode is enabled to prompt the staff in the scene to assist in removing the obstacle that is not a real person.
The method for handling a robot encountering an obstacle according to claim 1, wherein the method further comprises:

After the first prompt mode or the second prompt mode is enabled, if the real person still does not avoid the robot or the obstacle of the non-human person is still not eliminated, then upgrade the first prompt mode or the second prompt mode to continue prompt.
The method for handling an obstacle of a robot according to claim 1, wherein the detecting whether the obstacle on the current walking route of the robot is a real person comprises:

According to the image of the obstacle acquired by the image acquisition device, determine whether the obstacle is a humanoid;

If the obstacle is a human-like face, collect key feature point information in the human-like face image for the human-like face, where the key feature point information in the human-like face image includes the plane of the key feature points location information;

According to the plane position information of the key feature point, obtain the three-dimensional position information of the key feature point;

According to the three-dimensional position information of the key feature points, the face fitting surface is obtained;

According to the distance between the key feature point and the face fitting surface, it is determined whether the obstacle is a real person.
The method for dealing with obstacles of a robot according to claim 3, wherein the determining whether the obstacle is a real person according to the distance between the key feature point and the face fitting surface comprises:

determining the sum of the distances between the key feature points and the face fitting surface;

If the sum of the distances is greater than the dynamic depth information threshold, it is determined that the human-like body is a real person; otherwise, it is determined that the human-like human body is not a human being.
The method for handling a robot encountering an obstacle according to claim 1, wherein the enabling the first prompt mode to prompt the real person to avoid the robot comprises:

determine whether the real person is a young child;

If the real person is a young child, forming a terrain feature animation of the surrounding environment of the young child, and prompting the young child to avoid the robot through the terrain feature animation combined with voice;

If the real person is not a child, the real person who is not a child is prompted to avoid the robot through voice and the generated facial expression image.
The method for handling a robot encountering an obstacle according to claim 5, wherein forming a terrain feature animation of the surrounding environment of the young child, and prompting the young child to avoid the robot through the terrain feature animation combined with voice, comprising: :

Determine the prompt information of the current walking route of the robot by sensing the surrounding environment data of the young child;

Feature extraction is performed on the prompt information of the current walking route of the robot by a preset animation processing model to form the terrain feature animation;

Projecting the terrain feature animation onto the target projection surface;

The terrain feature animation projected onto the target projection surface is explained by using a preset child's voice audio, so as to guide the young child to avoid the robot.
The method for handling a robot encountering an obstacle according to any one of claims 1-6, wherein the method further comprises:

If after upgrading the first prompt mode or the second prompt mode, the real person still does not avoid the robot or the obstacle that is not a real person, adjust the current walking route according to the size of the obstacle in the a detour around an obstacle, controlling the robot to travel on an adjusted walking path to avoid the obstacle; or

After the real person avoids the robot, the voice prompt mode and/or the image prompt mode are turned off, or the warning prompt mode is turned off after the non-human obstacle is removed.
An obstacle handling device for a robot, comprising:

A stop operation module, used to control the robot to stop moving forward on the current walking route when it is detected that there is an obstacle on the current walking route of the robot;

a detection module for detecting whether the obstacles on the current walking route of the robot are real people;

a first prompting module, configured to enable a first prompting mode if the obstacle on the current walking route is a real person, so as to prompt the real person to avoid the robot;

The second prompting module is configured to enable a second prompting mode if the obstacle on the current walking route is not a real person, so as to prompt the staff in the scene to assist in removing the obstacle that is not a real person.
The device for handling an obstacle of a robot according to claim 8, wherein the device further comprises:

The third prompting module is used for, after the first prompting module enables the first prompting mode, or after the second prompting module enables the second prompting mode, if the real person still does not avoid the obstacle of the robot or the non-human person, the obstacle is still not eliminated , the upgrade prompt mode will continue to prompt.
The device for handling obstacles of a robot according to claim 8, wherein the detection module comprises:

a first determining unit, configured to determine whether the obstacle on the current walking route of the robot is a humanoid according to the image of the obstacle acquired by the image acquisition device;

The acquisition unit is used to collect key feature point information in the human-like face image for the human-like face if the obstacle on the robot's current walking route is a human-like face, wherein the key feature point information in the human-like face image includes: The plane position information of the key feature point;

A stereoscopic position information acquisition unit, configured to acquire the stereoscopic position information of these key feature points according to the plane position information of the key feature points in the human-like face image;

The fitting unit is used to obtain the face fitting surface according to the three-dimensional position information of the key feature points;

The judgment unit is used for judging whether the obstacle on the current walking route of the robot is a real person according to the distance between the key feature points in the human-like face image and the face fitting surface.
The device for handling an obstacle of a robot according to claim 10, wherein the judging unit comprises:

a calculation unit, used to calculate the sum of the distances between the key feature points in the human-like face image and the face fitting surface;

The second determining unit is configured to determine that the human-like face is a real person if the sum of the distances between the key feature points in the human-like face image and the face fitting surface is greater than the dynamic depth information threshold, and if the key feature in the human-like face image is a real person If the sum of the distances between the feature points and the face fitting surface is not greater than the dynamic depth information threshold, it is determined that this type of human body is not a real person.
The device for handling obstacles of a robot according to claim 8, wherein the first prompting module comprises:

The third determination unit is used to determine whether the real person on the current walking route of the robot is a young child;

The animation prompting unit is used to form a terrain feature animation of the surrounding environment of the child if the real person on the robot's current walking route is a child, and remind the child to avoid the robot through the terrain feature animation combined with voice;

The audio and video prompting unit is used to prompt the real person who is not a young child to avoid the robot through voice and generated facial expressions if the real person on the current walking route of the robot is not a child.
A device, the device comprises a memory and a processor, the memory stores a computer program, the computer program can run on the processor, and the processor implements the following steps when executing the computer program:

When detecting that there is an obstacle on the current walking route of the robot, controlling the robot to stop moving forward on the current walking route;

Detecting whether the obstacles on the current walking route of the robot are real people;

If the obstacle on the current walking route is a real person, enable a preset first prompt mode to prompt the real person to avoid the robot;

If the obstacle on the current walking route is not a real person, a preset second prompt mode is enabled to prompt the staff in the scene to assist in removing the obstacle that is not a real person.
The device according to claim 13, wherein the processor further implements the following steps when executing the computer program:

After the first prompt mode or the second prompt mode is enabled, if the real person still does not avoid the robot or the obstacle of the non-human person is still not eliminated, then upgrade the first prompt mode or the second prompt mode to continue prompt.
The device of claim 13, wherein the processor implements the following steps when executing the computer program:

The detecting whether the obstacle on the current walking route of the robot is a real person includes:

According to the image of the obstacle acquired by the image acquisition device, determine whether the obstacle is a humanoid;

If the obstacle is a human-like face, collect key feature point information in the human-like face image for the human-like face, where the key feature point information in the human-like face image includes the plane of the key feature points location information;

According to the plane position information of the key feature point, obtain the three-dimensional position information of the key feature point;

According to the three-dimensional position information of the key feature points, the face fitting surface is obtained;

According to the distance between the key feature point and the face fitting surface, it is determined whether the obstacle is a real person.
The device of claim 15, wherein the processor implements the following steps when executing the computer program:

According to the distance between the key feature point and the face fitting surface, judging whether the obstacle is a real person, including:

determining the sum of the distances between the key feature points and the face fitting surface;

If the sum of the distances is greater than the dynamic depth information threshold, it is determined that the human-like body is a real person; otherwise, it is determined that the human-like human body is not a human being.
The device of claim 15, wherein the processor implements the following steps when executing the computer program:

The enabling of the first prompt mode to prompt the real person to avoid the robot includes:

determine whether the real person is a young child;

If the real person is a young child, forming a terrain feature animation of the surrounding environment of the young child, and prompting the young child to avoid the robot through the terrain feature animation combined with voice;

If the real person is not a child, the real person who is not a child is prompted to avoid the robot through voice and the generated facial expression image.
The device of claim 17, wherein the processor implements the following steps when executing the computer program:

Forming a terrain feature animation of the surrounding environment of the young child, and prompting the young child to avoid the robot through the terrain feature animation combined with voice, including:

Determine the prompt information of the current walking route of the robot by sensing the surrounding environment data of the young child;

Feature extraction is performed on the prompt information of the current walking route of the robot by a preset animation processing model to form the terrain feature animation;

Projecting the terrain feature animation onto the target projection surface;

The terrain feature animation projected onto the target projection surface is explained by using a preset child's voice audio, so as to guide the young child to avoid the robot.
The device according to any one of claims 13-18, wherein the processor further implements the following steps when executing the computer program:

If after upgrading the first prompt mode or the second prompt mode, the real person still does not avoid the robot or the obstacle that is not a real person, adjust the current walking route according to the size of the obstacle in the a detour around an obstacle, controlling the robot to travel on an adjusted walking path to avoid the obstacle; or

After the real person avoids the robot, the voice prompt mode and/or the image prompt mode are turned off, or the warning prompt mode is turned off after the non-human obstacle is removed.
A computer-readable storage medium storing a computer program, the computer program implementing the steps of the method of claim 1 when executed by a processor.