WO2021248857A1

WO2021248857A1 - Obstacle attribute discrimination method and system, and intelligent robot

Info

Publication number: WO2021248857A1
Application number: PCT/CN2020/133969
Authority: WO
Inventors: 张雪元; 孙赟; 衡进; 秦文强
Original assignee: 特斯联科技集团有限公司
Priority date: 2020-06-08
Filing date: 2020-12-04
Publication date: 2021-12-16
Also published as: CN111872928B; CN111872928A

Abstract

An obstacle attribute discrimination method and system, and an intelligent robot. The obstacle attribute discrimination method comprises: obtaining an area video captured by a camera on an intelligent robot for a monitoring area; inputting the captured area video into a pre-trained obstacle attribute discrimination model; and when it is determined that an obstacle exists in the area video, generating obstacle attribute information. By applying the obstacle attribute discrimination method, the working efficiency of the intelligent robot can be improved.

Description

Method, system and intelligent robot for judging obstacle attributes

Technical field

The invention relates to the technical field of intelligent robots, in particular to a method and system for judging the attributes of obstacles and an intelligent robot.

Background technique

With the rapid development of science and technology, the development of intelligent robots has also shown a rapid development trend. It has appeared in public places such as restaurants, banks, and halls. In the research of intelligent robots, one aspect is to improve the autonomy of intelligent robots, which means that they hope to be intelligent. The robot is further independent of humans and has a more friendly human-machine interface, which can automatically form the steps of the task and complete it automatically. On the other hand, it is to improve the adaptability of intelligent robots, improve the ability of intelligent robots to adapt to environmental changes, so that they have higher security and better ability to complete tasks.

At present, there have been many cases where intelligent robots are used to guide services in the lobby and front hall. When the current intelligent robots provide services to users, when the intelligent robots move in the service trajectory, when they encounter users, they will be regarded as obstacles. Continue to move forward after opening. At this time, the user needs to take the items in the hands of the intelligent robot actively, thereby reducing the service efficiency of the intelligent robot.

Summary of the invention

The embodiments of the present application provide a method, system and intelligent robot for judging the attributes of obstacles. In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not a general comment, nor is it intended to identify key/important elements or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

In the first aspect, an embodiment of the present application provides a method for judging the attributes of an obstacle, which is used in an intelligent robot, and the method includes:

Obtain the area video collected by the camera on the intelligent robot for the surveillance area;

Input the collected regional video into a pre-trained obstacle attribute discrimination model;

When it is determined that there is an obstacle in the regional video, obstacle attribute information is generated.

Optionally, after the obstacle attribute information is generated, the method further includes:

The movement of the intelligent robot is controlled based on the obstacle attribute information and the preset scene service mode.

Optionally, the scene service mode includes a task mode and a service mode.

Optionally, the controlling the movement of the intelligent robot based on the obstacle attribute information and a preset scene service mode includes:

When the scene service mode is a task mode and the obstacle attribute information is a human body, the intelligent robot is controlled to continue to move after bypassing the obstacle.

When the scene service mode is a service mode and the obstacle attribute information is a human body, the intelligent robot is controlled to move to the obstacle.

When the scene service mode is a service mode/task mode and the obstacle attribute information is an inanimate obstacle, the intelligent robot is controlled to continue to move after bypassing the obstacle.

Optionally, before the acquiring the regional video collected by the camera on the intelligent robot for the monitored area, the method further includes:

Acquiring a collection of sample images, the collection of sample images including images of human bodies and images of inanimate obstacles;

An obstacle attribute discrimination model is created, and the sample image set including a human body image and an inanimate obstacle image is input into the obstacle attribute discrimination model for training, and a trained obstacle attribute discrimination model is generated.

Optionally, the algorithm of the obstacle attribute discrimination model includes at least a 3D lidar human body recognition algorithm or a multi-sensor fusion algorithm.

In the second aspect, an embodiment of the present application provides an obstacle attribute discrimination system, the system includes:

The video acquisition module is used to acquire the regional video collected by the camera on the intelligent robot for the surveillance area;

The video input module is used to input the collected regional video into a pre-trained obstacle attribute discrimination model;

The attribute information generating module is used to generate obstacle attribute information when it is determined that there is an obstacle in the regional video.

In a third aspect, an embodiment of the present application provides an intelligent robot, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is adapted to be loaded by the processor and execute the above method steps .

The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

In the embodiment of this application, the intelligent robot first obtains the area video collected by the camera on the intelligent robot for the monitoring area, and then inputs the collected area video into the pre-trained obstacle attribute discrimination model. When the area video is determined When there is an obstacle, the obstacle attribute information is generated, and finally the intelligent robot is controlled to move based on the obstacle attribute information and the preset scene service mode. Since the intelligent robot can distinguish whether the obstacle ahead is a human or an ordinary obstacle, the intelligent robot can automatically realize the two modes of stopping moving and avoiding people when encountering people according to the preset scene service mode and obstacle attributes during the movement. The obstacle avoids the obstacle mode, thereby improving the service efficiency of the robot.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present invention.

Description of the drawings

The drawings here are incorporated into the specification and constitute a part of the specification, show embodiments in accordance with the present invention, and together with the specification are used to explain the principle of the present invention.

FIG. 1 is a schematic flowchart of an obstacle attribute discrimination method applied to an intelligent robot according to an embodiment of the present application;

2 is a schematic flowchart of another obstacle attribute identification method applied to an intelligent robot according to an embodiment of the present application;

Fig. 3 is a system schematic diagram of an obstacle attribute discrimination system provided by an embodiment of the present application;

Fig. 4 is a schematic structural diagram of an intelligent robot provided by an embodiment of the present application.

detailed description

The following description and drawings fully illustrate specific embodiments of the present invention to enable those skilled in the art to practice them.

It should be clear that the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present invention. Rather, they are merely examples of systems and methods consistent with some aspects of the present invention as detailed in the appended claims.

In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meaning of the above-mentioned terms in the present invention can be understood in specific situations. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more. "And/or" describes the association relationship of the associated objects, indicating that there can be three types of relationships, for example, A and/or B, which can mean: A alone exists, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects before and after are in an "or" relationship.

So far, there have been many cases where intelligent robots are used to guide services in the lobby and front hall. When the current intelligent robots provide services to users, when the intelligent robots move in the service trajectory, when they encounter users, they will When the obstacle continues to move forward after bypassing, the user is required to actively take the items in the hands of the intelligent robot, thereby reducing the service efficiency of the intelligent robot. To this end, the present application provides a method, system and intelligent robot for determining the attributes of obstacles to solve the above-mentioned related technical problems. In the technical solution provided by this application, since the intelligent robot can distinguish whether the front obstacle is a human or an ordinary obstacle, the intelligent robot can automatically stop moving and circumvent when encountering people according to the preset scene service mode and obstacle attributes during the movement. There are two modes of opening people and the mode of bypassing ordinary obstacles, thereby improving the service efficiency of the robot, which will be described in detail by using exemplary embodiments below.

The following describes in detail the obstacle attribute identification method provided by the embodiment of the present application with reference to accompanying drawings 1 to 2. This method can be implemented by relying on computer programs, and can be run on the obstacle attribute discrimination system based on the Von Neumann system. The computer program can be integrated in the application or run as an independent tool application. Among them, the obstacle attribute discrimination system in the embodiment of the present application is an intelligent robot.

Please refer to FIG. 1, which provides a schematic flowchart of an obstacle attribute discrimination method according to an embodiment of this application. As shown in Figure 1, the method of the embodiment of the present application may include the following steps:

S101: Obtain an area video collected by a camera on an intelligent robot for a monitored area;

Among them, intelligent robots are machines with various internal information sensors and external information sensors (such as vision, hearing, touch, smell), and they have thinking and service capabilities. The camera is an image acquisition device installed on the intelligent robot. In this application, the RGB camera and the depth camera are preferentially selected as the image acquisition cameras. The area is the area occupied by a place, which can be understood as a certain place, and the monitoring area is the area space that the camera on the intelligent robot can capture. Video usually refers to the storage format of various moving images.

Generally, regional video is a dynamic image composed of thousands of regional spatial image frames. The dynamic image is collected by a camera on an intelligent robot. The collected dynamic image may contain people or inanimate features. Other obstacles.

In a possible implementation manner, when the intelligent robot is in a working state, the RGB camera and the depth camera installed on the intelligent robot are used to collect images of the space area that can be photographed. When the intelligent robot collects images, it saves the collected image frames. As time increases, when the intelligent robot saves thousands of image frames in a period of time, the image frames of this period of time constitute a regional video.

S102: Input the collected regional video into a pre-trained obstacle attribute discrimination model;

Among them, the relevant explanation of the regional video can be found in step S101, which will not be repeated here. The obstacle attribute discrimination model is a mathematical model used to discriminate obstacle attributes according to the input regional video. This mathematical model uses an algorithm At least include 3D lidar human body recognition algorithm or multi-sensor fusion algorithm. The specific algorithm use can be set according to the actual scene, so I won't repeat it here. After the creation of the algorithm model is completed, the image containing the human body and inanimate obstacles can be used for training. After the completion, the model has the ability to detect obstacle attributes.

In a possible implementation, based on step S101, an area video taken by the intelligent robot during a period of time can be obtained. After the area video is obtained, the intelligent robot obtains a pre-trained obstacle attribute discrimination model for processing and analysis. After the analysis is complete Obtain the attribute information of the obstacle according to the obstacle in the video image.

S103: When it is determined that there is an obstacle in the regional video, generate obstacle attribute information.

Among them, the obstacle attribute information may include human attributes or other inanimate obstacle attributes.

In the embodiment of the present application, after the intelligent robot is identified by the obstacle attribute discrimination model, it generates the attribute information of the obstacle according to the identified obstacle, and finally controls the movement of the robot based on the attribute information and the preset intelligent robot scene service mode. Among them, the intelligent robot scene service mode includes service mode and task mode. The service mode is the mode in which intelligent robots serve people. For example, when an intelligent robot distributes a cup in its hand to everyone, it will stop to serve people whenever it encounters a person. . The task mode is the mode in which the intelligent robot provides services according to the specific task instructions issued by the user. For example, when the intelligent robot receives an instruction to deliver a cup to a certain person, the intelligent robot will only move to the target person and encounter other users or obstacles in the middle. Things will go around.

Further, when the scene service mode preset by the intelligent robot is the service mode and the obstacle is a human body, the intelligent robot will control itself to move to the front of the obstacle (that is, in front of the human body).

Further, when the scene service mode preset by the intelligent robot is the task mode and the obstacle is a human body, the intelligent robot will control itself to continue moving after bypassing the human body.

Further, when the preset scene service mode of the intelligent robot is mission mode/service mode and the obstacle is an inanimate obstacle, the intelligent robot will control itself to bypass the inanimate obstacle and continue to move forward.

Please refer to FIG. 2, which is a schematic flowchart of an obstacle attribute discrimination method provided by an embodiment of this application. In this embodiment, the obstacle attribute identification method is applied to an intelligent robot as an example. The obstacle attribute discrimination method may include the following steps:

S201: Collect a sample image set, where the sample image set includes human body images and inanimate obstacle images;

Among them, the so-called sample refers to a collection of data information composed of characters, words, and sentences that express its product performance, function, structural principle, and size parameters. In the embodiments of the present application, it specifically refers to a collection of sample images. It is an electronic upgraded version of the traditional paper samples, which can be disseminated through the Internet and displayed in front of users in a more novel and intuitive form. It has a visual and friendly human-computer interaction interface, which is rich in expressiveness and diverse in expression techniques. , Which makes the user's query faster, and the efficiency of searching sample data is higher.

Generally, sample collection is also called sample acquisition. Today, with the rapid development of the Internet industry, sample collection has been widely used in the Internet field. The accurate selection of the sample to be collected has a profound impact on the product. If the collected sample is not accurate enough, it may lead to The test results showed a large deviation, causing immeasurable losses to the product. Therefore, it is necessary to accurately collect sample information.

In the embodiment of the present application, a large number of images containing obstacles need to be collected first. The obstacles in the image may include human obstacles or other obstacles with inanimate features. After the collection is completed, a sample image set is generated. Obstacle images can be collected through the Internet, can also be obtained based on a gallery, or can be an image in a cloud server. The image acquisition method includes multiple forms, which are not limited here.

S202: Create an obstacle attribute discrimination model, and input the sample image set including human body images and inanimate obstacle images into the obstacle attribute discrimination model for training, and generate a trained obstacle attribute discrimination model;

In a possible implementation, in the training phase of the obstacle model, it is first necessary to use the 3D lidar human body recognition algorithm or multi-sensor fusion algorithm to create the obstacle attribute discrimination model, and after the creation is completed, the obstacle image collected in S101 Input into the obstacle attribute discriminant model for training. When the loss value of the obstacle attribute discriminant model reaches the minimum value, the trained obstacle attribute discriminant model is generated.

S203: Obtain an area video collected by a camera on the intelligent robot for the monitored area;

For details, please refer to step S101, which will not be repeated here.

S204: Input the collected regional video into a pre-trained obstacle attribute discrimination model;

S205: When it is determined that there is an obstacle in the regional video, generate obstacle attribute information;

For details, please refer to step S102, which will not be repeated here.

S206: Control the smart robot to move based on the obstacle attribute information and a preset scene service mode.

In a possible implementation, when the scene service mode preset by the intelligent robot is the service mode, the intelligent robot moves in the service mode, and collects real-time regional images through the depth camera installed on it during the movement. The video composed of image frames is continuously input into the obstacle discrimination model for recognition. When a human body is detected, it stops moving before moving to the human body.

In another possible implementation, when the scene service mode preset by the intelligent robot is the service mode, the intelligent robot moves in the service mode, and collects real-time regional images through the depth camera installed on it during the movement. When it detects that the human eye and the camera in the front area are in the same straight line, the intelligent robot moves forward to the front of the human body following the straight line that the human body looks at, and finally stops moving.

Specifically, when the intelligent robot judges whether the human eye and the camera are on the same straight line, it first obtains the human face through the camera on it, then obtains the eye area of the human face, and then collects the coordinates of the pupil at the current position. When the X-axis coordinate of the pupil and the X-axis coordinate point of the intelligent robot's depth camera are the same, it is determined that the human eye and the intelligent robot's depth camera are on the same straight line.

The following are system embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details that are not disclosed in the system embodiment of the present invention, please refer to the method embodiment of the present invention.

Please refer to FIG. 3, which shows a schematic structural diagram of an obstacle attribute discrimination system provided by an exemplary embodiment of the present invention. The obstacle attribute discrimination system can be implemented as all or part of an intelligent robot through software, hardware or a combination of the two. The system 1 includes a video acquisition module 10, a video input module 20, and an attribute information generation module 30.

The video acquisition module 10 is used to acquire the area video collected by the camera on the intelligent robot for the surveillance area;

The video input module 20 is configured to input the collected regional video into a pre-trained obstacle attribute discrimination model;

The attribute information generating module 30 is configured to generate obstacle attribute information when it is determined that there is an obstacle in the regional video.

It should be noted that, when the obstacle attribute discrimination system provided by the foregoing embodiment executes the obstacle attribute discrimination method, only the division of the aforementioned functional modules is used as an example for illustration. In actual applications, the aforementioned functional assignments can be divided according to needs. The function module is completed, that is, the internal structure of the device is divided into different function modules to complete all or part of the functions described above. In addition, the obstacle attribute discriminating system provided by the above-mentioned embodiment and the obstacle attribute discriminating method embodiment belong to the same concept. For the implementation process of the obstacle attribute discrimination method, please refer to the method embodiment, which will not be repeated here.

The serial numbers of the foregoing embodiments of the present application are for description only, and do not represent the superiority or inferiority of the embodiments.

The present invention also provides a computer-readable medium on which program instructions are stored, and when the program instructions are executed by a processor, the obstacle attribute discrimination method provided by the foregoing method embodiments is implemented.

The present invention also provides a computer program product containing instructions, which when running on a computer, causes the computer to execute the obstacle attribute identification method described in the foregoing method embodiments.

Please refer to FIG. 4, which provides a schematic structural diagram of an intelligent robot according to an embodiment of the present application. As shown in FIG. 4, the smart robot 1000 may include: at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

Among them, the communication bus 1002 is used to implement connection and communication between these components.

The user interface 1003 may include a display screen (Display) and a camera (Camera), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.

Among them, the network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

The processor 1001 may include one or more processing cores. The processor 1001 uses various excuses and lines to connect various parts of the entire electronic device 1000, and executes by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Various functions and processing data of the electronic device 1000. Optionally, the processor 1001 may use at least one of digital signal processing (Digital Signal Processing, DSP), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), and Programmable Logic Array (Programmable Logic Array, PLA). A kind of hardware form to realize. The processor 1001 may be integrated with one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a modem, and the like. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is used to render and draw the content that needs to be displayed on the display; the modem is used to process wireless communication. It is understandable that the above-mentioned modem may not be integrated into the processor 1001, but may be implemented by a chip alone.

The memory 1005 may include random access memory (Random Access Memory, RAM), and may also include read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a storage program area and a storage data area, where the storage program area may store instructions for implementing the operating system and instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), Instructions used to implement the foregoing method embodiments, etc.; the storage data area can store the data involved in the foregoing method embodiments, etc. Optionally, the memory 1005 may also be at least one storage system located far away from the foregoing processor 1001. As shown in FIG. 4, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an obstacle attribute discrimination application program.

In the smart robot 1000 shown in FIG. 4, the user interface 1003 is mainly used to provide an input interface for the user to obtain data input by the user; and the processor 1001 can be used to call the obstacle attribute discrimination application program stored in the memory 1005, And specifically perform the following operations:

In an embodiment, after the processor 1001 executes the generation of obstacle attribute information, it further executes the following operations:

In one embodiment, when the processor 1001 executes the movement of the intelligent robot based on the obstacle attribute information and the preset scene service mode, it specifically executes the following operations:

In an embodiment, the processor 1001 further executes the following operations before executing the acquisition of the area video collected by the camera on the smart robot for the surveillance area:

Acquiring a set of sample images, the set of sample images including images of human bodies and images of inanimate obstacles;

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The program can be stored in a computer readable storage medium. When executed, it may include the procedures of the above-mentioned method embodiments. Wherein, the storage medium can be a magnetic disk, an optical disc, a read-only storage memory or a random storage memory, etc.

The above-disclosed are only the preferred embodiments of the application, and of course the scope of rights of the application cannot be limited by this. Therefore, equivalent changes made in accordance with the claims of the application still fall within the scope of the application.

Claims

An obstacle attribute discrimination method applied to an intelligent robot, characterized in that the method includes:

Obtain the area video collected by the camera on the intelligent robot for the surveillance area;

Input the collected regional video into a pre-trained obstacle attribute discrimination model;

When it is determined that there is an obstacle in the regional video, obstacle attribute information is generated.
The method according to claim 1, wherein after said generating the obstacle attribute information, it further comprises:

The movement of the intelligent robot is controlled based on the obstacle attribute information and the preset scene service mode.
The method according to claim 2, wherein the scene service mode includes a task mode and a service mode.
The method according to claims 1-3, wherein the controlling the movement of the intelligent robot based on the obstacle attribute information and a preset scene service mode comprises:

When the scene service mode is a task mode and the obstacle attribute information is a human body, the intelligent robot is controlled to continue to move after bypassing the obstacle.
The method according to claims 1-3, wherein the controlling the movement of the intelligent robot based on the obstacle attribute information and a preset scene service mode comprises:

When the scene service mode is a service mode and the obstacle attribute information is a human body, the intelligent robot is controlled to move to the obstacle.
The method according to claims 1-3, wherein the controlling the movement of the intelligent robot based on the obstacle attribute information and a preset scene service mode comprises:

When the scene service mode is a service mode/task mode and the obstacle attribute information is an inanimate obstacle, the intelligent robot is controlled to continue to move after bypassing the obstacle.
The method according to claim 1, characterized in that, before acquiring the area video collected by the camera on the intelligent robot for the surveillance area, the method further comprises:

Acquiring a collection of sample images, the collection of sample images including images of human bodies and images of inanimate obstacles;

An obstacle attribute discrimination model is created, and the sample image set including a human body image and an inanimate obstacle image is input into the obstacle attribute discrimination model for training, and a trained obstacle attribute discrimination model is generated.
The method according to claim 7, wherein the algorithm of the obstacle attribute discrimination model at least comprises a 3D lidar human body recognition algorithm or a multi-sensor fusion algorithm.
An obstacle attribute discrimination system applied to an intelligent robot, characterized in that the system includes:

The video acquisition module is used to acquire the regional video collected by the camera on the intelligent robot for the surveillance area;

A video input module for inputting the collected regional video into a pre-trained obstacle attribute discrimination model;

The attribute information generating module is used to generate obstacle attribute information when it is determined that there is an obstacle in the regional video.
An intelligent robot, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is adapted to be loaded by the processor and executed as claimed in any one of claims 1 to 8. Method steps.