CN112659127A

CN112659127A - Multi-robot control method, device, system, storage medium and electronic equipment

Info

Publication number: CN112659127A
Application number: CN202011562250.6A
Authority: CN
Inventors: 黄晓庆; 张站朝; 马世奎
Original assignee: Cloudminds Robotics Co Ltd
Current assignee: Cloudminds Robotics Co Ltd
Priority date: 2020-12-25
Filing date: 2020-12-25
Publication date: 2021-04-16
Also published as: CN114193447A; WO2022134732A1; CN114193447B

Abstract

The present disclosure relates to the field of robotics, and in particular, to a method, an apparatus, a system, a storage medium, and an electronic device for controlling multiple robots. The method comprises the following steps: according to target tasks to be executed by the multiple robots, roles of the robots and a three-dimensional semantic map of environments where the robots are located, target subtasks are planned and distributed for the digital twin corresponding to the robots; planning a space path according to the target subtask to obtain a target motion path; controlling a digital twin to execute simulation operation in a three-dimensional semantic map through a behavior blueprint and evaluate according to a target subtask and a target motion path to obtain a simulation evaluation result; and if the target task is determined to be completed according to the simulation evaluation result, the behavior blueprint, the target motion path and the action data and/or the text data required for executing the behavior blueprint of the target subtask are sent to the robot through the plurality of digital twin bodies so as to control the robot to execute the behavior blueprint and complete the target task.

Description

Multi-robot control method, device, system, storage medium and electronic equipment

Technical Field

The present disclosure relates to the field of robotics, and in particular, to a multi-robot control method, apparatus, system, storage medium, and electronic device.

Background

With the continuous development and progress of the robot technology, people have more and more demands on the robot, and a single robot is difficult to complete complex and tedious work tasks, so that a plurality of robots are required to cooperate together to complete a project target task. Compared with a single robot, a robot system consisting of a plurality of robots can complete relatively complex target tasks through mutual cooperation among the plurality of robots. The robot system can adopt centralized control networking, namely comprises a server and one or more robots, and the server centrally controls all the robots so as to complete target tasks. In practice it has been found that: in such networking, after the server directly controls the robot to execute the target task, the target task may fail to be executed.

Disclosure of Invention

In order to solve the above problems, the present disclosure provides a multi-robot control method, apparatus, system, storage medium, and electronic device.

In a first aspect, the present disclosure provides a multi-robot control method, the method comprising:

according to target tasks to be executed by a plurality of robots, roles of the plurality of robots and a three-dimensional semantic map of environments where the plurality of robots are located, planning and distributing one or more target subtasks for a plurality of digital twin corresponding to the plurality of robots, planning and distributing one or more behavior blueprints corresponding to each target subtask, and executing action data and/or text data required by the behavior blueprints, wherein the roles represent a preset function set allowing the robots to execute, and the behavior blueprints comprise action sequences required by the robots to complete the target subtasks and logic judgment required by the execution of the action sequences;

performing space path planning according to the target subtask to obtain a target motion path of the digital twin body;

controlling a plurality of digital twin bodies to execute simulation operation in the three-dimensional semantic map through a behavior blueprint and evaluate according to the target subtask and the target motion path, and obtaining a simulation evaluation result, wherein the simulation evaluation result is used for representing whether the target task is completed;

under the condition that the target task is determined to be completed according to the simulation evaluation result, the behavior blueprint, the action data and/or the text data and the target motion path are synchronized to the robot through the digital twin body so as to control the robot to execute the behavior blueprint and complete the target task.

Optionally, the allocating one or more target subtasks for the digital twin planning corresponding to the robot according to target tasks to be performed by the plurality of robots, roles of the plurality of robots, and a three-dimensional semantic map of environments in which the plurality of robots are located includes:

decomposing the target task plan into one or more candidate subtasks according to the three-dimensional semantic map and the roles of the plurality of robots;

acquiring twin parameters of the digital twin, wherein the twin parameters comprise one or more of position information of the digital twin, residual energy of the digital twin, continuous operation time of the digital twin, operation state of the digital twin and tasks to be performed by the digital twin;

and acquiring a target subtask from the one or more candidate subtasks according to the twin parameters, and distributing the target subtask to the digital twin.

Optionally, the target task is divided into a plurality of candidate subtask sets according to different planning manners during task planning, each candidate subtask set includes one or more target subtasks, the digital twin is controlled to execute simulation operation in the three-dimensional semantic map through a behavior blueprint and to perform evaluation according to the target subtask and the target motion path, and obtaining a simulation evaluation result includes:

for each candidate subtask set of the plurality of candidate subtask sets, performing a simulation run and evaluation of each candidate subtask set, wherein performing the simulation run and evaluation of the candidate subtask set comprises: controlling the digital twin to execute simulation operation in the three-dimensional semantic map through a behavior blueprint and evaluate according to the target subtasks in the candidate subtask set and the target motion path corresponding to each target subtask to obtain a target task execution evaluation result;

taking a candidate subtask set with a globally optimal target task execution evaluation result as a target subtask set, and taking a candidate evaluation result corresponding to the target subtask set as the simulation evaluation result, wherein the globally optimal target task execution evaluation result represents that the robot energy consumed for completing the target task is the least, or the time consumed for completing the target task is the shortest, or the moving path of the robot for completing the target task is the shortest;

the synchronizing the behavior blueprint, the target motion path, and the action data and/or text data required for executing the behavior blueprint to the robot includes:

and synchronizing the behavior blueprint, the target motion path and action data and/or text data required for executing the behavior blueprint corresponding to the target subtask in the target subtask set to the robot.

Optionally, the method further comprises:

under the condition that the target task is determined to be not completed according to the simulation evaluation result, circularly executing a simulation adjustment step until the target task is determined to be completed according to the simulation evaluation result, and synchronizing a behavior blueprint, a target motion path and action data and/or text data required for executing the behavior blueprint corresponding to the target subtask completing the task target to the robot through the digital twin; wherein the simulation adjusting step comprises:

displaying the simulation evaluation result, the target subtask and the digital twin to a user;

receiving a task adjusting instruction input by a user aiming at one or more digital twins, and adjusting a target subtask of the digital twins according to the task adjusting instruction;

performing path planning according to the adjusted target subtask to obtain an adjusted target motion path of the digital twin;

and controlling the plurality of digital twin bodies to execute simulation operation in the three-dimensional semantic map through a behavior blueprint and evaluate according to the adjusted target subtask and the adjusted target motion path, and taking the result obtained by evaluation as a new simulation evaluation result.

Optionally, after obtaining a target subtask from the one or more candidate subtasks according to the twin parameter, the method further includes:

reacquiring new twin parameters of the digital twin;

determining whether the target subtask needs to be updated according to the new twin parameters;

and under the condition that the target subtasks need to be updated, re-executing the three-dimensional semantic map according to the target tasks to be executed by the multiple robots, the roles of the multiple robots and the environments where the multiple robots are located, distributing one or more target subtasks for the multiple digital twin plans corresponding to the multiple robots, and synchronizing the behavior blueprints, the target paths and the action data and/or text data required for executing the behavior blueprints to the robots.

Optionally, the reacquiring new twin parameters of the digital twin includes:

receiving an operation instruction input by a user aiming at the digital twin;

updating the twin parameters of the digital twin according to the operation instruction;

and taking the updated twin parameters as new twin parameters.

Optionally, determining whether the target subtask needs to be updated according to the new twin parameter includes:

determining that the target subtask needs to be updated when the new twin parameter satisfies a preset condition, wherein the preset condition includes one or more of the following conditions:

the residual energy of the digital twin body is less than or equal to a preset energy threshold;

the running state of the digital twin body is a preset state;

and the tasks to be executed of the digital twin body have tasks to be executed with task priorities higher than the target subtasks.

Optionally, in a case that the simulation evaluation result is confirmed to achieve the task goal, the method further includes:

acquiring a local three-dimensional semantic map required by the robot according to the three-dimensional semantic map, the target subtask and the target motion path;

and sending the local three-dimensional semantic map to the robot.

Optionally, the three-dimensional semantic map is obtained by:

sending an environment scanning instruction to the robot, wherein the environment scanning instruction is used for instructing the robot to scan the environment of a target task and obtain environment information;

receiving the environment information sent by the robot according to the environment scanning instruction; and acquiring the three-dimensional semantic map according to the environment information.

Optionally, the target task is obtained by any one of the following methods:

receiving a task instruction input by a user, and acquiring the target task according to the task instruction;

and receiving the environmental information sent by the robot, and acquiring the target task according to the environmental information.

Optionally, the space path planning includes three-dimensional path planning and moving path planning, and the target moving path includes a three-dimensional moving path obtained through the three-dimensional path planning and a moving track obtained through the moving path planning in a two-dimensional coordinate.

In a second aspect, the present disclosure provides a multi-robot control apparatus, the apparatus comprising:

the task allocation module is used for planning and allocating one or more target subtasks for a plurality of digital twin bodies corresponding to a plurality of robots according to target tasks to be executed by the robots, roles of the robots and three-dimensional semantic maps of environments where the robots are located, planning and allocating one or more behavior blueprints corresponding to each target subtask, and executing action data and/or text data required by the behavior blueprints, wherein the roles represent a preset function set which allows the robots to execute, and the behavior blueprints comprise action sequences required by the robots to complete the target subtasks and logic judgment required by the execution of the action sequences;

the path planning module is used for planning a space path according to the target subtask to obtain a target motion path of the digital twin;

the simulation evaluation module is used for controlling the plurality of digital twin bodies to execute simulation operation in the three-dimensional semantic map through a behavior blueprint and evaluate according to the target subtasks and the target motion path to obtain a simulation evaluation result, and the simulation evaluation result is used for representing whether the target task is completed or not;

and the task synchronization module is used for synchronizing the behavior blueprint, the target motion path and action data and/or text data required for executing the behavior blueprint to the robot through the digital twin body under the condition that the target task is determined to be completed according to the simulation evaluation result so as to control the robot to execute the behavior blueprint and complete the target task.

In a third aspect, the present disclosure provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method of the first aspect of the present disclosure.

In a fourth aspect, the present disclosure provides an electronic device comprising: a memory having a computer program stored thereon; a processor for executing the computer program in the memory to implement the steps of the method of the first aspect of the disclosure.

In a fifth aspect, the present disclosure provides a multi-robot control system, the system comprising a server, and a plurality of robots connected to the server; the server comprises a plurality of digital twins, each digital twins being a virtual three-dimensional physical model corresponding to one robot; wherein:

the server is configured to perform the steps of the method according to the first aspect of the present disclosure.

The robot is used for receiving the behavior blueprint planned by the server, the target path, and action data and/or text data required by executing the behavior blueprint, and executing the behavior blueprint.

Optionally, the robot comprises: a robot body and a digital twin replica; wherein:

the digital twin copy is used for interactively synchronizing with a digital twin of a server, synchronizing a behavior blueprint planned by the server, a target path and action data and/or text data required by executing the behavior blueprint, and synchronously controlling the robot body to execute the behavior blueprint;

the robot body is used for executing the behavior blueprint according to the control of the digital twin copy.

Optionally, the robot further comprises a perception processing component; wherein:

the perception processing component is used for scanning the environment of the target task and obtaining environment information;

the digital twin copy is used for receiving an environment scanning instruction sent by the server and controlling the perception processing component to scan according to the environment scanning instruction; and acquiring the environmental information scanned by the perception processing component and sending the environmental information to the server.

Optionally, the digital twin copy is further configured to synchronize state information of the robot to the digital twin so that the digital twin synchronously updates twin parameters of the digital twin according to the state information.

By adopting the technical scheme, target subtasks are planned and distributed for the digital twin corresponding to the robots according to target tasks to be executed by the robots, roles of the robots and a three-dimensional semantic map of the environment where the robots are located; planning a space path according to the target subtask to obtain a target motion path; controlling a digital twin to execute simulation operation in a three-dimensional semantic map through a behavior blueprint and evaluate according to a target subtask and a target motion path to obtain a simulation evaluation result; and if the target task is determined to be completed according to the simulation evaluation result, the behavior blueprint, the target motion path and the action data and/or the text data required for executing the behavior blueprint of the target subtask are sent to the robot through the plurality of digital twin bodies so as to control the robot to execute the behavior blueprint and complete the target task. Therefore, the robot can be ensured to complete the target task, and the accuracy of multi-robot control is improved.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows.

Drawings

The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure without limiting the disclosure. In the drawings:

fig. 1 is a schematic structural diagram of a multi-robot control system provided in an embodiment of the present disclosure;

fig. 2 is a flowchart of a multi-robot control method provided by an embodiment of the present disclosure;

FIG. 3 is a schematic structural diagram of another multi-robot control system provided in the embodiments of the present disclosure;

FIG. 4 is a schematic structural diagram of another multi-robot control system provided in the embodiments of the present disclosure;

fig. 5 is a schematic structural diagram of a multi-robot control device according to an embodiment of the disclosure;

fig. 6 is a schematic structural diagram of a second multi-robot control device provided in the embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of a third multi-robot control device provided in the embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of a fourth multi-robot control device provided in the embodiment of the present disclosure;

fig. 9 is a schematic structural diagram of a fifth multi-robot control device provided in the embodiment of the disclosure;

fig. 10 is a block diagram of an electronic device provided by an embodiment of the disclosure;

fig. 11 is a block diagram of another electronic device provided by embodiments of the present disclosure.

Detailed Description

The following detailed description of specific embodiments of the present disclosure is provided in connection with the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present disclosure, are given by way of illustration and explanation only, not limitation.

In the description that follows, the terms "first," "second," and the like are used for descriptive purposes only and are not intended to indicate or imply relative importance nor order to be construed.

First, an application scenario of the present disclosure will be explained. The method and the device can be applied to the technical field of robots, in particular to the field of robot control of centralized control networking. The robot system composed of a plurality of robots can adopt centralized control networking, and exemplarily, the robot system can comprise a server and one or more robots, and mutual cooperation among the plurality of robots is realized through the coordination control of the server, so that relatively complex target tasks are completed. However, in the related art, the server divides the target tasks to obtain target subtasks of each robot, and directly issues the target subtasks to the robots so as to control the robots to execute the target tasks. Under the condition, the rationality of the divided target subtasks cannot be guaranteed, and if the divided target subtasks are unreasonable, the robot cannot complete the target tasks. For example, if the target task is too complex, for example, more robots need to participate, or the environment information for task execution is complex and variable, the target subtask divided by the server may be unreasonable probabilistically, and the target task may not be completed.

In order to solve the above problems, the present disclosure provides a multi-robot control method, apparatus, system, storage medium, and electronic device, the method including: according to target tasks to be executed by the multiple robots, roles of the robots and a three-dimensional semantic map of environments where the robots are located, target subtasks are planned and distributed for the digital twin corresponding to the robots; space path planning is carried out according to the target subtasks to obtain a target motion path of the digital twin body; controlling a digital twin to execute simulation operation in a three-dimensional semantic map through a behavior blueprint and evaluate according to a target subtask and a target motion path to obtain a simulation evaluation result; and if the target task is determined to be completed according to the simulation evaluation result, the behavior blueprint, the target motion path and the action data and/or the text data required for executing the behavior blueprint for completing the target subtask are sent to the robot through the plurality of digital twin bodies so as to control the robot to execute the behavior blueprint and complete the target task, so that the robot can be ensured to complete the target task, and the accuracy of controlling the plurality of robots is improved.

The following detailed description of specific embodiments of the present disclosure is provided in connection with the accompanying drawings.

Fig. 1 is a schematic structural diagram of a multi-robot control system according to an embodiment of the present disclosure, and as shown in fig. 1, the multi-robot control system includes a server 101, and a plurality of robots 102 (i.e., a robot 1021, a robot 1022, robots 1023 and …, and a robot 102n) connected to the server 101, where: the server 101 may include a plurality of digital twins 103 (i.e.,

digital twins

1031, 1032, 1033, …, 103 n). Each of the digital twins corresponds to a robot, illustratively digital twins 1031 to robot 1021, digital twins 1032 to robot 1022, and digital twins 1033 to robot 1023.

The server 101 may be a cloud server, a desktop computer, or other electronic devices with a memory and a processor. The server 101 and the robot 102 may be connected via a wired network or a wireless network.

The robot 102 is a physical robot, and may include a homogeneous multi-robot or a heterogeneous multi-robot. The isomorphic multi-robot represents a plurality of robots with the same hardware equipment or the same capability, and different robots can execute the same type of task; while heterogeneous multi-robots represent multiple robots with disparate hardware devices or different capabilities, different robots can perform different types of tasks.

The digital twin 103 may be a virtual three-dimensional physical model of the robot 102 created in a digital manner, and the digital twin 103 is connected to the corresponding robot 102 and may realize real-time interactive synchronization. On one hand, parameter instructions of the digital twin 103 can be transmitted to the robot 102, so as to adjust the behavior and state of the robot 102; on the other hand, the digital twin 103 has twin parameters, which may be the same as the state parameters of the robot 102 corresponding to the digital twin 103, and the twin parameters of the digital twin 103 and the state parameters of the corresponding robot 102 are kept consistent through real-time interactive synchronization of the digital twin 103 and the robot 102, so that the digital twin 103 realistically reflects the real state of the robot 102.

In the multi-robot control system, the server 101 may acquire the state parameters of the robot 102 through the digital twin 103, or may transmit related parameter commands to the robot 102 through the digital twin 103 to control the robot 102 to execute a target task.

Fig. 2 is a flowchart of a multi-robot control method provided in an embodiment of the present disclosure, and as shown in fig. 2, an execution subject of the method may be a server in the multi-robot control system, where the method includes:

s201, planning and distributing one or more target subtasks for a plurality of digital twin bodies corresponding to a plurality of robots according to target tasks to be executed by the plurality of robots, roles of the plurality of robots and a three-dimensional semantic map of environments where the plurality of robots are located, planning one or more behavior blueprints corresponding to each target subtask, and executing action data and/or text data required by the behavior blueprints.

The role of the robot represents a preset function set allowed to be executed by the robot, and the function set can be a complete set or a subset of functions that can be supported by hardware capabilities of the robot.

The behavior blueprint comprises an action sequence required to be executed by the robot to complete the target subtask and logic judgment required by executing the action sequence.

The three-dimensional semantic map is an environmental semantic scene described by natural language, and can describe a target object in a three-dimensional space, and a spatial position and semantic attributes of the target object, wherein the semantic attributes include, but are not limited to, a size, a shape, a weight, a category, a material, a color and the like of the target object. For example, a meeting room environment can be described by a three-dimensional semantic map, wherein the target objects can include floors, walls, doors, windows, tables and lamps, the semantic attributes of the target objects can include information of size, material, color and the like, and the three-dimensional semantic map also describes the spatial position information of each target object in the meeting room environment. In another example, an office environment may also be described by a three-dimensional semantic map, and the office environment may include target objects such as a first-floor office, a second-floor office, stairs from the first floor to the second floor, a refrigerator on the first floor, a water dispenser, a conference room, and a desk on the second floor.

The three-dimensional semantic map can be preset or constructed through robot scanning. For example, the task environment can be scanned by a tool and manually marked to form a three-dimensional semantic map, and the three-dimensional semantic map is preset. For another example, the robot can be controlled to scan the task environment by using the perception processing component of the robot, and the task environment is automatically marked to form a three-dimensional semantic map.

In this step, a preset function set executed by the robot may be determined according to the role of the robot, so as to assign a target subtask corresponding to the function set to the digital twin corresponding to the robot.

Illustratively, the three-dimensional semantic map is a home map, which comprises a living room, a dining room, a kitchen and a bedroom, and the robot with three different roles is in the multi-robot control system: the sweeping robot, the window cleaning robot and the garbage conveying robot are characterized in that under the condition that a target task is to sweep a large space in a room, the target subtask distributed to the sweeping robot can be a sweeping subtask, and the sweeping subtask comprises the steps of sweeping the floors of a living room, a dining room, a kitchen and a bedroom; the target subtask assigned to the window-cleaning robot may be a window-cleaning subtask including actuation of the window-cleaning robot to wipe the living room, the dining room, the kitchen, and the bedroom; the target subtask allocated to the garbage carrying robot may be a garbage cleaning subtask, which includes carrying the garbage generated by the sweeping robot to a district garbage centralized storage place and dumping the garbage into a garbage can.

And S202, planning a space path according to the target subtask to obtain a target motion path of the digital twin body.

The space path planning means that the movement planning of the robot is realized on the basis of the three-dimensional semantic map. Under the condition that the environmental information of the three-dimensional semantic map is in a known state and meets various evaluation standards such as certain time, distance or energy, an optimal or suboptimal collision-free route from a current point to a final point is found for each robot; on the basis of global space path planning, the types of possible conflicts among the robots are predicted in the optimal or suboptimal paths selected from different starting points to target points, and obstacle conflicts, deadlock elimination and the like are achieved.

In the same example, the target motion path planned for the sweeping robot may be, according to the above three target subtasks, a living room- > a bedroom- > a dining room- > a kitchen- > a doorway; in order to avoid the conflict of the robot paths, the target motion path planned for the window-cleaning robot can be a restaurant- > kitchen- > living room- > bedroom; the target movement path planned for the garbage cleaning robot can be stopped at the door for waiting, after the cleaning robot finishes a cleaning sub-task and transfers the garbage generated by cleaning into the body of the garbage cleaning robot, the garbage cleaning robot goes out of the door and moves to a district garbage centralized storage place, and the garbage is placed into a garbage bin and then returns to the door for stopping.

In another example, when there is a target object in the target subtask, the target object corresponding to the target subtask may be determined, and the target position of the target object may be determined in a three-dimensional semantic map, and based on the three-dimensional semantic map, a moving path of each digital twin from the current position to the target position may be planned, where the moving path may be a shortest path without obstacles, so that the moving path may be used as a target motion path of the digital twin.

Optionally, the space path planning may include a three-dimensional path planning and a moving path planning, and accordingly, the target motion path may include a three-dimensional motion path obtained through the three-dimensional path planning and a moving track obtained through the moving path planning in a two-dimensional coordinate.

For example, the movement track of the robot moving from the living room to the bedroom under the two-dimensional coordinates can be obtained through movement path planning, and the three-dimensional path planning can obtain a three-dimensional movement path of the robot for lifting an arm from a current position to a window in order to wipe the window.

Optionally, after a moving path of each digital twin body from the current position to the target position is planned, a collision which may occur during the moving process of the robot may be predicted, for example, if moving paths of two robots intersect, it may be predicted that there exists a certain collision probability between the two robots, at this time, the moving path of the robot having the collision probability may be adjusted to avoid the occurrence of the collision, and the adjusted moving path is used as the target moving path of the digital twin body.

And S203, controlling the plurality of digital twin bodies to execute simulation operation in the three-dimensional semantic map through the behavior blueprint and evaluate according to the target subtasks and the target motion path, and obtaining a simulation evaluation result.

In this step, the plurality of digital twin bodies may be controlled to perform one or more simulation operations on the entire target task in the virtual three-dimensional semantic map through the behavior blueprint and perform evaluation, so as to obtain a simulation evaluation result.

The simulation evaluation result can be used for representing whether the target task is completed. Illustratively, the target task is to move the target object from a position a to a position B, and if the target object is moved from the position a to the position B through the simulation evaluation, the simulation evaluation result is that the target task is completed; on the contrary, if the target object moves from the position a to the position C through the simulation evaluation, and the position C and the position B are two positions with a long distance, the simulation evaluation result is that the target task is not completed.

For example, in the above example where the target task is indoor large sweeping, the simulation evaluation result may be obtained by performing simulation one or more times on the digital twin bodies respectively corresponding to the sweeping robot, the window cleaning robot, and the garbage cleaning robot according to the allocated target subtasks and target movement paths. The obtained simulation evaluation result can represent that the large sweeping task is completed or not completed.

And S204, under the condition that the target task is determined to be completed according to the simulation evaluation result, synchronizing the behavior blueprint, the target motion path and the motion data and/or text data required for executing the behavior blueprint to the robot through the digital twin so as to control the robot to execute the behavior blueprint and complete the target task.

In this step, the behavior blueprint, the target motion path, and the motion data and/or text data required for executing the behavior blueprint may be first distributed to the digital twin, and the digital twin performs interactive synchronization with the corresponding robot, so as to synchronize the behavior blueprint, the target motion path, and the motion data and/or text data required for executing the behavior blueprint to the robot, so as to control the robot to execute the behavior blueprint and complete the target task.

By adopting the method, the target subtasks are planned and distributed for the digital twin corresponding to the robots according to the target tasks to be executed by the robots, the roles of the robots and the three-dimensional semantic map of the environment where the robots are located; space path planning is carried out according to the target subtasks to obtain a target motion path of the digital twin body; controlling a digital twin to execute simulation operation in a three-dimensional semantic map through a behavior blueprint and evaluate according to a target subtask and a target motion path to obtain a simulation evaluation result; and if the target task is determined to be completed according to the simulation evaluation result, the behavior blueprint, the target motion path and the action data and/or the text data required for executing the behavior blueprint for completing the target subtask are sent to the robot through the plurality of digital twin bodies so as to control the robot to execute the behavior blueprint and complete the target task, so that the robot can be ensured to complete the target task, and the accuracy of controlling the plurality of robots is improved.

Optionally, under the condition that the target task is determined to be completed according to the simulation evaluation result, a local three-dimensional semantic map required by the robot can be obtained according to the three-dimensional semantic map, the target subtask and the target motion path; and sending the local three-dimensional semantic map to the robot.

The local three-dimensional semantic map may be a partial map associated with the target subtask and the target motion path, while a map not associated with the target subtask and the target motion path need not be sent to the robot.

Therefore, the storage space and the computing capacity occupied by the robot for storing the complete three-dimensional semantic map can be reduced, and the execution efficiency of the robot can be improved by executing the corresponding target subtasks through the local three-dimensional semantic map.

In some other embodiments of the present disclosure, the step S201 may assign the target subtasks to the digital twin by:

firstly, the target task plan is decomposed into one or more candidate subtasks according to the three-dimensional semantic map and the roles of the multiple robots.

In this step, the target task may be decomposed into one or more candidate subtasks according to the current position and the target position of the target object in the target task in the three-dimensional semantic map. For example, there may be three different roles of robots in a robotic system: robot a 1: the wheeled robot with the dexterous arm can execute subtasks of grabbing objects, pushing, pulling, turning, placing, moving on the flat ground and the like; robot B1: the quadruped robot with the grabbing hands can execute the subtasks of going up and down stairs and also can execute the subtasks of grabbing articles; robot C1: the delivery robot may perform a subtask of item transportation on a flat ground, and may perform a subtask of transporting a specified item between different locations. The target task is to take a bottle of beverage from the refrigerator on the first floor to the office desk on the second floor, and the target task plan can be decomposed into three target subtasks: the objective subtask 1, taking the beverage from the refrigerator of the first floor and delivering the beverage to the stair opening of the first floor; target subtask 2: conveying the beverage from the first-floor stair opening to the second-floor stair opening; target subtask 3: the beverage is transported from the second floor landing onto the office table.

Secondly, twin parameters of the digital twin are acquired.

Wherein the twin parameters may include one or more of position information of the digital twin, remaining energy of the digital twin, continuous operation duration of the digital twin, operation status of the digital twin, and tasks to be performed by the digital twin;

the twin parameters of the digital twin may be the same as the corresponding state parameters of the robot, and when the state parameters of the robot change, the twin parameters may be synchronized with the corresponding twin parameters of the digital twin.

The position information of the digital twin body may be a current position of the robot corresponding to the digital twin body on the three-dimensional semantic map, and similarly, the position information of the robot a1 may be a stair entrance; the position information of the robot B1 may be a second floor landing; the position information of the robot C1 may be a doorway of a second floor office. If a plurality of robots of the same type are located at different positions, the target subtask may be assigned to the robot closest to the target subtask according to the distance between the positions of the plurality of robots and the position of the target object of the target subtask.

The residual energy of the digital twin may be used to represent the residual electric quantity of the robot corresponding to the digital twin, and it may be determined whether the robot corresponding to the digital twin can complete the target subtask by the residual energy. For example, 5% of the remaining energy, the robot may charge as quickly as possible to replenish the energy and will not perform new subtasks any more.

The operation states of the digital twin bodies can be used for representing the operation states of the robots corresponding to the digital twin bodies, and the operation states can include a normal state, a fault state, a charging state, a fatigue state and other states. The fault state can represent that the robot has faults so that subtasks cannot be executed, and at the moment, subtasks are not distributed to the digital twin; the charging state can represent that the robot is charging, and at this time, a high-priority subtask can be allocated to the digital twin instead of a low-priority subtask, and the high-priority subtask may be a subtask which can not be executed by other digital twin or a subtask priority set in advance; the fatigue state may indicate that the time for the robot to continuously execute the subtasks is greater than or equal to a first preset time threshold, at this time, a second preset time may be set for the digital twin, and a new subtask may not be allocated to the digital twin within the second preset time.

The tasks to be executed of the digital twin body can be used for representing whether the digital twin body has subtasks to be executed and subtask priorities of the subtasks to be executed.

And finally, acquiring a target subtask from the one or more candidate subtasks according to the twin parameters, and distributing the target subtask to the digital twin.

In this step, a suitable target subtask may be assigned to the digital twin according to one or more of the digital twin parameters. For example, a target subtask may be assigned to a digital twin according to the type and location information of the digital twin.

In the same example, in the case where the robots a1, B1, and C1 are all in a normal state, the remaining power is greater than 90%, and there is no task to be performed, the subtask 1 (taking a beverage from the refrigerator on the first floor and delivering the beverage to the entrance of the first floor) can be assigned to the robot a1 (a wheeled robot with dexterous hands) if the target task is to take a bottle of beverage from the refrigerator on the first floor onto the office table on the second floor; assign subtask 2 (transport beverage from first floor landing to second floor landing) to robot B1 (quadruped robot with grabbing hand); subtask 3 (transport of beverage from second floor landing onto office desk) is assigned to robot C1 (delivery robot).

In this way, each digital twin may be assigned an appropriate target subtask according to twin parameters of the digital twin, such that the target task is completed by the cooperation of one or more robots.

In some other embodiments of the present disclosure, the target task may be divided into a plurality of candidate subtask sets according to different planning manners when performing task planning, and each candidate subtask set may include one or more target subtasks. Illustratively, the target task may be divided into a first candidate subtask set, a second candidate subtask set, or a third candidate subtask set, wherein the first candidate subtask set may include a target subtask 11, a target subtask 12, and a target subtask 13; the second set of candidate subtasks may include a target subtask 21, a target subtask 22, and a target subtask 23; the third set of candidate subtasks may include a target subtask 31, a target subtask 32, and a target subtask 33; space path planning can be performed on the target subtasks in each candidate subtask combination, and a target motion path corresponding to each target subtask is obtained.

Thus, the step S203 can be implemented as follows:

first, for each candidate subtask set of the plurality of candidate subtask sets, a simulation run and evaluation of each candidate subtask set is performed.

Wherein performing the simulation run and evaluation of the set of candidate subtasks includes: and controlling the digital twin to execute simulation operation in a three-dimensional semantic map through a behavior blueprint and evaluate according to the target subtasks in the candidate subtask set and the target motion path corresponding to each target subtask to obtain a candidate evaluation result and a task execution evaluation result of the candidate subtask set.

Secondly, taking a candidate subtask set with a global optimal task execution evaluation result as a target subtask set, and taking a candidate evaluation result corresponding to the target subtask set as a simulation evaluation result.

The task execution evaluation result is globally and optimally characterized, robot energy consumed for completing the target task is the least, or time consumed for completing the target task is the shortest, or a moving path of the robot of the target task is the shortest.

Finally, under the condition that the target task is determined to be completed according to the simulation evaluation result, the behavior blueprint, the target motion path, and the action data and/or the text data required for executing the behavior blueprint corresponding to the target subtask in the target subtask set can be synchronized to the robot through the digital twin.

Thus, by the above method, an optimal target subtask allocation mode can be obtained through simulation evaluation, so that the power consumption of the robot is minimum in the execution of the target task, or the completion time of the target task is shortest.

In addition, the simulation evaluation of each candidate subtask set can be performed in parallel, so that the efficiency of the simulation evaluation can be improved.

In some other embodiments of the present disclosure, in a case that it is determined that the target task is not completed according to the simulation evaluation result, cyclically executing a simulation adjustment step until the target task is completed according to the simulation evaluation result, and synchronizing a behavior blueprint, a target motion path, and action data and/or text data required for executing the behavior blueprint corresponding to a target subtask that completes the task target to the robot through the digital twin; wherein, the simulation adjusting step may include:

first, the simulation evaluation result, the target subtask, and the digital twin are presented to a user.

The user may be a robot service trainer or a user who issues a target task. The presented information may include the corresponding relationship between each target subtask and the digital twin, so that the user can clearly determine the task allocation condition of each digital twin. The simulation evaluation result may include specific conditions of task execution, such as the duration of task execution, which step the task failed to execute, the reason for the failure of task execution, and the like.

Secondly, receiving a task adjusting instruction input by a user aiming at one or more digital twins, and adjusting the target subtask of the digital twins according to the task adjusting instruction.

The user may adjust the target subtask of each digital twin according to the current task allocation, and the task adjustment instruction may include an adjustment of the target subtask of one or more digital twins, e.g. adjusting the target subtask 1 of the digital twin 1 to the digital twin 2 and adjusting the target subtask 2 of the digital twin 2 to the digital twin 1.

And thirdly, planning a space path according to the adjusted target subtask to obtain an adjusted target motion path of the digital twin.

And finally, controlling a plurality of digital twin bodies to execute simulation operation in a three-dimensional semantic map through a behavior blueprint and evaluate according to the adjusted target subtask and the adjusted target motion path, and taking the result obtained by evaluation as a new simulation evaluation result.

Therefore, under the condition that the target task is determined to be not completed according to the simulation evaluation result, the allocation of the target subtasks can be manually adjusted, the target task is determined to be completed through the adjusted simulation evaluation, and then the target subtasks are issued to the robot and executed by the robot, so that the task is successfully executed at one time, the invalid task execution is avoided, and the energy loss of the robot is reduced.

Further, under the condition that the target task is determined to be not completed according to the simulation evaluation result, or under the condition that the target task is determined to be still not completed according to the simulation evaluation result after the preset adjustment times of the simulation adjustment step are circularly executed, the user can modify the target task, and carry out target subtask planning, space path planning and simulation evaluation again according to the modified target task.

For example, the user can change the original mode of taking a bottle of beverage from a refrigerator on the first floor to be sent to an office on the second floor into the mode of receiving a cup of water from a water dispenser on the second floor to be sent to the office on the second floor.

Therefore, after the target subtask allocation is manually adjusted for multiple times, if the target task cannot be completed, the target task can be adjusted, invalid task execution is avoided, and energy loss of the robot is reduced.

In some further embodiments of the present disclosure, after acquiring a target subtask from one or more candidate subtasks according to a twin parameter, since a state parameter of the robot may be changed, a twin parameter of the digital twin may also be changed, in order to adjust task allocation in time when the twin parameter is changed, the method may further include:

first, new twin parameters of the digital twin are reacquired.

On one hand, the state parameters of the robot can change automatically along with the executed tasks or actions of the robot, for example, after the robot executes the tasks for a long time, the running state of the robot can be changed from a normal state to a charging state or a fatigue state; when some parts of the robot are damaged, the running state can be changed into a fault state; the position information of the robot may vary according to the movement of the robot. Due to the real-time interaction synchronization between the digital twin body and the robot, the twin body parameters of the digital twin body also change synchronously along with the change of the state parameters of the robot. Therefore, the new twin parameters of the digital twin can be periodically inquired and obtained in the step.

On the other hand, the state parameter of the robot may be modified by a user through the robot service client, and the user may be a robot service trainer, for example, to modify the type or operation state of the robot, or to assign a new task, so as to cause a twin parameter of a digital twin corresponding to the robot to change. At this time, the new twin parameters of the digital twin can be reacquired by: receiving an operation instruction input by a user aiming at the digital twin; updating the twin body parameters of the digital twin body according to the operation instruction; and taking the updated twin parameters as new twin parameters.

Further, if a new task to be executed is obtained through the operation instruction, the priority of the task to be executed can be the highest priority, so that the operation instruction of the user is guaranteed to be completed preferentially. For example, if the user finds that the robot has low electric quantity, an operation instruction of a charging task is sent, the robot needs to be controlled preferentially to complete the charging task, and a new task is not executed any more; and after the charging task is completed, executing a new task. In this way, it can be ensured that tasks requested by the user are preferentially performed.

Secondly, determining whether the target subtask needs to be updated according to the new twin parameters.

In this step, the following two ways may be used to determine whether the target subtask needs to be updated:

in a first mode, it may be determined that the target subtask needs to be updated when the new twin parameter meets a preset condition, where the preset condition includes one or more of the following conditions:

preset condition 1: the remaining energy of the digital twin is less than or equal to a preset energy threshold. The preset energy threshold may be 5% or 10%, and if the remaining energy of the digital twin is less than or equal to the preset energy threshold, it may be characterized that the robot corresponding to the digital twin cannot complete the target subtask. The robot can be charged as soon as possible to replenish energy and no longer perform new subtasks.

Preset condition 2: the operation state of the digital twin body is a preset state. Wherein the preset state may include a fault state, a charging state, or a fatigue state.

Preset condition 3: tasks to be executed of the digital twin body have tasks to be executed with task priorities higher than the target subtasks.

And secondly, distributing a new target subtask to the digital twin according to the new twin parameter, and determining that the target subtask needs to be updated if the new target subtask is inconsistent with the target subtask distributed last time.

If the new target subtask is confirmed to be the same as the target subtask allocated last time through one of the two modes, the target subtask does not need to be updated, and the robot continues to execute the current target subtask.

And finally, under the condition that the target subtasks need to be updated, re-executing the steps of planning and distributing one or more target subtasks for a plurality of digital twin bodies corresponding to a plurality of robots according to the target tasks to be executed by the robots, the roles of the robots and the three-dimensional semantic maps of the environments where the robots are located, and synchronizing the behavior blueprints, the target paths and the action data and/or text data required for executing the behavior blueprints to the robots.

In this step, if the target subtask needs to be updated, the steps S201 to S204 may be re-executed, so as to re-perform the target subtask allocation, the spatial path planning, the simulation evaluation, and the task synchronization.

The above-mentioned new twin parameters of the digital twin may be acquired after synchronizing the behavior blueprint, the target path, and the motion data and/or text data required for executing the behavior blueprint to the robot, or may be acquired in a simulation evaluation phase.

Therefore, an end-to-end complete continuous closed-loop optimization target task flow can be formed through the method, and the target subtask of the digital twin can be adjusted in time under the condition that the twin parameters of the digital twin are changed, so that the target task is completed.

In some other embodiments of the present disclosure, the three-dimensional semantic map may be obtained by:

firstly, an environment scanning instruction is sent to the robot, and the environment scanning instruction is used for instructing the robot to scan the environment of the target task and obtaining environment information.

In this step, one or more robots in the robot system may be selected to scan the environment of the target task, thereby obtaining environment information. The accuracy of the environmental scan may be determined by the scan acquisition accuracy of the robot, which may be 3 or 5 centimeters, for example. The robot may send the scanned environmental information to the server.

Then, receiving environment information sent by the robot according to the environment scanning instruction; and acquiring a three-dimensional semantic map according to the environment information.

It should be noted that the three-dimensional semantic map constructed according to the environment information can be used for the positioning and navigation functions of the robot performing the target subtasks. The precision of the three-dimensional semantic map may be higher than or equal to the scanning and collecting precision of the robot, for example, if the scanning and collecting precision of the robot is 3 cm, the precision of the three-dimensional semantic map may also be 3 cm, and may also be 1 cm or 2 cm.

Therefore, the three-dimensional semantic map can be constructed through robot scanning, so that extra tools are not needed for scanning and construction, and the accurate three-dimensional semantic map can be conveniently obtained to execute a target task.

Further, in the task execution process, new environment information sent by the robot can be received, and the three-dimensional semantic map is updated according to the new environment information. Therefore, under the condition that the environment changes, the three-dimensional semantic map can be updated in time, so that the target task can be completed smoothly.

In addition, the target task can be obtained by any one of the following modes:

and the first task acquisition mode is used for receiving a task instruction input by a user and acquiring a target task according to the task instruction.

Wherein, a task instruction of a user can be received by a certain robot of the robot system, for example, a user in an office says to the robot: "please help me take a bottle of beverage", after receiving the task instruction, according to the task instruction, can obtain the target task as: take bottles of beverages from the first floor refrigerator to the second floor office.

In addition, the task instruction of the user may also be received through a robot service client, where the robot service client may be an electronic device such as a terminal, a PAD, or a computer, and may be, for example, a robot server APP installed on the electronic device. Therefore, the robot service client can receive the task instruction input by the user and acquire the target task according to the task instruction.

Therefore, the task instruction of the user can be completed through the method, and the requirement of the user on the robot is met.

And a task obtaining mode II of receiving the environment information sent by the robot and obtaining the target task according to the environment information.

The target task can be obtained according to the change of the environmental information, for example, the target task of wiping the ground can be obtained when the water is found on the ground through the environmental information sent by the robot.

Thus, by the mode, the target task can be automatically executed based on the environment information, and the automatic operation of the robot system is realized.

Fig. 1 is a schematic structural diagram of a multi-robot control system provided in an embodiment of the present disclosure, as shown in fig. 1, the multi-robot control system includes a server 101, and one or more robots 102 connected to the server; the server includes one or more digital twins, each digital twins being a three-dimensional physical model corresponding to a robot; wherein:

the server 101 is configured to execute the steps of the multi-robot control method provided in any of the above embodiments.

And the robot 102 is configured to receive the behavior blueprint planned by the server 101, the target path, and the action data and/or the text data required for executing the behavior blueprint, and execute the behavior blueprint.

By adopting the multi-robot control system, a server plans and distributes target subtasks for the digital twin corresponding to the robots according to target tasks to be executed by the robots, roles of the robots and a three-dimensional semantic map of environments where the robots are located; space path planning is carried out according to the target subtasks to obtain a target motion path of the digital twin body; controlling a digital twin to execute simulation operation in a three-dimensional semantic map through a behavior blueprint and evaluate according to a target subtask and a target motion path to obtain a simulation evaluation result; and if the target task is determined to be completed according to the simulation evaluation result, the behavior blueprint, the target motion path and the action data and/or the text data required for executing the behavior blueprint for completing the target subtask are sent to the robot through the plurality of digital twin bodies so as to control the robot to execute the behavior blueprint and complete the target task, so that the robot can be ensured to complete the target task, and the accuracy of controlling the plurality of robots is improved.

Further, fig. 3 is a schematic structural diagram of another multi-robot control system provided in the embodiment of the present disclosure, and as shown in fig. 3, the robot 102 may include: a digital twin replica 104 and a robot body 105; wherein:

and the digital twin copy 104 can be used for interactive synchronization with a digital twin body of the server, synchronizing a behavior blueprint planned by the server, a target path, and action data and/or text data required for executing the behavior blueprint, and synchronously controlling the robot body to execute the behavior blueprint, wherein the manner of synchronously controlling the robot body can be to send a control instruction to the robot body 105.

And the robot body 105 is used for executing a behavior blueprint according to the control of the digital twin copy 104.

In this way, the execution of the target subtask is achieved by the digital twin copy and the robot ontology.

Still further, fig. 4 is a schematic structural diagram of another multi-robot control system provided in the embodiment of the present disclosure, and as shown in fig. 4, the robot 102 further includes a perception processing component 106; wherein:

the perception processing component 106 is configured to scan an environment of the target task and obtain environment information;

the digital twin copy 104 is configured to receive an environment scanning instruction sent by the server 101, and control the sensing processing component 106 to scan according to the environment scanning instruction; and acquiring the environmental information scanned by the sensing processing component 106, and sending the environmental information to the server 101.

Therefore, the robot realizes the environment scanning function through the perception processing component, so that the scanned environment information is sent to the server, and the server can construct a three-dimensional semantic map according to the environment information for executing a target task.

Further, the digital twin copy 104 is also used for synchronizing the state information of the robot to the digital twin so that the digital twin synchronously updates the twin parameters of the digital twin according to the state information.

Therefore, the robot realizes the real-time interactive synchronization of the twin parameters of the digital twin and the corresponding state parameters of the robot through the real-time interactive synchronization of the digital twin copy and the digital twin of the server, so that the digital twin can vividly reflect the real state of the robot.

Fig. 5 is a multi-robot control apparatus provided in an embodiment of the present disclosure, and as shown in fig. 5, the apparatus includes:

a task allocation module 501, configured to plan and allocate one or more target subtasks for a plurality of digital twin corresponding to a plurality of robots according to target tasks to be executed by the plurality of robots, roles of the plurality of robots, and a three-dimensional semantic map of environments in which the plurality of robots are located, plan one or more behavior blueprints corresponding to each target subtask, and execute action data and/or text data required by the behavior blueprints, where the role represents a preset function set that the robot is allowed to execute, and the behavior blueprints include an action sequence that the robot needs to execute to complete a target subtask and a logic judgment that the action sequence needs to be executed;

a path planning module 502, configured to perform spatial path planning according to the target subtask to obtain a target motion path of the digital twin;

the simulation evaluation module 503 is configured to control the plurality of digital twin bodies to perform simulation operation in the three-dimensional semantic map through a behavior blueprint and evaluate according to the target subtask and the target motion path, so as to obtain a simulation evaluation result, where the simulation evaluation result is used to represent whether the target task is completed;

and a task synchronization module 504, configured to synchronize the behavior blueprint, the target path, and motion data and/or text data required for executing the behavior blueprint to the robot through the digital twin when it is determined that the target task is completed according to the simulation evaluation result, so as to control the robot to execute the behavior blueprint and complete the target task.

Optionally, the task allocation module 501 is configured to:

acquiring twin parameters of the digital twin, wherein the twin parameters comprise one or more of position information of the digital twin, residual energy of the digital twin, continuous operation duration of the digital twin, operation state of the digital twin and tasks to be executed of the digital twin;

and acquiring a target subtask from the one or more candidate subtasks according to the twin parameter, and distributing the target subtask to the digital twin.

Optionally, the target task is divided into a plurality of candidate subtask sets according to different planning manners during task planning, each of the candidate subtask sets includes one or more target subtasks, and the simulation evaluation module 503 is configured to:

for each candidate subtask set of a plurality of candidate subtask sets, performing simulation running and evaluation of the candidate subtask set, wherein the performing simulation running and evaluation of the candidate subtask set comprises: controlling the digital twin to execute simulation operation and evaluate in the three-dimensional semantic map through a behavior blueprint according to the target subtasks in the candidate subtask set and the target motion path corresponding to each target subtask to obtain a candidate evaluation result and a task execution evaluation result of the candidate subtask set;

and taking the candidate subtask set with the globally optimal task execution evaluation result as a target subtask set, and taking the candidate evaluation result corresponding to the target subtask set as the simulation evaluation result, wherein the globally optimal task execution evaluation result represents that the robot energy consumed for completing the target task is the least, or the time consumed for completing the target task is the shortest, or the moving path of the robot for completing the target task is the shortest.

The task synchronization module 504 is configured to: and synchronizing the behavior blueprint, the target motion path and action data and/or text data required for executing the behavior blueprint corresponding to the target subtask in the target subtask set to the robot.

Optionally, the apparatus further comprises:

a simulation adjusting module 601, configured to cyclically execute a simulation adjusting step when it is determined that the target task is not completed according to the simulation evaluation result, and synchronize, through the digital twin, a behavior blueprint, a target motion path, and action data and/or text data required for executing the behavior blueprint, which correspond to a target subtask that completes the task target, to the robot after it is determined that the target task is completed according to the simulation evaluation result; wherein, the simulation adjusting step comprises:

receiving a task adjusting instruction input by a user aiming at one or more digital twins, and adjusting the target subtasks of the digital twins according to the task adjusting instruction;

performing space path planning according to the adjusted target subtask to obtain an adjusted target motion path of the digital twin;

and controlling the plurality of digital twins to execute simulation operation in the three-dimensional semantic map through the behavior blueprint and evaluate according to the adjusted target subtasks and the adjusted target motion path, and taking the result obtained by evaluation as a new simulation evaluation result.

Optionally, the apparatus further comprises:

a task updating module 701, configured to reacquire new twin parameters of the digital twin; determining whether the target subtask needs to be updated according to the new twin parameters; and under the condition that the target subtasks need to be updated, re-executing a step of distributing one or more target subtasks for the digital twin plans corresponding to the robots according to the target tasks to be executed by the robots, the roles of the robots and the three-dimensional semantic maps of the environments where the robots are located, and synchronizing the behavior blueprints, the target paths and action data and/or text data required for executing the behavior blueprints to the robots.

Optionally, the task updating module 701 is configured to receive an operation instruction input by a user for the digital twin; updating the twin body parameters of the digital twin body according to the operation instruction; and taking the updated twin parameters as new twin parameters.

Optionally, the task updating module 701 is configured to determine that the target subtask needs to be updated when the new twin parameter meets a preset condition, where the preset condition includes one or more of the following:

the residual energy of the digital twin body is less than or equal to a preset energy threshold value, and the preset energy threshold value represents that the residual energy of the digital twin body cannot complete the target subtask;

the running state of the digital twin body is a preset state;

the tasks to be executed of the digital twin body have tasks to be executed with task priorities higher than the target subtask.

Optionally, the task synchronization module 504 is further configured to obtain a local three-dimensional semantic map required by the robot according to the three-dimensional semantic map, the target subtask, and the target motion path; and sending the local three-dimensional semantic map to the robot.

Optionally, the apparatus further comprises:

a three-dimensional semantic map obtaining module 801, configured to send an environment scanning instruction to the robot, where the environment scanning instruction is used to instruct the robot to scan an environment of a target task and obtain environment information; and receiving the environment information sent by the robot according to the environment scanning instruction, and acquiring the three-dimensional semantic map according to the environment information.

Optionally, the apparatus further comprises:

a target task obtaining module 901, configured to receive a task instruction input by a user, and obtain the target task according to the task instruction; and receiving the environmental information sent by the robot, and acquiring the target task according to the environmental information.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Fig. 10 is a block diagram illustrating an electronic device 1000 in accordance with an example embodiment. As shown in fig. 10, the electronic device 1000 may include: a processor 1001 and a memory 1002. The electronic device 1000 may also include one or more of a multimedia component 1003, an input/output (I/O) interface 1004, and a communications component 1005.

The processor 1001 is configured to control the overall operation of the electronic device 1000, so as to complete all or part of the steps in the multi-robot control method. The memory 1002 is used to store various types of data to support operation of the electronic device 1000, such as instructions for any application or method operating on the electronic device 1000 and application-related data, such as contact data, messaging, pictures, audio, video, and so forth. The Memory 1002 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic Memory, a flash Memory, a magnetic disk, or an optical disk. The multimedia components 1003 may include screen and audio components. Wherein the screen may be, for example, a touch screen and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may further be stored in memory 1002 or transmitted through communication component 1005. The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 1004 provides an interface between the processor 1001 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 1005 is used for wired or wireless communication between the electronic device 1000 and other devices. Wireless Communication, such as Wi-Fi, bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, which is not limited herein. The corresponding communication component 1005 may thus include: Wi-Fi module, Bluetooth module, NFC module, etc.

In an exemplary embodiment, the electronic Device 1000 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the multi-robot control method described above.

In another exemplary embodiment, there is also provided a computer-readable storage medium including program instructions which, when executed by a processor, implement the steps of the multi-robot control method described above. For example, the computer readable storage medium may be the memory 1002 including program instructions executable by the processor 1001 of the electronic device 1000 to perform the multi-robot control method described above.

Fig. 11 is a block diagram illustrating an electronic device 1100 in accordance with an example embodiment. For example, the electronic device 1100 may be provided as a server. Referring to fig. 11, electronic device 1100 includes a processor 1122, which can be one or more in number, and a memory 1132 for storing computer programs executable by processor 1122. The computer programs stored in memory 1132 may include one or more modules that each correspond to a set of instructions. Further, the processor 1122 may be configured to execute the computer program to perform the multi-robot control method described above.

Additionally, the electronic device 1100 may also include a power component 1126 and a communication component 1150, the power component 1126 may be configured to perform power management of the electronic device 1100, and the communication component 1150 may be configured to enable communication, e.g., wired or wireless communication, of the electronic device 1100. In addition, the electronic device 1100 may also include an input/output (I/O) interface 1158. The electronic device 1100 may operate based on an operating system stored in the memory 1132, such as Windows Server^TM，Mac OSX^TM，Unix^TM，Linux^TMAnd so on.

In another exemplary embodiment, there is also provided a computer-readable storage medium including program instructions which, when executed by a processor, implement the steps of the multi-robot control method described above. For example, the computer-readable storage medium may be the above-described memory 1132 including program instructions executable by the processor 1122 of the electronic device 1100 to perform the above-described multi-robot control method.

In another exemplary embodiment, a computer program product is also provided, which comprises a computer program executable by a programmable apparatus, the computer program having code portions for performing the multi-robot control method described above when executed by the programmable apparatus.

The preferred embodiments of the present disclosure are described in detail with reference to the accompanying drawings, however, the present disclosure is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solution of the present disclosure within the technical idea of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.

It should be noted that the various features described in the above embodiments may be combined in any suitable manner without departing from the scope of the invention. In order to avoid unnecessary repetition, various possible combinations will not be separately described in this disclosure.

In addition, any combination of various embodiments of the present disclosure may be made, and the same should be considered as the disclosure of the present disclosure, as long as it does not depart from the spirit of the present disclosure.

Claims

1. A multi-robot control method, the method comprising:

and under the condition that the target task is determined to be completed according to the simulation evaluation result, synchronizing the behavior blueprint, the target motion path and action data and/or text data required for executing the behavior blueprint to the robot through the digital twin so as to control the robot to execute the behavior blueprint and complete the target task.

2. The method of claim 1, wherein the assigning one or more target subtasks to the digital twin plan corresponding to the robot based on the target tasks to be performed by the plurality of robots, the roles of the plurality of robots, and the three-dimensional semantic map of the environment in which the plurality of robots are located comprises:

3. The method according to claim 1, wherein the target task is divided into a plurality of candidate subtask sets according to different planning manners during task planning, each candidate subtask set includes one or more target subtasks, the controlling the digital twin to perform simulation operation and evaluation in the three-dimensional semantic map through a behavior blueprint according to the target subtask and the target motion path includes:

4. The method of claim 1, further comprising:

5. The method of claim 2, wherein after obtaining a target subtask from the one or more candidate subtasks based on the twin parameters, the method further comprises:

reacquiring new twin parameters of the digital twin;

6. The method of claim 5, wherein the reacquiring new twin parameters of the digital twin comprises:

receiving an operation instruction input by a user aiming at the digital twin;

and taking the updated twin parameters as new twin parameters.

7. A multi-robot control apparatus, characterized in that the apparatus comprises:

8. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 6.

9. An electronic device, comprising:

a memory having a computer program stored thereon;

a processor for executing the computer program in the memory to carry out the steps of the method of any one of claims 1 to 6.

10. A multi-robot control system is characterized in that the system comprises a server and a plurality of robots connected with the server; the server comprises a plurality of digital twins, each digital twins being a virtual three-dimensional physical model corresponding to one robot; wherein:

the server for performing the steps of the method of any one of claims 1 to 6;