WO2022116716A1 - 云端机器人系统、云服务器、机器人控制模块和机器人 - Google Patents

云端机器人系统、云服务器、机器人控制模块和机器人 Download PDF

Info

Publication number
WO2022116716A1
WO2022116716A1 PCT/CN2021/124506 CN2021124506W WO2022116716A1 WO 2022116716 A1 WO2022116716 A1 WO 2022116716A1 CN 2021124506 W CN2021124506 W CN 2021124506W WO 2022116716 A1 WO2022116716 A1 WO 2022116716A1
Authority
WO
WIPO (PCT)
Prior art keywords
robot
digital twin
data
module
physical
Prior art date
Application number
PCT/CN2021/124506
Other languages
English (en)
French (fr)
Inventor
黄晓庆
张站朝
马世奎
王斌
董文锋
Original Assignee
达闼机器人股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 达闼机器人股份有限公司 filed Critical 达闼机器人股份有限公司
Publication of WO2022116716A1 publication Critical patent/WO2022116716A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models

Definitions

  • Embodiments of the present invention relate to the field of robotics, and in particular, to a cloud robot system, a cloud server, a robot control module, and a robot.
  • cloud robots have been more and more widely used. Among them, in some application scenarios that are dangerous, dirty, repetitive and difficult to implement, the requirements for cloud robots are also higher, resulting in a market demand for intelligent robots that can replace humans in function.
  • the embodiments of the present invention provide a cloud robot system, a cloud server, a robot control module and a robot, which are used to solve the problem that the cloud robot implementation solution in the prior art is not intelligent enough.
  • a cloud robot system including a cloud server and a robot control module, where the cloud server includes a robot access and data exchange module, a knowledge and data intelligence module, an artificially enhanced machine intelligence module,
  • the digital twin runs the core module and the robot big data module
  • the robot control module is located in the physical robot, and the robot control module and the cloud server communicate through a dedicated network;
  • the robot access and data exchange module is used for robot service process registration and robot access authentication, and for receiving multi-source data sent by the robot control module, and for data exchange, fusion and distribution;
  • the knowledge and data intelligence module is used to provide a multi-domain knowledge map, a robot behavior library and a three-dimensional environment semantic map for robot services;
  • the digital twin operation core module includes a digital twin world and a digital twin
  • the robot control module includes a digital twin copy, wherein the digital twin world is constructed based on the three-dimensional environment semantic map, and the digital twin is the same as the digital twin.
  • the physical model with the same physical properties of the physical robot, and the digital twin is a copy of the digital twin running on the cloud server; the digital twin is used in the digital twin world based on the robot
  • the multi-domain knowledge graph of the service, the robot behavior and action library, and the multi-source data perform training and online operation of robot skills and applications, and the digital twin copy controls synchronously according to the robot skills and applications performed by the digital twin.
  • the physical robot performs the robotic skills and applications;
  • the artificially enhanced machine intelligence module supports the digital twin operation core module to perform training and online operation of robot skills and applications through language AI, visual AI, motion AI, multimodal AI and artificially enhanced AI;
  • the robot big data module is used for storing and analyzing the multi-source data, and feeding back the analyzed multi-source data to the digital twin operation core module for training and online operation of the robot skills and applications;
  • the robot control module is further configured to send multi-source data to the robot access and data exchange module.
  • the cloud server further includes a robot business application service platform, which is used to configure the physical robot and provide the download of the robot service.
  • the configuring the physical robot includes:
  • the application scene is configured according to the three-dimensional environment semantic map.
  • the cloud server further includes a robot open platform for providing a robot service development interface for developers to develop the robot service.
  • the robot service is an application developed and trained based on the digital twin
  • the robot service development includes digital twin development, robot behavior and action editing, and robot business behavior blueprint editing.
  • the digital twin operation core module is further configured to: during the training process of the digital twin performing the robot skills and applications in the digital twin world, if the digital twin When the numerical evaluation of the completion of the robot skills and applications performed by the body exceeds the first preset threshold, it is determined that the training of the robot skills and applications is completed. If it is determined that the training of the robot skills and applications is completed, the training completed Robot skills and applications are loaded into the robot control module for synchronous trial operation;
  • the robot control module is also used for: loading and synchronizing the robot skills and applications completed by the trial run training;
  • the digital twin operation core module is also used for: if the numerical evaluation of the completion of the robot skills and applications completed by the robot control module trial operation training exceeds the second preset threshold, the corresponding robot skills and applications are compared.
  • the services are published to the robot business application service platform.
  • the digital twin operation core module further includes a first game engine for loading the digital twin and the digital twin world, running and updating the digital twin world, and running all describe the behavior and actions of the digital twin;
  • the robot control module also includes a second game engine for running the digital twin
  • the first game engine and the second game engine are used to jointly drive the behaviors and actions of the digital twin and the digital twin to execute synchronously.
  • the digital twin operation core module is further used for: synchronizing the behavior and actions of the digital twin to the digital twin copy on the robot control module through the dedicated network;
  • the digital twin replica synchronously controls the physical robot to execute the behavior and action according to the behavior and action of the digital twin.
  • the robot control module is further configured to: send the current environment change information obtained by the sensor of the entity robot and the behavior and action change information of the entity robot itself to the digital twin operation core module, So that the digital twin and the physical robot keep the behavior and movement synchronized.
  • the multi-domain knowledge graph includes a semantic network of relationships between entities related to robotic services, the semantic network includes information and knowledge, the information is used to describe external objective facts, the Knowledge is the induction and summary of external objective laws;
  • the robot behavior action library includes human behaviors and actions learned by the robot through imitation
  • the three-dimensional environment semantic map is the semantic data of the three-dimensional environment where the entity robot is located, and the three-dimensional environment semantic map is obtained by the following method: obtaining the three-dimensional environment data by fusing the multi-source data, and obtaining the three-dimensional environment data based on the three-dimensional environment data. Segmentation performs map modeling to construct the three-dimensional environmental semantic map.
  • the constructing the three-dimensional environment semantic map includes:
  • a multi-semantic fusion 3D environment semantic map is constructed.
  • the language AI includes automatic speech recognition, natural language understanding and speech synthesis
  • the visual AI includes face recognition, human body recognition, portrait recognition, object recognition and environmental scene recognition
  • the motion AI includes external force sensing perception, autonomous movement and navigation, and body movements
  • the multimodal AI refers to the ability to have the language AI, visual AI and motion AI, as well as the ability to combine multiple factors at the same time, wherein the The multi-factor combined output includes the input of the language AI, visual AI and motion AI, as well as voice output and motion output;
  • the artificially enhanced AI is used to provide positive incentive input for system reinforcement learning through manual intervention.
  • the language AI, visual AI, motion AI and multi-modal AI are all running online.
  • the artificially enhanced AI is further used to: if an abnormal situation of the robot service occurs, receive the operation of the digital twin by the service trainer within his control authority.
  • the robot big data module is further configured to store and analyze one or more of system operation and service log data, user data, artificially enhanced operation data and system performance data.
  • the multi-source data includes one or more of audio and video data obtained by sensors of the physical robot, three-dimensional environment point cloud data, robot behavior and motion data, and multi-modal interaction data. variety.
  • the robot big data module is further used for:
  • the numerical evaluation includes the actual recognition rate of AI algorithms and models, the satisfaction of man-machine dialogue responses, the service response time, and the efficiency and stability of the robot business behavior blueprint;
  • the robot big data module is further used for: classifying the target conclusion of the numerical evaluation to form prior knowledge, related business and related data.
  • a cloud server for controlling a physical robot, including a robot access and data exchange module, a knowledge and data intelligence module, an artificially enhanced machine intelligence module, a digital twin operation core module and A robot big data module, the cloud server and the entity robot communicate through a dedicated network;
  • the robot access and data exchange module is used for robot service process registration and robot access authentication, as well as receiving multi-source data sent by the entity robot, and performing data exchange, fusion and distribution;
  • the knowledge and data intelligence module is used to provide a multi-domain knowledge map, a robot behavior library and a three-dimensional environment semantic map for robot services;
  • the digital twin operation core module includes a digital twin world and a digital twin, wherein the digital twin world is constructed based on the three-dimensional environment semantic map, and the digital twin is a physical model with the same physical attributes as the physical robot;
  • the digital twin is used to perform training and online operation of robot skills and applications in the digital twin world based on the multi-domain knowledge graph of the robot service, the robot behavior library, and the multi-source data to synchronize controlling the physical robot to perform the robot skills and application tasks;
  • the artificially enhanced machine intelligence module supports the digital twin operation core module for training and online operation of robot skills and applications through language AI, visual AI, motion AI, multimodal AI and artificially enhanced AI; the robot big data
  • the module is used for storing and analyzing the multi-source data, and feeding back the analyzed multi-source data to the digital twin operation core module for training and online operation of the robot skills and applications.
  • a robot control module is provided, and communication between the robot control module and a cloud server is performed through a dedicated network;
  • the robot control module includes a digital twin copy, which is a copy of the digital twin body running on the cloud server; the digital twin copy controls synchronously according to the robot skills and applications performed by the digital twin body. Physical robots perform said robotic skills and applications;
  • the robot control module is further configured to send multi-source data to the cloud server, so that the digital twin is executed in the digital twin world based on the multi-domain knowledge graph of the robot service, the robot behavior and action library, and the multi-source data. Training and online operation of robotic skills and applications to synchronously control the physical robot to execute the robotic skills and applications through the digital twin.
  • a robot including the robot control module as described above.
  • a digital twin world is constructed on a cloud server, and a digital twin body with the same physical attributes as the physical robot is used in the digital twin world to train and run the robot online, and control the virtual digital twin body to realize the control of the real robot.
  • the synchronous control of the real robot reduces the difficulty and cost of the operation of the entity robot to complete the business application, and adopts artificial enhancement AI to introduce manual operation as the positive incentive input of the system reinforcement learning, which supports the training and application of the robot skills and applications by the digital twin.
  • Online operation, and the multi-source data collected by the physical robot is also fed back to the cloud server for training and online operation of robot skills and applications, realizing a dynamic closed-loop and continuously evolving intelligent cloud robot system.
  • FIG. 1 is a schematic diagram of an application of a cloud robot system provided by an embodiment of the present invention
  • FIG. 2 is a schematic structural diagram of a cloud robot system provided by an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of a framework of a cloud robot system provided by an embodiment of the present invention.
  • FIG. 4 is a schematic flowchart of the operation of a robot service provided by an embodiment of the present invention.
  • FIG. 5 is a schematic flowchart of a robot service development provided by an embodiment of the present invention.
  • FIG. 6 is a schematic flowchart of an artificially enhanced AI operation provided by an embodiment of the present invention.
  • FIG. 7 is a schematic structural diagram of a cloud server provided by an embodiment of the present invention.
  • FIG. 8 is a schematic structural diagram of a robot control module provided by an embodiment of the present invention.
  • FIG. 9 is a schematic structural diagram of a robot provided by an embodiment of the present invention.
  • FIG. 1 is a schematic diagram of an application of the cloud robot system provided by the embodiment of the present invention.
  • the communication between the cloud server 10 and the physical robot 20 is performed through a dedicated network 30 .
  • Various robot services are trained on the cloud server 10, and the cloud server 10 controls the physical robot 20 to execute the trained robot services.
  • robot service refers to performing preset actions in different application scenarios to complete preset functions, such as welcome reception, mobile grabbing, security patrols, and distribution.
  • a service needs to be composed of an application, and a number of skill sets make up the logic of the application. For example, playing ping-pong, cutting the ball, pulling the ball, etc.
  • the service refers to the service that the physical robot can provide the service of playing table tennis sparring.
  • the physical robot grasping items is a skill, and the physical robot can use the grasping skills to complete the application of delivering coffee to people, and the physical robot can complete the reception service of serving tea and pouring water.
  • FIG. 2 is a schematic structural diagram of a cloud robot system provided by an embodiment of the present invention.
  • the cloud robot system 100 includes a cloud server 10 and a robot control module 21 .
  • the cloud server 10 includes a robot access and data exchange module 11 , a knowledge and data intelligence module 12 , an artificially enhanced machine intelligence module 13 , a digital twin operation core module 14 and a robot big data module 15 .
  • the robot control module 21 is located in the physical robot 20 . Communication between the robot control module 21 and the cloud server 10 is performed through the dedicated network 30 . Communication through the private network 30 can ensure the security of the communication between the robot control module 21 and the cloud server 10 .
  • the robot access and data exchange module 11 is used to register the robot service process and the robot access authentication, receive multi-source data sent by the robot control module 21, and perform data exchange, fusion and distribution.
  • the service process refers to the service process of the program, that is, the microservice.
  • the knowledge and data intelligence module 12 is used to provide a multi-domain knowledge map, a robot behavior library and a three-dimensional environment semantic map for robot services.
  • the digital twin operation core module 14 includes the digital twin world and the digital twin body, and the robot control module 21 includes the digital twin copy.
  • the digital twin world is constructed based on the three-dimensional environment semantic map, the digital twin is a physical model with the same physical properties as the physical robot, and the digital twin copy is a copy of the digital twin running on the cloud server.
  • the digital twin is used to perform training and online operation of robot skills and applications based on the multi-domain knowledge graph, robot behavior library and multi-source data of robot services in the digital twin world. According to the robot skills and applications performed by the digital twin, the digital twin synchronously controls the physical robot to perform robot skills and applications.
  • the artificially enhanced machine intelligence module 13 supports the digital twin operation core module 14 to perform training and online operation of robot skills and applications through multimodal AI and artificially enhanced AI.
  • the robot big data module 15 is used to store and analyze multi-source data, and feed back the analyzed multi-source data to the digital twin operation core module 14 for training and online operation of robot skills and applications.
  • the robot control module 21 is also used for sending multi-source data to the robot access and data exchange module 11 .
  • a digital twin world is constructed on a cloud server, and a digital twin body with the same physical attributes as the physical robot is used in the digital twin world to train and run the robot online, and control the virtual digital twin body to realize the control of the real robot.
  • the synchronous control of the real robot reduces the difficulty and cost of operating the entity robot to complete the business application, and adopts artificial enhancement AI to introduce manual operation as the positive incentive input of the system reinforcement learning, and supports the training and online training of robot skills and applications by the digital twin.
  • the multi-source data collected by the physical robot is also fed back to the cloud server for training and online operation of robot skills and applications, realizing a dynamic closed-loop and continuously evolving intelligent cloud robot system.
  • FIG. 3 is a schematic diagram of a framework of a cloud robot system provided by an embodiment of the present invention.
  • the cloud robot system adopts a distributed computing architecture of "cloud (brain)-network (nerve)-end (body)".
  • the “cloud” is located in the cloud server
  • the network refers to the private network
  • the "end” is located in the physical robot.
  • the artificially enhanced machine intelligence module of the cloud brain organically integrates multimodal AI and artificially enhanced AI, such as robot language AI (Artificial Intelligence, artificial intelligence) capabilities, visual AI capabilities, motion AI capabilities, and environmental cognitive capabilities. It forms the perception and cognitive capabilities of the cloud brain, and combines human prior knowledge and data intelligence to realize advanced human-like intelligence such as logical reasoning and intelligent decision-making. Run the core module through the digital twin, let the digital twin of the physical robot run in the virtual digital twin world, and execute robot skills and applications. All the behaviors and actions of the digital twin will synchronously control the digital twin copy running in the robot control module of the physical robot through a dedicated network.
  • robot language AI Artificial Intelligence, artificial intelligence
  • visual AI capabilities visual AI capabilities
  • motion AI capabilities motion AI capabilities
  • environmental cognitive capabilities combines human prior knowledge and data intelligence to realize advanced human-like intelligence such as logical reasoning and intelligent decision-making.
  • Run the core module through the digital twin let the digital twin of the physical robot run in the virtual digital twin world, and execute robot skills and applications. All the
  • the instructions and data sequences executed by the digital twin copy will drive the physical robot to synchronize all the behaviors and actions of the digital twin. , to complete the target tasks of the physical robot in the application scenario, so as to make the entire cloud robot system more intelligent, allowing users to use the physical robot to provide intelligent services for various industries in a simple, safe and reliable way. Among them, the actions of one or more robots constitute a meaningful behavior.
  • the cloud server includes, in addition to the robot access and data exchange module, knowledge and data intelligence module, artificially enhanced machine intelligence module, digital twin operation core module, and robot big data module of the above-mentioned embodiment, the robot business Application service platform and robot open platform.
  • the robot access and data exchange module is used for robot service process registration and robot access authentication, as well as receiving multi-source data sent by the robot control module, and performing data exchange, fusion and distribution.
  • the multi-source data includes one or more of audio and video data acquired by the sensors of the physical robot, three-dimensional environment point cloud data, robot behavior and motion data, and multi-modal interaction data.
  • the robot behavior and action data are mainly robot joint motion frame data.
  • the sensors of the robot acquire data in various ways, including vision, ultrasound, and laser.
  • Multimodal interaction generally refers to human-computer interaction through text, voice, vision, action, environment, etc., to fully simulate the interaction between people.
  • the exchange of data refers to the exchange of multi-source data upstream of the physical robot and data downstream from the cloud server (such as control instructions, voice data, update data, etc.), such as sending the upstream data of the physical robot to the robot big data module, Send the downlink data from the cloud server to the physical robot.
  • the cloud server such as control instructions, voice data, update data, etc.
  • Data distribution refers to distributing the upstream data on the cloud server to one or more services for different processing or analysis.
  • a service is a program that provides various function calls for other programs, such as a program, routine or process running in the background of the operating system.
  • audio and video data can be distributed to visual processing services, and at the same time, visual monitoring by users and operators is also required.
  • Data fusion refers to the processing of data from different sources or different structures to form a standard data interface or to use a standard data structure to represent it.
  • a standard data interface is formed by adding data descriptions, including but not limited to interface identifiers. , SessionID (session ID), interface type, interface sequence, version, initiator, receiver, initiator module, receiver module, data identification ID, etc.
  • SessionID session ID
  • interface type interface type
  • interface sequence version
  • initiator receiver
  • initiator module e.g., version of the same category and different sources
  • standard data structures can be used to represent them.
  • the knowledge and data intelligence module is used to provide multi-domain knowledge graph, robot behavior library and 3D environment semantic map for robot services.
  • the multi-domain knowledge graph and robot behavior library of robot service belong to the human prior knowledge base.
  • the 3D environment semantic map is perceived and recognized by the physical robot through various sensors.
  • the multi-domain knowledge graph includes the semantic network of the relationship between entities related to robot services.
  • the semantic network includes information and knowledge.
  • Information is used to describe external objective facts, and knowledge is the induction and summary of external objective laws.
  • multi-domain knowledge graphs include, but are not limited to, knowledge graphs for various vertical fields and industries, and corpora for general natural language understanding, such as character relationship knowledge graphs, hotel industry knowledge graphs, real estate industry knowledge graphs, and Chinese historical knowledge graphs Wait.
  • the robot behavior action library includes human behaviors and actions learned by robots through imitation, including but not limited to human actions that physical robots can learn through imitation, such as grasping target objects, autonomous positioning and navigation, raising hands, bending over, shaking hands, etc.
  • the 3D environment semantic map is the semantic data of the 3D environment where the entity robot is located.
  • the 3D environment semantic map is a semantic-level data service provided to the 3D environment where the entity robot is located. It describes the environment and relationship of the objective physical world in the way of human natural language, and it is recognized by the entity robot in various application scenarios.
  • the digital representation of the understood 3D environment semantics helps physical robots perceive and recognize the physical world, and is used to train virtual digital twins (i.e., digital twin robots, which will be described in detail later).
  • the 3D environment semantic map can be obtained by fusing multi-source data to obtain 3D environment data, and performing map modeling through semantic segmentation based on the 3D environment data to construct a 3D environment semantic map.
  • semantic segmentation is aimed at 3D environment data fused with multiple features.
  • a three-dimensional environmental semantic map can be constructed by combining deep learning-based application scene recognition, object detection and recognition, geometric model representation, spatial semantic relationship and semantic annotation to construct a multi-semantic fusion three-dimensional environmental semantic map.
  • the three-dimensional environmental semantic map is stored and accessed by means of a database.
  • the core modules of the digital twin operation include the digital twin world and the digital twin body, and correspondingly, the robot control module located on the physical robot includes the digital twin copy (which will be explained in detail later).
  • the digital twin world is constructed based on a three-dimensional environment semantic map
  • the digital twin is a physical model with the same physical properties as the physical robot
  • the digital twin copy is a copy of the digital twin running on the cloud server.
  • the digital twin is used to perform training and online operation of robot skills and applications based on the multi-domain knowledge graph, robot behavior library and multi-source data of robot services in the digital twin world. Application, synchronously control the physical robot to perform robot skills and applications.
  • the embodiment of the present invention adopts a digital twin with the same physical attributes as the physical robot, and realizes the low-cost training and trial-and-error process of the robot service.
  • the digital twin world is constructed through the fusion of multiple sensors of the robot, so that the digital twin of the robot can be trained and run online in real time in the digital twin world, and the synchronous control of the physical robot can be realized through the control of the virtual digital twin, which reduces the need for the completion of the physical robot.
  • Manipulation requirements for robotic services are constructed through the fusion of multiple sensors of the robot, so that the digital twin of the robot can be trained and run online in real time in the digital twin world, and the synchronous control of the physical robot can be realized through the control of the virtual digital twin, which reduces the need for the completion of the physical robot.
  • the behaviors and actions of the digital twin are synchronized to the digital twin on the robot control module through a dedicated network, and the digital twin synchronously controls the physical robot to perform actions and actions according to the actions and actions of the digital twin.
  • a first preset threshold can be preset as a numerical evaluation threshold for the completion of training of robot skills and applications
  • a second preset threshold can be preset as the completion of the trial operation of robot skills and applications The numerical evaluation threshold of the situation.
  • the numerical evaluation includes the actual recognition rate of AI algorithms and models, the satisfaction of man-machine dialogue responses, service response time, and the efficiency and stability of the robot's business behavior blueprint.
  • the robot big data module is further used for: classifying the target conclusion of the numerical evaluation to form prior knowledge, related business and related data. Among them, the numerical evaluation can be completed by the robot big data module, which will be explained in detail later.
  • the numerical evaluation exceeds the first preset threshold, it means that the training of robot skills and applications is completed. After the training is completed, the robot skills and applications will be synchronously tested on the physical robot.
  • the chemical evaluation exceeds the second preset threshold, the service corresponding to the robot skill and application will be released and put into online operation. If the numerical evaluation of the robot service that has been trained and put into online operation does not exceed the second preset threshold in subsequent evaluations, retraining and updating of the robot's skills and applications will be triggered.
  • the core module of digital twin operation is an environment service that runs continuously online in the cloud.
  • the digital twin is a 1:1 geometric appearance modeling of the geometric shape, structure and appearance of the physical robot, and the simulation of each movable intelligent joint of the physical robot (including but not limited to motors, accelerators, damping parameters, etc. ) can support the design model update, 3D reconstruction and other methods to realize the physical model.
  • physical simulation of the sensors of the physical robot is required.
  • the physical simulation includes physical gravity simulation, physical collision simulation, and the application of physical materials to express its own physical properties such as friction and light reflection. The above physical properties will affect the behavior of the robot in a specific environment.
  • the digital twin world is a 3D semantic map data service that is a virtual mirror of the physical world where the physical robot lives. It is a digital representation of the 3D environmental semantics that the physical robot can recognize and understand in various application scenarios, helping the robot to perceive and recognize the physical world.
  • the changes in the environment acquired by various sensors of the physical robot will also be synchronized to the digital twin world.
  • the digital twin world is also used to train various digital twins in the background (offline) to ensure that the physical robot has the best operating strategies and behaviors and actions when it runs online.
  • the artificially enhanced machine intelligence module supports the digital twin operation core module for training and online operation of robot skills and applications through language AI, visual AI, motion AI, multimodal AI, and artificially enhanced AI.
  • language AI includes automatic speech recognition, natural language understanding and speech synthesis
  • visual AI includes but is not limited to face recognition, human body recognition, portrait recognition, various object recognition, environmental scene recognition and other visual perception
  • motion AI includes external force sensing perception , autonomous movement and navigation, and various body movements, etc.
  • multimodal AI refers to the ability to combine the above-mentioned language, visual and external perception input, voice output, motion output and other factors in addition to the above single AI capabilities.
  • Artificially enhanced AI is used to provide positive incentive input for system reinforcement learning through manual intervention. During manual intervention, the language AI, visual AI, motion AI, and multi-modal AI are all online and are a kind of human.
  • artificial enhancement ensures the uncertainty caused by the inexplicability of current AI, and fundamentally guarantees security and robustness.
  • the above-mentioned one-way AI capabilities and multi-modal AI capabilities are used to support the digital twin in the core module of digital twin operation in the form of API (Application Programming Interface) or SDK (Software Development Kit, software development kit). and digital twin world, and support robot business application management.
  • API Application Programming Interface
  • SDK Software Development Kit, software development kit
  • natural language AI includes AI capabilities such as speech recognition, natural language understanding, dialogue knowledge base, knowledge graphs in industry fields, and speech synthesis.
  • Vision AI supports AI capabilities such as face recognition, human body recognition, object recognition, and visual positioning and navigation.
  • Motion AI supports robot autonomous positioning, navigation and movement, autonomous obstacle avoidance, robot self-balancing, robot vision-guided grasping and beckoning and other common actions, dancing, and robot behavior and movement training and generation capabilities.
  • the 3D environment semantic map function refers to the recognition and cognition of the application scene, 2D/3D object recognition, 3D pose in the world coordinate system, 3D reconstruction and semantic segmentation of the physical environment, and semantic description of the scene. Three-dimensional semantic cognitive ability.
  • the above AI capabilities can be achieved by employing various deep learning algorithms, machine learning algorithms, deep reinforcement learning algorithms, kinematic planning algorithms, and the like.
  • the deep learning algorithms may include Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Deep Neural Networks (DNN), FastR-CNN ), YOLO (You Only Look Once), Single Shot MultiBox Detector (SSD), Long Short-Term Memory Network (LSTM, Long Short-Term Memory), Deep Bidirectional Language Model (Embeddings from LanguageModels, ELMO), Bidirectional Encoder Representation from Transformers, Transformers-based Bidirectional Encoder Representation from Transformers (BERT), Generative Pre-Training (GPT), etc.
  • CNN Convolutional Neural Networks
  • RNN Recurrent Neural Networks
  • DNN Deep Neural Networks
  • FastR-CNN FastR-CNN
  • YOLO You Only Look Once
  • SSD Single Shot MultiBox Detector
  • LSTM Long Short-Term Memory Network
  • the artificially augmented AI is further used to: if there is an abnormal situation of the robot service, receive the operation of the digital twin from the service trainer within the control authority.
  • receive the operation of the digital twin from the service trainer within the control authority. The specific operation of the service trainer will be described in further detail later.
  • the robot big data module is used to store and analyze multi-source data, and feed the analyzed multi-source data to the digital twin operation core module for training and online operation of robot skills and applications. Further, the robot big data module is also used to store and analyze one or more of system operation and service log data, user data, artificially enhanced operation data and system performance data.
  • the user data refers to user identity information, multi-dimensional portrait attributes, and the like.
  • the artificially enhanced operation data refers to the recorded data recorded by the system when the service is manually operated, or the identification data generated during the manual operation.
  • the above-mentioned data analysis mainly refers to that for the above-mentioned stored data, the robot big data module further performs data extraction, data conversion, data loading, data classification, data labeling, anomaly detection and data cleaning on the stored data, and obtains the processed data.
  • Data real-time analysis and offline analysis of the processed data, and numerical evaluation of the operation of each robot service in the cloud robot system.
  • the numerical evaluation is used to trigger the core module of digital twin operation to retrain and update robot skills and applications.
  • data extraction, data transformation and data loading are also called ETL (Extract-Transform-Load), which refers to the process of extracting, transforming, and loading data from the source to the destination.
  • the numerical evaluation includes the actual recognition rate of AI algorithms and models, the satisfaction of man-machine dialogue responses, service response time, and the efficiency and stability of the robot business behavior blueprint. For example, when an entity robot performs services, based on the feedback of various data, it is judged that the behavior and actions of the entity robot are not like humans, and the actual recognition rate of the evaluation AI algorithm and model is not high (for example, the specific recognition rate value is calculated), The numerical evaluation results are fed back to the core module of digital twin operation to retrain and update robot skills and applications. In addition, the numerical evaluation is also used to evaluate the business behavior blueprint of the robot to determine whether the behavior logic represented by the robot business behavior blueprint is optimal.
  • the analysis of the data also includes the generation of user portraits and related knowledge according to the human-computer dialogue during human-computer interaction.
  • the feedback information for each robot service of the cloud robot system is formed, which triggers the retraining and updating of the robot skills and applications (especially its algorithms and models), so that the entire cloud robot system constitutes a complete A closed-loop system that is continuously optimized.
  • the robot open platform is used to provide a robot service development interface for developers to develop robot services.
  • robot service is an application based on digital twin development and training.
  • Robot service development includes digital twin development, robot behavior and action editing, and robot business behavior blueprint editing.
  • the robot business behavior blueprint represents the behavior logic of the robot.
  • the robot open platform has various robot development kits, such as integrated development environment, digital twin model building, behavior and action editor, blueprint editor, behavior task orchestration, etc., and provides robot service development and training for digital twins.
  • robot development kits such as integrated development environment, digital twin model building, behavior and action editor, blueprint editor, behavior task orchestration, etc.
  • the robot business application service platform is used to configure the physical robot and provide the download of the robot service.
  • the configuration of the physical robot is mainly to configure the digital twin model of the physical robot, the robot name, role, personality, application scenarios and dialogues, language parameters, network parameters, the list of user faces to be recognized, and one of the corresponding robot services. one or more, wherein the application scenarios are configured according to the three-dimensional environment semantic map.
  • the robot business application service platform is also used to publish the corresponding services for the robot skills and applications that have been trained for the physical robots to download and apply.
  • roles include receptionist, patrolman, deliveryman, etc., which can be configured according to actual application requirements
  • personalities include fast, slow, etc.
  • Service places can obtain 3D environmental semantic map scenarios in advance according to the application scenarios of the robot, such as business halls, communities , hotels, etc.
  • the robot business application service platform includes a robot business management module and a robot application market.
  • the robot business management module is used to implement the above configuration function, that is, for robot business application scenarios in various industries, and the robot business management module Configure related attributes for roles and corresponding robot services, among which robot services mainly include but are not limited to the following robot services, such as welcome reception, mobile grabbing, security patrols, and delivery skills.
  • the robot application market is mainly used to support directly downloading robot services from it to the core module of digital twin operation for trial operation or operation.
  • the physical robot includes a robot body and a robot control module.
  • the robot body includes at least one or more intelligent flexible actuators, various sensors and local computing units.
  • the intelligent flexible actuator highly integrates high torque density servo motor, motor driver, high-precision encoder, and precision reducer into a small and flexible whole for robot joints.
  • Various sensors include but are not limited to: Lidar, Ultrasonic Radar, Millimeter Radar, 3D Depth Vision Camera, RGB Camera, Simultaneous Localization And Mapping (SLAM) camera, Inertial Measurement Unit, IMU), air detector, temperature and humidity detector, etc.
  • the local computing unit is mainly used to realize preprocessing, motion control and execution, such as preprocessing, perceptual detection and recognition of environmental data collected by various sensors (that is, multi-source data), and at the same time, motion control of robot joints to complete the robot
  • preprocessing perceptual detection and recognition of environmental data collected by various sensors (that is, multi-source data)
  • perceptual detection and recognition of environmental data collected by various sensors that is, multi-source data
  • motion control of robot joints to complete the robot
  • the executive function of behaviors and actions such as movement, body movements, etc.
  • the robot control module is located in the physical robot, and the communication between the robot control module and the cloud server is carried out through a dedicated network.
  • the robot control module is further configured to: send the current environment change information obtained by the sensor of the physical robot and the behavior and action change information of the physical robot to the core module of the digital twin operation, so that the digital twin can keep the physical robot and the digital twin. Behavior and actions are synchronized.
  • the robot control module also sends the multi-source data preprocessed by the local computing unit to the robot access and data exchange module for data exchange, fusion and distribution by the robot access and data exchange module.
  • the robot control module can also download the published robot service from the robot application market, so that the physical robot can execute the robot service.
  • the robot control module includes an interrelated communication unit and a computational processing unit.
  • the communication unit supports WiFi, 4G, 5G, Ethernet and other network communication methods, connects to the cloud server through a dedicated network, and forms a secure connection channel and network isolation domain with the cloud server.
  • the robot control module is also connected with the robot body for controlling and transmitting data to the robot body. All behaviors and actions of the digital twin on the robot control module will be executed in full synchronization with the robot body. On-screen support for the robot control module displays the behavior and movements of the digital twin.
  • game engine technology is employed in the digital twin runtime core module.
  • the digital twin running core module also includes a first game engine for loading the digital twin and the digital twin world, running and updating the digital twin world, and running the behaviors and actions of the digital twin.
  • the robot control module also includes a second game engine for running the digital twin. The first game engine and the second game engine are used to jointly drive the behaviors and actions of the digital twin and the digital twin to execute synchronously.
  • a game engine is a complex system composed of multiple subsystems, including modeling, animation (movement of a physical robot mapped by a digital twin), lighting, special effects, physics systems, collision detection, file management, network features, editing tools, and plug-ins Wait.
  • one is the character's skeletal animation motion system, which uses the built-in bones to drive objects to move; the other is the model animation motion system, which is directly deformed on the basis of the model.
  • Light and shadow refers to the way the light source in the application scene affects the people and objects in it.
  • Basic optical principles such as refraction and reflection, as well as advanced effects such as dynamic light sources and color light sources, are implemented through the game engine.
  • the game engine provides a physics system to make the movement of objects follow fixed physical laws. For example, when a character jumps, the default gravity value of the system will determine how high he can jump, how fast he falls, and the flight path of the object. , the way the robot moves bumps are also determined by the physical system.
  • Collision detection is a core part of the physics system and can detect the physical edges of objects in the digital twin world.
  • collision detection prevents them from passing through each other, thus ensuring that when the object hits a wall, it does not pass through the wall and does not knock over the wall, because the collision detection will be based on Properties between objects and walls determine their position and interaction.
  • Rendering means when the target 3D model is completed, the material maps will be assigned to the model according to different faces, which is equivalent to covering the bones of the robot physical model, and finally the model, animation, light and shadow, special effects, etc. are rendered through the rendering engine. All effects are calculated in real time and displayed on the screen.
  • the digital twin and the digital twin world have a higher degree of running simulation, and can more realistically simulate the environment of the physical robot and the application scenario in which it is located.
  • the behaviors and actions of these robot digital twins will be synchronized to the digital twin copy on the robot control module through the robot safety private network, and the digital twin copy will run and output commands to control the physical robot synchronously.
  • the physical robot obtains the current actual environment changes and the robot's own state through the sensor will also be reported to the cloud digital twin, so that the cloud digital twin and the physical robot can keep the behavior and status synchronized.
  • FIG. 4 is a schematic flowchart of the operation of a robot service provided by an embodiment of the present invention. As shown in Figure 4, it includes the following steps:
  • Step 401 Configuration management and monitoring.
  • the user that is, the administrator
  • the robot service configures the physical robot and monitors the designated robot service by logging into the robot business management module. And can choose and configure the robot service from the robot application market.
  • Step 402 Trial operation and download of the robot business behavior blueprint and behaviors and actions.
  • the robot business behavior blueprint and all the used behavior and action data are used for digital twin training on the core module of digital twin operation.
  • the threshold is preset, it is determined that the training of robot skills and applications is completed. If it is determined that the training of robot skills and applications is completed, the trained robot skills and applications are loaded into the robot control module for synchronous trial operation. When the trial operation process and the completion of the target task, the robot service involving the robot business behavior blueprint and all used behavior and action data can be released, and the released blueprint and behavior and action data can be downloaded to the robot control module superior.
  • Step 403 Synchronize the behavior and actions of the core module of the digital twin and the physical robot.
  • the digital twin runs the first game engine of the core module and the second game engine of the robot control module to drive the behavior of the digital twin and the digital twin to execute synchronously.
  • Step 404 Human-robot interaction and physical robot interaction with the environment.
  • the logic operation of the physical robot based on the business behavior blueprint of the robot is driven by the instructions run by the digital twin copy on the robot control module, and the physical robot performs multi-modal interaction with the user and the physical environment of the current application scenario.
  • Step 405 Entity robot event and status feedback.
  • the user's voice input, events caused by changes in the current environment, and state changes of the physical robot are received through various sensors of the physical robot. These emerging events and states are fed back into the digital twin and digital twin world of the cloud server's digital twin running core modules.
  • Step 406 Synchronize the behaviors and actions of the core module of the digital twin operation based on the feedback and the physical robot.
  • the cloud server based on the intelligent decision-making response of time-serialized events and state changes, affects the digital twin through the digital twin running the core module, and synchronizes behaviors and actions to the digital twin copy of the robot control module, thereby synchronizing the control entity
  • the bot completes the response behavior.
  • FIG. 5 is a schematic flowchart of a robot service development provided by an embodiment of the present invention. As shown in Figure 5, it includes the following steps:
  • Step 501 The developer registers and logs in on the robot open platform.
  • the developer refers to the developer of the robot service.
  • Step 502 Create a number of specified physical robots through the integrated development environment of the robot development kit
  • Step 503 Based on the behavior and action editor of the robot development kit, open the
  • Step 504 Based on the blueprint editor of the robot development kit, develop a robot business behavior blueprint.
  • Step 505 The digital twin performs several simulation trainings of robot skills and applications in the digital twin world, and continues trial and error until the running process of the training and the numerical evaluation of the completion of the target task exceed the first preset threshold.
  • Step 506 Package the robot service and blueprint through the integrated development environment of the robot development kit.
  • Step 507 Push or load the digital twin and the robot business behavior blueprint to the robot control module for synchronous trial operation.
  • Step 508 When the numerical evaluation of the trial operation process and the completion of the target task exceeds the second preset threshold, submit the robot service for management review.
  • Management review can be performed by reviewers of the cloud server. In particular, it can be carried out by reviewers of the robotics application market.
  • Step 509 If the robot service passes the review, it will be published in the robot application market of the robot business application service platform.
  • robot services can be downloaded and used by physical robots.
  • FIG. 6 is a schematic flowchart of an artificially enhanced AI operation provided by an embodiment of the present invention. As shown in Figure 6, it includes the following steps:
  • Step 601 the robot service client intervenes to control.
  • the robot's service trainer visually monitors the physical robot currently in service, including monitoring its digital twin and digital twin world.
  • an abnormal service situation such as loss of positioning, service timeout, body overtemperature, etc.
  • manual intervention and control can be performed through multi-modal fusion AI and artificial enhancement AI, mainly including: But not limited to: Directly operate the digital twin through voice input devices, keyboard and mouse, VR glasses and other devices.
  • Step 602 Human intelligence and artificial intelligence manipulation.
  • the manual intervention of the service trainer will automatically cover the current operation functions on the digital twin driven by the robot business behavior blueprint and multi-modal fusion AI. Therefore, during the manual intervention operation, the cloud robot system will determine whether the current execution behavior and action can be replaced manually. If it cannot be replaced manually, a prompt message will be given, and the service trainer will determine whether to intervene in the operation according to the prompt message. When the manual operation is completed, the cloud robot system will also enter the processing logic of the original blueprint or the response process of multi-modal fusion AI, and the system can also give corresponding prompt information.
  • Step 603 Manual intervention operation storage, behavior synchronization operation and events, and status feedback.
  • the operation based on manual intervention will also run the core module through the digital twin to keep the digital twin and the digital twin copy on the robot control module running synchronously.
  • the cloud robot system identifies and records the manual intervention operation and connects it to the robot big data module.
  • the human-computer interaction between the physical robot and the user, as well as events and state changes caused by environmental changes will also be fed back to the digital twin synchronously, and visual feedback will be provided through the client of the service trainer to affect the current operation behavior of manual intervention.
  • Step 604 Data analysis triggers retraining and optimization to form empirical knowledge and data accumulation.
  • the numerical evaluation of the current manual intervention operation mainly includes: improved evaluation of language AI, visual AI, motion AI, and multi-modal fusion AI, triggering retraining of corresponding algorithms and models to improve algorithm and model capabilities. It is also possible to optimize and improve the blueprint logic and process for completing service functions through big data analysis. Based on historical statistics, big data analysis can also obtain empirical knowledge, data, and routine habits of robot behavior and movements.
  • Step 605 Update the robot business behavior blueprint, behavior and action data.
  • the digital twin and the digital twin copy of the robot control module will be synchronously updated, and will also drive the online service of the physical robot.
  • Step 606 Respond to real-time events and state changes based on the update.
  • FIG. 7 is a schematic structural diagram of a cloud server provided by an embodiment of the present invention.
  • the cloud server 10 is used to control the physical robot, which includes a robot access and data exchange module 11, a knowledge and data intelligence module 12, an artificially enhanced machine intelligence module 13, a digital twin operation core module 14 and a robot large
  • the data module 15, the cloud server 10 and the physical robot communicate through a dedicated network.
  • the robot access and data exchange module 11 is used to perform robot service process registration and robot access authentication, as well as receive multi-source data sent by the entity robot, and perform data exchange, fusion and distribution.
  • the knowledge and data intelligence module 12 is used to provide a multi-domain knowledge map, a robot behavior library and a three-dimensional environment semantic map for robot services.
  • the digital twin operation core module 14 includes a digital twin world and a digital twin, wherein the digital twin world is constructed based on a three-dimensional environment semantic map, and the digital twin is a physical model with the same physical attributes as the physical robot; the digital twin is used in the digital twin world. Based on the multi-domain knowledge graph of robot service, robot behavior and action library and multi-source data, the training and online operation of robot skills and applications can be performed in order to synchronously control physical robots to perform robot skills and applications.
  • the artificially enhanced machine intelligence module supports the digital twin operation core module for training and online operation of robot skills and applications through language AI, visual AI, motion AI, multimodal AI and artificially enhanced AI; the robot big data module 15 It is used to store and analyze multi-source data, and feed the analyzed multi-source data to the digital twin operation core module 14 for training and online operation of robot skills and applications.
  • the specific structure and function of the cloud server 10 are the same as those of the cloud server 10 in the aforementioned cloud robot system 100 , and reference may be made to the foregoing description, which will not be repeated here.
  • FIG. 8 is a schematic structural diagram of a robot control module provided by an embodiment of the present invention.
  • the robot control module 21 communicates with the cloud server through a dedicated network.
  • the robot control module 21 includes a digital twin copy 211, which is a copy of the digital twin on the cloud server; the digital twin 211 synchronously controls the physical robot according to the robot skills and applications performed by the digital twin Perform robotics skills and applications.
  • the robot control module 21 is also used to send multi-source data to the cloud server, so that the digital twin can perform training and application of robot skills and applications based on the multi-domain knowledge graph, robot behavior library and multi-source data of the robot service in the digital twin world. Runs online to synchronously control the physical robot to perform robot skills and applications through the digital twin 211 of the robot control module 21 .
  • the specific structure and function of the robot control module 21 are the same as those of the robot control module 21 in the aforementioned cloud robot system 100 , and reference may be made to the foregoing description, which will not be repeated here.
  • FIG. 9 is a schematic structural diagram of a robot provided by an embodiment of the present invention.
  • the robot 40 includes the robot control module 21 in the embodiment shown in FIG. 8 .
  • the specific structure and function of the robot control module 21 are the same as those of the robot control module 21 in the aforementioned cloud robot system 100 , and reference may be made to the foregoing description, which will not be repeated here.
  • modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment.
  • the modules or units or components in the embodiments may be combined into one module or unit or component, and they may be divided into multiple sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method so disclosed may be employed in any combination, unless at least some of such features and/or procedures or elements are mutually exclusive. All processes or units of equipment are combined.
  • Each feature disclosed in this specification may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Manipulator (AREA)

Abstract

一种云端机器人系统、云服务器、机器人控制模块和机器人。云端机器人系统包括云服务器和机器人控制模块，云服务器包括机器人接入与数据交换模块、知识和数据智能模块、人工增强机器智能模块、数字孪生运行核心模块和机器人大数据模块，机器人控制模块位于实体机器人，机器人控制模块和云服务器之间通过专用网络进行通信。

Description

云端机器人系统、云服务器、机器人控制模块和机器人
相关申请的交叉引用
本公开要求在2020年12月1日提交中国专利局、申请号为202011386136.2、名称为“云端机器人系统、云服务器、机器人控制模块和机器人”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本发明实施例涉及机器人技术领域，具体涉及一种云端机器人系统、云服务器、机器人控制模块和机器人。
背景技术
目前，在机器人的实现方式中，云端机器人得到了越来越广泛的应用。其中，在危险的、肮脏的、重复性的以及实现较为困难的一些应用场景中，对云端机器人的要求也更高，产生了功能上可以替代人类的智能机器人的市场需求。
现有技术中的云端机器人实现方案不够智能，尚不能满足市场需求。如何构建一种更为智能的云端机器人系统架构，是目前亟待解决的问题。
发明内容
鉴于上述问题，本发明实施例提供了一种云端机器人系统、云服务器、机器人控制模块和机器人，用于解决现有技术中存在的云端机器人实现方案不够智能的问题。
根据本发明实施例的一个方面，提供了一种云端机器人系统，包括云服务器和机器人控制模块，所述云服务器包括机器人接入与数据交换模块、知识和数据智能模块、人工增强机器智能模块、数字孪生运行核心模块和机器人大数据模块，所述机器人控制模块位于实体机器人，所述机器人控制模块和所述云服务器之间通过专用网络进行通信；其中，
所述机器人接入与数据交换模块用于进行机器人服务进程注册和机器人接入认证，以及接收所述机器人控制模块发送的多源数据，并进行数据的交换、融合和分发；
所述知识和数据智能模块用于提供机器人服务的多领域知识图谱、机器人行为动作库和三维环境语义地图；
所述数字孪生运行核心模块包括数字孪生世界和数字孪生体，所述机器人控制模块包括数字孪生副本，其中，所述数字孪生世界基于所述三维环境语义地图构建，所述数字孪生体为与所述实体机器人物理属性相同的物理模型，所述数字孪生副本为运行在所述云服务器上的所述数字孪生体的副本；所述数字孪生体用于在所述数字孪生世界中基于所述机器人服务的多领域知识图谱、所述机器人行为动作库和所述多源数据执行机器人技能和应用的训练和在线运行，所述数字孪生副本根据所述数字孪生体执行的机器人技能和应用，同步控制所述实体机器人执行所述机器人技能和应用；
所述人工增强机器智能模块通过语言AI、视觉AI、运动AI、多模态AI和人工增强AI，支持所述数字孪生运行核心模块进行机器人技能和应用的训练和在线运行；
所述机器人大数据模块用于存储和分析所述多源数据，将分析后的多源数据反馈给所述数字孪生运行核心模块用于所述机器人技能和应用的训练和在线运行；
所述机器人控制模块还用于向所述机器人接入与数据交换模块发送多源数据。
在一种可选的方式中，所述云服务器还包括机器人业务应用服务平台，用于对所述实体机器人进行配置，以及提供机器人服务的下载。
在一种可选的方式中，所述对所述实体机器人进行配置，包括：
配置所述实体机器人的数字孪生体模型、机器人名称、角色、性格、应用场景及对话、语言参数、网络参数、待识别的用户人脸清单和对应的机器人服务中的一种或多种，其中所述应用场景根据所述三维环境语义地图配置。
在一种可选的方式中，所述云服务器还包括机器人开放平台，用于提供机器人服务开发接口以供开发者进行所述机器人服务开发。
在一种可选的方式中，所述机器人服务为基于所述数字孪生体开发和训练的应用，所述机器人服务开发包括数字孪生体开发、机器人行为和动作编辑和机器人业务行为蓝图编辑。
在一种可选的方式中，所述数字孪生运行核心模块还用于：在所述数字孪生体在所述数字孪生世界中执行所述机器人技能和应用的训练过程中，若所述数字孪生体执行所述机器人技能和应用的完成情况的数值化评价超过第一预设阈值时，确定所述机器人技能和应用的训练完成，若确定所述机器人技能和应用的训练完成，将训练完成的机器人技能和应用加载到所述机器人控制模块进行同步试运行；
所述机器人控制模块还用于：加载并同步试运行训练完成的机器人技能和应用；
所述数字孪生运行核心模块还用于：若所述机器人控制模块试运行训练完成的机器人技能和应用的完成情况的数值化评价超过第二预设阈值时，将所述机器人技能和应用所对应的服务发布至所述机器人业务应用服务平台。
在一种可选的方式中，所述数字孪生运行核心模块还包括第一游戏引擎，用于加载所述数字孪生体和所述数字孪生世界，运行和更新所述数字孪生世界，以及运行所述数字孪生体的行为和动作；
所述机器人控制模块还包括第二游戏引擎，用于运行所述数字孪生副本；
所述第一游戏引擎和所述第二游戏引擎用于共同驱动所述数字孪生体和所述数字孪生副本的行为和动作同步执行。
在一种可选的方式中，所述数字孪生运行核心模块进一步用于：将所述数字孪生体的行为和动作通过所述专用网络同步给所述机器人控制模块上的数字孪生副本；
所述数字孪生副本根据所述数字孪生体的行为和动作，同步控制所述实体机器人执行所述行为和动作。
在一种可选的方式中，所述机器人控制模块还用于：将所述实体机器人的传感器获取的当前环境变化信息和实体机器人自身行为和动作变化信息发送给所述数字孪生运行核心模块，以使所述数字孪生体与所述实体机器人保持行为和动作同步。
在一种可选的方式中，所述多领域知识图谱包括与机器人服务相关的实体之间关系的语义网络，所述语义网络包括信息和知识，所述信息用于描述外部客观事实，所述知识是外部客观规律的归纳和总结；
所述机器人行为动作库包括机器人通过模仿学习到的人类行为和动作；
所述三维环境语义地图是实体机器人所处的三维环境的语义数据，所述三维环境语义地图通过如下方式获得：将所述多源数据进行融合获得三维环境数据，基于所述三维环境数据通过语义分割进行地图建模，构建所述三维环境语义地图。
在一种可选的方式中，所述构建所述三维环境语义地图，包括：
结合基于深度学习的应用场景识别、物体检测识别、几何模型表示、空间语义关系和语义标注，构建多语义融合的三维环境语义地图。
在一种可选的方式中，所述语言AI包括自动语音识别、自然语言理解和语音合成；所述视觉AI包括人脸识别、人体识别、人像识别、物体识别和环境场景识别；所述运动AI包括外力传感感知、自主移动和导航、肢体动作；所述多模态AI是指具有所述语言 AI、视觉AI和运动AI的能力，以及同时具有多因素结合输出的能力，其中所述多因素结合输出包括所述语言AI、视觉AI和运动AI的输入以及语音输出、运动输出；
所述人工增强AI用于：通过人工介入操作为系统强化学习提供正向激励输入，在人工介入操作时，所述语言AI、视觉AI、运动AI和多模态AI均是在线运行状态。
在一种可选的方式中，所述人工增强AI进一步用于：若出现机器人服务异常情况，接收所述服务训练师在其操控权限内对所述数字孪生体的操作。
在一种可选的方式中，所述机器人大数据模块还用于存储和分析系统运行和服务日志数据、用户数据、人工增强的操作数据和系统性能数据中的一种或多种。
在一种可选的方式中，所述多源数据包括通过所述实体机器人的传感器获取的音视频数据、三维环境点云数据、机器人行为和动作数据和多模态交互数据中的一种或多种。
在一种可选的方式中，所述机器人大数据模块进一步用于：
对存储的数据进行数据抽取、数据转换、数据装载、数据分类、数据标注、异常检测和数据清洗，得到处理后的数据；
对所述处理后的数据进行实时分析和离线分析，对所述云端机器人系统中各个所述机器人技能和应用的运行进行数值化评价，所述数值化评价用于确定所述机器人技能和应用的训练是否完成，以及触发所述数字孪生运行核心模块对所述机器人技能和应用进行重新训练和更新。
在一种可选的方式中，所述数值化评价包括AI算法和模型的实际识别率、人机对话回复的满意度、服务响应时长和机器人业务行为蓝图的高效性和稳定性；
所述机器人大数据模块进一步用于：对数值化评价的目标结论进行分类，形成先验知识、相关业务和相关数据。
根据本发明实施例的另一方面，提供了一种云服务器，用于控制实体机器人，包括机器人接入与数据交换模块、知识和数据智能模块、人工增强机器智能模块、数字孪生运行核心模块和机器人大数据模块，所述云服务器和所述实体机器人之间通过专用网络进行通信；其中，
所述机器人接入与数据交换模块用于进行机器人服务进程注册和机器人接入认证，以及接收所述实体机器人发送的多源数据，并进行数据交换、融合和分发；
所述知识和数据智能模块用于提供机器人服务的多领域知识图谱、机器人行为动作库和三维环境语义地图；
所述数字孪生运行核心模块包括数字孪生世界和数字孪生体，其中，所述数字孪生世界基于所述三维环境语义地图构建，所述数字孪生体为与所述实体机器人物理属性相同的物理模型；所述数字孪生体用于在所述数字孪生世界中基于所述机器人服务的多领域知识图谱、所述机器人行为动作库和所述多源数据执行机器人技能和应用的训练和在线运行，以同步控制所述实体机器人执行所述机器人技能和应用务；
所述人工增强机器智能模块通过语言AI、视觉AI、运动AI、多模态AI和人工增强AI，支持所述数字孪生运行核心模块进行机器人技能和应用的训练和在线运行；所述机器人大数据模块用于存储和分析所述多源数据，将分析后的多源数据反馈给所述数字孪生运行核心模块用于所述机器人技能和应用的训练和在线运行。
根据本发明实施例的另一方面，提供了一种机器人控制模块，所述机器人控制模块和云服务器之间通过专用网络进行通信；
所述机器人控制模块包括数字孪生副本，所述数字孪生副本为运行在所述云服务器上的数字孪生体的副本；所述数字孪生副本根据所述数字孪生体执行的机器人技能和应用，同步控制实体机器人执行所述机器人技能和应用；
所述机器人控制模块还用于向所述云服务器发送多源数据，以使所述数字孪生体在数字孪生世界中基于机器人服务的多领域知识图谱、机器人行为动作库和所述多源数据执行机器人技能和应用的训练和在线运行，以通过所述数字孪生副本同步控制所述实体机器人执行所述机器人技能和应用。
根据本发明实施例的另一方面，提供了一种机器人，所述机器人包括如上所述的机器人控制模块。
本发明实施例通过在云服务器构建数字孪生世界，在数字孪生世界中采用与实体机器人物理属性相同的数字孪生体进行机器人的训练和在线运行，通过对虚拟的数字孪生体的控制实现对实体机器人的同步控制，降低了对实体机器人完成业务应用的操控难度和成本，并采用人工增强AI引入人工操作作为系统强化学习的正向激励输入，支持所述数字孪生体对机器人技能和应用的训练和在线运行，同时实体机器人采集的多源数据也反馈回云端服务器，用于机器人技能和应用的训练和在线运行，实现动态闭环、持续进化的智能云端机器人系统。
上述说明仅是本发明实施例技术方案的概述，为了能够更清楚了解本发明实施例的技术手段，而可依照说明书的内容予以实施，并且为了让本发明实施例的上述和其它目 的、特征和优点能够更明显易懂，以下特举本发明的具体实施方式。
附图说明
附图仅用于示出实施方式，而并不认为是对本发明的限制。而且在整个附图中，用相同的参考符号表示相同的部件。在附图中：
图1是本发明实施例提供的云端机器人系统的应用示意图；
图2是本发明实施例提供的云端机器人系统的结构示意图；
图3是本发明实施例提供的云端机器人系统的框架示意图；
图4是本发明实施例提供的机器人服务运行的流程示意图；
图5是本发明实施例提供的机器人服务开发的流程示意图；
图6是本发明实施例提供的人工增强AI运行的流程示意图；
图7是本发明实施例提供的云服务器的结构示意图；
图8是本发明实施例提供的机器人控制模块的结构示意图；
图9是本发明实施例提供的机器人的结构示意图。
具体实施方式
下面将参照附图更详细地描述本发明的示例性实施例。虽然附图中显示了本发明的示例性实施例，然而应当理解，可以以各种形式实现本发明而不应被这里阐述的实施例所限制。
从机器人向人类的智能化发展角度来看，如果要制造一个跟人脑一样聪明的电子大脑，该电子大脑将会是巨大的，不可能在单体机器人上实现。此外，由于单体机器人所能接触的数据有限，无法完成需要有大数据训练的机器学习和深度学习。人工智能的深度学习必须由大量机器人提供数据，汇聚到云端，由云端的巨大的“机器大脑”来完成，这进一步说明机器人的部分感知和认知系统必须放在云端，这是智能机器人发展的必然方向。
基于此，本发明实施例提供了一种云端机器人系统，图1是本发明实施例提供的云端机器人系统的应用示意图。如图1所示，云服务器10和实体机器人20之间通过专用网络30进行通信。各项机器人服务在云服务器10训练完成，并由云服务器10控制实体机器人20执行训练好的各项机器人服务。其中，机器人服务是指在不同应用场景中执行预设动作，完成预设功能，例如迎宾接待、移动抓取、安防巡逻和配送等。服务需要由 应用组成，而若干技能组合成应用的逻辑。例如，会打乒乓球入削球、拉球等，属于技能，应用是指由实体机器人实际去打乒乓球。服务是指实体机器人可以提供打乒乓球陪练的服务。再如，实体机器人抓物品属于技能，实体机器人使用抓物品的技能可以完成送咖啡给人的应用，则实体机器人可以完成端茶倒水的接待服务。
图2是本发明实施例提供的云端机器人系统的结构示意图。如图2所示，该云端机器人系统100包括云服务器10和机器人控制模块21。云服务器10包括机器人接入与数据交换模块11、知识和数据智能模块12、人工增强机器智能模块13、数字孪生运行核心模块14和机器人大数据模块15。机器人控制模块21位于实体机器人20。机器人控制模块21和云服务器10之间通过专用网络30进行通信。通过专用网络30通信可保证机器人控制模块21和云服务器10之间的通信安全。
机器人接入与数据交换模块11用于进行机器人服务进程注册和机器人接入认证，以及接收机器人控制模块21发送的多源数据，并进行数据的交换、融合和分发。这里，服务进程是指程序的服务进程，也即微服务。
知识和数据智能模块12用于提供机器人服务的多领域知识图谱、机器人行为动作库和三维环境语义地图。
数字孪生运行核心模块14包括数字孪生世界和数字孪生体，机器人控制模块21包括数字孪生副本。数字孪生世界基于三维环境语义地图构建，数字孪生体为与实体机器人物理属性相同的物理模型，数字孪生副本为运行在所述云服务器上的数字孪生体的副本。数字孪生体用于在数字孪生世界中基于机器人服务的多领域知识图谱、机器人行为动作库和多源数据执行机器人技能和应用的训练和在线运行。数字孪生副本根据数字孪生体执行的机器人技能和应用，同步控制实体机器人执行机器人技能和应用。
人工增强机器智能模块13通过多模态AI和人工增强AI，支持数字孪生运行核心模块14进行机器人技能和应用的训练和在线运行。
机器人大数据模块15用于存储和分析多源数据，将分析后的多源数据反馈给数字孪生运行核心模块14用于机器人技能和应用的训练和在线运行。
机器人控制模块21还用于向机器人接入与数据交换模块11发送多源数据。
本发明实施例通过在云服务器构建数字孪生世界，在数字孪生世界中采用与实体机器人物理属性相同的数字孪生体进行机器人的训练和在线运行，通过对虚拟的数字孪生体的控制实现对实体机器人的同步控制，降低了对实体机器人完成业务应用的操控难度 和成本，并采用人工增强AI引入人工操作作为系统强化学习的正向激励输入，支持支持数字孪生体对机器人技能和应用的训练和在线运行，同时实体机器人采集的多源数据也反馈回云端服务器，用于机器人技能和应用的训练和在线运行，实现动态闭环、持续进化的智能云端机器人系统。
下面对云端机器人系统进行进一步说明。图3是本发明实施例提供的云端机器人系统的框架示意图。如图3所示，云端机器人系统采用“云(大脑)-网(神经)-端(身体)”的分布式计算架构。其中，“云”位于云端服务器，网是指专用网络，“端”位于实体机器人。
其中，云端大脑的人工增强机器智能模块将机器人语言AI(Artificial Intelligence，人工智能)能力、视觉AI能力、运动AI能力及环境认知能力等多模态融合的AI和人工增强AI进行有机融合，形成云端大脑的感知和认知能力，结合人类先验知识和数据智能，实现逻辑推理和智能决策等高级类人智能。通过数字孪生运行核心模块，让实体机器人的数字孪生体在虚拟的数字孪生世界中运行，并执行机器人技能和应用。数字孪生体的所有行为和动作会通过专用网络同步控制运行在实体机器人的机器人控制模块中的数字孪生副本，数字孪生副本执行的指令和数据序列将驱动实体机器人同步数字孪生体的所有行为和动作，完成该实体机器人在应用场景下的目标任务，从而让整个云端机器人系统更加智能，让用户以简单、安全可靠的方式使用实体机器人为各行业提供智能化服务。其中，一个或多个机器人的动作构成一个有意义的行为。
在本实施例中，云服务器除了包括上述实施例的机器人接入与数据交换模块、知识和数据智能模块、人工增强机器智能模块、数字孪生运行核心模块和机器人大数据模块以外，还包括机器人业务应用服务平台和机器人开放平台。
下面分别对实体机器人以及云端服务器的各个模块和平台的功能进行进一步详细说明。
1.机器人接入与数据交换模块
机器人接入与数据交换模块用于进行机器人服务进程注册和机器人接入认证，以及接收机器人控制模块发送的多源数据，并进行数据的交换、融合和分发。多源数据包括通过实体机器人的传感器获取的音视频数据、三维环境点云数据、机器人行为和动作数据和多模态交互数据中的一种或多种。机器人行为和动作数据主要为机器人关节运动帧数据。机器人的传感器获取数据的方式包括视觉、超声波、激光等多种方式。多模态交 互一般是指通过文字、语音、视觉、动作、环境等多种方式进行人机交互，充分模拟人与人之间的交互方式。
其中，数据的交换是指实体机器人上行的多源数据和从云服务器下行的数据(例如控制指令、语音数据、更新数据等)进行交换，例如将实体机器人上行的数据发送给机器人大数据模块，将云服务器下行的数据发送给实体机器人。
数据的分发是指将上行的数据在云服务器分发给一个或多个服务进行不同处理或分析。其中，服务是为其他程序提供各种功能调用的程序，例如运行于操作系统后台的程序、例程或进程。比如音视频数据，可以分发给视觉处理服务，同时也需要由用户和运营方进行可视监控。
数据的融合是指将不同来源或者不同结构的数据，经过处理形成标准数据接口或者采用标准的数据结构表示。例如，对于不同来源和结构的数据，例如音视频类非结构化数据、机器人行为和动作数据、三维环境语义地图数据等，通过增加数据描述形成标准数据接口，添加的描述包括但不限于接口标识、SessionID(会话ID)、接口类型、接口序列、版本、发起方、接收方、发起模块、接收模块、数据标识ID等。对于同类别、不同来源的数据，例如激光雷达点云数据和视觉相机深度点云数据，可以采用标准的数据结构表示。
2.知识和数据智能模块
知识和数据智能模块用于提供机器人服务的多领域知识图谱、机器人行为动作库和三维环境语义地图。机器人服务的多领域知识图谱和机器人行为动作库属于人类先验知识库。三维环境语义地图由实体机器人通过各种传感器感知和认知到的。
其中，多领域知识图谱包括与机器人服务相关的实体之间关系的语义网络，语义网络包括信息和知识，信息用于描述外部客观事实，知识是外部客观规律的归纳和总结。具体而言，多领域知识图谱包括但不限于面向各个垂直领域和行业的知识图谱以及通用自然语言理解的语料库，例如人物关系知识图谱、酒店行业知识图谱、地产行业知识图谱、中国历史知识谱图等。
机器人行为动作库包括机器人通过模仿学习到的人类行为和动作，包括但不限于实体机器人可通过模仿学习到的人类动作，例如抓取目标物体、自主定位导航、举手、弯腰、握手等。
三维环境语义地图是实体机器人所处的三维环境的语义数据。三维环境语义地图是 提供给实体机器人所处的三维环境的语义级数据服务，是用人类自然语言的方式来描述客观物理世界的环境及关系的，是实体机器人在各种应用场景下可认知理解的三维环境语义的数字化表示，帮助实体机器人感知和认知物理世界，并被用来训练虚拟的数字孪生体(也即数字孪生机器人，后文将详细阐述)。
在一些实施例中，三维环境语义地图可通过如下方式获得：将多源数据进行融合获得三维环境数据，基于三维环境数据通过语义分割进行地图建模，构建三维环境语义地图。其中，语义分割针对的是多特征融合的三维环境数据。
在一些实施例中，可通过如下方式构建三维环境语义地图：结合基于深度学习的应用场景识别、物体检测识别、几何模型表示、空间语义关系和语义标注，构建多语义融合的三维环境语义地图。三维环境语义地图通过数据库的方式存储和被访问。
3.数字孪生运行核心模块
数字孪生运行核心模块包括数字孪生世界和数字孪生体，相应的，位于实体机器人上的机器人控制模块包括数字孪生副本(后文将详细阐述)。其中，数字孪生世界基于三维环境语义地图构建，数字孪生体为与实体机器人物理属性相同的物理模型，数字孪生副本为运行在所述云服务器上的数字孪生体的副本。数字孪生体用于在数字孪生世界中基于机器人服务的多领域知识图谱、机器人行为动作库和多源数据执行机器人技能和应用的训练和在线运行，数字孪生副本根据数字孪生体执行的机器人技能和应用，同步控制实体机器人执行机器人技能和应用。
本发明实施例采用与实体机器人物理属性相同的数字孪生体，实现了机器人服务的低成本训练和试错过程。通过机器人多种传感器融合构建数字孪生世界，让机器人的数字孪生体在数字孪生世界中训练和实时在线运行，通过对虚拟的数字孪生体控制实现对实体机器人的同步控制，降低了对实体机器人完成机器人服务的操控要求。
具体的，数字孪生体的行为和动作通过专用网络同步给机器人控制模块上的数字孪生副本，数字孪生副本根据数字孪生体的行为和动作，同步控制实体机器人执行行为和动作。
对于机器人技能和应用的训练，可以预设一个第一预设阈值作为机器人技能和应用的训练完成情况的数值化评价阈值，并预设一个第二预设阈值作为机器人技能和应用的试运行完成情况的数值化评价阈值。在数字孪生体在数字孪生世界中执行机器人技能和应用的训练过程中，若数字孪生体执行机器人技能和应用的完成情况的数值化评价超过 第一预设阈值时，确定机器人技能和应用的训练完成，若确定机器人技能和应用的训练完成，将训练完成的机器人技能和应用加载到机器人控制模块进行同步试运行。然后，由机器人控制模块加载并同步试运行训练完成的机器人技能和应用。若机器人控制模块试运行训练完成的机器人技能和应用的完成情况的数值化评价超过第二预设阈值时，数字孪生运行核心模块将机器人技能和应用所对应的服务发布。
数值化评价包括AI算法和模型的实际识别率、人机对话回复的满意度、服务响应时长和机器人业务行为蓝图的高效性和稳定性。所述机器人大数据模块进一步用于：对数值化评价的目标结论进行分类，形成先验知识、相关业务和相关数据。其中，数值化评价可由机器人大数据模块完成，将在后文详细阐述。数值化评价超过第一预设阈值时代表机器人技能和应用训练完成，训练完成后的机器人技能和应用将在实体机器人上同步试运行，若试运行训练完成的机器人技能和应用的完成情况的数值化评价超过第二预设阈值时，将机器人技能和应用所对应的服务发布，可投入线上运行。若训练完成并投入线上运行的机器人服务在后续的评价中，数值化评价未超过第二预设阈值，则将触发对该机器人技能和应用的重新训练和更新。
可以理解的是，数字孪生运行核心模块是一个在云端持续在线运行的环境服务。其中，数字孪生体是通过对实体机器人的几何形状、结构及外观进行1:1几何外观建模，对实体机器人每个可活动的智能关节的仿真(包括但不限于电机、加速器、阻尼参数等)构建的虚拟机器人，可支持设计模型更新、三维重建等方法实现物理模型。此外，还需要对实体机器人的传感器进行物理仿真。其中，物理仿真包括物理重力仿真，物理碰撞仿真，应用物理材质来表达摩擦力、光反射等自身物理属性，上述物理属性将影响机器人在特定环境下的行为。
数字孪生世界是实体机器人所处物理世界的虚拟镜像的三维语义地图数据服务，是实体机器人在各种应用场景下可认知理解的三维环境语义的数字化表示，帮助机器人感知、认知物理世界，为云服务器的机器人实时在线运行服务提供可交互的数字语义化环境。同时，实体机器人的各种传感器获取环境的变化，也将同步到数字孪生世界。数字孪生世界也被用于后台(离线)训练各种数字孪生体，保证在实体机器人上线运行时具有最佳运行策略和行为和动作。
4.人工增强机器智能模块
人工增强机器智能模块通过语言AI、视觉AI、运动AI、多模态AI和人工增强AI， 支持数字孪生运行核心模块进行机器人技能和应用的训练和在线运行。
其中语言AI包括自动语音识别、自然语言理解和语音合成；视觉AI包括但不限于人脸识别、人体识别、人像识别以及各种物体识别、环境场景识别等视觉感知；运动AI包括外力传感感知、自主移动和导航以及各种肢体动作等；多模态AI是指在除上述单项AI能力以外，同时具有上述语言、视觉和外力感知的输入以及语音输出、运动输出等多因素结合输出的能力。人工增强AI用于通过人工介入操作为系统强化学习提供正向激励输入，在人工介入操作时，所述语言AI、视觉AI、运动AI和多模态AI均是在线运行状态，是一种人机协同的方式，同时人工增强保证了当前AI不可解释性造成的不确定性，从根本上保证安全性和健壮性。上述各种单向AI能力以及多模态AI能力均以API(Application Programming Interface，应用程序接口)或SDK(Software Development Kit，软件开发工具包)方式用于支撑数字孪生运行核心模块中数字孪生体和数字孪生世界，以及支撑机器人业务应用管理。通过人工增强AI使云端机器人系统逐渐向人类智能逼近，云服务器的重要决策可以设定需由人工做出决策，使云端大脑接受人类控制。
其中，自然语言AI包括语音识别、自然语言理解、对话知识库、行业领域的知识图谱以及语音合成等AI能力。视觉AI支持人脸识别、人体识别、物体识别、视觉定位导航等AI能力。运动AI支持机器人自主定位、导航移动、自主避障、机器人自平衡、机器人视觉引导的抓取和招手等常用动作、跳舞、机器人的行为和动作训练和生成等能力。三维环境语义地图功能是指应用场景的识别认知、2D/3D物体识别、世界坐标系下的三维位姿、物理环境3D重建和语义分割、对场景语义描述等能力，共同形成对物理环境的三维语义认知能力。
在一些实施例中，可通过采用各种深度学习算法、机器学习算法、深度强化学习算法、运动学规划算法等，实现上述AI能力。其中深度学习算法可包括卷积神经网络(Convolutional Neural Networks，CNN)、循环神经网络(Recurrent Neural Network，RNN)、深度神经网络(Deep Neural Networks，DNN)、基于快速区域卷积网络(FastR-CNN)、YOLO(You Only Look Once)、单级多框预测(Single Shot MultiBox Detector，SSD)、长短期记忆网络(LSTM，Long Short-Term Memory)、深层双向语言模型(Embeddings from LanguageModels，ELMO)、Bidirectional Encoder Representation from Transformers，基于Transformers的双向编码器表示(Bidirectional Encoder Representation fromTransformers，BERT)、生成式预训练(Generative Pre-Training，GPT)等。
在一些实施例中，人工增强AI进一步用于：若出现机器人服务异常情况，接收服务训练师在其操控权限内对数字孪生体的操作。服务训练师的具体操作将在后文进一步详细阐述。
5.机器人大数据模块
机器人大数据模块用于存储和分析多源数据，将分析后的多源数据反馈给数字孪生运行核心模块用于机器人技能和应用的训练和在线运行。进一步的，机器人大数据模块还用于存储和分析系统运行和服务日志数据、用户数据、人工增强的操作数据和系统性能数据中的一种或多种。其中，用户数据是指用户身份信息、多维度画像属性等。人工增强的操作数据是指系统记录的人工操作服务时的记录数据，或人工操作时产生的标识数据。
上述对数据的分析，主要是指对于存储的上述数据，机器人大数据模块进一步对存储的数据进行数据抽取、数据转换、数据装载、数据分类、数据标注、异常检测和数据清洗，得到处理后的数据；对处理后的数据进行实时分析和离线分析，对云端机器人系统中各个机器人服务的运行进行数值化评价，数值化评价用于触发数字孪生运行核心模块对机器人技能和应用进行重新训练和更新。其中，数据抽取、数据转换和数据装载也被称为ETL(Extract-Transform-Load)，是指将数据从来源端经过抽取(extract)、转换(transform)、加载(load)至目的端的过程。
具体的，所述数值化评价包括AI算法和模型的实际识别率、人机对话回复的满意度、服务响应时长和机器人业务行为蓝图的高效性和稳定性。例如，对于实体机器人执行服务时，基于各个数据的反馈，判断该实体机器人的行为和动作不像人类，则评价AI算法和模型的实际识别率不高(例如计算出具体的识别率数值)，将该数值化评价结果反馈给数字孪生运行核心模块，以对机器人技能和应用进行重新训练和更新。此外，数值化评价还用于评价该机器人业务行为蓝图，判断机器人业务行为蓝图所代表的行为逻辑是否是最优的，若不是则反馈给数字孪生运行核心模块进行蓝图的优化。
对数据的分析还包括根据人机交互时的人机对话产生用户画像和相关的知识。
通过上述对数据的分析，以此形成对云端机器人系统各个机器人服务的反馈信息，触发该机器人技能和应用(特别是其算法和模型)的重新训练和更新，从而使整个云端机器人系统构成一个完整的、持续优化的闭环系统。
6.机器人开放平台
机器人开放平台用于提供机器人服务开发接口以供开发者进行机器人服务开发。其中，机器人服务为基于数字孪生体开发和训练的应用，机器人服务开发包括数字孪生体开发、机器人行为和动作编辑和机器人业务行为蓝图编辑。机器人业务行为蓝图代表机器人的行为逻辑。
机器人开放平台上具有各种机器人开发套件，例如集成开发环境、数字孪生体模型构建、行为和动作编辑器、蓝图编辑器、行为任务编排等，并提供数字孪生体的机器人服务开发与训练。
通过机器人开放平台，让开发者可以基于实体机器人的数字孪生体进行机器人服务的开发和训练，以及机器人智能的持续进化。通过面向机器人服务开发者提供可视化编排的交互接口，来调用各种机器人、移动智能设备或者自动驾驶车等完成对应业务场景服务功能，让机器人服务开发变得简单和快捷。
7.机器人业务应用服务平台
机器人业务应用服务平台用于对实体机器人进行配置，以及提供机器人服务的下载。对实体机器人进行配置，主要是配置实体机器人的数字孪生体模型、机器人名称、角色、性格、应用场景及对话、语言参数、网络参数、待识别的用户人脸清单和对应的机器人服务中的一种或多种，其中应用场景根据三维环境语义地图配置。当机器人技能和应用在数字孪生运行核心模块训练完成后，机器人业务应用服务平台还用于发布完成训练的机器人技能和应用所对应的服务，以供实体机器人下载和应用。其中，角色包括接待员、巡逻员、配送员等，可根据实际应用需求配置，性格包括快、慢等，服务场所可预先根据该机器人的应用场景获取三维环境语义地图场景，如营业厅、社区、酒店等。
在一些实施例中，机器人业务应用服务平台包括机器人业务管理模块和机器人应用市场，机器人业务管理模块用于实现上述配置功能，也即面向各行业的机器人业务应用场景，通过机器人业务管理模块对机器人角色和对应的机器人服务等进行相关属性配置，其中机器人服务主要包括但不限于以下机器人服务，如迎宾接待、移动抓取、安防巡逻和配送技能等。机器人应用市场主要用于支持直接从其中下载机器人服务到数字孪生运行核心模块进行试运行或运行。
8.实体机器人
实体机器人包括机器人本体和机器人控制模块。
其中，机器人本体包括至少1个或多个智能柔性执行器、多种传感器和本地计算单 元。
智能柔性执行器将高扭矩密度伺服电机、电机驱动器、高精度编码器、精密减速器高度集成为一个小巧灵活整体，用于机器人关节。多种传感器包括但不限于：激光雷达、超声波雷达、毫米雷达、3D深度视觉相机，RGB相机，双目同步定位与地图构建(Simultaneous Localization And Mapping，SLAM)相机，惯性测量单元(Inertial Measurement Unit，IMU)，空气检测器、温湿度检测器等。
本地计算单元主要用于实现预处理、运动控制和执行，例如对多种传感器采集的环境数据(也即多源数据)进行预处理、感知检测和识别，同时对机器人关节进行运动控制，完成机器人移动、肢体动作等行为和动作的执行功能。
机器人控制模块位于实体机器人，机器人控制模块和云服务器之间通过专用网络进行通信。
在一些实施例中，机器人控制模块还用于：将实体机器人的传感器获取的当前环境变化信息和实体机器人自身行为和动作变化信息发送给数字孪生运行核心模块，以使数字孪生体与实体机器人保持行为和动作同步。
此外，机器人控制模块还将本地计算单元预处理后的多源数据发送给机器人接入与数据交换模块，供机器人接入与数据交换模块进行数据的交换、融合和分发。机器人控制模块也可以从机器人应用市场下载已发布的机器人服务，从而使实体机器人可以执行该机器人服务。
机器人控制模块包括相互关联的通信单元和计算处理单元。通信单元支持WiFi、4G、5G、Ethernet等网络通信方式，通过专用网络连接到云服务器，并形成与云服务器的安全连接通道和网络隔离域。机器人控制模块也与机器人本体连接，用于对机器人本体进行控制和数据传输。机器人控制模块上数字孪生副本的所有行为和动作将与机器人本体保持完全同步执行。机器人控制模块的屏幕上支持展示数字孪生副本的行为和动作。
在一些实施例中，在数字孪生运行核心模块中采用了游戏引擎技术。数字孪生运行核心模块还包括第一游戏引擎，用于加载数字孪生体和数字孪生世界，运行和更新数字孪生世界，以及运行数字孪生体的行为和动作。机器人控制模块还包括第二游戏引擎，用于运行数字孪生副本。第一游戏引擎和第二游戏引擎用于共同驱动数字孪生体和数字孪生副本的行为和动作同步执行。
游戏引擎是由多个子系统共同构成的复杂系统，包括建模、动画(数字孪生体映射 的实体机器人的运动)、光影、特效、物理系统、碰撞检测、文件管理、网络特性、编辑工具和插件等。
其中，动画包括两种：一种是角色的骨骼动画运动系统，用内置的骨骼带动物体产生运动；一种是模型动画运动系统，是在模型的基础上直接进行变形。
光影是指应用场景中的光源对处于其中的人和物的影响方式。折射、反射等基本光学原理以及动态光源、彩色光源等高级效果都通过游戏引擎实现。
游戏引擎提供物理系统，使物体的运动遵循固定的物理规律，例如，当角色跳起的时候，系统内定的重力值将决定他能跳多高，以及下落的速度有多快，物体的飞行轨迹、机器人移动的颠簸方式也都由物理系统决定。
碰撞探测是物理系统的核心部分，可以探测数字孪生世界中各物体的物理边缘。当两个3D物体撞在一起时，碰撞探测可以防止它们相互穿过，从而确保当物体撞在墙上的时候，不会穿墙而过，也不会把墙撞倒，因为碰撞探测会根据物体和墙之间的特性确定两者的位置和相互作用关系。通过上述物理系统和碰撞探测，可以实现对数字孪生体物理特性及动力控制的仿真模拟。
渲染是指：当目标3D模型制作完毕后，会按照不同的面把材质贴图赋予模型，这相当于为机器人物理模型的骨骼蒙上皮肤，最后再通过渲染引擎把模型、动画、光影、特效等所有效果实时计算出来并展示在屏幕上。
本发明实施例中，通过采用游戏引擎提供各种工具和运行的服务器环境，加载数字孪生体和数字孪生世界，并运行数字孪生体的所有行为和动作，以及运行和更新数字孪生世界，可以使数字孪生体和数字孪生世界的运行仿真度更高，能较为真实的模拟实体机器人和其所处的应用场景的环境。这些机器人数字孪生体的行为和动作将通过机器人安全专用网络同步给机器人控制模块上的数字孪生副本，由数字孪生副本运行输出指令同步控制实体机器人。同时实体机器人通过传感器获得当前实际环境变化以及机器人自身状态的也将被采用上报到云端数字孪生体上来，使得让云端数字孪生体与实体机器人保持行为和状态同步。
下面对云端机器人系统的运行机制和操作流程进行进一步详细说明。
1.机器人业务管理和机器人服务的运行机制和流程
图4是本发明实施例提供的机器人服务运行的流程示意图。如图4所示，包括如下步骤：
步骤401：配置管理和监控。
本步骤中，机器人服务的使用者(也即管理员)通过登录机器人业务管理模块进行实体机器人的配置和监控指定的机器人服务。并可以从机器人应用市场选择并配置机器人服务。
步骤402：机器人业务行为蓝图和行为和动作的试运行和下载。
本步骤中，将机器人业务行为蓝图和所有使用到的行为和动作数据在数字孪生运行核心模块上进行数字孪生体训练，若数字孪生体执行机器人技能和应用的完成情况的数值化评价超过第一预设阈值时，确定机器人技能和应用的训练完成，若确定机器人技能和应用的训练完成，将训练完成的机器人技能和应用加载到机器人控制模块进行同步试运行当试运行过程和目标任务完成情况的数值化评价超过第二预设阈值时，可以发布涉及该机器人业务行为蓝图和所有使用到的行为和动作数据的机器人服务，并可以将发布后的蓝图和行为和动作数据下载到机器人控制模块上。
步骤403：数字孪生运行核心模块和实体机器人的行为和动作同步。
本步骤中，通过数字孪生运行核心模块的第一游戏引擎和机器人控制模块的第二游戏引擎，驱动数字孪生体和数字孪生副本的行为同步执行。
步骤404：人机交互和实体机器人与环境交互。
本步骤中，通过机器人控制模块上的数字孪生副本运行的指令驱动实体机器人基于机器人业务行为蓝图的逻辑运行，实体机器人与当前应用场景的用户和物理环境进行多模态的交互。
步骤405：实体机器人事件和状态反馈。
本步骤中，通过实体机器人的各种传感器接收用户的语音输入，和当前环境变化的引起的事件，以及实体机器人的状态变化。这些出现的事件和状态被反馈到云服务器的数字孪生运行核心模块的数字孪生体和数字孪生世界中。
步骤406：基于反馈的数字孪生运行核心模块和实体机器人的行为和动作同步。
本步骤中，云服务器基于时间序列化的事件和状态变化的智能决策响应，通过数字孪生运行核心模块影响数字孪生体，并将行为和动作同步到机器人控制模块的数字孪生副本，进而同步控制实体机器人完成响应行为。
2.机器人服务开发流程
图5是本发明实施例提供的机器人服务开发的流程示意图。如图5所示，包括如下 步骤：
步骤501：开发者在机器人开放平台上进行注册并登陆。
其中，开发者是指机器人服务的开发者。
步骤502：通过机器人开发套件的集成开发环境创建指定实体机器人的数
字孪生模型。
步骤503：基于机器人开发套件的行为和动作编辑器，通过导入或编辑开
发数字孪生体的基础行为和动作。
开发数字孪生体的基础行为和动作时支持多次嵌套组合。
步骤504：基于机器人开发套件的蓝图编辑器，开发机器人业务行为蓝图。
开发机器人业务行为蓝图时支持导入子蓝图和多蓝图嵌套组合等。
步骤505：数字孪生体在数字孪生世界进行若干次机器人技能和应用的仿真训练不断试错，直到训练的运行过程和目标任务完成情况的数值化评价超过第一预设阈值。
步骤506：通过机器人开发套件的集成开发环境对机器人服务和蓝图进行打包。
步骤507：将数字孪生体、机器人业务行为蓝图推送或加载到机器人控制模块，进行同步试运行。
步骤508：当试运行过程和目标任务完成情况的数值化评价超过第二预设阈值时，将该机器人服务提交管理审核。
管理审核可由云端服务器的审核人员进行。特别是，可由机器人应用市场的审核人员进行。
步骤509：该机器人服务通过审核，则发布到机器人业务应用服务平台的机器人应用市场中。
在机器人应用市场，机器人服务可被实体机器人下载使用。
3.人工增强AI运行流程
图6是本发明实施例提供的人工增强AI运行的流程示意图。如图6所示，包括如下步骤：
步骤601：机器人服务客户端介入操控。
机器人的服务训练师通过服务客户端，在对当前处于服务状态中的实体机器人进行可视化监控，包括监控其数字孪生体和数字孪生世界。当出现服务异常情况时(如丢失定位、服务超时、本体过温等异常)，在当前服务训练师的操控权限以内，可以通过多模 态融合AI和人工增强AI进行人工的介入操控，主要包括但不限于：通过语音输入设备、键盘鼠标、VR眼镜等设备直接操作数字孪生体。
步骤602：人类智能和人工智能操控。
服务训练师的人工介入操作将自动覆盖当前由机器人业务行为蓝图以及多模态融合AI驱动的在数字孪生体上的操作功能。因此，在人工介入操作时，云端机器人系统将判断当前执行行为和动作是否可被人工替代，如不能被人工替代，则给出提示信息，由服务训练师根据该提示消息确定是否介入操作。当人工操作完成时，云端机器人系统也会进入原有的蓝图的处理逻辑或多模态融合AI的响应过程，系统也可以给出相应的提示信息。
可以理解的是，通过人工增强AI中人工介入完成的若干操作，并不是让实体机器人完全按照人工介入的操作完成目标任务，而仅仅是当前操作由人工替代。当人工操作结束，则需要回到原有的蓝图上，以使实体机器人按照原有的蓝图执行并完成目标任务。其中，人工介入操作的策略被称为人类智能(Human intelligence，HI)。
步骤603：人工介入操作存储、行为同步运行和事件、状态反馈。
基于人工介入操作也将通过数字孪生运行核心模块让数字孪生体和机器人控制模块上的数字孪生副本保持行为同步运行。云端机器人系统将人工介入操作进行标识记录，接入到机器人大数据模块中。实体机器人与用户的人机交互以及环境变化引起的事件和状态变化也会同步反馈到数字孪生体，通过服务训练师的客户端进行可视化反馈，以此影响当前人工介入的操作行为。
步骤604：数据分析触发重新训练和优化，形成经验知识和数据积累。
基于机器人大数据模块的大数据分析能力，给出当前人工介入操作的数值化评价。对人工介入操作的数值化评价主要包括：对语言AI、视觉AI运动AI以及多模态融合AI相关的改进性评价，触发对应算法和模型的重新训练，以提升算法和模型能力。还可以通过大数据分析对完成服务功能的蓝图逻辑和流程的优化和改进建议。根据历史统计数据，大数据分析还可以获得机器人行为和动作的经验知识、数据和常规习惯。
步骤605：更新机器人业务行为蓝图、行为和动作数据。
基于上述对机器人业务行为蓝图的优化改进以及行为和动作数据的更新，将同步更新到数字孪生体和机器人控制模块的数字孪生副本上，也将驱动实体机器人在线服务。
步骤606：基于更新响应实时事件和状态变化。
基于更新后的语言AI、视觉AI、运动AI以及多模态融合AI的算法和模型，以及形成的经验知识、数据和常规习惯，来响应当前实体机器人的实时事件和状态变化等。
图7是本发明实施例提供的云服务器的结构示意图。如图7所示，该云服务器10用于控制实体机器人，其包括机器人接入与数据交换模块11、知识和数据智能模块12、人工增强机器智能模块13、数字孪生运行核心模块14和机器人大数据模块15，云服务器10和实体机器人之间通过专用网络进行通信。
其中，机器人接入与数据交换模块11用于进行机器人服务进程注册和机器人接入认证，以及接收实体机器人发送的多源数据，并进行数据交换、融合和分发。
知识和数据智能模块12用于提供机器人服务的多领域知识图谱、机器人行为动作库和三维环境语义地图。
数字孪生运行核心模块14包括数字孪生世界和数字孪生体，其中，数字孪生世界基于三维环境语义地图构建，数字孪生体为与实体机器人物理属性相同的物理模型；数字孪生体用于在数字孪生世界中基于机器人服务的多领域知识图谱、机器人行为动作库和多源数据执行机器人技能和应用的训练和在线运行，以同步控制实体机器人执行机器人技能和应用。
所述人工增强机器智能模块通过语言AI、视觉AI、运动AI、多模态AI和人工增强AI，支持所述数字孪生运行核心模块进行机器人技能和应用的训练和在线运行；机器人大数据模块15用于存储和分析多源数据，将分析后的多源数据反馈给数字孪生运行核心模块14用于机器人技能和应用的训练和在线运行。
该云服务器10的具体结构和功能和前述云端机器人系统100中的云服务器10相同，可参考前文描述，此处不再赘述。
图8是本发明实施例提供的机器人控制模块的结构示意图。该机器人控制模块21和云服务器之间通过专用网络进行通信。如图8所示，机器人控制模块21包括数字孪生副本211，数字孪生副本211为云服务器上的数字孪生体的副本；数字孪生副本211根据数字孪生体执行的机器人技能和应用，同步控制实体机器人执行机器人技能和应用。
机器人控制模块21还用于向云服务器发送多源数据，以使数字孪生体在数字孪生世界中基于机器人服务的多领域知识图谱、机器人行为动作库和多源数据执行机器人技能和应用的训练和在线运行，以通过机器人控制模块21的数字孪生副本211同步控制实体机器人执行机器人技能和应用。
该机器人控制模块21的具体结构和功能和前述云端机器人系统100中的机器人控制模块21相同，可参考前文描述，此处不再赘述。
图9是本发明实施例提供的机器人的结构示意图。如图9所示，该机器人40包括图8所示的实施例中的机器人控制模块21。该机器人控制模块21的具体结构和功能和前述云端机器人系统100中的机器人控制模块21相同，可参考前文描述，此处不再赘述。
在此提供的算法或显示不与任何特定计算机、虚拟系统或者其它设备固有相关。各种通用系统也可以与基于在此的示教一起使用。根据上面的描述，构造这类系统所要求的结构是显而易见的。此外，本发明实施例也不针对任何特定编程语言。应当明白，可以利用各种编程语言实现在此描述的本发明的内容，并且上面对特定语言所做的描述是为了披露本发明的最佳实施方式。
在此处所提供的说明书中，说明了大量具体细节。然而，能够理解，本发明的实施例可以在没有这些具体细节的情况下实践。在一些实例中，并未详细示出公知的方法、结构和技术，以便不模糊对本说明书的理解。
类似地，应当理解，为了精简本发明并帮助理解各个发明方面中的一个或多个，在上面对本发明的示例性实施例的描述中，本发明实施例的各个特征有时被一起分组到单个实施例、图、或者对其的描述中。然而，并不应将该公开的方法解释成反映如下意图：即所要求保护的本发明要求比在每个权利要求中所明确记载的特征更多的特征。
本领域技术人员可以理解，可以对实施例中的设备中的模块进行自适应性地改变并且把它们设置在与该实施例不同的一个或多个设备中。可以把实施例中的模块或单元或组件组合成一个模块或单元或组件，以及可以把它们分成多个子模块或子单元或子组件。除了这样的特征和/或过程或者单元中的至少一些是相互排斥之外，可以采用任何组合对本说明书(包括伴随的权利要求、摘要和附图)中公开的所有特征以及如此公开的任何方法或者设备的所有过程或单元进行组合。除非另外明确陈述，本说明书(包括伴随的权利要求、摘要和附图)中公开的每个特征可以由提供相同、等同或相似目的的替代特征来代替。
应该注意的是上述实施例对本发明进行说明而不是对本发明进行限制，并且本领域技术人员在不脱离所附权利要求的范围的情况下可设计出替换实施例。在权利要求中，不应将位于括号之间的任何参考符号构造成对权利要求的限制。单词“包含”不排除存在未列在权利要求中的元件或步骤。位于元件之前的单词“一”或“一个”不排除存在 多个这样的元件。本发明可以借助于包括有若干不同元件的硬件以及借助于适当编程的计算机来实现。在列举了若干装置的单元权利要求中，这些装置中的若干个可以是通过同一个硬件项来具体体现。单词第一、第二、以及第三等的使用不表示任何顺序。可将这些单词解释为名称。上述实施例中的步骤，除有特殊说明外，不应理解为对执行顺序的限定。

Claims (20)

  1. 一种云端机器人系统，其特征在于，包括云服务器和机器人控制模块，所述云服务器包括机器人接入与数据交换模块、知识和数据智能模块、人工增强机器智能模块、数字孪生运行核心模块和机器人大数据模块，所述机器人控制模块位于实体机器人，所述机器人控制模块和所述云服务器之间通过专用网络进行通信；其中，
    所述机器人接入与数据交换模块用于进行机器人服务进程注册和机器人接入认证，以及接收所述机器人控制模块发送的多源数据，并对所述多源数据进行交换、融合和分发；
    所述知识和数据智能模块用于提供机器人服务的多领域知识图谱、机器人行为动作库和三维环境语义地图；
    所述数字孪生运行核心模块包括数字孪生世界和数字孪生体，所述机器人控制模块包括数字孪生副本，其中，所述数字孪生世界基于所述三维环境语义地图构建，所述数字孪生体为与所述实体机器人物理属性相同的物理模型，所述数字孪生副本为运行在所述云服务器上的所述数字孪生体的副本；所述数字孪生体用于在所述数字孪生世界中基于所述机器人服务的多领域知识图谱、所述机器人行为动作库和所述多源数据执行机器人技能和应用的训练和在线运行，所述数字孪生副本根据所述数字孪生体执行的机器人技能和应用，同步控制所述实体机器人执行所述机器人技能和应用；
    所述人工增强机器智能模块通过语言AI、视觉AI、运动AI、多模态AI和人工增强AI，支持所述数字孪生运行核心模块进行机器人技能和应用的训练和在线运行；
    所述机器人大数据模块用于存储和分析所述多源数据，将分析后的所述多源数据反馈给所述数字孪生运行核心模块用于所述机器人技能和应用的训练和在线运行。
  2. 根据权利要求1所述的系统，其特征在于，所述云服务器还包括机器人业务应用服务平台，用于对所述实体机器人进行配置，以及提供机器人服务的下载。
  3. 根据权利要求2所述的系统，其特征在于，所述对所述实体机器人进行配置，包括：
    配置所述实体机器人的数字孪生体模型、机器人名称、角色、性格、应用场景及对话、语言参数、网络参数、待识别的用户人脸清单和对应的机器人服务中的一种或多种，其中所述应用场景根据所述三维环境语义地图配置。
  4. 根据权利要求1所述的系统，其特征在于，所述云服务器还包括机器人开放平台，用于提供机器人服务开发接口以供开发者进行所述机器人服务开发。
  5. 根据权利要求4所述的系统，其特征在于，所述机器人服务为基于所述数字孪生体开发和训练的应用，所述机器人服务开发包括数字孪生体开发、机器人行为和动作编辑和机器人业务行为蓝图编辑。
  6. 根据权利要求2所述的系统，其特征在于，所述数字孪生运行核心模块还用于：在所述数字孪生体在所述数字孪生世界中执行所述机器人技能和应用的训练过程中，若所述数字孪生体执行所述机器人服务的完成情况的数值化评价超过第一预设阈值时，确定所述机器人技能和应用的训练完成，若确定所述机器人技能和应用的训练完成，将训练完成的机器人技能和应用加载到所述机器人控制模块，以使所述机器人控制模块还同步试运行训练完成的机器人技能和应用；
    所述数字孪生运行核心模块还用于：若所述机器人控制模块试运行训练完成的机器人技能和应用的完成情况的数值化评价超过第二预设阈值时，将所述机器人技能和应用所对应的服务发布至所述机器人业务应用服务平台。
  7. 根据权利要求1所述的系统，其特征在于，所述数字孪生运行核心模块还包括第一游戏引擎，用于加载所述数字孪生体和所述数字孪生世界，运行和更新所述数字孪生世界，以及运行所述数字孪生体的行为和动作；
    所述机器人控制模块还包括第二游戏引擎，用于运行所述数字孪生副本；
    所述第一游戏引擎和所述第二游戏引擎用于共同驱动所述数字孪生体和所述数字孪生副本的行为和动作同步执行。
  8. 根据权利要求1所述的系统，其特征在于，所述数字孪生运行核心模块还用于：将所述数字孪生体的行为和动作通过所述专用网络同步给所述机器人控制模块上的数字孪生副本；
    所述数字孪生副本根据所述数字孪生体的行为和动作，同步控制所述实体机器人执行所述行为和动作。
  9. 根据权利要求1所述的系统，其特征在于，所述多源数据包括所述实体机器人的传感器获取的当前环境变化信息和实体机器人自身行为和动作变化信息，所述数字孪生运行核心模块用于根据所述多源数据控制所述数字孪生体与所述实体机器人保持行为和动作同步。
  10. 根据权利要求1所述的系统，其特征在于，所述多领域知识图谱包括与机器人服务相关的实体之间关系的语义网络，所述语义网络包括信息和知识，所述信息用于描述 外部客观事实，所述知识是外部客观规律的归纳和总结；
    所述机器人行为动作库包括机器人通过模仿学习到的人类行为和动作；
    所述三维环境语义地图是实体机器人所处的三维环境的语义数据；
    所述三维环境语义地图通过如下方式获得：将所述多源数据进行融合获得三维环境数据，基于所述三维环境数据通过语义分割进行地图建模，构建所述三维环境语义地图。
  11. 根据权利要求10所述的系统，其特征在于，所述构建所述三维环境语义地图，包括：
    结合基于深度学习的应用场景识别、物体检测识别、几何模型表示、空间语义关系和语义标注，构建多语义融合的三维环境语义地图。
  12. 根据权利要求1所述的系统，其特征在于，所述语言AI包括自动语音识别、自然语言理解和语音合成；所述视觉AI包括人脸识别、人体识别、人像识别、物体识别和环境场景识别；所述运动AI包括外力传感感知、自主移动和导航、肢体动作；所述多模态AI是指具有所述语言AI、视觉AI和运动AI的能力，以及同时具有多因素结合输出的能力，其中所述多因素结合输出包括所述语言AI、视觉AI和运动AI的输入以及语音输出、运动输出；
    所述人工增强AI用于：通过人工介入操作为系统强化学习提供正向激励输入，在人工介入操作时，所述语言AI、视觉AI、运动AI和多模态AI均是在线运行状态。
  13. 根据权利要求12所述的系统，其特征在于，所述人工增强AI还用于：若出现机器人服务异常情况，接收并执行操作指令。
  14. 根据权利要求1所述的系统，其特征在于，所述机器人大数据模块还用于存储和分析系统运行和服务日志数据、用户数据、人工增强的操作数据和系统性能数据中的一种或多种。
  15. 根据权利要求1所述的系统，其特征在于，所述多源数据包括通过所述实体机器人的传感器获取的音视频数据、三维环境点云数据、机器人行为和动作数据和多模态交互数据中的一种或多种。
  16. 根据权利要求1、14或15所述的系统，其特征在于，所述机器人大数据模块还用于：
    对所述多源数据进行数据抽取、数据转换、数据装载、数据分类、数据标注、异常检测和数据清洗，得到处理后的数据；
    对所述处理后的数据进行实时分析和离线分析，对所述云端机器人系统中各个所述机器人服务的运行进行数值化评价，所述数值化评价用于确定所述机器人技能和应用的训练是否完成；
    所述数字孪生运行核心模块还用于在所述数值化评价满足预设条件的情况下，对所述机器人技能和应用进行重新训练和更新。
  17. 根据权利要求16所述的系统，其特征在于，
    所述数值化评价包括AI算法和模型的实际识别率、人机对话回复的满意度、服务响应时长和机器人业务行为蓝图的高效性和稳定性；
    所述机器人大数据模块还用于：对数值化评价的目标结论进行分类，形成先验知识、相关业务和相关数据。
  18. 一种云服务器，用于控制实体机器人，其特征在于，包括机器人接入与数据交换模块、知识和数据智能模块、人工增强机器智能模块、数字孪生运行核心模块和机器人大数据模块，所述云服务器和所述实体机器人之间通过专用网络进行通信；其中，
    所述机器人接入与数据交换模块用于进行机器人服务进程注册和机器人接入认证，以及接收所述实体机器人发送的多源数据，并对所述多源数据交换、融合和分发；
    所述知识和数据智能模块用于提供机器人服务的多领域知识图谱、机器人行为动作库和三维环境语义地图；
    所述数字孪生运行核心模块包括数字孪生世界和数字孪生体，其中，所述数字孪生世界基于所述三维环境语义地图构建，所述数字孪生体为与所述实体机器人物理属性相同的物理模型；所述数字孪生体用于在所述数字孪生世界中基于所述机器人服务的多领域知识图谱、所述机器人行为动作库和所述多源数据执行机器人技能和应用的训练和在线运行，以同步控制所述实体机器人执行所述机器人技能和应用；
    所述人工增强机器智能模块通过语言AI、视觉AI、运动AI、多模态AI和人工增强AI，支持所述数字孪生运行核心模块进行机器人技能和应用的训练和在线运行；
    所述机器人大数据模块用于存储和分析所述多源数据，将分析后的所述多源数据反馈给所述数字孪生运行核心模块用于所述机器人技能和应用的训练和在线运行。
  19. 一种机器人控制模块，其特征在于，所述机器人控制模块和云服务器之间通过专用网络进行通信；
    所述机器人控制模块包括数字孪生副本，所述数字孪生副本为运行在所述云服务器 上的数字孪生体的副本；所述数字孪生副本根据所述数字孪生体执行的机器人服务，同步控制实体机器人执行所述机器人服务；
    所述机器人控制模块还用于向所述云服务器发送多源数据，以使所述数字孪生体在数字孪生世界中基于机器人服务的多领域知识图谱、机器人行为动作库和所述多源数据执行机器人技能和应用的训练和在线运行，以通过所述数字孪生副本同步控制所述实体机器人执行所述服务。
  20. 一种机器人，其特征在于，所述机器人包括如权利要求19所述的机器人控制模块。
PCT/CN2021/124506 2020-12-01 2021-10-18 云端机器人系统、云服务器、机器人控制模块和机器人 WO2022116716A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011386136.2 2020-12-01
CN202011386136.2A CN112668687B (zh) 2020-12-01 2020-12-01 云端机器人系统、云服务器、机器人控制模块和机器人

Publications (1)

Publication Number Publication Date
WO2022116716A1 true WO2022116716A1 (zh) 2022-06-09

Family

ID=75400739

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/124506 WO2022116716A1 (zh) 2020-12-01 2021-10-18 云端机器人系统、云服务器、机器人控制模块和机器人

Country Status (2)

Country Link
CN (1) CN112668687B (zh)
WO (1) WO2022116716A1 (zh)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115080797A (zh) * 2022-06-28 2022-09-20 电子科技大学 一种基于知识图谱的数字孪生车间多尺度关联方法
CN115659838A (zh) * 2022-11-14 2023-01-31 中国电力科学研究院有限公司 一种电力系统主设备数字孪生混合模型构建方法及装置
CN115860366A (zh) * 2022-11-17 2023-03-28 桂林电子科技大学 一种社区机器人智慧协调管控方法、系统和可读存储介质
CN115994458A (zh) * 2023-03-23 2023-04-21 华南理工大学 一种虚实集成的多智能体集群系统仿真方法
CN116214527A (zh) * 2023-05-09 2023-06-06 南京泛美利机器人科技有限公司 一种增强人机协作适应性的三体协同智能决策方法和系统
CN116704156A (zh) * 2023-04-28 2023-09-05 北京优酷科技有限公司 模型生成方法、电子设备及模型生成系统
WO2024078003A1 (zh) * 2022-10-10 2024-04-18 华为云计算技术有限公司 模拟训练方法、装置及计算设备集群
CN115860366B (zh) * 2022-11-17 2024-05-14 桂林电子科技大学 一种社区机器人智慧协调管控方法、系统和可读存储介质

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112668687B (zh) * 2020-12-01 2022-08-26 达闼机器人股份有限公司 云端机器人系统、云服务器、机器人控制模块和机器人
CN115237113B (zh) * 2021-08-02 2023-05-12 达闼机器人股份有限公司 机器人导航的方法、机器人、机器人系统及存储介质
CN113626003A (zh) * 2021-08-16 2021-11-09 杭州群核信息技术有限公司 云端机器人解决方案系统
CN113687718A (zh) * 2021-08-20 2021-11-23 广东工业大学 一种人-机集成的数字孪生系统及其构建方法
CN113959444A (zh) * 2021-09-30 2022-01-21 达闼机器人有限公司 用于无人设备的导航方法、装置、介质及无人设备
CN114310870A (zh) * 2021-11-10 2022-04-12 达闼科技（北京）有限公司 智能体的控制方法、装置、电子设备及存储介质
CN114080905B (zh) * 2021-11-25 2022-12-06 杭州乔戈里科技有限公司 基于数字双胞胎的采摘方法及云采摘机器人系统
CN114371174A (zh) * 2021-12-17 2022-04-19 中国电子科技集团公司第四十一研究所 一种用于工业产线的视觉孪生检测装置及方法
CN114387643A (zh) * 2021-12-28 2022-04-22 达闼机器人有限公司 机器人控制方法、系统、计算机设备及存储介质
CN114372356B (zh) * 2021-12-29 2023-02-28 达闼机器人股份有限公司 基于数字孪生的人工增强方法、装置及介质
CN114290333B (zh) * 2021-12-29 2024-02-27 中国电信股份有限公司 一种泛在机器人系统、构建方法及装置、设备、介质
CN114559433B (zh) * 2022-03-17 2024-01-12 达闼机器人股份有限公司 机器人控制方法、装置、存储介质、机器人及云端服务器
CN114722050B (zh) * 2022-06-10 2022-09-30 辰星（天津）自动化设备有限公司 机器人系统的数据同步方法、及机器人系统
CN115268667A (zh) * 2022-07-18 2022-11-01 北京数字冰雹信息技术有限公司 一种网页端数字孪生三维场景控制方法及系统
CN116370954B (zh) * 2023-05-16 2023-09-05 北京可以科技有限公司 游戏方法和游戏装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108235697A (zh) * 2017-09-12 2018-06-29 深圳前海达闼云端智能科技有限公司 一种机器人动态学习方法、系统、机器人以及云端服务器
CN110866588A (zh) * 2019-11-08 2020-03-06 中国科学院软件研究所 一种实现智能虚拟数字动物的可学习能力模型个性化的训练学习方法与系统
CN111273892A (zh) * 2020-02-13 2020-06-12 济南浪潮高新科技投资发展有限公司 一种基于云端技术和边缘计算实现智能机器人的方法
CN112668687A (zh) * 2020-12-01 2021-04-16 达闼机器人有限公司 云端机器人系统、云服务器、机器人控制模块和机器人

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200304375A1 (en) * 2019-03-19 2020-09-24 Microsoft Technology Licensing, Llc Generation of digital twins of physical environments
CN110430260B (zh) * 2019-08-02 2022-05-13 安徽领云物联科技有限公司 一种基于大数据云计算支撑的机器人云平台及工作方法
CN111897332B (zh) * 2020-07-30 2022-10-11 国网智能科技股份有限公司 一种语义智能变电站机器人仿人巡视作业方法及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108235697A (zh) * 2017-09-12 2018-06-29 深圳前海达闼云端智能科技有限公司 一种机器人动态学习方法、系统、机器人以及云端服务器
CN110866588A (zh) * 2019-11-08 2020-03-06 中国科学院软件研究所 一种实现智能虚拟数字动物的可学习能力模型个性化的训练学习方法与系统
CN111273892A (zh) * 2020-02-13 2020-06-12 济南浪潮高新科技投资发展有限公司 一种基于云端技术和边缘计算实现智能机器人的方法
CN112668687A (zh) * 2020-12-01 2021-04-16 达闼机器人有限公司 云端机器人系统、云服务器、机器人控制模块和机器人

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115080797A (zh) * 2022-06-28 2022-09-20 电子科技大学 一种基于知识图谱的数字孪生车间多尺度关联方法
WO2024078003A1 (zh) * 2022-10-10 2024-04-18 华为云计算技术有限公司 模拟训练方法、装置及计算设备集群
CN115659838A (zh) * 2022-11-14 2023-01-31 中国电力科学研究院有限公司 一种电力系统主设备数字孪生混合模型构建方法及装置
CN115659838B (zh) * 2022-11-14 2023-02-28 中国电力科学研究院有限公司 一种电力系统主设备数字孪生混合模型构建方法及装置
CN115860366A (zh) * 2022-11-17 2023-03-28 桂林电子科技大学 一种社区机器人智慧协调管控方法、系统和可读存储介质
CN115860366B (zh) * 2022-11-17 2024-05-14 桂林电子科技大学 一种社区机器人智慧协调管控方法、系统和可读存储介质
CN115994458A (zh) * 2023-03-23 2023-04-21 华南理工大学 一种虚实集成的多智能体集群系统仿真方法
CN115994458B (zh) * 2023-03-23 2023-07-18 华南理工大学 一种虚实集成的多智能体集群系统仿真方法
CN116704156A (zh) * 2023-04-28 2023-09-05 北京优酷科技有限公司 模型生成方法、电子设备及模型生成系统
CN116214527A (zh) * 2023-05-09 2023-06-06 南京泛美利机器人科技有限公司 一种增强人机协作适应性的三体协同智能决策方法和系统
CN116214527B (zh) * 2023-05-09 2023-08-11 南京泛美利机器人科技有限公司 一种增强人机协作适应性的三体协同智能决策方法和系统

Also Published As

Publication number Publication date
CN112668687B (zh) 2022-08-26
CN112668687A (zh) 2021-04-16

Similar Documents

Publication Publication Date Title
WO2022116716A1 (zh) 云端机器人系统、云服务器、机器人控制模块和机器人
CN111432989B (zh) 人工增强基于云的机器人智能框架及相关方法
Caccavale et al. Kinesthetic teaching and attentional supervision of structured tasks in human–robot interaction
US11120365B2 (en) For hierarchical decomposition deep reinforcement learning for an artificial intelligence model
JP7085788B2 (ja) ロボットの動的学習方法、システム、ロボット及びクラウドサーバ
CN107671857B (zh) 用于服务机器人操作演示与算法验证的三维仿真平台
AU2020201118A1 (en) Methods and systems for managing dialogs of a robot
CN108297098A (zh) 人工智能驱动的机器人控制系统及方法
Walker et al. Virtual, augmented, and mixed reality for human-robot interaction: A survey and virtual design element taxonomy
WO2022134775A1 (zh) 数字孪生模型的运行方法、装置和电子设备
CN115916477A (zh) 机器人演示学习的技能模板分发
Scheutz et al. Systematic integration of cognitive and robotic architectures
US11833691B2 (en) Hybrid robotic motion planning system using machine learning and parametric trajectories
Seib et al. Team homer@ unikoblenz—approaches and contributions to the robocup@ home competition
Liu et al. Manipulating complex robot behavior for autonomous and continuous operations
Zhu Closing the Perception-action Loop: Towards General-purpose Robot Autonomy
Duan et al. Intelligent Robot: Implementation and Applications
Miyake et al. Proposal for the Implementation of Spatial Common Ground and Spatial AI using the SSCP (Spatial Simulation-based Cyber-Physical) Model
Temsamani et al. A multimodal AI approach for intuitively instructable autonomous systems: a case study of an autonomous off-highway vehicle
US11945117B2 (en) Anticipating user and object poses through task-based extrapolation for robot-human collision avoidance
US11731279B2 (en) Systems and methods for automated tuning of robotics systems
Thalmann et al. Virtual and real humans interacting in the virtual world
US11712804B2 (en) Systems and methods for adaptive robotic motion control
Tian et al. Research on Multi-Sensor Fusion of Layered Intelligent System for Indoor Mobile Robot
Qi et al. Avr-akg: using virtual reality for domain knowledge generation from first-person demonstration

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21899750

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 101123)