JP2021534988A - Systems, equipment, and methods for robot learning and execution of skills - Google Patents

Abstract

Systems, devices, and methods for robot learning and execution of skills are described. The robotic apparatus may include a memory, a processor, a sensor, and one or more moving components (eg, operating and / or transporting elements). The processor may be operably coupled to memory, moving elements, and sensors and configured to acquire information about the environment, including one or more objects located within the environment. In some embodiments, the processor may be configured to learn skills by demonstration, exploration, user input, and the like. In some embodiments, the processor may be configured to perform skills and / or perform arbitration between different behaviors and / or actions. In some embodiments, the processor may be configured to learn environmental constraints. In some embodiments, the processor may be configured to learn using a generic model of skill.

Description

Cross-reference of related applications
[0001] This application has priority over US Provisional Patent Application No. 62 / 723, 694 entitled "SYSTEMS, APPARATUS, AND METHODS FOR ROBOTIC LEARNING AND EXECUTION OF SKILLS" filed on August 28, 2018. Is to insist.

[0002] This application claims priority to US Provisional Patent Application No. 62 / 723,694 and is filed on February 25, 2017, "METHOD AND SYSTEM FOR ROBOTIC LEARNING OF EXECUTION PROCESSES RELATED TO PERCEPTUALLY. US Provisional Patent Application No. 62/463,628 entitled "CONSTRAINED MANIPULATION SKILLS" and "METHOD AND SYSTEM FOR ROBOTIC EXECUTION OF A PERCEPTUALLY CONSTRAINED MANIPULATION SKILL LEARNED VIA HUMAN INTERACTION" filed on February 25, 2017. International named "SYSTEMS, APPARATUS, AND METHODS FOR ROBOTIC LEARNING AND EXECUTION OF SKILLS" filed on February 23, 2018, claiming priority over US Provisional Patent Application No. 62 / 463,630. PCT Application No. PCT / US2018 / 019520, a partial continuation application, filed on June 28, 2019, is the US patent application named "SYSTEMS, APPARATUS, AND METHODS FOR ROBOTIC LEARNING AND EXECUTION OF SKILLS". It also claims priority over 16 / 456,919.

[0003] This application is also a partial continuation of International PCT Application No. PCT / US2018 / 019520.

[0004] The disclosures of each of the applications mentioned above are incorporated herein by reference in their entirety.

Government support
[0005] The present invention has been made with the support of the United States Government under Phase I of the Small Business Innovation Program, under authorization numbers 1621651 and 1738375, granted by the United States National Science Fund. The US Government has certain rights to the invention.

Technical field
[0006] The present disclosure generally relates to systems, devices, and methods for robot learning and execution of skills. More specifically, the present disclosure relates to a robotic device capable of learning and executing skills in an unstructured environment.

background
[0007] Robots can be used to perform and automate various tasks. Robots can perform tasks by moving through an environment such as an office building or hospital. The robot may be equipped with wheels, trucks, or other moving components that allow the robot to move autonomously around the environment. However, a robot without an arm or other manipulator cannot manipulate objects in the environment. Therefore, these robots have limited ability to perform tasks, for example, if such robots cannot pick up and deliver an object without a person manipulating the object at the point of pick-up or delivery. There is.

[0008] A robot, including an arm or other manipulator, may be able to pick up an object and deliver it to a location without humans. For example, a robot with an arm equipped with an end effector, such as a gripper, can use the gripper to pick up one or more objects from different locations and deliver these objects to new locations, all human. It is possible without help. These robots can be used to automate certain tasks, thereby allowing human operators to focus on other tasks. However, most commercial robots do not include a manipulator due to the challenges and complexity of programming the manipulator's actions.

[0009] Also, most commercial robots are designed to operate in a structured environment (eg, factories, warehouses, etc.). Unstructured environments (eg, human-related environments such as hospitals and homes) can pose additional challenges for programming robots. In an unstructured environment, the robot cannot rely on complete knowledge of its surroundings, but must be able to recognize and adapt to changes in its surroundings. Therefore, in an unstructured environment, the robot must continuously acquire information about the environment in order to be able to make autonomous decisions and perform tasks. Often, the movement of the robot (eg, the movement of the arm or end effector in the environment) is also constrained by objects and other obstacles in the environment, further increasing the perceptual and operational challenges of the robot. Given the uncertain and dynamic nature of unstructured environments, robots are generally not pre-programmed to perform tasks.

[0010] Therefore, it is possible to recognize and adapt to dynamic, unstructured environments, and to perform tasks within those environments, without relying on pre-programmed operational skills. There is a need for a robotic system that can be done.

Overview
[0011] Systems, devices, and methods for robot learning and execution of skills are described. In some embodiments, the device comprises a set of memory, processor, operating elements, and sensors. The processor is operably coupled to a set of memory, operating elements, and sensors, as well as using a subset of sensors from the set of sensors to obtain a representation of the environment and multiple markers within the representation of the environment. Is to identify that each marker from multiple markers is associated with a physical object from multiple physical objects located in the environment, from multiple markers in the representation of the environment. Presenting information indicating the location of each marker in the environment, receiving a selection of a set of markers from multiple markers associated with a set of physical objects from multiple physical objects, and manipulating in the environment. Acquiring sensory information related to an operation element for each position from multiple positions related to the movement of the element, and the movement of the operation element becomes a physical interaction between the operation element and a set of physical objects. Relevant, acquiring, and generating a model that is configured to specify the behavior of the operating element to perform a physical interaction between the operating element and a set of physical objects, based on sensory information. , May be configured to do.

[0012] In some embodiments, the operating element may include multiple joints and end effectors. In some embodiments, the operating element may include a base or a transport element.

[0013] The set of physical objects may be stationary or moving objects. In some embodiments, the set of physical objects may include fixtures, storage containers, structures (eg, doors, handles, furniture, walls) and the like. In some embodiments, the set of physical objects may include humans (eg, patients, doctors, nurses, etc.).

[0014] In some embodiments, the plurality of markers are reference markers and the representation of the environment is a visual representation of the environment.

[0015] In some embodiments, two or more markers from multiple markers may be associated with one physical object from a set of physical objects. Alternatively or additionally, in some embodiments, one marker from multiple markers may be associated with two or more physical objects from a set of physical objects.

[0016] In some embodiments, the method is to use a set of sensors to obtain a representation of the environment and to identify multiple markers within the representation of the environment, from multiple markers. Each marker is associated with a physical object from multiple physical objects located in the environment, identifying and presenting information indicating the position of each marker from multiple markers in the representation of the environment. After the above presentation, receiving a selection of a set of markers from multiple markers, associated with a set of physical objects from multiple physical objects, and multiple positions related to the movement of the operating element in the environment. To acquire sensory information related to an operating element for each position from, and the movement of the operating element is related to the physical interaction between the operating element and a set of physical objects. Informedly, it involves generating a model configured to specify the behavior of the operating element to perform a physical interaction between the operating element and a set of physical objects.

[0017] In some embodiments, the method further comprises receiving a selection of a subset of first features from a set of features, which here is based on sensor data associated with the subset of first features. And the model is generated without being based on sensor data related to a subset of the second feature from the set of features that is not included in the first set of features.

[0018] In some embodiments, the method is to use a set of sensors to obtain a representation of the environment and to identify multiple markers within the representation of the environment, from the plurality of markers. Each marker is associated with a physical object from multiple physical objects located in the environment, identifying and presenting information indicating the position of each marker from multiple markers in the representation of the environment. Performs a physical interaction between an operating element and a set of physical objects in response to receiving a selection of a set of markers from multiple markers associated with the set of physical objects from multiple physical objects. Identifying a model associated with, wherein the operating element comprises multiple joints and end effectors, and is associated with identifying and performing physical interactions using the model. Includes the generation of trajectories of operating elements that define the behavior of joints and end effectors.

[0019] In some embodiments, the above method indicates displaying the trajectory of the operating element to the user within a representation of the environment, receiving input from the user after display, and approving the trajectory of the operating element. It further comprises performing the actions of operating elements (eg, multiple joints, end effectors, transport elements) in response to input to perform physical interactions.

[0020] In some embodiments, the model is (i) associated with a set of conserved markers and (ii) a set of conserved markers, operating elements (eg, multiple joints, end effectors, transport). The element) is associated with sensory information indicating at least one of the position, orientation, or configuration of the operating element at a point along the conserved trajectory of the element. The method of generating the trajectory of the operating element is, for example, to calculate a conversion function between a set of markers and a set of stored markers, and pointwise, using the conversion function, at least the position or orientation of the operating element. Transforming one and, point by point, a planned configuration of one or more components of an operating element (eg, multiple joints, end effectors, transport elements) of the above components along a conserved trajectory. It may include determining on the basis of the configuration and, on a point-by-point basis, determining a portion of the orbit between the point and the continuum point based on the planned configuration of the above component for that point.

[0021] In some embodiments, the robotic device is configured to learn and perform skills associated with transport elements. The transport element allows the robotic device to move, for example, a set of wheels, a set of trucks, a set of crawlers, or from one room to a second room, or within an environment such as between multiple floors of a building. It may be any other suitable device. When the robotic device is moved from a first place to a second place, for example through a doorway or by an object such as a human, the robotic device is equipped with transport elements, the environment around the robotic device, and /. Alternatively, it may be configured to learn skills related to transport elements by acquiring sensory information related to another component of the robotic device. Sensory information may be recorded at specific points in time (eg, keyframes) during the operation of the robotic device. The robot device operates a transport element (or other component of the robot device (eg, an operating element)) to perform movement of the robot device from a first location to a second location based on sensory information. May be configured to generate a model configured to specify.

[0022] In some embodiments, the robotic device is configured to learn a specialized skill using a generic version of the skill. The robot device can start executing a general-purpose skill in a particular environment and pause execution when the robot device reaches the part of the execution that requires specialization of the skill in that particular environment. The robot device can then encourage the user to demonstrate that part of the skill. The robot device can continue to execute the skill and / or prompt the user to demonstrate a particular part of the skill until the skill is completed. The robot device then adapts the generic skill based on a demonstration of a particular part of the skill to generate a specialized model for performing the skill.

[0023] In some embodiments, the robotic device is configured to learn environmental constraints. A robot device can record information about its surrounding environment or objects within that environment and obtain general knowledge about the environment. Robotic devices can apply this general knowledge to a set of models related to different skills performed in an environment.

[0024] In some embodiments, the robotic device is capable of using behavioral arbitration to behave continuously and autonomously in a dynamic environment such as the human social environment. Robotic devices may have various resources or components (eg, operating elements, transport elements, heads, cameras, etc.) and arbitration to determine when and how to use these resources. The algorithm can be applied. Arbitration algorithms can specify a set of rules that allow different actions or behaviors to be prioritized based on the information captured by the robot device. The robotic device may be configured to continually determine (and thus use different resources or components) from performing different actions and behaviors as new information is captured by the robotic device. As part of the behavior arbitration, the robot device may be configured to handle the interruption of an action and switch to a new action. By continuously monitoring the robot device itself and its surrounding environment and performing continuous arbitration, the robot device can constantly engage in socially appropriate behavior over time.

[0025] In some embodiments, the robot device receives social contextual information related to the environment and combines that information with the navigation map of the environment (eg, overlaying the social contextual information on the navigation map). It is composed. The robot device may be given social context information by a user (eg, a human operator), or the robot device may capture social context information through interactions within the environment. As the robot device engages in human interaction in the environment over time, the robot device can improve social contextual information. In some embodiments, the robotic device may be configured to associate certain types of behaviors or actions with these locations based on social contextual information associated with specific locations in the environment.

[0026] In some embodiments, the robotic device is configured to interact with a human operator by a network connection to a remote device operated in the field or by a human operator. Human operators, referred to herein as "robot supervisors," use a variety of software-based tools to inform robot devices about their surroundings and / or to take specific actions. You can instruct the robot device. These actions may include, for example, navigation behavior, operational behavior, head behavior, voice, lights, and the like. The robot supervisor collects information during the learning or execution of the action, and / or the ability of the robot device to use that information to arbitrate in different actions in the future, or a particular action. Robotic devices can be controlled to tag specific behaviors as good or bad so that they can improve their ability to perform.

Other systems, processes, and features will be apparent to those of skill in the art by considering the drawings and detailed description below. It is intended that all such additional systems, processes, and features are contained within the scope of the present specification, are within the scope of the invention, and are protected by the appended claims. To.

A brief description of the drawing
It will be appreciated by those skilled in the art that the drawings are intended primarily for illustration purposes and are not intended to limit the scope of the invention-specific matters described herein. The drawings are not necessarily to a constant scale, and in some cases, various aspects of the invention-specific matters disclosed herein are exaggerated in the drawings to facilitate understanding of the different features. That is, it may be shown enlarged. In the drawings, similar reference numerals generally refer to similar features (eg, functionally similar and / or structurally similar elements).

[0029] FIG. 3 is a block diagram illustrating a configuration of a system including a robotic device according to some embodiments. [0030] FIG. 3 is a block diagram showing a configuration of a robotic device according to some embodiments. [0031] FIG. 3 is a block diagram illustrating a configuration of a control unit associated with a robotic device, according to some embodiments. [0032] It is a schematic diagram of the operation element of the robot device according to some embodiments. [0033] It is a schematic diagram of a robot device according to some embodiments. [0034] It is a schematic diagram of an object in an environment seen by a robot device according to some embodiments. [0034] It is a schematic diagram of an object in an environment seen by a robot device according to some embodiments. [0035] It is a schematic diagram of the object in the environment seen by the robot device by some embodiments. [0035] It is a schematic diagram of the object in the environment seen by the robot device by some embodiments. [0036] FIG. 3 is a flow diagram illustrating a method of environmental sensing or scanning performed by a robotic device according to some embodiments. [0037] FIG. 6 is a flow diagram illustrating a method of learning and executing skills performed by a robotic device, according to some embodiments. [0038] It is a flow diagram which shows the method of learning the skill performed by a robot device by some embodiments. [0039] It is a flow diagram which shows the method of performing the skill performed by a robot device by some embodiments. [0040] FIG. 6 is a block diagram illustrating a system architecture for robot learning and execution, including user actions, according to some embodiments. [0041] It is a flow diagram which shows the operation of the robot device in an environment by some embodiments. [0042] FIG. 6 is a flow diagram illustrating a method of requesting and receiving input from a user, such as a robot supervisor, according to some embodiments. [0043] FIG. 6 is a flow diagram illustrating a method of learning skills and environmental constraints performed by a robotic device, according to some embodiments. [0044] It is a flow diagram which shows the method of learning a skill from a general-purpose skill model performed by a robot device by some embodiments. [0045] FIG. 6 is a flow diagram illustrating a method of learning environmental constraints performed by a robotic device, according to some embodiments. [0046] It is a block diagram which shows an example of the component of the robot device which performs the behavior arbitration by some embodiments. [0047] Schematic representation of layers of a map of an environment generated by a robotic device, according to some embodiments. [0048] FIG. 3 is a block diagram illustrating a configuration of a control unit associated with a robotic device, according to some embodiments. [0049] In some embodiments, the flow of information provided by and received from a map by the robot device for a map held by the robot device is shown. [0050] Shown is a flow diagram showing different learning modes of a robotic device according to some embodiments. [0051] A flow diagram showing exemplary learning behavior of a robotic device according to some embodiments is shown. [0051] A flow diagram showing exemplary learning behavior of a robotic device according to some embodiments is shown. [0051] A flow diagram showing exemplary learning behavior of a robotic device according to some embodiments is shown.

Detailed explanation
[0052] This specification describes systems, devices, and methods for robot learning and execution of skills. In some embodiments, the systems, devices, and methods described herein relate to robotic devices capable of learning skills through human demonstrations and interactions and performing the learned skills in an unstructured environment. ..

Overview
[0053] In some embodiments, the systems, devices, and methods described herein relate to robots capable of learning skills (eg, operational skills) by a Learning from Demonstration (“LfD”) process. In this LfD process, the human performs the action by kinesthetic teaching (eg, physically and / or by remote control, instructing the robot to perform the action) and / or by the human himself. By demonstrating the action to the system. Such systems, devices, and methods do not require the robot to be pre-programmed with operational skills, and the robot is designed to be adaptable and able to learn the skills by observation. For example, robots can acquire and execute operational skills using machine learning techniques. After learning the skill, the robot can perform the skill in different environments. The robot can learn and / or execute skills based on visual data (eg, perceived visual information). Alternatively or additionally, the robot can use tactile data (eg, torque, force, and other non-visual information) to learn and / or perform skills. Robot learning can be done in the factory before the robot is deployed or in the field (eg, in the hospital) after the robot is deployed. In some embodiments, the robot can be trained and / or adapted to operate in an environment by a user who is not trained in robotics and / or programming. For example, a robot can have a learning algorithm that takes advantage of natural human behavior and can include tools that can instruct the user to perform a demonstration process.

[0054] In some embodiments, the robot can be designed to interact with and cooperate with humans to perform tasks. In some embodiments, the robot can take advantage of common social behaviors to behave in a socially predictable and acceptable manner by humans. Movable robots can also be designed to navigate within an environment while interacting with humans in that environment. For example, a robot may be programmed to speak out a specific word to navigate around a human, to set aside for human passage, and to intentionally communicate using the line of sight during navigation. Can be done. In some embodiments, the robot provides sensors within its surrounding environment that allow the robot to sense and track humans and use that information to generate gaze and other social behaviors. You may prepare.

[0055] In some embodiments, the robot can be designed to suggest options for achieving goals or performing actions during the LfD process. For example, a robot can propose several different options to achieve a goal (eg, picking up an object), and which of these options is effective in achieving that goal. It can indicate whether it is most likely to be targeted and / or efficient. In some embodiments, the robot can adapt the skill based on user input (eg, indicating the relevant features that the user includes in the skill model).

[0056] In some embodiments, the robotic device learns and / or learns skills in an unstructured environment (eg, dynamic and / or human environment) in which the robotic device does not have complete information of the environment in advance. Can be done. The unstructured environment may include, for example, indoor and outdoor situations, and may include one or more people or other objects that are mobile within the environment. Robotic devices that can be adapted and operated in an unstructured environment, such as the robotic devices and / or systems described herein, are unstructured because the most natural or real-world environment is unstructured. It can provide significant improvements over existing robotic devices that are not adaptable to the environment. Unstructured environments include indoor situations (eg, buildings, offices, houses, rooms, etc.) and / or other types of closed spaces (eg, planes, trains, and / or other types of mobile compartments). , As well as outdoor conditions (eg, parks, beaches, gardens, fields). In certain embodiments, the robotic devices described herein can operate in an unstructured hospital environment.

[0057] FIG. 1 is a high level block diagram showing a system 100 according to some embodiments. The system 100 can be configured to learn and execute skills, such as operational skills, in an unstructured environment. The system 100 may be implemented as a single device or may be implemented across multiple devices connected to the network 105. For example, as shown in FIG. 1, the system 100 includes one or more computing devices such as, for example, one or more robotic devices 102 and 110, a server 120, and one or more additional computing devices 150. obtain. Although four devices are shown, it is understood that the system 100 may include any number of computational devices, including computational devices not specifically shown in FIG.

[0058] The network 105 is implemented as a wired and / or wireless network and is used to operably combine computing devices including robotic devices 102 and 110, a server 120, and one or more computing devices 150. It may be any type of network (eg, local area network (LAN), wide area network (WAN), virtual network, telecommunications network). As described in more detail herein, in some embodiments, for example, the computing device is a computer connected to each other by an Internet Service Provider (ISP) and the Internet (eg, Network 105). be. In some embodiments, the connection between any two computing devices can be defined by the network 105. As shown in FIG. 1, for example, a connection can be defined between the robot device 102 and any one of the robot device 110, the server 120, or one or more computing devices 150. In some embodiments, these computing devices are capable of communicating with each other (eg, transmitting and / or receiving data), as well as intermediate and / or alternative networks (not shown in FIG. 1). It is possible to communicate with the network 105 via the network 105. Such an intermediate network and / or an alternative network may be the same type and / or a different type of network as the network 105. Each computing device may be any type of device configured to transmit data on the network 105 for transmission and / or receive data from one or more of the other computing devices.

[0059] In some embodiments, the system 100 includes a single robot device (eg, robot device 102). The robot device 102 senses information about the environment, learns skills through human demonstrations and interactions, interacts with the environment, and / or learns environmental constraints through human demonstrations and inputs, and / or in that environment. It can be configured to perform those skills. In some embodiments, the robot device 102 may perform self-search and / or require user input to learn additional information about skills and / or the environment. A more detailed diagram of an example robot device is shown in FIG.

[0060] In another embodiment, the system 100 includes a plurality of robotic devices (eg, robotic devices 102 and 110). The robot device 102 can transmit data to and / or receive data from the robot device 110 via the network 105. For example, the robot device 102 can transmit information sensed by the robot device 102 regarding the environment (for example, the location of an object) to the robot device 110, and can receive information about the environment from the robot device 110. Also, robotic devices 102 and 110 can send and / or receive information from each other in order to learn and / or execute skills. For example, the robot device 102 can learn a skill in an environment and send a model representing the learned skill to the robot device 110, and when the robot device 110 receives the model, it uses the same model. Or you can perform the skill in a different environment. The robot device 102 may be located at the same or different location as the robot device 110. For example, robotic devices 102 and 110 may be in the same room in a building (eg, a hospital building) so that they can learn and / or perform skills together (eg, move heavy or large objects). It may be located. Alternatively, the robot device 102 may be located on the first floor of a building (eg, a hospital building) and the robot device 110 may be located on the second floor of a building, the two robot devices relating to different floors. They can communicate with each other to convey information (eg, where objects are on these floors, where resources can exist, etc.).

[0061] In some embodiments, the system 100 includes one or more robotic devices (eg, robotic devices 102 and / or 110) and a server 120. The server 120 may be a dedicated server that manages the robot device 102 and / or 110. The server 120 may be located at the same or different location as the robot devices 102 and / or 110. For example, the server 120 may be located in the same building as one or more robotic devices (eg, a hospital building) or may be managed by a local administrator (eg, a hospital administrator). Alternatively, the server 120 may be located in a remote location (eg, a location associated with the manufacturer or supplier of the robotic device).

[0062] In some embodiments, the system 100 includes one or more robotic devices (eg, robotic devices 102 and / or 110) and an additional computing device 150. The computing device 150 may be any suitable processing device configured to activate and / or perform a particular function. For example, in a hospital situation, the computing device 150 may be a diagnostic and / or therapeutic device capable of connecting to the network 105 and communicating with other computing devices including the robot device 102 and / or 110.

[0063] In some embodiments, one or more robotic devices (eg, robotic devices 102 and / or 110) are configured to communicate with the server 120 and / or the computing device 150 via the network 105. May be done. The server 120 may include one or more components that are remote from the robot device and / or located near the robot device on the premises. Computational device 150 may include one or more components that are remote from the robotic device, located close to the robotic device on the premises, and / or incorporated into one robotic device. The server 120 and / or the computing device 150 may include a user interface that allows a user (eg, a nearby user or a robot supervisor) to control the operation of the robot device. For example, the user can suspend and / or modify the execution of one or more actions performed by the robot device. These actions may include, for example, navigation behavior, operational behavior, head behavior, voice / light, and / or other components of the robotic device. In some embodiments, the robot supervisor can remotely monitor the robot device and control its movement for safety purposes. For example, a robot supervisor commands a robot device to stop or modify the execution of an action to avoid endangering a human or damaging the robot device or another object in the environment. can do. In some embodiments, nearby users can instruct or demonstrate actions to the robotic device. In some embodiments, the robotic device may be configured to seek user intervention at a particular point in time during the execution of the action. For example, when a robot device cannot see specific information about the robot device itself and / or the environment around the robot device, the robot device completes an action or navigates to a specific location. When the orbit of the robot device cannot be determined, such as when the robot device is pre-programmed to prompt for user input (eg, during learning using an interactive learning template, as described further below). , User intervention can be requested. In some embodiments, the robotic device can request feedback from the user at specific points during learning and / or execution. For example, what information should the robot device collect (eg, information related to operating or transporting elements, information related to the surrounding environment) and / or when information should be collected (eg, keyframes being demonstrated). You can urge the user to specify the timing). Alternatively or additionally, the robot device may require the user to tag the past or present behavior of the robot device as a good or bad example of the action in a particular context. The robot device may be configured to use this information to improve future execution of the action in that particular context.

Systems and devices
[0064] FIG. 2 schematically shows a robot device 200 according to some embodiments. The robot device 200 includes a control unit 202, a user interface 240, at least one operating element 250, and at least one sensor 270. Additionally, in some embodiments, the robotic device 200 optionally comprises at least one transport element 260. The control unit 202 includes a memory 220, a storage 230, a processor 204, a graphics processor 205, a system bus 206, and at least one input / output interface (“I / O interface”) 208. The memory 220 includes, for example, a random access memory (RAM), a memory buffer, a hard drive, a database, an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), and a read-only memory (ROM). ) And / or others. In some embodiments, the memory 220 stores instructions that cause the processor 204 to perform modules, processes, and / or functions related to environment sensing or scanning, skill learning, and / or skill execution. The storage 230 may be, for example, a hard drive, a database, cloud storage, a network-attached storage device, or another data storage device. In some embodiments, the storage 230 stores sensor data including state information about, for example, one or more components of the robotic device 200 (eg, operating element 250), learned models, marker position information, and the like. be able to.

[0065] Processor 204 of control unit 202 may be any suitable processing device configured to inspect the environment, learn skills, and / or activate and / or perform functions related to skill execution. good. For example, processor 204 performs a skill by generating a model for the skill based on sensor information, or by using the model to generate a trajectory to perform the skill, as further described herein. It may be configured as follows. More specifically, processor 204 may be configured to execute modules, functions, and / or processes. In some embodiments, the processor 204 is a general purpose processor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), and / or the like. May be good.

[0066] The graphics processor 205 may be any suitable processing device configured to activate and / or perform one or more display functions (eg, functions associated with the display device 242). In some embodiments, the graphics processor 205 may be a low power graphics processing unit such as, for example, a dedicated graphics card or an integrated graphics processing unit.

[0067] System bus 206 may be any suitable component that allows other components of processor 204, memory 220, storage 230, and / or control unit 202 to communicate with each other. One or more I / O interfaces 208 connected to system bus 206 are internal components of control unit 202 (eg, processor 204, memory 220, storage 230) and user interface 240, one or more operating elements. It may be any suitable component that allows communication with external input / output devices such as 250, one or more transport elements 260, and one or more sensors 270.

[0068] The user interface 240 is configured to receive input and send output to other devices and / or users operating the device (eg, a user operating the robot device 200). May include. For example, the user interface 240 receives a display device 242 (eg, display, touch screen, etc.), an audio device 244 (eg, microphone, speaker), and optionally an input and / or outputs to the user. It may include one or more additional input / output devices (“I / O devices”) 246 configured to generate.

[0069] The one or more operating elements 250 may be any suitable component capable of manipulating and / or interacting with stationary and / or moving objects, including, for example, humans. In some embodiments, the one or more operating elements 250 can provide translation along one or more axes and / or rotation about one or more axes. May include a plurality of segments coupled to each other by. The one or more operating elements 250 may optionally include end effectors that can engage and / or otherwise interact with objects in the environment. For example, the operating element may include a gripping mechanism capable of releasably engaging (eg, gripping) an object in the environment for picking up and / or transporting the object. Other examples of end effectors include, for example, one or more vacuum engagement mechanisms, one or more magnetic engagement mechanisms, one or more suction mechanisms, and / or combinations thereof. In some embodiments, the one or more operating elements 250 can be stowed within the housing of the robot device 200 when not in use to reduce the dimensions of the robot device. You may. In some embodiments, the one or more operating elements 250 are heads or other humanoids configured to interact with one or more objects in the environment, including the environment and / or humans. It may include components. In some embodiments, the one or more operating elements 250 may include a transport element or base (eg, one or more transport elements 260). A detailed diagram of an example of the operation element is shown in FIG.

[0070] The one or more transport elements 260 may be any suitable component configured for movement, such as wheels or trucks. One or more transport elements 260 may be provided at the base of the robot device 200 to allow the robot device 200 to move around in the environment. For example, the robot device 200 may include a plurality of wheels that allow the robot device 200 to navigate, for example, in a building such as a hospital. One or more transport elements 260 are designed and / or dimensioned to facilitate movement through narrow spaces and / or restricted spaces (eg, small aisles, corridors, small rooms such as storage). can do. In some embodiments, one or more transport elements 260 may be rotatable about an axis and / or may be movable relative to each other (eg, along a track). good. In some embodiments, the transport element 260 can be stowed within the base of the robot device 200 when not in use to reduce the dimensions of the robot device. You may. In some embodiments, the one or more transport elements 260 may be part of, or part of, an operating element (eg, one or more operating elements 250). ..

[0071] The sensor 270 may be any suitable component that allows the robot device 200 to capture information about the environment and / or objects in the environment around the robot device 200. One or more sensors 270 may include, for example, an imaging device (eg, a camera such as an RGB-D (red-green-blue-depth) camera or a webcam), an audio device (eg, a microphone), an optical sensor (eg, eg). , Light detection ranging (ie, rider) sensor, color detection sensor), proprioception sensor, position sensor, tactile sensor, force or torque sensor, temperature sensor, pressure sensor, motion sensor, sound detector, etc. may be included. For example, one or more sensors 270 may include at least one imaging device, such as a camera, for capturing visual information about the object and the environment around the robot device 200. In some embodiments, the one or more sensors 270 may include tactile sensors (eg, sensors capable of transmitting force, vibration, contact, and other non-visual information to the robot device 200).

[0072] In some embodiments, the robotic device 200 may have humanoid features (eg, head, torso, arms, legs, and / or base). For example, the robot device 200 may include eyes, nose, mouth, and a face with other humanoid features. These humanoid features can form part of one or more manipulation elements and / or may be part of one or more manipulation elements. Although not schematically shown, the robot device 200 connects, operates, and / or drives different parts of the actuator, motor, coupler, connector, power supply (eg, built-in battery), and / or robot device 200. Other components may also be included.

[0073] FIG. 3 is a block diagram schematically showing the control unit 302 according to some embodiments. The control unit 302 may include components similar to the control unit 202 and may be structurally and / or functionally similar to the control unit 202. For example, control unit 302 may be structurally and / or functionally similar to processor 204, memory 220, one or more I / O interfaces 208, system bus 206, and storage 230, respectively, graphic processor 305. , Memory 320, one or more I / O interfaces 308, system bus 306, and storage 330.

[0074] Memory 320 causes processor 304 to perform modules, processes, and / or functions shown as active sensing 322, marker identification 324, learning and model generation 326, orbit generation and execution 328, and success monitoring 329. Save instructions that can be. Active sensing 322, marker identification 324, learning and model generation 326, orbit generation and execution 328, and success monitoring 329 were tied to hardware components (eg, sensors, operating elements, I / O devices, processors, etc.). It can be implemented as one or more programs and / or applications. Active sensing 322, marker identification 324, learning and model generation 326, orbit generation and execution 328, and success monitoring 329 may be implemented by a single robot device or multiple robot devices. For example, the robot device may be configured to implement active sensing 322, marker identification 324, and orbit generation and execution 328. As another example, the robotic device may be configured to implement active sensing 322, marker identification 324, optionally learning and model generation 326, and orbit generation and execution 328. As another example, the robotic device may be configured to implement active sensing 322, marker identification 324, orbit generation and execution 328, and optionally success monitoring 329. Although not shown, the memory 320 may also store programs and / or applications related to the operating system and general robotic operations (eg, power management, memory allocation, etc.).

[0075] Storage 330 stores information related to skill learning and / or execution. The storage 330 stores, for example, state information 331, one or more models 334, object information 340, and machine learning library 342. The state information 331 may include information about the state of the robot device (eg, robot device 200) and / or the environment in which the robot device is operating (eg, a building (eg, a hospital, etc.)). In some embodiments, the state information 331 may indicate the location of the robotic device in an environment, such as a room, floor, closed space, and the like. For example, the state information 331 may include a map 332 of the environment and may indicate the location of the robot device in the map 332. The state information 331 may also include one or more locations (or a marker representing and / or an object associated with) one or more objects in the environment (eg, in map 332). Therefore, the state information 331 can identify the location of the robot device with respect to one or more objects. Objects can include any type of physical object located in the environment, including objects that define a space or opening (eg, a surface or wall that defines a doorway). The object may be stationary or movable. Examples of objects in an environment, such as a hospital, include equipment, fixtures, equipment, appliances, furniture, and / or humans (eg, nurses, doctors, patients, etc.).

[0076] In some embodiments, the state information 331 may include a representation or map of the environment as shown in FIG. The map of the environment may include, for example, a navigation layer, a static semantic layer, a social layer, and a dynamic layer. For example, information learned by a robot device from demonstrations, user inputs, sensed sensor information, etc. can be sent to multiple layers of different maps, which will be referred to later by this robot device (and / or other robot devices). Can be organized for. For example, a robotic device, as further described herein, depends on information learned about different objects (eg, doors) in the environment and has different behaviors (eg, before passing through a doorway). You can decide how to do the arbitration between (waiting for the door of the closed door to open, asking for help to open the door before going through the closed door).

[0077] Object information 340 may include information related to one or more physical objects in the environment. For example, object information may include information that identifies or quantifies different features of an object, such as location, color, shape, and surface features. The object information can also identify codes, symbols, and other markers associated with the physical object (eg, quick response, ie, "QR" code, barcode, tag, etc.). In some embodiments, the object information may include information that characterizes an object in the environment (eg, the doorway or passage is closed, the door handle is a type of door handle, etc.). Object information can allow the control unit 302 to identify one or more physical objects in the environment.

[0078] The state information 331 and / or the object information 340 may be an example of environmental constraints as used and described herein. Environmental constraints may include information about the environment that can be presented or seen in various dimensions. For example, environmental constraints can vary by location, time, social interaction, specific context, and so on. Robotic devices can learn environmental constraints from user input, skill demonstrations, environmental interactions, self-search, skill execution, and so on.

[0079] The machine learning library 342 may include multiple modules, processes, and / or functions associated with different algorithms for machine learning and / or model generation of different skills. In some embodiments, the machine learning library may include methods such as a hidden Markov model (ie, "HMM"). An example of Python's existing machine learning library is scikit-learn. The storage 330 may also include, for example, additional software libraries related to robot simulation, motion planning and control, kinematics teaching and perception, and the like.

[0080] One or more skill models 334 are models generated to perform different actions and represent skills learned by a robotic device. In some embodiments, each model 334 is associated with a set of markers associated with different physical objects in an environment. The marker information 335 can indicate which marker is associated with a particular model 334. Each model 334 can also be associated with sensory information 336 collected, for example, by one or more sensors in a robotic device during kinesthetic teaching and / or other demonstrations of the skill. The sensory information 336 may optionally include operating element information 337 associated with the operating element of the robot device when the robot device performs an action during the demonstration. The operating element information 337 includes, for example, the position and configuration of the joint, the position and configuration of the end effector, the position and configuration of the base or the transport element, and / or the force and torque acting on the joint, the end effector, the transport element, and the like. It can be. The operating element information 337 may be recorded at a specific time point (eg, keyframe) during the skill demonstration and / or execution, or throughout the skill demonstration and / or execution. Sensory information 336 may also include information associated with the environment in which the skill is demonstrated and / or performed (eg, the location of a marker within the environment). In some embodiments, each model 334 may be associated with a success criterion 339. Success criteria 339 can be used to monitor skill performance. In some embodiments, the success criterion 339 may include information associated with visual and tactile data sensed using one or more sensors (eg, camera, force / torque sensor, etc.). Success criteria 339 are, for example, visually detecting the movement of an object, detecting the force acting on a component of a robot device (eg, the weight of the object), and engaging the component of the robot device with the object (eg, the weight of the object). For example, it may be linked to detecting (change in pressure or force acting on a surface). Examples of using tactile data in robotic learning of operational skills are available herein at http://ieeexplore.ieee.org/document/7463165/, 2016 IEEE Haptics Symposium (HAPTICS) (Philadelphia, It is described in a paper entitled "Learning Haptic Affordances from Demonstration and Human-Guided Exploration" published by Chu et al., Published in PA, 2016, pp.119-125). An example of using visual data for robotic learning of operational skills is Autonomous, available from https://doi.org/10.1007/s10514-015-9448-x ("Akgun Paper"), incorporated herein by reference. It is described in a paper entitled "Simultaneously Learning Actions and Goals from Demonstration" written by Akgun et al., Published in Robots (Volume 40, Issue 2, February 2016, pp.211-227).

[0081] Optionally, the sensory information 336 may include transport element information 338. The transport element information 338 is associated with the movement (eg, navigation through a doorway, transport of an object, etc.) of the robot device's transport element (eg, one or more transport elements 260) as the robot device undergoes a skill demonstration. May be done. Transport element information 338 may be recorded at a particular point in time during the demonstration and / or execution of the skill (such as a keyframe associated with the skill), or during the demonstration and / or execution of the skill.

[0082] In some embodiments, an initial set of environmental constraints (eg, state information 331, object information 340, etc.) and / or skills (eg, one or more models 334) is, for example, remote management. May be provided to the robotic device by a person or supervisor. The robotic device may be environmentally constrained and / or based on the interaction, demonstration, etc. of the robotic device itself with the environment or humans within the environment (eg, patients, nurses, doctors, etc.) and / or by additional user input. Skill knowledge can be adapted and / or increased. Alternatively or additionally, the robot supervisor may use the robot device, or one or more other robot devices (eg, similar or, one or more in the same environment (eg, hospital)). Robots regarding environmental constraints and / or skills based on new information collected by (robot devices) and / or provided to robot supervisors by outside parties (eg, suppliers, managers, manufacturers, etc.) You can update your device knowledge. Such updates may be provided periodically and / or continuously as new information about the environment or skills is provided to the robot device and / or the robot supervisor.

[0083] FIG. 4 schematically shows an operating element 350 according to some embodiments. The operating element 350 may form part of a robotic device such as, for example, the robotic device 102 and / or 200. The operating element 350 may be mounted as an arm comprising two or more segments 352 coupled together by a joint 354. The joint 354 may allow one or more degrees of freedom. For example, the joint 354 can provide translation along one or more axes and / or rotation about one or more axes. In certain embodiments, the operating element 350 may have seven degrees of freedom provided by the joint 354. Although four segments 352 and four joints 354 are shown in FIG. 4, one of skill in the art will appreciate that the operating elements may include different numbers of segments and / or joints.

[0084] The operating element 350 includes an end effector 356 that can be used to interact with objects in the environment. For example, the end effector 356 can be used to engage with and / or manipulate different objects. Alternatively or additionally, end effectors 356 can be used to interact with mobile or dynamic objects, including, for example, humans. In some embodiments, the end effector 356 may be a gripper capable of releasably engaging or gripping one or more objects. For example, an end effector 356 implemented as a gripper can pick up an object and move it from a first location (eg, storage) to a second location (eg, office, room, etc.).

[0085] The plurality of sensors 353, 355, 357, and 358 may be arranged on a plurality of different components of the operating element 350 (eg, segments 352, joints 354, and / or end effectors 356). Sensors 353, 355, 357, and 358 may be configured to measure sensory information, including environmental information and / or operational element information. Examples of sensors include position encoders, torque and / or force sensors, contact and / or tactile sensors, imaging devices such as cameras, temperature sensors, pressure sensors, optical sensors and the like. In some embodiments, the sensor 353 located on the segment 352 may be a camera configured to capture visual information about the environment. In some embodiments, the sensor 353 located on the segment 352 is an accelerometer configured to allow measurement of the acceleration of the segment 352 and / or calculation of the moving speed and / or position of the segment 352. But it may be. In some embodiments, the sensor 355 placed on the joint 354 may be a position encoder configured to measure the position and / or configuration of the joint 354. In some embodiments, the sensor 355 located on the joint 354 may be a force or torque sensor configured to measure the force or torque exerted on the joint 354. In some embodiments, the sensor 358 located on the end effector 356 may be a position encoder and / or a force or torque sensor. In some embodiments, the sensor 357 located on the end effector 356 may be a contact or tactile sensor configured to measure the engagement of the end effector 356 with an object in the environment. Alternatively or additionally, one or more of the sensors 353, 355, 357, and 358 may be configured to record information about one or more objects and / or markers in the environment. For example, the sensor 358 placed on the end effector 356 may be configured to track the location of the object in the environment and / or the position of the object relative to the end effector 356. In some embodiments, one or more of the sensors 353, 355, 357, and 358 can also track whether an object, such as a human, has moved in the environment. Sensors 353, 355, 357, and 358 provide sensory information recorded by these sensors to computational devices (eg, on-board control units (eg, control units 202 and / or 302, etc.)) located in the robot device. Alternatively, sensors 353, 355, 357, and 358 can send sensory information to a remote computing device (eg, a server (eg, server 120, etc.)).

[0086] The operating element 350 is a coupling element 359 that allows the operating element 350 to be detachably coupled to a robot device, such as one of the robot devices described herein. May be optionally included. In some embodiments, the operating element 350 may be coupled in place on the robotic device and / or at multiple locations on the robotic device (eg, on the body of the robotic device, as shown in FIG. 5). It may be possible to connect to the right side or the left side). The coupling element 359 may include any type of mechanism capable of coupling the operating element 350 to the robotic device, such as, for example, mechanical mechanisms (eg, fasteners, latches, mounts), magnetic mechanisms, friction fittings, and the like. ..

[0087] FIG. 20 is a block diagram schematically showing a control unit 1702 of a robot system according to some embodiments. The control unit 1702 may include components similar to other control units described herein (eg, control units 202 and / or 302). For example, control unit 1702 may be structurally and / or functionally similar to the processors, memory, one or more I / O interfaces, system buses, and storage of control units 202 and / or 302, respectively. Includes graphics processor 1705, memory 1720, one or more I / O interfaces 1708, system bus 1706, and storage 1730. The control unit 1702 may be located on the robot device and / or at a remote server connected to one or more robot devices.

[0088] Memory 1720 includes active sensing 1722, skill and behavior learning 1724, and action execution 1726, optionally including resource arbitration 1728, data tracking and analysis 1758, modules, processes. , And / or stored instructions that could cause the processor 1704 to perform the function. Active sensing 1722, skill and behavior learning 1724, action execution 1726, resource arbitration 1728, and data tracking and analysis 1758 are tied to hardware components (eg, sensors, operating elements, I / O devices, processors, etc.). It can be implemented as one or more programs and / or applications. Active sensing 1722, skill and behavior learning 1724, action execution 1726, resource arbitration 1728, and data tracking and analysis 1758 may be implemented by a single robot device or multiple robot devices. In certain embodiments, active sensing 1722 may include active sensing or scanning of the environment, as described herein. In other embodiments, the Active Sensing 1722 is an active scanning of an environment and / or an environment, one or more objects within that environment (including, for example, humans within the environment), and / or a robotic device or. It may include detection or sensing of information related to one or more states related to the system.

[0089] Similar to storage 330, storage 1730 stores information related to an environment and / or objects within that environment, as well as learning and / or execution of skills (eg, tasks and / or social behavior). .. The storage 1730 stores, for example, state information 1732, one or more skill models 1734, object information 1740, machine learning libraries 1742, and / or one or more environmental constraints 1754. Optionally, the storage 1730 may also store tracking information 1756 and / or one or more arbitration algorithms 1758.

[0090] The state information 1732 includes information about the state of the robot device (such as any of the robot devices described herein) and / or the state of the environment in which the robot device is operating. In some embodiments, state information 1732 may include a map of the environment, along with additional static and / or dynamic information related to objects in the environment. For example, state information 1732 provides a navigation map of a building with static and / or dynamic information about objects (eg, equipment, equipment, etc.) in the building, as well as human and / or social conditions within the building. May include with social contextual information related to. FIG. 19 provides an exemplary map of a building or a schematic diagram of representation 1600. Representation 1600 includes a navigation layer 1610, a static semantic layer 1620, a social layer 1630, and a dynamic layer 1640. The navigation layer 1610 has one or more walls 1614, one or more stairs 1616, and one or more floors with other elements incorporated into the building (eg, passages, openings, boundaries). Provided is a general layout or map of a building from which the number of 1612 can be identified. The static semantic layer 1620 identifies objects and / or spaces within a building, such as one or more rooms 1622, one or more objects 1624, or one or more doors 1626. The static semantic layer 1620 can identify which one or more rooms 1622 or other spaces the robot device can or cannot access. In some embodiments, the static semantic layer 1620 can provide a three-dimensional map of an object located within a building. The social layer 1630 provides social context information 1632. Social context information 1632 includes information related to humans in a building, such as past interactions between one or more robotic devices and one or more humans. Social context information 1632 can be used to track interactions between one or more robotic devices and one or more humans, and these interactions can be used with one robotic device and one person. Existing models of skills involved in one or more interactions can be generated and / or adapted. For example, social contextual information 1632 can indicate that a person is usually in a particular location, so that a robotic device with knowledge of that information can be near that person's location. Can adapt to the execution of skills that require movement. Dynamic layer 1640 provides information about one or more objects and other elements within a building that can move and / or change over time. For example, the dynamic layer 1640 can track one or more movements 1644 and / or one or more changes 1646 associated with one or more objects 1642. In certain embodiments, the dynamic layer 1640 can monitor the expiration date of an object 1642 and identify when the object 1642 has expired.

[0091] Representation 1600 may be available, for example, to control unit 1602 of a robotic device and / or may be managed by control unit 1602. The control unit 1602 may include components similar to other control units described herein (eg, control units 202, 302, and / or 1702). The control unit 1602 stores the representation 1600 and state information 1604 containing information related to one or more robotic devices (eg, the configuration of elements of a robotic device, the location of that robotic device in a building, etc.). It may include storage (similar to other storage elements described herein), such as storage 330 and / or 1730. The control unit 1602 may be located on the robot device and / or at a remote server connected to one or more robot devices. As one or more robotic devices collect information about their surrounding environment, one or more robotic devices may update and retain state information 1604 containing information related to building representation 1600. It may be configured in.

[0092] In some embodiments, the control unit 1602 may optionally also include a data tracking and analysis element 1606. The data tracking and analysis element 1606, for example, performs data tracking and / or analysis of information collected by one or more robotic devices (eg, information contained within representation 1600 and / or other state information 1604). It may be a computational element configured as such (for example, a processor). For example, the data tracking and analysis element 1606 is configured to manage inventory (eg, expiration tracking, monitoring and recording of inventory usage, ordering new inventory, analyzing and recommending new inventory, etc.). You may. In hospital situations, the data tracking and analysis element 1606 can control the use and / or maintenance of medical supplies and / or equipment. In some embodiments, the data tracking and analysis element 1606 can generate aggregated data displays (eg, reports, charts, etc.) using which they comply with formal laws, rules, and / or standards. can do. In some embodiments, the data tracking and analysis element 1606 may be configured to analyze human-related information such as patients in hospitals. For example, a robotic device may collect information about one or more patients in a hospital and perform analysis and / or summarization for use in various functions, including, for example, diagnostic testing and / or screening. The information may be configured to be routed to the data tracking and analysis element 1606. In some embodiments, the data tracking and analysis element 1606 accesses information from third-party systems (eg, hospital electronic medical records, security system data, insurance data, vendor data, etc.) and / Or obtain such information and use and / or use that data with or without the data collected by one or more robotic devices or without the data collected by one or more robotic devices. By analyzing, it may be configured to perform data tracking and / or analysis functions.

As shown in FIG. 20, one or more skill models 1734 can be used to perform learning and / or execution of various actions or skills, including tasks and / or behaviors. Is. One or more models 1734 may resemble one or more models 334 as described herein with reference to FIG. For example, one or more models 1734 execute one or more objects involved in the execution of a skill (eg, an object operated by a robot device, an object with which the robot device interacts during the execution of the skill, the skill). However, it may include information related to the object) considered by the robot device. As mentioned above, examples of objects include stationary and / or movable objects such as surfaces that define equipment, equipment, humans, and / or openings (eg, doorways). Information related to one or more objects may include, for example, a marker that identifies the object and / or the features of the object. Additional or alternative, one or more models 1734 may include sensory information collected by the robotic device, for example, during skill learning and / or execution, or active sensing. As described above, sensory information is information related to one or more components (eg, operating elements, transport elements) of the robot device when learning and / or executing the skill, and / or the skill learns and / or learns. Or it may include information related to the environment in which it is performed (eg, the location of one or more objects in the environment, social contextual information, etc.). In some embodiments, the model 1734 is associated with success criteria such as visual and / or tactile data sensed using one or more sensors of the robotic device, indicating successful execution of the skill, for example. May be good.

[0094] Object information 1740 may include information related to one or more physical objects (eg, location, color, shape, surface features, and / or identification code). The machine learning library 1742 may include multiple modules, processes, and / or functions related to model generation of different algorithms and / or different skills of machine learning.

[0095] Environmental constraints 1754 include information related to objects and / or conditions in the environment that may limit the operation of robotic devices in the environment. For example, environmental constraints 1754 can be information related to the size, composition, and / or location of objects in the environment (eg, equipment containers, rooms, doorways, etc.) and / or specific areas (eg, rooms, passageways, etc.). ) May contain information indicating that access is restricted. Environmental constraints 1754 can affect the learning and / or execution of one or more actions in the environment. As such, the environmental constraint 1754 can be part of each model of skill performed within the context that includes the environmental constraint.

[0096] Tracking information 1756 performs information related to representations of the environment (eg, representation 1600) and / or data tracking and analysis elements such as, for example, data tracking and analysis element 1606, or data tracking and analysis 1758. Includes information obtained from third-party systems that is tracked and / or analyzed by processor 1704 (eg, hospital electronic medical records, security system data, insurance data, vendor data, etc.). Examples of follow-up information 1756 include inventory data, supply chain data, point-in-time data, and / or patient data, as well as aggregated data aggregated from such data.

[0097] One or more arbitration algorithms 1758 include algorithms for making arbitrations or choices between different actions to be performed (eg, how to use different resources or components of a robotic device). The arbitration algorithm 1758 may be a rule or, in some embodiments, may be learned as further described with reference to FIG. These algorithms can be used by the robot device to choose from different actions if the robot device has multiple resources and / or objectives to manage. For example, a robot device operating in an unstructured environment and / or a dynamic environment (eg, an environment containing humans) may require several different behaviors or actions from the robot device at any point in time. Can be exposed to conditions. In such cases, the robotic device may assign different priorities to various actions based on pre-defined rules (eg, emotional state, environmental constraints, socially defined constraints, etc.) 1 It may be configured to choose from a plurality of different actions based on one or more arbitration algorithms 1758. In one embodiment, the arbitration algorithm 1758 can assign different scores or values to multiple actions based on the information collected by the robotic device regarding the surrounding environment, current state, and / or other factors.

[0098] Similar to one or more I / O interfaces 208 and / or 308, one or more I / O interfaces 1708 may include internal components of control unit 1702 and user interfaces, operating elements, transport elements. And / or any suitable component that allows communication with external devices such as computing devices. One or more I / O interfaces 1708 may include a network interface 1760 capable of connecting the control unit 1702 to a network (eg, network 105 as shown in FIG. 1). The network interface 1760 monitors and / or controls the control unit 1702 (which may be located on the robot device or on another network device that communicates with one or more robot devices) and one or more robot devices. To do so, it allows communication with remote devices such as computing devices that can be used by robot supervisors. The network interface 1760 may be configured to provide wireless and / or wired connections to the network.

[0099] FIG. 5 schematically shows a robot device 400 according to some embodiments. The robot device 400 includes a head 480, a body 488, and a base 486. The head 480 may be connected to the torso 488 by segments 482 and one or more joints (not shown). The segments 482 may be movable and / or flexible to allow the head 480 to move relative to the torso 488. The head 480, segment 482, etc. can be examples of one or more operating elements and have similar functionality and / or structure to one or more other operating elements described herein. Can include.

[0100] Head 480 includes one or more imaging devices 472 and / or other sensors 470. The imaging device 472 and / or other sensors 470 (eg, rider sensor, motion sensor, etc.) allow the robot device 400 to scan the environment and acquire a representation of that environment (eg, a visual representation or other semantic representation). Can be made possible. In some embodiments, the imaging device 472 may be a camera. In some embodiments, the imaging device 472 may be mobile so that it can be used to focus on different areas of the environment around the robot device 400. The imaging device 472 and / or another sensor 470 can collect sensory information and send the sensory information to a computing device or processor (eg, control unit 202 or 302, etc.) mounted on the robot device 400. In some embodiments, the head 480 of the robotic device 400 has a human shape and may include one or more human features (eg, eyes, nose, mouth, ears, etc.). In such embodiments, the imaging device 472 and / or the other sensor 470 may be implemented as one or more human features. For example, the imaging device 472 may be mounted as an eye on the head 480.

[0101] In some embodiments, the robotic device 400 uses an imaging device 472 and / or other sensor 470 to provide information about an object in the environment (eg, physical structure, device, article, human, etc.). You can ask and scan the environment. The robot device 400 can engage in active sensing, or the robot device 400 can initiate sensing or scanning in response to a trigger (eg, user input, detection event or change in the environment). can.

[0102] In some embodiments, the robotic device 400 can engage in adaptive sensing capable of performing sensing based on stored knowledge and / or user input. For example, the robot device 400 can identify an area in an environment in which an object should be sought and scanned based on prior information that the robot device 400 has about the object. With reference to FIG. 6A, the robot device 400 can scan a scene (eg, an area with a room) and obtain a representation 500 of that scene. In representation 500, the robot device 400 identifies that the first object 550 is located in areas 510 and the second object 560 is located in areas 510 and 530. The robot device 400 can use that information to locate objects 550 and 560 in future scans, so that the robot device 400 can use it in a map of the environment it has stored internally. The location of objects 550 and 560 can be preserved. For example, when the robot device 400 returns to the scene and scans the scene for the second time, the robot device 400 may acquire different views of the scene, as shown in FIG. 6B. When performing this second scan, the robot device 400 can acquire the representation 502 of the scene. To locate objects 550 and 560 in representation 502, robotic device 400 may refer to information previously stored by robotic device 400 with respect to the location of these objects when acquiring representation 500 of the scene. can. The robot device 400 can take into account that the location of the robot device 400 in the environment may have changed and recognize that objects 550 and 560 may be located in different areas of representation 502. can. Based on this information, the robot device 400 can know that it sees the area 510 for the object 550, but sees the areas 520 and 540 for the object 560. By using previously stored information about the location of objects 550 and 560, the robot device 400 seeks objects 550 and 560 (eg, by zooming in and slowly moving the camera within the area). ) It can automatically identify areas that should be carefully scanned.

[0103] In some embodiments, it can also be seen that the robotic device 400 performs more careful sensing or scanning of different areas of the scene based on human input. For example, a human can indicate to a robot device 400 that a particular area of a scene contains one or more objects of interest, which the robot device 400 may use to identify those objects. Areas can be scanned more carefully. In such an embodiment, the robot device 400 may include a display with a keyboard or other input device and / or an input / output device 440 such as a touch screen, as schematically shown in FIG.

[0104] In some embodiments, the robotic device 400 can scan the environment and identify that an object, such as a human being, is moving within the environment. For example, as shown in FIGS. 7A and 7B, the object 650 may remain stationary while the object 660 may be moving in the environment. FIG. 7A shows a scene representation 600 showing that the object 660 is in areas 610 and 630, and FIG. 7B shows a scene representation 602 showing that the object 660 is in areas 620 and 640. In both representations 600 and 602, the object 650 may stay in the same place in the area 610. The robot device 400 can identify that the object 660 has moved in the scene and adjust the action of the robot device 400 accordingly. For example, if the robot device 400 was to interact with an object 660, the robot device 400 would change its trajectory (eg, move closer to the object 660) and / or with an operating element or object 660. You can change the trajectory of other components that are configured to interact. Alternatively or additionally, if the robot device 400 was to interact with the object 650 (and / or another object in the scene), the robot device 400 plans a course for interacting with the object 650. In the meantime, the movement of the object 660 can be taken into account. In some embodiments, the robot device 400 can engage in active sensing so that the actions of the robot device 400 can be coordinated in near real time.

[0105] As schematically shown in FIG. 5, the base 486 may optionally include one or more transport elements mounted as wheels 460. Wheels 460 allow the robot device 400 to move around in an environment (eg, a hospital). The robot device 400 also includes at least one operating element mounted as an arm 450. The arm 450 may be structurally and / or functionally similar to other operating elements described herein (eg, operating element 350). The arm 450 may be fixedly attached to the body 488 of the robot device 400, or optionally, the operating element 450 may attach to the coupling portion 484 of the robot device 400 (eg, coupling element). 359) may be detachably coupled to the body 488. The coupling portion 484 engages the coupling element 359 and the on-board computing device (eg, control unit 202 or 302) powers and / or controls the components of the arm 450 and operates. To provide an electrical connection between the arm 450 and the on-board computing device so that it can receive information collected by sensors located on the element 450 (eg, sensors 353, 355, 357, and 358). It may be configured in.

[0106] Optionally, the robot device 400 may also include one or more additional sensors 470 located on a segment 482, a body 488, a base 486, and / or other parts of the robot device 400. .. The one or more sensors 470 may be, for example, an imaging device, a force or torque sensor, a motion sensor, an optical sensor, a pressure sensor, and / or a temperature sensor. The sensor 470 can allow the robot device 400 to capture visual and non-visual information about the environment.

Method
[0107] FIGS. 8-11 are flow diagrams illustrating method 700, according to some embodiments, which may be performed by a robotic system (eg, robotic system 100) comprising one or more robotic devices. For example, all or part of the method 700 may be performed by a single robotic device, such as one of the robotic devices described herein. Alternatively, all of the method 700 may be performed sequentially by a plurality of robotic devices, each of which in turn performs a portion of the method 700. Alternatively, all or part of the method 700 may be performed simultaneously by a plurality of robotic devices.

[0108] As shown in FIG. 8, the robot device can scan the environment and obtain a representation of the environment at 702. The robot device can scan the environment using one or more sensors (eg, one or more sensors 270 or 470, and / or one or more imaging devices 472). In some embodiments, the robotic device can scan the environment with a mobile camera, and the position and / or focus of the camera is adjusted to capture multiple areas within the scene of the environment. Can be done. At 704, based on the information collected during sensing, the robot device can analyze the data to identify one or more markers within the representation of the captured environment. The marker may be associated with one or more objects in the scene marked with a visual marker or a reference marker (eg, a visible marker such as a QR code®, barcode, tag, etc.). .. Alternatively or additionally, the robot device uses object information (eg, object information 340) stored in a memory (eg, storage 330) on the robot device to recognize one or more objects in the environment. The marker associated with the object can be identified. The object information may include information indicating a plurality of different features of the object, such as location, color, shape, and surface features. In certain embodiments, the object information can be organized as a numerical value representing a plurality of different features of the object, which may be referred to as a feature space.

[0109] After identifying one or more markers, at 706, the robotic device can optionally present one or more markers within the representation of the environment. In some embodiments, the representation of the environment may be a visual representation, for example, an expanded view of the environment. In such an embodiment, the robot device can display a visual representation of the environment, for example on a display screen, and display the location of one or more markers within the visual representation of the environment. Alternatively or additionally, the representation of the environment may be a semantic representation of the environment having the location of one or more markers indicated by the semantic markers within the environment.

[0110] In some embodiments, the robotic device can present a representation of the environment to the user, having one or more markers, optionally at 708, eg, a user interface or other type. I / O devices can be used to encourage users to approve or reject one or more markers in a representation of the environment. If the user does not approve one or more markers (708: NO), method 700 returns to 702 and the robot device can obtain a second representation of the environment by rescanning the environment. .. If the user approves one or more markers (708: YES), method 700 proceeds to 708, where the robot device stores information (eg, location, features, etc.) associated with the marker in memory (eg, location, features, etc.). , Storage 330). For example, the robot device can store the location of one or more markers in an internal map of the environment (eg, map 332).

[0111] In some embodiments, the robotic device identifies one or more markers in 704 and, as is, in 710, 1 without prompting the user to approve the one or more markers. The location of one or more markers and / or other information associated with the markers may be initiated. In such an embodiment, the robot device can analyze the location of one or more markers before storing those locations. For example, the robot device was previously stored with respect to the location of one or more markers (eg, acquired during a previous scan of the environment and / or entered into the robot device by the user or computing device). By having information and comparing the location of one or more markers with previously stored information, accuracy checks and / or changes in marker location can be identified. Specifically, if previously stored information indicates that a particular marker should be located at a different location than the location identified by the robot device, the robot device is an additional environment. By initiating a scan, the location of the marker can be verified before it is saved. Alternatively or additionally, the robot device can send a notification to the user that the location of the marker has changed. In such cases, the robot device can save the new location of the marker, but can also save a message indicating that the location of the marker has changed. In that case, the user or computing device can later review the message to match the change in marker location.

[0112] As shown in FIG. 9, the method 700 optionally proceeds to 712, where the robotic device is represented in the environment, eg, using a user interface or other type of I / O device. The user can be prompted to select a set of markers from one or more identified markers. At 714, the user can make a selection and the robot device can receive the selection from the user. Alternatively, in some embodiments, the robot device can automatically select a set of markers instead of prompting the user to make a selection. Robotic devices may be programmed to select markers based on certain pre-defined or learned rules and / or conditions. For example, a robotic device may select a marker associated with a particular type of object (eg, supply) during a particular time of the day or when there is less traffic in the building. Can be instructed. In the latter case, the robotic device will be less crowded in the building by actively moving through the building (eg, patrol and monitoring passages and rooms) and by sensing or scanning the environment. Can be determined. In doing so, the robotic device can know to select a marker associated with a particular object when there is less traffic in the building than most other times.

[0113] After the robot device receives a selection of a set of markers from the user and / or automatically selects a set of markers, method 700 proceeds to learn 716 skills or execute 718 skills. Can be done.

[0114] For any particular skill, it is possible for the robot device to be taught the skill before the robot device performs or performs the skill. For example, in order to acquire operational skills, the robot device may allow a user (eg, a nearby user or a remotely located robot supervisor) or another robot device to demonstrate the skill to the robot device. It can be taught by (for example, kinesthetic teaching). For example, an operating element, such as an arm of a robotic device, can move through a set of waypoints to interact with an object. As another example, the movable base of a robotic device (eg, a base with transport elements such as wheels, trucks, crawlers, etc.) is placed by the user (eg, a joystick, user interface, or other type of physical or virtual control device). Can be navigated around objects in the environment by using, or by physically guiding or moving (eg, pulling, pushing) the robot device.

[0115] In the case of kinesthetic teaching, the user can physically demonstrate the skill to the robot device. Training or teaching can be performed in mass production situations (eg, manufacturing environments, etc.) where the robotic device can be taught using an aggregate model that represents the general-purpose movement of the skill. Alternatively or additionally, teaching can be performed in the field after the robot device has been deployed (eg, in a hospital) so that the robot device can learn to perform skills in a particular field environment. In some embodiments, the robotic device can be taught in a field situation and then these by transmitting information related to the learned skill to one or more different robotic devices. Other robotic devices can also have knowledge of the taught skills when operating in the same field situation. Such embodiments may be useful when multiple robotic devices are deployed in a single field. In that case, each robot device can send and receive information to the other robot device so that the other robot device can collectively learn a set of skills related to its field environment.

[0116] In the learning mode shown in FIG. 10, the method 700 proceeds from 720 to 724, where the user can teach the skill to the robot device using the LfD teaching process. In one embodiment, the skill is defined as grasping an object at a particular location, picking it up, moving it to another location, and lowering it to another location. Can be done. In another embodiment, the skill may be involved in interacting with the environment around the robot device (eg, the door).

[0117] In 720, the user (or another robotic device) has an operating element (eg, operating element 250, 350, or 450) and / or a transport element (eg, one or more transport elements 260, 460). The operation can be instructed to the robot device including. For example, a user may interact with a particular skill (eg, interacting with a human, engaging with an object and / or manipulating an object, and / or interacting with one or more humans and / or the surrounding environment). Demonstrations related to execution can teach the operating elements of the robot device (and / or other components of the robot device (eg, transport elements)). In certain embodiments, the user can demonstrate to the robot device how to interact with the door and / or how to navigate through the door. The user can demonstrate how the robot device interacts with the door handle, for example, using one or more of its operating elements. For example, an operating element of a robotic device, such as an arm, may be guided by a series of movements with respect to the door handle (eg, entered by visual input or identified by a reference marker). Demonstration shows that the robotic device pushes open the door, eg, by moving one or more transport elements, while the door handle is turned, or after the door handle is turned, using, for example, an operating element. Can be instructed. As further described below, by using a combination of movements of one or more operating elements and one or more transport elements, for future interaction with that door and / or similar doors. You can build a model to perform a skill or behavior.

[0118] In some embodiments, the robotic device is instructed by a locally present user (eg, by a user who is physically moving the robotic device and / or giving input to the robotic device). It is possible to be done. In some embodiments, the robot device can be mentored by a user (eg, a robot supervisor) located remote from the robot device, eg, using a remote or cloud interface.

[0119] While instructing the operating element (and / or other components of the robot device) to move, the user can use the state of the operating element (eg, joint configuration, joint force and / or joint torque, end effector configuration, etc. When to capture information about the location of the end effector), another component of the robotic device, and / or the environment (eg, the location of the object associated with the selected marker, and / or other objects in the environment). Can be shown on a robot device. For example, at 722, the robot device may receive a signal from the user at a waypoint or keyframe in operation of the operating element to capture information about the operating element and / or the environment. In response to receiving a signal, at 724, the robot device can capture information about operating elements, other components of the robot device, and / or the environment at its keyframes. The operating element information may include, for example, joint configuration, joint torque, end effector position, and / or end effector torque. The environmental information may include, for example, the position of the selected marker relative to the end effector and indicate to the robot device when an object in the environment may have moved. If the operation is still ongoing (728: NO), the robot device can wait to capture information about the operating element and / or the environment at a further number of keyframes. In some embodiments, the robotic device may be programmed to capture keyframe information without receiving a signal from the user. For example, while the operating element is being moved by the user, the robot device monitors changes in the segments and joints of the operating element, and when those changes exceed a threshold, or when the orientation of the segment or joint orbit changes. At that time, the robot device can autonomously select a keyframe at that time and record information about the operating elements and / or the environment at that keyframe.

[0120] During the operation of the operating element (and / or other component of the robot device (eg, the transport element)), the robot device at 730 does not receive a signal from the user and the sensory information (eg, the operating element). , Other elements of the robotic device (eg, transport elements), and / or information about the environment) can also be recorded continuously or periodically. For example, as the user moves the operating elements throughout the demonstration, the robot device can record information about the trajectories of segments and joints, as well as their configurations. During the demonstration, the robot device may be informed about one or more environmental constraints (eg, static information about one or more objects in the environment (eg, location or size of doorways, fixture containers, etc.) and / or. Dynamic information about one or more objects in the environment (eg, the level of traffic in the room, the movement of the user around the robot device, etc.) can also be recorded. In some embodiments, the sensory information recorded by the robotic device during the demonstration may be added and / or modified to one or more layers of the map of the environment, eg, as shown in FIG. can.

[0121] In some embodiments, the robotic device may include an audio device (eg, 244) such as a microphone, and keyframe demarcation may be controlled by voice commands. For example, a user can indicate to a robot device that they are planning to demonstrate by saying "I will guide you." By saying "start here", the demonstration can start when the user points to the first keyframe. You can indicate an intermediate keyframe by saying "go here". You can then indicate the last keyframe that marks the end of the demonstration by saying "end here". A good example of demonstration teaching is provided in Akgun's paper.

[0122] In some embodiments, while demonstrating a skill to a robotic device, the user has one or more inputs to the robotic device, for example, to generalize any one or more parts of the skill. And can indicate which one or more parts of the skill are more specific to a particular environment or situation. For example, moving equipment from a first place to a second place (eg, a room) and unloading those equipment to a second place on a robot device while demonstrating to a robot device a second from the first place. The action of navigating to a location is generic, while the action of unloading equipment to a second location is unique and, in order to be carried out, environment-specific information (eg, equipment). The user can indicate to the robot device that he needs more specific tags or markers to unload). When the robot device later uses the model for that skill to move equipment between different locations, the robot device requests specific information about unloading before performing the skill, and / Or you can know to scan that particular information (eg, request and / or scan information about a particular tag to determine its location before unloading).

[0123] In some embodiments, the user exhibits a part of the skill that is generic or unique after demonstrating the skill to the robot device, and / or the user has previously indicated that it is generic or unique. You can change the skill part. In some embodiments, the robotic device can determine whether a portion of a set of skills is generic or unique after learning the set of skills. In some embodiments, the robotic device can recommend to the user that a portion of the skill set is generic or unique and request confirmation from the user. Based on the user's confirmation, the robot device may store this information for reference during learning and / or execution of future skills associated with this set of skills. Alternatively, the robotic device can automatically determine and classify multiple parts with different skills as generic or unique without user input.

[0124] When the motion or demonstration is complete (728: YES), the robot device is based on a subset of all recorded sensory information (eg, operational element information, transport element information, environmental information), and the demonstrated skill. Can generate a model of. For example, at 732, the robot device can optionally prompt the user to select features related to skill learning, eg, using a user interface or other type of I / O device, at 734. The robot device can receive a selection of features from the user. Alternatively or additionally, the robot device can know to select a particular feature to be used to generate the model, based on previous instructions from the user. For example, a robotic device can recognize that object pick-up is being demonstrated, for example based on sensory information, for example to generate a skill model based on past demonstrations of the same or different object pick-up. One or more features of sensory information (eg, joint configuration, joint torque, end effector torque) that should be included as related features can be automatically selected.

[0125] At 736, the robot device can generate a model of the skill with the selected features. The model may be generated using a stored machine learning library or algorithm (eg, machine learning library 342). In some embodiments, the model has multiple parameters such as, for example, some hidden states, a feature space (eg, features contained in the feature vector), and an emission distribution for each state modeled as a Gaussian distribution. It may be expressed as an HMM algorithm including. In some embodiments, the model is, for example, a kernel type (eg, linear, radial, polynomial, S-shaped), cost parameter or cost function, weight (eg, equality, class equilibrium), loss type or loss function (eg, equality, class equilibrium). It may be represented as a support vector machine, i.e., an "SVM" model, which may include parameters such as, for example, a hinge, a square hinge), and an answer type or question type (eg, double, primal). The model may be associated with relevant sensory information and / or other sensory information recorded by the robotic device during the demonstration of the skill. The model can also be associated with marker information indicating a set of markers manipulated during the demonstration of the skill and / or features associated with one or more physical objects associated with those markers. At 738, the robot device can store the model in memory (eg, storage 230 or 330).

[0126] In some embodiments, demonstration-related information (eg, sensory information, models, etc.) is used to add and / or add to a map or layer of representation of the environment, such as representation 1600 shown in FIG. Or you can make changes. For example, FIG. 21 shows

[0127] Optionally, at 740, the robot device can determine whether the user will perform another demonstration of the skill. If another demonstration takes place (740: YES), method 700 returns to 720, where the user (or other robot device) can instruct the robot device for additional demonstrations. When the demonstration is complete (740: NO), Method 700 can optionally return to the beginning and perform a new scan of the environment. Alternatively, in some embodiments, method 700 may be terminated.

[0128] In another embodiment, the skill may be a navigation behavior such as navigation between two locations, or navigation around and / or through an object in the environment. Similar to learning skills using operational elements, as described herein, at 720, the user (or other robotic device) has a transport element (eg, one or more transport elements 260, Navigation behavior can be taught to robotic devices including the set of 460). For example, the user uses a joystick or other control device to control the movement of a set of transport elements (and / or other components of the robot device (eg, operating elements)) and / or the robot device navigates. Such elements can be physically guided (eg, pushing or pulling a robotic device) so that they can behave. While controlling the movement of the set of transport elements (and / or other components of the robotic device), the user can use the state of the set of transport elements (eg, the angle of each transport element, the configuration of each transport element, if to each other. Information about the robot device's other components, and / or the environment (eg, the location of the robot device in the map, the location of objects in the environment, and / or boundaries), if movable, such as the spacing between transport elements. It is possible to send a signal to the robot device when it is necessary to capture sensory information such as. The user moves the set of transport elements at the moving keyframe of the set of transport elements (and / or other components of the robot device), eg, at the start, at the end, and / or in the first direction. A signal can be sent to the robot device at the transition point between moving a set of transport elements in a second direction. In response to receiving a signal from the user at 722, the robot device keyframes a snapshot of the movement, including a set of transport elements, other components of the robot device, and / or information about the environment. be able to. Alternatively or additionally, the robotic device is configured to autonomously select a moving point in time for a set of transport elements (and / or other components of the robotic device) to capture a snapshot of the movement. You may. For example, a robot device is a set of transport elements (eg, when the user takes control of the robot device (eg, by controlling the robot device that sue the control device, or by pushing the robot device)). The angle and / or configuration can be monitored and the robot device can automatically take a snapshot of that information each time it detects a change in angle and / or configuration. In some embodiments, the robotic device is such that the user continuously collects information about a set of transport elements, other components of the robotic device, and / or the environment while controlling the movement of the robotic device. It may be configured in.

[0129] Similar to learning skills using operating elements as described herein, the robotic device is of the transport element (and / or robotic device) until the movement is complete (728: YES). It is possible to continue to capture sensory information related to the movement of other components). The robotic device, at 732-734, optionally receives a selection of features related to learning navigation behavior and / or autonomously identifies the relevant features in the collected sensory information (eg, certain identification). By recognizing that the navigation skill of is being demonstrated and identifying the above features associated with learning that skill). The robot device can then generate a model for navigation behavior at 736 based on sensory information related to the relevant features and will be used later at 738 to generate the trajectory of the robot device to perform. The model and sensory information can be stored so that it can be done.

[0130] An example of navigation behavior is navigating through a doorway. Robotic devices can be configured to navigate through standard doorways in a building, while robotic devices navigate through non-standard doorways (eg, small doorways or odd-shaped doorways). Sometimes you can't. Therefore, the robot device may encourage the user to demonstrate to the robot device how the robot device can safely navigate through the doorway. The robot device can navigate to one side of the doorway and then let the user demonstrate to the robot device how to pass through the doorway to the other side. During the demonstration, the robot device may include the location of the robot device in the map and / or sensor data (eg, door boundaries, transport and / or operation element configurations, and others as described herein. Information such as (sensory information) can be passively recorded. In some embodiments, the robotic device can learn how to navigate through the doorway using an interactive learning template, as described further herein with reference to FIG.

[0131] In some embodiments, the user may be in the field near the robot device. In other embodiments, the user may be located in a remote location and by network connection to a robotic device (eg, as shown in FIG. 1 and described above), a computing device (eg, server 120, compute). The robot device may be controlled from the device 150).

[0132] In some embodiments, as described herein, the robotic device can actively scan its surrounding environment to monitor changes in the environment. Thus, during learning and / or execution, the robotic device can engage in continuous sensing or scanning of the environment and accordingly update the representation of the environment and the environmental information stored by the robotic device.

[0133] In some embodiments, the robotic device can be configured to learn socially appropriate behaviors, i.e., actions that take into account human interaction. For example, a robot device can be configured to learn operational or navigation skills that occur around humans. Robotic devices can learn socially appropriate behavior through demonstrations by human operators (eg, nearby users or remote robot supervisors). In some embodiments, the robot device behaves in an interactive context (eg, by demonstrating to the robot device how a human operator intervenes in the autonomous execution of a skill by the robot device and performs the skill). May be configured to learn. When the robot device detects intervention by a human operator (eg, a nurse pushing the robot device to one side), the robot device is controlled by the human operator to provide information related to the demonstration and the perceptual context in which the demonstration is performed. It may be configured to switch to a learning mode of passive recording. The robot device can use this information to generate a modified model of the skill and associate the model with the appropriate social context in which the robot device should perform the skill later. As an example, a robotic device can detect that a human operator has intervened in the autonomous execution of a navigation plan and can switch to a learning model controlled by the human operator in response to the detection of the intervention. .. Where humans are, how the navigation plan should be modified (for example, instead of waiting for the aisle to open, the robot device should be by the side of the aisle to allow humans to pass through). ) When demonstrated by a human operator, the robot device can record information about its surroundings and the behavior of the robot device itself. Further details regarding interactive learning are described herein with reference to FIG.

[0134] In the execution mode shown in FIG. 11, the robotic device is optionally previously generated by the robotic device in learning mode at 750, eg, using a user interface or other type of I / O device. You can encourage the user to select a model, such as a model that has been created. The robot device can receive a selection of models at 752. In some embodiments, the robot device can receive a model selection from the user, or, instead, the robot device automatically selects a model based on specific rules and / or conditions. Can be done. For example, a robot device is programmed to select a model at a particular time or during the day of the week when it is in a particular area of a building (eg, when it is in a particular room or floor). May be done. Alternatively or additionally, the robotic device may know to select a particular model based on the selected set of markers. At 754, the robot device can generate a trajectory and determine whether to move closer to the selected marker before performing the skill on the selected marker. For example, the robot device should be placed in a more desirable location (eg, closer or closer to the marker) to perform the skill based on the selected set of markers and the selected model. In addition, it is possible to determine whether the robot device should move (so that it faces the marker from a particular angle). The robot device may make this decision based on the sensory information recorded during the demonstration of the skill. For example, the robot device may recognize that the robot device has been placed closer to the marker when demonstrating the skill and may adjust its position accordingly.

[0135] If the robot device decides to move relative to the selected marker (754: YES), the robot device moves its position (eg, adjusts its location and / or orientation) at 756. ), Method 700 can return to 702, where the robotic device scans the environment again to obtain a representation of the environment. Method 700 can go through various steps back to 754. If the robot device decides not to move relative to the selected marker (754: NO), the robot device can generate, for example, an action trajectory for the operating element of the robot device.

[0136] Specifically, in 758, the robotic device has a selected set of markers and a selected model referred to herein as a "conserved marker" or a "conserved marker set". Transformation (eg, translation) with the associated set of markers (eg, one or more markers selected when the robot device learned the skill, i.e., generated the selected model). You can calculate the function that does. For example, a robotic device may use a set of first markers that are present at one or more specific locations and / or one or more orientations for a portion of the robotic device, such as an end effector. The skill can be taught, and later the robotic device will attach a set of second markers to the operating element that are present in one or more different locations and / or in one or more different orientations. You can use it to perform that skill. In such a case, the robot device may have one or more positions and / or one or more orientations of the first set of markers and one or more positions and / or 1 of the second set of markers. It is possible to calculate a conversion function that performs a conversion between one or more orientations.

[0137] In 760, the robot device uses a calculated transformation function at each keyframe recorded when the skill is taught to use some of the operating elements (eg, the operating element's end effector). The position and orientation can be converted. Optionally, at 762, the robotic device can take into account any environmental constraints, such as the characteristics of objects and / or areas within the environment, such as size, configuration, and / or location. Robotic devices can limit the movement of operating elements based on environmental constraints. For example, when the robot device recognizes that it is trying to perform a skill in the equipment room, the robot device avoids some of the operating elements coming into contact with walls or other physical structures within the equipment storage. Therefore, the size of the equipment room can be taken into account when converting the position and orientation of the operating elements. The robotic device can also take into account other environmental constraints associated with the equipment room, such as the size of the container in the equipment room, the location of the shelves in the equipment room, and so on. The robotic device is information about one or more environmental constraints prior to performing the skill in situations with one or more environmental constraints, as further described herein with reference to FIG. Is given and / or one or more environmental constraints can be taught.

[0138] Optionally, at 762, the robotic device can use inverse kinematic equations or algorithms to determine, at 762, the composition of the joints of the operating element on a keyframe-by-keyframe basis. The position and orientation of the set of end effectors and markers may be provided in the task space (eg, the Cartesian space in which the robot device is operating) and the orientation of the joints may be in the joint or constitutive space (eg, the robot device is the point). It can be provided in n-dimensional space associated with the configuration of the operating element, where n is the number of degrees of freedom of the operating element. In some embodiments, the inverse kinematics calculation is guided by joint composition information recorded when the robot device is taught the skill (eg, joint composition recorded during a teaching demonstration using operating elements). Can be done. For example, inverse kinematics calculations can be given seed values (eg, given the first guess of the calculation, or biased) using the joint composition recorded at each key frame. be. Additional conditions may also be imposed on the inverse kinematics calculation, for example, requiring that the calculated joint configuration do not deviate from the joint configuration of the adjacent key frame by more than a predetermined amount. In 764, the robotic device may plan a trajectory between joint configurations from one key frame to the next, for example in joint space, in order for the operating element to generate a complete trajectory to perform the skill. can. Optionally, the robotic device can take into account the environmental constraints as described herein.

[0139] In some embodiments, the robot device may plan the trajectory of the operating element in task space after the robot device has transformed the position and orientation of some of the operating elements (eg, end effectors). can. In such an embodiment, method 700 can proceed directly from 760 to 764.

[0140] In 766 and 768, the robotic device may optionally present the trajectory to the user and prompt the user to approve or reject the trajectory, for example using a user interface or other I / O device. can. Alternatively, the robotic device may approve or reject the trajectory based on internal rules and / or conditions and by analyzing relevant sensory information. If the orbit is rejected (768: NO), the robot device can optionally modify one or more parameters of the selected model at 770 and a second at 758-764. Orbits can be generated. The parameters of the model can be modified, for example, by selecting different features (eg, different sensory information) to include in the model generation. In some embodiments where the model is an HMM model, the robot device can change the parameters of the model based on the determined success or failure, where the robot device is different with different parameters. Track the log-likelihood of the model and select the model with the higher log-likelihood than the other models. In some embodiments where the model is an SVM model, the robotic device modifies feature space or configuration parameters (eg, kernel type, cost parameter or cost function, weights) as described herein. By doing so, the parameters can be changed.

[0141] If the orbit is approved (768: YES), the robotic device can move the operating element at 772 to perform the generated orbit. While the operating element is performing the planned trajectory, the robot device at 774, eg, using one or more sensors on the operating element and other components of the robot device, eg, the operating element and / or Sensory information such as environmental information can be recorded and / or stored.

[0142] Optionally, at 774, the robot device may determine whether the skill execution was successful (eg, whether the interaction with the object meets pre-defined success criteria). can. For example, a robotic device scans the environment and includes, for example, the location of one or more objects and / or the position or orientation of those objects with respect to an operating element or another component of the robotic device. It is possible to determine the current state of the robot and whether or not the current state meets pre-defined success criteria. Predefined success criteria and / or learned success criteria may be provided by the user, or in some embodiments, by different robotic devices and / or computational devices. Predefined success criteria and / or learned success criteria may indicate information about different characteristics of the environment and / or robotic device associated with success. In some embodiments, the user can also provide an input to indicate to the robot device that the execution was successful.

[0143] In certain cases where the skill is defined as grasping and picking up an object at a particular location, the success of the skill is one or more markers associated with the object. Sufficient force or torque has been experienced by the joints of the end effectors or operating elements (eg, wrist joints), detecting that they have a specific relationship to each other and / or to the robotic device (eg, wrist joints). Or experienced) (which means that the operating element supports the weight of the object and thus picks it up) can be taught and / or specified as detecting. If the execution is unsuccessful (776: NO), the robotic device may optionally modify the parameters of the model at 770 and / or generate a new trajectory at 758-764. If the execution was successful (776: YES), data related to the successful interaction (eg, data indicating that the execution was successful and how it was successful) may be recorded, method 700. You can optionally go back to the beginning and do a new scan of your environment. Alternatively, in some embodiments, method 700 may be terminated.

[0144] In some embodiments, the robotic device is involved in one or more movements of an operating element, a transport element, and / or another component of the robotic device (eg, head, eyes, sensors, etc.). Configured to perform the skill. The robotic device may be configured to plan the trajectory of the skill, similar to those described above with respect to the operating elements. For example, at 750 to 752, the robot device can prompt and receive user selection of the skill model, or can autonomously select the skill model. In 758, the robot device is identified with a set of markers currently identified in the environment and a set of saved markers associated with the selected model (eg, when the robot device learns a skill). It is possible to calculate a function that makes a conversion to one or more markers stored with the skill's model). Such transformations may include keyframe transformations associated with transport elements or other components of the robotic device. At 760, the robot device can use the calculated function to transform the configuration of one or more components of the robot device at each key frame. At 764, the robotic device can plan the trajectory of one or more components of the robotic device between each transformed keyframe. While performing keyframe conversion and / or orbit planning, the robotic device can optionally take into account any environmental constraints associated with the situation in which the skill is being performed. At 772, the robotic device can perform the operation of one or more components of the robotic device according to the planned trajectory. In some embodiments, the robotic device determines the joint configuration at 762, presents the planned trajectory to the user at 766, and / or performs other optional steps as shown in FIG. You may. An example of a skill involved in the movement of a transport element may be opening a door. A user (eg, a nearby user or a remotely located robot supervisor) can use the base of the robot (eg, by pushing open the door) to instruct the robot device to open the door. .. The user, for example, by using a control device such as a joystick or by physically instructing the robot device in the action of opening the door using the base (eg, pushing or pulling the robot device). Can be instructed. The robot device can detect and record information and use that information to generate a skill model while the user is instructing the movement. The robot device later uses its skill model to perform actions to open that door or other door in the environment, for example by transforming the keyframes associated with the transport element during the demonstration of the user's skill. Can be executed.

[0145] FIG. 12 is a block diagram showing a system architecture for robot learning and execution, including actions performed by the user, according to some embodiments. The system 800 can be configured for robot learning and execution. The system 800 may include one or more robotic devices, such as one of the robotic devices described herein, active sensing 822, marker identification 824, learning and model generation 826, orbital. Modules, processes, and / or functions shown in FIG. 12 as generation and execution 828, and success monitoring 829 can be performed. Active sensing 822, marker identification 824, learning and model generation 286, orbit generation and execution 828, and success monitoring 829 are performed by a robotic device as described with reference to method 700 shown in FIGS. 8-11. It may correspond to one or more steps. For example, the active sensing 822 may include step 702 of method 700, the marker identification 824 may include one or more of steps 704 to 710 of method 700, and the training and model generation 826 may include one of steps 712-738. The orbital generation and execution 828 may include one or more of steps 712, 714, 718, and 750-774, and the success monitoring 829 may include one or more of steps 774 and 776. A plurality may be included.

[0146] The system 800 may include one or more cameras 872, an arm 850 (including a gripper 856 and one or more sensors 870), a display device 842, and a microphone 844, for example. Can connect (eg, communicate) with a device. The system 800 can receive input from a user associated with one or more user actions 880 using a display device 842, a microphone 844, and / or another I / O device (not shown). User action 880 may be, for example, 882 where the user approves a marker or requests a rescan of the environment, 884 where the user selects one or more markers, 886 where the user selects relevant information to generate a model. 888 selecting a model for the user to perform the skill, 890 approving the trajectory for the user to perform the skill, 892 confirming the success of the skill performed by the user, the user acquiring the skill by kinesthetic learning It may include 894 to teach.

[0147] For active sensing 822, system 800 scans the environment with one or more cameras 872 and has sensory information about the environment, including information associated with one or more markers in the environment. To record. With respect to the marker identification 824, the system 800 may identify one or more markers in the environment by analyzing sensory information and receive one or more inputs from the user, eg, using a display device 842. Yes (indicating 822 that the user approves the marker or requests a rescan of the environment). For learning and model generation 826, system 800 receives sensory information collected by one or more sensors 870 on one or more cameras 872 and / or arm 850 and uses that information to use the information of the skill. You can generate a model. As part of the learning and model generation 826, the system 800 can receive one or more inputs from the user (one or one to teach a skill, for example using a display device 842 and / or a microphone 844). A user-selected 884 with a set of multiple markers, a user-selected 886 with specific features of recorded sensory information for use in model generation, and / or a user demonstrating a skill 894. show). With respect to orbit generation and execution 828, the system 800 can generate the planned orbit and control the movement of the arm 850 to execute the orbit. As part of orbit generation and execution 828, the system 800 can receive one or more inputs from the user, eg, using display device 842 (user selected 888 model for orbit generation). , And / or 890 where the user approves or rejects the generated trajectory). With respect to the success monitoring 829, the system 800 can determine whether the skill execution was successful by analyzing the sensory information recorded by one or more sensors 870 during the skill execution. As part of the success monitoring 829, the system 800 can receive one or more inputs from the user, for example using the display device 842 and / or the microphone 844 (the user confirms that the execution was successful 892). Shows).

[0148] The connection between the particular one or more devices and / or the system 800 and the one or more devices is shown in FIG. 12, any of the embodiments described herein. Also, an additional device (not shown) may be able to receive information from and / or transmit information to system 800 by communicating with system 800. Understood.

[0149] FIGS. 13-17 are flow diagrams illustrating methods 1300 and 1400 that can be performed by a robotic system (eg, robotic system 100) comprising one or more robotic devices according to embodiments described herein. Is. For example, methods 1300 and / or 1400 can be performed by a single robot device and / or multiple robot devices.

[0150] As shown in FIG. 13, the robotic device is configured to operate in execution mode at 1301. In execution mode, the robot device can autonomously plan and execute actions within the environment. At 1304, the robotic device scans the environment and / or collects information about the environment and / or objects within the environment in order to determine which actions to perform and / or plan how to perform the actions. Can be, at 1305, this information can be used to construct and / or modify a representation or map of the environment (eg, representation 1600). The robotic device can repeatedly (eg, at predetermined times and / or time intervals) or continuously scan the environment and update the representation of the environment based on the information it collects about the environment. Similar to the above method, the robot device uses one or more sensors (eg, one or more sensors 270 or 470, and / or one or more imaging devices 472) to provide information about the environment. Can be collected.

[0151] In some embodiments, the robotic device can repeatedly and / or continuously scan the environment at 1304 for data tracking and / or analysis at 1307. For example, a robot device may be autonomously or remotely driven (eg, by a robot supervisor) to move through an environment (eg, a building such as a hospital) and collect data about the environment, objects within the environment, and so on. can do. This information can be used, for example, by data tracking and analysis elements (eg, data tracking and analysis element 1606) that manage hospital equipment and / or equipment. In some embodiments, the robotic device provides information about various humans (eg, patients), tracks the behavior of such patients (eg, adherence to drugs and / or treatment), performs diagnostic tests, and / Or can be collected to perform screening etc.

[0152] In some embodiments, the robotic device can repeatedly and / or continuously scan the environment at 1304 for modeling and learning purposes at 1309. For example, a robot device collects data about its environment and / or objects within that environment, including, for example, humans and human behavior in response to robot actions, and uses that information to create new behavior. And / or actions can be developed. In some embodiments, the robotic device can collect a large amount of information about an environment, which information is used by the robotic device to further improve and / or generate a model specific to that environment. obtain. In some embodiments, the robotic device can use this information to further adapt the robotic device to a particular environment (eg, skill model and / or behavior generation and / or behavior within that environment. This information can be provided to the robot supervisor (eg, a remote user) (by making modifications). The robot supervisor repeats (eg, at specific time intervals) and / or continually the information collected by the robot device and / or other robot devices, and / or the parameters of the model used by the robot. Can be fine-tuned (eg in real time). In some embodiments, the robot supervisor can repeatedly and / or continuously adapt the robot device to a particular environment by this active exchange of information with the robot device. For example, a robot supervisor can modify a planned path in which a robot device may be in use to navigate through the environment, which in turn can then move its transport element. The information and / or one or more models used by the robot device to generate can be modified. These changes, both provided by the robot supervisor and / or made by the robotic device, include one or more layers of the map of the environment (eg, Map 1600) (eg, the social or semantic layers of the map, etc.). ) Can be supplied.

[0153] At 1306, the robotic device can determine which one or more actions should be performed in the environment, i.e., arbitrate with respect to a set of resources or components capable of performing a particular action. .. Robotic devices can be selected between different actions based on one or more arbitration algorithms (eg, one or more arbitration algorithms 1758). Robotic devices can perform arbitration autonomously and / or using user input. For example, when the robot device is unable to determine the current state of one or more of its components and / or objects in the environment, or when the robot device determines which action to perform, and. / Or the robot device may be configured to require user input when it is not possible to plan how to perform the selected skill. In some embodiments, if the robotic device is familiar with a particular situation (eg, has previously learned and / or performed an action in that situation), it will autonomously from within the set of actions. If selected, the robot device may be configured to require user input when new situations are encountered. When the robot device encounters a new situation, the robot device can require the user to select the appropriate action to be taken by the robot device, or, instead, the robot device selects the action and You can prompt the user to confirm their choice of action.

[0154] In some embodiments, a human operator (eg, a robot supervisor) monitors the robot device and signals the robot device when the human operator wants to intervene in performing an action on the robot device. You can also send it. The human operator may be located near the robot device and / or at a remote computing device connected via a network to the robot device, or a nearby device that may be used to monitor the robot device. For example, a human operator may use new skills or behaviors for safety reasons (eg, to prevent harm to the environment around a human or robotic device) and / or to prevent damage to the robotic device. When wanting to teach a robot device, a human operator may decide to intervene in performing an action by the robot device.

[0155] In response to the determination that user input is required (eg, due to an unfamiliar situation or in response to a signal from the user) (1312: YES), the robot device optionally At 1314, the user can be prompted to provide user input. For example, a robot device may cause a user to be prompted for user input on an on-board display or a display located on a remote device (eg, using a remote or cloud interface). At 1315, the robot device can receive user input and perform arbitration based on the user input. If no user input is required (1312: NO), the robot device continues to autonomously perform arbitration.

[0156] After the robot device has selected an action to be performed, the robot device can plan and perform the action at 1308. As mentioned above, an action involves an operational action (eg, involved in an operational element such as an operational element described herein) or a transport element (eg, involved in a transport element such as a transport element described herein). ) Can be associated with tasks such as movement and / or social behavior. While performing the action, the robot device can continue to scan its surroundings at 1304. When the robot device detects a change in its current state and / or state of the environment (eg, the location of an object in the environment), the robot device at 1310 determines whether the robot device should interrupt the execution of the action. The change can be evaluated to determine. For example, a robotic device responds to the detection of physical engagement of one or more of its components with a human or other object in the environment (eg, by a human operator with operating elements, transport elements, etc.). It may decide to interrupt the execution of the action when it detects the presence of a human or other object (when in contact) or in the immediate vicinity. Additionally or additionally, the robot device may decide to interrupt the execution of the action in response to receiving a signal from the user (eg, the robot supervisor). In some embodiments, the robot device requires the user to demonstrate a portion of the action at a particular point in time during the execution of the action, for example, as the robot device is defined in the interactive learning template. When gained, it may be preconfigured to interrupt the execution of the action. Further details regarding learning using the interactive learning template are described herein with reference to FIG.

[0157] If the robot device decides to suspend the execution of the action (1310: YES), the robot device can determine at 1312 whether user input is required. As mentioned above, if the robot device determines that user input is required (1312: YES), the robot device optionally prompts user input at 1314-1315 and / or the user. You can receive input. The robotic device can then return to sensing or scanning the surrounding environment at 1304, performing arbitration on a set of resources at 1306, and / or performing an action at 1308. Optionally, as shown in FIGS. 14 and 15, when the robot device determines that user input is required, the robot device switches to learning mode at 1402 and then learns the skill at 1404. Or you can proceed to the learning of environmental constraints in 1406. If the robot device determines that no user input is required (1312: NO), the robot device performs ambient sensing or scanning at 1304, arbitration with respect to a set of resources at 1306, and / or. You can return to the execution of the action at 1308. If the robot device determines that the action does not need to be interrupted (1310: NO), the robot device can continue executing the action at 1308.

[0158] In some embodiments, the robot device determines when the robot device may need to solicit user input, for example, in order to help the robot device perform a particular task or behavior. It may be configured to do so. Such assistance may be requested both during the execution of a particular action and / or while the robot device is navigating through the environment. The robot device uses its trained model, as well as information about the environment and objects within that environment, to determine in 1312 when the robot device needs to prompt user input and how to request that user input. It may be configured to do so. For example, a robotic device uses the type of door in its environment and the knowledge of different users to seek help and how to do so (eg, to different users, different types of doors (eg). , A door with a keypad that requires specific permission or knowledge), and certain types of users who may be more likely to provide help than others. You can decide to know to ask (eg, nurse vs. doctor).

[0159] In some embodiments, for example, a user regarding whether skill selection and / or execution was appropriate and / or successful for user input received by a robotic device in 1315. Feedback from may be included. For example, the user can tag an action (eg, a task or behavior) performed by the robot device (or previously performed by the robot device) as a good or bad example of the action. The robot device stores this feedback (eg, as a success criterion associated with this action and / or other action) and uses it to select and / or execute that action or other action in the future. Can be adjusted.

[0160] In some embodiments, the robotic device is configured to engage in socially appropriate behavior. Robotic devices can be designed to operate in a human environment. When moving around a human, from 1306 to 1308, the robotic device may be configured to constantly plan how the action can be perceived by the human. For example, when the robot device is moving, the robot device can look for a human at 1304, monitor its surrounding environment, and engage in social interaction with the human it encounters at 1308. .. When the robot device is not performing any task (eg, stationary), the robot device continues to look for a human at 1304 and monitor its surrounding environment, at 1306 where the human Based on how the presence of the robot device can be perceived, it can be determined whether the robot device may need to perform one or more socially appropriate behaviors. In such embodiments, the robotic device uses a basic framework (eg, one or more arbitration algorithms) to plan and execute socially appropriate behavior given a particular context or situation. Can be configured. The robot device manages multiple resources (eg, operating elements, transport elements, humanoid components such as head or eyes, sound generators, etc.) and behaves appropriately based on one or more of these resources. Can be configured to produce.

[0161] FIG. 18 provides an example of a component of a robotic device that performs arbitration on a mechanism of interest (eg, an eye element). For example, the robot device may have a camera, or other element that can be perceived by a human as having a line of sight (eg, an eye element). A person near the robot device can recognize the direction in which the eye element is directed as the robot device is paying attention. Thus, the robotic device may be configured to determine where the eye element is directed when the robotic device is in motion and / or in the vicinity of a human. The robotic device can constantly arbitrate the use of eye elements so that the robotic device maintains socially appropriate behavior when and / or in the presence of humans.

[0162] As shown in FIG. 18, different components of a robotic device may require a line-of-sight target (eg, directing an eye element in a particular direction). A first component related to active sensing (eg, a camera, laser, or other sensor) can determine at 1510 that the object is within the field of view of the eye element. The first component can further determine at 1514 that the object is human. In response to the determination that the object is human, the first component makes a request 1516 to the central resource manager 1508 (eg, a control unit (eg, one or more) in order to direct the eye element to the human face. Can be sent to the control units 202, 302 and / or 1702) and / or the components of the control unit). A second component related to a navigation action (eg, a camera, laser, or other sensor) is such that at 1520, a particular location (eg, an intermediate or final destination in the navigation) is within the field of view of the eye element. You can decide that. In response to the determination that the location is in view, the second component can send a request 1522 to the central resource manager 1508 to direct the eye element to the location. A third component related to the operating action (eg, a sensor, camera, or other sensor on the operating element) is that at 1530, the robot device is engaged with the object (eg, the robot device is engaging with the object). It can be determined that it is being carried and that a human has come into contact with the robot device. In response to the determination that the robot device is engaged with the object, the third component can send a request 1532 to the central resource manager 1508 to direct the eye element to the object.

[0163] The central resource manager 1508 can receive requests 1516, 1522, and 1532 from the components of the robotic device and, at 1540, perform arbitration to determine in which direction the eye element should be directed. When deciding in which direction the eye element should be directed, the central resource manager 1508 can take into account defined rules or constraints (eg, socially relevant constraints). These defined rules include, for example, not to rotate the eye element twice for a predetermined period (eg, about 5 seconds), at least for a predetermined shortest period (eg, about 5 seconds). In the meantime, it may include orienting the eye element in a certain direction, which may differ between non-human objects and humans, moving the eye element after a predetermined long period of time, and so on. The defined rules can be encoded in an arbitration algorithm for managing gaze resources.

[0164] In some embodiments, the central resource manager 1508 can arbitrate gaze resources based on other information gathered about social context and environment. For example, a robot device may be configured to associate a behavior with a particular location (eg, a person traverses a doorway or aisle before attempting to navigate through a crowded doorway or aisle. Wait, operate with lesser sound in quiet areas (eg, refrain from using sound and / or speech functionality). Robotic devices should capture social contextual information related to such locations and add it to the representation of the environment used for navigation (as described herein with reference to FIG. 19). It may be configured. In some embodiments, it is also possible for a human operator to provide social contextual information to the robot device, and as the robot device learns more about the environment in which it operates, the robot device over time. This information can be adapted.

[0165] FIGS. 15 to 17 show a flow chart of a robot device operating in the learning mode. As described above, the robot device operates in execution mode and, for example, in 1312, can switch to operation in learning mode when the robot device determines that it requires user input. Alternatively or additionally, the robot device may be configured to operate in learning mode, eg, when the robot device is first deployed in a new environment (eg, new area, building, etc.). The robot device determines that the robot device can switch to operation in execution mode until the user indicates to the robot device that the robot device can switch to operation in execution mode, and / or the robot device can switch to operation in execution mode. May operate in learning mode.

[0166] When operating in learning mode, the robot device can learn skills at 1404 and / or environmental constraints at 1406. The robotic device may be configured to learn the skill with or without an existing model of the skill (eg, a generic model of the skill). When learning a skill without using an existing model (eg, learning a skill without relying on prior knowledge), the robotic device is a skill as described herein with reference to FIG. After being taught a demonstration of, you can generate a model of the skill. When learning a skill using an existing model (for example, a general-purpose model of the skill), the robot device starts executing the skill using the existing model, and the user demonstrates a part of the skill. User input can be requested when the robot device requires it. An existing model of a skill is part of an interactive learning template (ie, a template for instructing a robot device to learn a skill using user input at a specified point in time during the execution of the skill). It may be present or may form a part thereof.

[0167] Figure 16 shows the process of learning a skill using an interactive learning template. Robotic devices can be deployed in certain situations (eg, buildings). The robot device can select an existing model of skill at 1410. The existing model of the skill may be a general-purpose model of the skill that is not specific to the environment or situation in which the robot device is operating. At 1412, the robotic device can generate a plan to perform a skill using a skill model according to a process similar to the process described herein with reference to FIG. At 1414, the robot device can initiate the execution of a skill by performing one or more steps or portions of the skill that do not require user input and / or specialization. When you reach a part of the skill (for example, the part where the robot device cannot decide how to perform it, or the part of the skill that requires specialization in the situation where the robot device is performing the skill) (1416: YES), The robotic device can encourage the user to provide a demonstration of that part of the skill at 1418. In some embodiments, the interactive learning template can indicate to the robot device when the robot device should encourage the user to demonstrate. Alternatively or additionally, for example, a robot device is unable to determine the state of an object in the environment and / or performs certain parts of the skill given the constraints imposed by the environment. The robot device can autonomously determine that it requires user input in order for the robot device to be able to perform some of its skills if it is unable to generate a plan. In some embodiments, the user (eg, the robot supervisor) also monitors the performance of the robot device's skill and signals the robot device when the user wants to demonstrate some of the skill to the robot device. Can be sent. Upon receiving the signal from the user, the robot device can decide to proceed to 1418.

[0168] At 1420, the user can instruct the robot device to move. For example, the user can demonstrate some of the skills by moving one or more components of the robot device (eg, operating elements, transport elements, etc.). While instructing the robot device to move, the user may optionally indicate to the robot device at 1422 when information about one or more of the components of the robot device and / or the state of the environment should be captured. can. For example, the robot device can receive a signal from the user to capture sensory information, including information about operating elements, transporting elements, and / or the environment, at key frames during operation of the robot device. Alternatively, the robot device can autonomously determine when sensory information should be captured. For example, while the robot device is being moved by the user, the robot device can monitor changes in one or more of its components, and when those changes exceed a threshold or change direction into the orbit of the component. When this occurs, the robot device can autonomously select that time point as a key frame and record information about the robot device and / or the environment at that key frame. In response to receiving a signal from the user or a decision to capture autonomous sensory information, the robot device can capture sensory information at 1424 using one or more of its sensors. During operation of the robot device, the robot device may also record sensory information at 1430, either continuously or periodically, without receiving a signal from the user.

[0169] Upon completion of the motion or demonstration (1426: YES), the robotic device can optionally receive a selection of features associated with learning some of the demonstrated skills at 1432. In some embodiments, the robotic device can autonomously make feature selections and / or prompt the user to confirm the selections made by the robotic device in 1432. At 1436, the robot device can store information related to the demonstration. If the skill execution is not complete (1417: NO), the robot device will continue to execute the skill at 1414 and, if necessary, encourage the user to further demonstrate the skill portion at 1418. Can be done. The robot device can continue to iterate through the interactive learning process until the execution of the action is complete (1417: YES), at which point the robot device performed the skill at 1438. You can generate a model of a skill that has a context-specific part. The robot device can then learn another skill and / or environmental constraint. Alternatively, if the robot device does not need to learn another skill and / or environmental constraint, the robot device switches to run mode at 1301 to perform environmental sensing or scanning, arbitration, and so on. / Or the execution of the action can be started.

[0170] Using the interactive learning template, the robot device is taught and / or provided with an initial set of models for the skill. An initial set of models may be developed before the robotic device is deployed in the field in a particular environment (eg, a hospital). For example, an initial set of this model may be developed under factory conditions or at a training site and made available to robotic devices. Once deployed in the field, the robotic device can adapt or specialize an initial set of models to the environment, for example, through interactive learning sessions as described herein. Also, with the development of new models (eg, off-site at factories or training sites), new models may be made available to robotic devices, for example by network connectivity. Therefore, by the systems and methods described herein, users and / or entities will develop new models of skills and robotic devices, even after these robotic devices have been deployed in the field. It will be possible to continue to provide to.

[0171] An example of an interactive learning session can be involved in adapting a generic model for moving items into a room. The robotic device is equipped with a general-purpose model and can be deployed in the field in a hospital. Once deployed in the hospital, the robotic device can autonomously initiate the execution of the skill to move the item into the patient's room at 1412-1414. For example, a robot device can autonomously navigate to a patient's room using a hospital map. Once the robot device has navigated to the patient's room, the robot device may decide that the robot device needs to demonstrate where the user should unload the item in the patient's room. Yes (1416: YES). At 1418, the robot device can prompt the user to move it to a specific unloading location. At 1420, the user can control the operation of the robot device, for example using a joystick or other type of control device. As mentioned above, the user may be in the field near the robot device or may be located in a remote location. While the user moves the robot device to the unloading location, the robot device can capture sensory information about its current state and / or its surrounding environment at 1424 and / or 1430. Upon arriving at the unloading location, the robotic device can switch back to autonomous execution at 1414. For example, a robot device may have a known arm movement (eg, an operating element in a general position that locates an item within a container mounted on the robot device, uses the operating element to grab the item, and releases the item. (Positioning) can be executed. The robot device can make a second determination that the robot device requires the user to demonstrate some of the skills (eg, unloading an item on a shelf) (1416: YES). The robotic device can prompt the user for another demonstration at 1418, and the user can move the operating element so that the item is in place on the shelf. At 1424 and / or 1430, the robotic device can recapture sensory information during the operation of the operating element. At 1414, the robotic device can regain control and autonomously open the gripper of the operating element to lower the item onto the shelf and then retract the operating element to return to its resting position. The robot device can then determine that execution is complete (1417: YES) and, in 1438, generate a specialized model of the skill based on the information captured by the robot device during the two user demonstrations. can.

[0172] Other examples of interactive learning sessions may include adapting navigation actions to navigate through certain passages and / or doorways, for example, as described above.

[0173] In some embodiments, the user (or robotic device) may modify the skill's model after the skill has been demonstrated and / or after the skill's model has been generated. For example, the user can iterate over the keyframes captured during the demonstration of the skill and decide whether to retain, modify, and / or delete these keyframes. By modifying and / or deleting one or more keyframes, the user can modify the model of the skill generated based on these keyframes. Examples of using keyframe-based demonstration iterations and adaptive versions are in the Proceedings of the 7th Annual ACM / IEEE International Conference on Human-Robot Interaction (2012), pp.391-98, incorporated herein by reference. Published in a paper entitled "Trajectories and keyframes for kinesthetic teaching: a human-robot interaction perspective", published by Akgun et al. As another example, an interactive graphical user interface (“GUI”) is used to determine how much distribution is acceptable in relation to the position, orientation, etc. of components of a robotic device during planned execution. As the user can show, the keyframes captured during the demonstration of the skill can be displayed to the user. An example of using a GUI with a keyframe is available at http://sim.ece.utexas.edu/static/papers/kurenkov_iros2015.pdf, which is incorporated herein by reference, at the IEEE / RSJ International Conference on Intelligent. It appears in a paper entitled "An Evaluation of GUI and Kinesthetic Teaching Methods for Constrained-Keyframe Skills", written by Kurenkov et al., Published in Robots and Systems (2015). These examples allow the skill model to be modified after the skill has been learned. The systems and methods described herein allow robotic devices to plan and execute specific parts of a skill while leaving other parts of the skill to user demonstrations while the skill is currently in progress. Further learning templates are provided to enable. Also, when and / or skills should be collected by the systems and methods described herein, eg, autonomously or based on information provided to the robotic device prior to the learning process. Robotic devices are provided that can determine if some demonstrations should be requested.

[0174] Figure 17 shows the process of learning environmental constraints. As described above, the robot device can be configured to learn environmental constraints, which may be applicable to a set of skills to be learned and / or performed in situations that include the environmental constraints. .. At 1450, the robot device can optionally prompt the user to select the type of constraint and / or the object associated with that constraint. The type of constraint may be, for example, a barrier (eg, a wall, a surface, a boundary), a size and / or dimension, an exclusion zone, the location of an object, a time constraint, and the like. At 1452, the robot device can receive a choice of constraint type and / or object. Optionally, at 1454, the user (or other robotic device) can instruct the robotic device in actions to demonstrate environmental constraints. For example, the user can demonstrate the size of the container to the robot device by moving the operating elements of the robot device along one or more edges of the equipment container. As another example, the user can demonstrate the location of the shelves by moving the operating elements of the robot device along the surface of the shelves. During operation of the robot device, the robot device can capture information related to the constraint at 1456 using one or more sensors (eg, camera, laser, tactile sensation, etc.). At 1458, the robotic device can store capture information related to environmental constraints for use with models of skills performed in situations that include environmental constraints. In some embodiments, the robotic device can include information related to environmental constraints in each model of skill performed in the above situations. Alternatively, the robotic device may be configured to refer to information related to environmental constraints when planning and / or performing the skill in the above situations. In some embodiments, environmental constraints may be added to the representation of the environment (eg, representation 1600).

[0175] In some embodiments, the robotic device can learn environmental constraints without the need for user demonstrations or robotic device movements. For example, at 1452, after receiving a constraint type and / or object selection, the robot device may be configured at 1456 to scan the environment for relevant information related to the environmental constraint. In some embodiments, the robotic device captures and / or determines that it is relevant to an environmental constraint so that the user can see and / or modify which information is relevant to the environmental constraint. Can be presented to the user. The robot device can then store relevant information at 1458.

[0176] In some embodiments, the robotic device can use transport elements (eg, wheels or trucks) to move around in an environment and learn one or more environmental constraints. For example, a robot device can learn to navigate a corridor or aisle, for example, receiving a demonstration and / or performing a skill, and the corridor is congested during a particular time frame. In some embodiments, the robotic device can learn various environmental constraints and / or behaviors and associate them with different conditions (eg, time, place, etc.). For example, a robot device is based on information collected by one or more of its sensors, user input, and / or information derived from information detected (eg, during a demonstration and / or execution of a skill). It can be recognized that the corridor is crowded. As another example, the robot device responds "hello" or "good night" at various times of the day when the user enters a particular room. You can decide what to do.

[0177] In some embodiments, the robotic device can autonomously or interactively learn environmental constraints. For example, at 1458, a robotic device can acquire an initial set of environmental constraints by moving through the environment (eg, going through a corridor or aisle). The user (eg, robot supervisor or local user) can review the constraint and, at 1459, determine whether the constraint should be adjusted by directly modifying the constraint and / or by demonstrating. For example, some constraints are provided through interaction with the robot device (eg, demonstration), while other constraints (eg, the location of the object (eg, the distance the shelf extends from the wall)) (eg, the distance the shelf extends from the wall). It may be provided by input to a user interface (eg, user interface 240, etc.).

[0178] FIG. 21 shows an example of information entering and exiting map 2120 (eg, a map of a building (eg, a hospital, etc.)). Map 2120 may be stored and / or retained by one or more robotic devices, such as robotic devices of any of the robotic devices as described herein. Map 2120 may resemble map 1600 as shown in FIG. For example, map 2120 may include one or more layers 2122, such as navigation layers, static layers, dynamic layers, and social layers.

[0179] Map 2120 can provide information that can be used by the robotic device while operating in learning mode 2102 or execution mode 2112. For example, a robotic device operating in learning mode 2102 may access map 2120 (eg, state information 331, 1732, object information 340, 1740, one or more environmental constraints 1754, social context). Information about one or more environmental constraints (similar to 1632, etc.), one or more objects, and / or one or more social contexts can be obtained. Such information, as described herein, determines the location of one or more objects, identifies one or more properties of one or more objects, or one or more. Robotic devices can analyze multiple environmental constraints, select one or more skill models to use, prompt one or more users for one or more inputs, and so on. Can be made possible. Additional or alternative, a robotic device operating in run mode 2112 may access map 2120 to accommodate one or more environmental constraints, one or more objects, and / or one or more. To obtain information about a social context and use it to evaluate the environment, to perform arbitration between different behaviors or skills, to determine which skills to perform, and / or to identify. Behaviors or skills can be adapted to suit the environment of.

[0180] The robotic device operating in learning mode 2102 can provide information in addition to and / or information in the map 2120 to change the information in the map 2120. For example, the robot device is based on analysis of detected information 2104 (eg, information collected by one or more sensors of the robot device) and / or derived information 2106 (eg, for example, detected information 2104). The information derived by the robot device) can be incorporated into the map 2120. The robotic device can incorporate such information by adding the information to the map 2120 and / or adapting the existing information in the map 2120. Additionally, the robotic device operating in run mode 2112 is added to the information in the map 2120 and / or information that modifies the information in the map 2120 (eg, detected information 2114, derived information). 2116) can be provided.

[0181] As an example, a robot device undergoing a demonstration of how to navigate through a doorway (eg, such as the demonstration shown in FIGS. 10 and 16) is on one or more layers 2122 of map 2120. The information provided can be collected during the demonstration. The doorway may be a specific doorway present at various locations throughout the building, for example, as shown in Map 2122. The characteristics and / or properties of the doorway can be derived by the robot device based on the information recorded and / or detected by various sensors on the robot device. For example, a robot device can detect the size of a doorway and determine that the doorway is a narrow doorway. The robotic device can add this information about the doorway to the map 2120, for example, using one or more semantic labels, as raw sensor information or processed sensor information, as derived rules. The robotic device can also use this information to adapt the context layer of the map (eg, the social or behavioral layer of the map (eg, the social layer 1630 shown in FIG. 19)). For example, a robot device is based on the fact that the doorway is a narrow doorway, for example, instead of passing through the doorway following and / or in parallel with the user (or one or more other robotic devices). Or passing through the doorway when it is opened by one or more other robotic devices (eg, keeping the doorway open, and / or performing various actions on the robotic device). Specific one or more behaviors and / or one or more, including helping the robot device navigate through the doorway (by teaching), searching for a specific user familiar with the doorway, etc. You can decide to engage in multiple actions.

[0182] In some embodiments, the map 2120 may be centrally retained for one or more robotic devices described herein. For example, the map 2120 may be stored on a remote computing device (eg, a server) and centrally hosted, for example, to a group of robotic devices working together in a hospital. Map 2120 displays information from these robotic devices (eg, detected information 2104, 2114, and derived information 2106, 2116) when a group of robotic devices operates in learning mode 2102 and / or execution mode 2112. May be updated as it is received and provided to each robotic device as needed to perform actions, behaviors, etc. This information is used when other robotic devices encounter similar environments, as individual robotic devices in the group learn new information so that one or more layers 2122 of map 2120 are adapted. / Or may be shared with those robotic devices when performing similar skills (eg, actions, behaviors). In some embodiments, a local copy of Map 2120 may be stored on each robotic device, which is a unit of Map 2120 (eg, at predetermined intervals, during breaks or downtime). Can be updated or synchronized with a copy that is retained. Periodically update the map (using newly collected information such as, for example, detected information 2104, 2114, and derived information 2106, 2116) and / or share the map among multiple robotic devices. By doing so, each robotic device has access to a more comprehensive map that interacts with its surroundings and / or provides the robotic device with more accurate information for performing skills within its surroundings. Can be done.

[0183] In some embodiments, the map 2120 is such that information (eg, one or more environmental constraints, one or more rules, one or more skills, etc.) is (eg, learning 2102 and). / Or may be a mixed initiative map that can be learned (during run 2112) and / or provided by the user (eg, using user input 2130). For example, a robot device may include information directly given by a user (eg, a robot supervisor or local user), information learned through demonstrations and / or executions, or demonstrations and / or executions (eg, in real time). Alternatively, the map 2120 can be constructed by incorporating information interactively provided in combination with user input (as required by the robotic device for interaction retroactively). For example, the user can indicate to the robot device that the robot device should travel more slowly through an area on the map 2120 (eg, a narrow or congested passage). In some embodiments, the robot device presents a map to the user, who can draw an area on the map to indicate that the robot should travel through that area more slowly. The robotic device may attempt to navigate the area on the map 2120 multiple times and learn that the area should be avoided (eg, due to overcrowding) at certain times of the day. Alternatively or additionally, the robotic device may encounter new situations (eg, aisles are separated) that the robotic device is not accustomed to coping with autonomously. In such cases, the robot device obtains more information about the area and / or determines how to navigate the area around the user (eg, the robot supervisor and / or the local user). ) And can communicate with. When requesting user input, the robot device asks broadly about the situation and / or an example that the user labels (eg, to allow the robot device to autonomously derive appropriate behavior). A list of may be provided. In some embodiments, the robotic device discovers and / or derives information about an area without user input and proposes one or more rules associated with that area to the user for confirmation. be able to. In some embodiments, the user may modify the proposed rule and / or other information when he sees it and / or other information from the robot device before approving it. can. FIG. 24, described below, provides a more detailed description of such a process.

[0184] FIGS. 22-25 show one or more of the robotic devices described herein, including, for example, robotic devices (eg, robotic devices 102, 110, 200, 400, etc.). It is a flow diagram showing different means capable of learning one or more environmental constraints, etc. In some embodiments, a robotic device that operates according to the method described herein is a robotic device. , Always learning (eg, operating in learning mode). For example, a robot device is continuously informed as it navigates the environment and / or engages in a particular behavior or action. Can be collected, stored, analyzed, and / or updated, and adapted and / or added to a library of learned information (eg, skills, behaviors, environmental constraints, maps, etc.). In the embodiment of the above, the robot device can switch between the operation in the learning mode and the operation in the execution mode. While operating in the learning mode 2202, the robot device described in the present specification is demonstrated 2204. A robotic device such that can be learned by execution 2206, by search 2208, by derivation / user input 2210, and / or any combination thereof.

[0185] In some embodiments, the robotic device can be learned by demonstration 2204, for example, as shown and described above with reference to FIGS. 8-10. For example, a robot device can learn a new skill or environmental constraint by demonstrating a skill using the LfD teaching process. Robotic devices can analyze and extract information from past skill demonstrations in the human environment. As an example, a user can demonstrate to a robot device how to move an operating element (eg, an arm) of the robot device in an equipment room containing humans and the robot device. From one or more demonstrations, the robot device may limit the movement of the robot device when planning and / or performing future skills for the movements of one or more existing demonstrations. Multiple environmental constraints can be specified. Robotic devices use existing demonstrations (eg, sequences of one or more keyframes), each providing a path through a node and n-dimensional space of motion, to generate motion and graph (environmental constraints). Can be constructed). Then, if the robot device is presented with a new environment and needs to adapt its skills to that new environment, the robot device efficiently samples from the existing set of demonstrated movements using the constructed graph. And can plan movements for new environments. By incorporating the learning of such environmental constraints into the initial demonstration of the skill, the robotic device can quickly adapt to the new environment without the need for new environmental constraints to be defined. In the new environment, the robot device scans the environment and / or scans the environment for additional information about one or more objects within the environment (eg, humans, fixtures, doors, etc.) and then existing skills. Models can be used to plan motion trajectories through new environments without significantly deviating from the existing set of demonstrated motions.

[0186] In some embodiments, the robotic device can be learned by demonstration 2204 and user input 2210. For example, a robot device can engage in interactive learning with a user, for example using an interactive learning template. As mentioned above, we have provided an initial set of robotic devices and / or skill models when using interactive learning. An initial set of models may be developed by other robotic devices in factory situations and / or by the robotic device itself in one or more situations. While operating in one particular environment, the robot device can perform an interactive learning session, in which the robot device autonomously performs certain parts of the skill while leaving other parts to user demonstrations. Can) to adapt or specialize an initial set of this skill. Further details of such an interactive learning process will be described with reference to FIG.

[0187] In some embodiments, the robotic device can learn while performing the skill 2206. For example, a robot device can collect, store, analyze, and / or update information as the robot device performs behaviors, actions, etc. (eg, 774 in FIG. 11 and 1304, 1305 in FIG. 13). , And 1307). In certain embodiments, the robotic device can perform skills that require the robotic device to move through aisles several times a day. The robot device can learn that there is more traffic in the aisle at a particular time of the day. The robot device is then used, for example, to avoid passing through the passage during those times of heavy traffic (eg, by taking another route and / or at times of low traffic). The behavior can be adapted (by waiting to pass through the passage).

[0188] In some embodiments, the robotic device can be learned by search 2208. FIG. 23 is a flow chart showing an exemplary method of learning by exploration. At 2302, the robotic device can scan the environment and / or collect information about the environment and / or objects within the environment. Optionally, based on the information collected by the robot device, the robot device can decide to explore the environment by performing the skill. To execute a skill, the robot device can select an existing skill model at 2304, plan the execution of the skill using the skill model at 2306, and execute the skill at 2308. For example, a robot device can scan the environment and identify doorways. The robotic device can determine the execution of a skill, including the step of opening a door and the step of moving through a doorway. At 2304, the robotic device can select existing skill models previously provided and / or learned to navigate through the doorway. In 2306, the robotic device uses existing skills (eg, according to a process similar to the process described with reference to FIG. 11) to generate a plan to open a door and move through the doorway. Can be done. Then, at 2308, the robot device can perform the skill according to the generated plan. At 2310, while performing the skill, the robot device can collect information about the environment and / or perform the skill and compare that information with the information collected during the previous interaction with the doorway. Based on this comparison, the robot device can evaluate whether its interaction with the doorway was successful or unsuccessful (eg, based on the success criteria described above). The robot device can optionally generate a skill model specific to its doorway at 2312, based on the execution of its skill.

[0189] In some embodiments, the robotic device can initiate the execution of a skill and determine that user input is required to further complete the skill. For example, as described above with reference to FIG. 16, the robot device starts executing the skill, and when the robot device reaches the part of the skill where the execution method cannot be determined, the execution of the skill is advanced. User input can be requested.

[0190] In some embodiments, the robotic device can perform a skill and determine that the execution has failed. For example, a robotic device can plan and / or perform movements through a doorway, but through the doorway (eg, due to an attempt to open a door or to navigate across a narrow doorway). It can be detected that it could not be moved. The robotic device decides to scan the environment again (2314: YES) and can replan and re-execute the skill at 2306-2308. In some embodiments, the robotic device selects another skill at 2304 (eg, a more specific skill for a narrow doorway) and then replans and replans based on that other skill. Can be executed. Until the robot device determines that it has met a purpose (eg, successfully performed a skill a given number of times or in a given number of ways, learned enough information about the environment, etc.). Can continue learning by exploration by rediscovering, replanning, and re-executing skills. At 2316, the robot device stores the information it collects about the environment and / or objects within the environment, and optionally, based on the performance of the skill within the environment, the model it adapts or generates. be able to.

[0191] In some embodiments, the robotic device can engage in learning by search 2208 using user input 2210. For example, a robotic device may perform one or more skills in a particular environment in order to explore and / or learn more information about that environment. While the robot device is exploring, the user (eg, a remote supervisor or local user) is recognized by the user (eg, directly by being in the environment and / or through the robot device). Inputs can be provided to the robot device based on the information learned about and / or the environment. The user can then provide the robot device with one or more inputs that further guide the search for the environment. For example, a robot device that encounters a doorway can plan one or more actions for navigating through the doorway. When performing those actions, the robot device may fail to navigate through its doorway. The user may see the failure of the attempt by the robot device and provide the robot device with one or more inputs to guide the robot device in a second attempt to navigate through the doorway. For example, the user decides that the doorway is a narrow doorway and that the robot device should first ask a nearby user or robot device to open the door before navigating through the doorway. Can be shown on a robot device. Upon receiving user input, the robot device can ask the user to open the door before navigating through the doorway.

[0192] In some embodiments, the robotic device learns by user input 2210 by receiving information about the environment and / or objects in the environment from the user, or by receiving an initial set of models, rules, etc. from the user. can do. In some embodiments, the robotic device first learns a skill by demonstration (eg, as described above with reference to FIG. 10) and adapts the skill based on user input. You may learn. For example, a robot device can learn the skill of moving equipment from a first place to a second place and then unloading the equipment to a second place. During a user demonstration of the skill, the user can indicate to the robot device that unloading equipment at a second location requires specific information specific to the environment or situation. The user can later provide the robot device with an input that specifies specific information that the robot device should observe while unloading the equipment with respect to different equipment and / or different locations. For example, the user can provide information about a tag or marker that identifies where the robot device should unload equipment.

[0193] In some embodiments, the robot device learns behavior or rules (eg, one or more arbitration algorithms 1858 as described above) based on stored information and / or user input 2210. can do. FIG. 24 is a flow chart of an exemplary method of robot learning of behavior and / or rules. At 2402, the robot device can analyze stored information (eg, one or more maps, one or more models, one or more user inputs). At 2404, the robotic device can derive behaviors and / or rules based on the analysis of the stored information. For example, a user (eg, a robot supervisor or a local user) can specify rules for a robot device so that it is more interactive when more people are near the robot device. Based on the analysis of stored information, the robot device may determine that more humans are near it during a particular time of the day (eg, around lunch time). Therefore, the robot device can further derive that it should be more interactive during that time of the day. In some embodiments, the robotic device can automatically implement this behavior as a new rule. Alternatively, the robotic device can be proposed at 2406 to be more interactive at that time of the day as a new rule for the user. At 2408, the robot device can receive user input in response to rule proposals. For example, the user may view the rule on the local and / or remote display of the robot device and choose to approve or reject it. Based on user input, the robot device can modify its behavior and / or rules at 2410. For example, if the user indicates that the rule can be approved, the robot device can save the new rule and implement it in the future. Alternatively, if the rule is acceptable but the user indicates that it involves further tweaks or changes (eg, to be more interactive only in the cafeteria, around lunch time), the robot device , The rule can be adapted and the adapted rule can be saved for future implementation.

[0194] In some embodiments, the robotic device can learn skills and / or environmental constraints by adapting existing skills and / or environmental constraints based on user input. FIG. 25 is a flow diagram of an exemplary method of adapting skills and / or environmental constraints based on user input. The robot device can receive user input at 2502. The robot device can associate user input with one or more skills and / or environmental constraints in 2504. Such associations are, for example, further input by the user (eg, the user specifies the skill and / or environmental constraints associated with the input), and / or derivation by a robotic device (eg, such input). , The robot device determines that it is related to a particular skill or environmental constraint, based on this input and / or one or more rules, attributes, etc. related to this particular skill or environmental constraint. For example, the user may provide the robot device with new information for identifying an object such as a QR code®, bar code, tag, etc. The user may provide this new information. However, it can be specified that it is about a particular object, and the robot device can associate new information with this object. The robot device transfers new information about that object to the interaction with that object. It can be further associated with one or more skills involved (eg, grab or move skills). The robotic device optionally modifies the skill or environmental constraint based on the input at 2506-2508. And / or generation can be performed.

[0195] In some embodiments, the robotic device can allow the user to retrospectively label information related to skills and / or environmental constraints. For example, the robot can present skills or environmental constraints to the user, for example using a user interface (eg, user interface 240) and / or a remote interface. The user may choose to add, modify, and / or remove labels associated with the skill or environmental constraint, such input may be received by the robot device at 2502. The robot device can then associate the input with a skill or environmental constraint at 2504 and modify and / or generate a new skill or environmental constraint based on the input at 2506-2508.

Example
[0196] It will be appreciated that this disclosure may include up to all from any one of the following examples:

[0197] Example 1: A device having a set of memory, processor, operating element, and sensor, wherein the processor is operably coupled to the set of memory, operating element, and sensor, and from the set of sensors. Using a subset of the sensors of, to get a representation of the environment and to identify multiple markers within the representation of the environment, where each marker from multiple markers is a plurality of physics located within the environment. Identifying the physical object from the physical object, presenting information indicating the position of each marker from multiple markers in the representation of the environment, and the physical object from multiple physical objects. Receives a selection of a set of markers from multiple markers associated with the set, and obtains sensory information related to the operating element for each position from multiple positions related to the movement of the operating element in the environment. That is, the movement of the operating element is related to the physical interaction between the operating element and the set of physical objects, and the physics of the operating element and the set of physical objects based on the sensory information. A device configured to generate and perform a model configured to specify the behavior of an operating element to perform a physical interaction.

[0198] Example 2: The device of Example 1, wherein the set of physical objects includes a human.

[0199] Example 3: The apparatus of Example 1 or 2, wherein the operating element comprises an end effector configured to engage a subset of physical objects from a set of physical objects.

[0200] Example 4: A subset of sensors is a subset of the first sensor, the processor is configured to acquire sensory information using a subset of the second sensor from a set of sensors, second. The apparatus according to any one of Examples 1 to 3, wherein the subset of sensors of the first sensor is different from the subset of the first sensor.

[0201] Example 5: A sensor, wherein the operating element comprises a plurality of movable components connected by a plurality of joints, and a set of sensors is configured to measure the force acting on the joints from the plurality of joints. Or any one of Examples 1 to 4, comprising at least one of a sensor configured to detect engagement of a movable component from a plurality of movable components with a physical object from a set of physical objects. The device described in.

[0202] Example 6: The device of Example 5, wherein the set of sensors further comprises a sensor configured to measure the position of a joint or movable component with respect to a portion of the device.

[0203] Example 7: The apparatus of Example 5, wherein the set of sensors further comprises at least one of an optical sensor, a temperature sensor, a voice capture device, and a camera.

[0204] Example 8: A set of sensors comprising (i) a plurality of joints and (ii) an end effector configured to move a physical object from a set of physical objects. Examples include a sensor configured to measure the force exerted on at least one of an end effector or a joint from multiple joints coupled to the end effector as the end effector moves a physical object. The device according to 1.

[0205] Example 9: Sensory information includes sensor data associated with a set of features, the processor is further configured to receive a selection of a subset of the first features from the set of features, and the processor is the second. Generate a model based on sensor data related to a subset of features and not based on sensor data related to a subset of second features from a set of features that are not included in the first set of features. The apparatus according to any one of Examples 1 to 8, which is configured to be the same.

[0206] Example 10: In response to a selection made by the user, the user is asked to select at least one feature from the set of features so that the processor receives a selection of a subset of the first features. The device of Example 9, wherein the processor is further configured to prompt.

[0207] Example 11: The apparatus according to any one of Examples 1 to 10, wherein the plurality of markers are reference markers and the representation of the environment is a visual representation of the environment.

[0208] Example 12: The processor is further configured to store the model and model-related information in memory, the model-related information including (i) a set of markers and (ii) sensory information. The apparatus according to any one of Examples 1 to 11.

[0209] Example 13: The operating element comprises a plurality of joints, and the sensory information indicates the current state of each joint from the plurality of joints for each position from the plurality of positions related to the movement of the operating element. The apparatus according to the first embodiment, which comprises information.

[0210] Example 14: The processor selects at least one marker from a plurality of markers after the above presentation so that the processor receives a selection of a set of markers in response to a selection made by the user. The apparatus according to any one of Examples 1 to 13, further configured to urge the user to do so.

[0211] Example 15: Any of Examples 1-14, wherein the processor is configured to acquire a representation of the environment by sensing or scanning a region of interest in the environment using a set of sensors. The device according to one.

[0212] Example 16: Obtaining a representation of an environment using a set of sensors and identifying a plurality of markers within the representation of the environment, wherein each marker from the plurality of markers is within the environment. To identify and identify the physical object from multiple physical objects located in, and to present information indicating the position of each marker from multiple markers in the representation of the environment, and after the above presentation, multiple. Receiving a selection of a set of markers from multiple markers associated with a set of physical objects from a physical object in the environment, and by position from multiple positions related to the movement of the operating element in the environment. Acquiring sensory information related to an operating element, in which the movement of the operating element is related to the physical interaction between the operating element and a set of physical objects, and the operation based on the sensory information. A method of generating and having a model configured to specify the behavior of an operating element for performing a physical interaction between an element and a set of physical objects.

[0213] Example 17: The sensory information comprises sensor data associated with a set of features, further comprising the method receiving a selection of a subset of first features from the set of features, and generating above. However, based on the sensor data related to the subset of the first feature, and not based on the sensor data related to the subset of the second feature from the set of features, which is not included in the first set of features. 16. The method of Example 16, comprising generating a model.

[0214] Example 18: Encouraging the user to select at least one feature from a set of features so that the selection of a subset of the first features is received in response to the selection made by the user. The method according to Example 17, further comprising.

[0215] Example 19: The method according to any one of Examples 16-18, wherein the plurality of markers are reference markers and the representation of the environment is a visual representation of the environment.

[0216] Example 20: The operating element comprises a plurality of joints, and the sensory information indicates the current state of each joint from the plurality of joints for each position from the plurality of positions related to the movement of the operating element. The method according to any one of Examples 16 to 19, which comprises information.

[0217] Example 21: In response to a selection made by the user, the user is urged to select at least one marker from a plurality of markers after the presentation so that the selection of a set of markers is received. The method according to any one of Examples 16 to 20, further comprising the above.

[0218] Example 22: Any one of Examples 16-21, wherein acquiring a representation of the environment involves sensing or scanning a region of interest in the environment using a set of sensors. The method described in.

[0219] Example 23: A non-temporary processor-readable medium that stores code representing an instruction executed by a processor, wherein the code uses a set of sensors to obtain a representation of the environment. Identifying multiple markers within a representation of the environment, where each marker from multiple markers is associated with a physical object from multiple physical objects located within the environment. Presenting information indicating the position of each marker from multiple markers within the representation of, and selecting a set of markers from multiple markers associated with a set of physical objects from multiple physical objects. Identifying a model associated with performing a physical interaction between an operating element and a set of physical objects in response to receipt, wherein the operating element comprises multiple joints and end effectors. It has code that causes the processor to use the model to generate trajectories of operating elements that define the behavior of multiple joints and end effectors related to performing physical interactions. , Non-temporary processor readable medium.

[0220] Example 24: Implementation, wherein the code that causes the processor to identify the model associated with performing the physical interaction comprises the code that causes the processor to prompt the user to identify the model. The non-transient processor-readable medium according to Example 23.

[0221] Example 25: The code that causes the processor to identify the model associated with performing the physical interaction comprises a code that causes the processor to identify the model based on the selection of a set of markers. The non-transient processor-readable medium according to Example 23 or 24.

[0222] Example 26: Displaying the trajectory of an operating element to the user in a representation of the environment, receiving input from the user after display, and physics in response to an input indicating approval of the trajectory of the operating element. The non-temporary processor according to any one of Examples 23-25, further comprising a code that causes the processor to perform the actions of multiple joints and end effectors to perform a targeted interaction. Readable medium.

[0223] Example 27: The orbit is the first orbit and the non-transient processor-readable medium modifies the set of parameters associated with the model in response to an input that does not indicate the approval of the orbit of the operating element. 1 A non-transient processor-readable medium described in one.

Example 28: The orbit is the first orbit, the model is the first model, and the non-temporary processor readable medium responds to an input that does not indicate approval of the orbit of the operating element. To generate a second model based on sensor data related to a different set of features than the set of features used to generate the first model, and to use the second model, The non-temporary processor-readable medium according to any one of Examples 23-26, further comprising a code that causes the processor to generate a second trajectory of the operating element.

[0225] Example 29: Performing the movements of a plurality of joints and end effectors in order to perform a physical interaction, acquiring sensory information related to performing the physical interaction, and based on the sensory information. One of Examples 23-28, further comprising a code that causes the processor to determine whether the execution of the physical interaction meets pre-specified and / or learned success criteria. Non-temporary processor readable medium as described in.

[0226] Example 30: Generating a signal indicating a successful physical interaction and physics in response to a decision that the execution of the physical interaction meets pre-specified and / or learned success criteria. Generating a modified model by modifying the model based on sensory information in response to a decision that the execution of a physical interaction does not meet pre-defined and / or learned success criteria, and using the modified model. 29. The non-transitory processor readable medium of Example 29, further comprising a code that causes the processor to generate a second trajectory of the operating element.

[0227] Example 31: The model positions the operating element at a point along the stored trajectory of the operating element in relation to (i) a set of conserved markers and (ii) a set of conserved markers. Or the processor is associated with sensory information indicating at least one of the orientations and (iii) sensory information indicating the composition of multiple joints at points along the conserved trajectory to generate the trajectory of the operating element. The code to be done is to calculate the conversion function between the set of markers and the set of stored markers, and to use the conversion function pointwise to convert at least one of the positions or orientations of the operating elements. For each point, determine the planned composition of multiple joints based on the composition of multiple joints at points along the conserved trajectory, and for each point, plan for multiple joints at that point. The non-temporary processor readable according to any one of Examples 23-30, comprising code that causes the processor to determine a portion of the orbit between that point and the continuum based on the configuration. Medium.

[0228] Example 32: The position of the operational element at a point along the conserved trajectory of the operational element in which the model is (i) associated with a set of conserved markers and (ii) a set of conserved markers. Or the code associated with the sensory information indicating at least one of the orientations and causing the processor to generate the trajectories of the operating elements is to calculate the conversion function between the set of markers and the set of stored markers. , Pointwise, using a transformation function to transform at least one of the positions or orientations of the operating element, pointwise, determining the planned configuration of multiple joints, and pointwise, pointwise. In any one of Examples 23-30, including a code that causes the processor to determine a portion of the orbit between that point and the continuum based on the planned configuration of the plurality of joints with respect to. The non-temporary processor readable medium described.

[0229] Example 33: Determining whether to change the location of the operating element based on the distance between the location of the first operating element and the location of the physical object from the set of physical objects. Code that causes the processor to move the operational element from the first location to a second location closer to the location of the physical object in response to the decision to change the location of the operational element. Further, the code that causes the processor to generate the trajectory of the operating element, after the above movement, causes the processor to generate the trajectory based on the location of the physical object with respect to the second location of the operating element. The non-temporary processor-readable medium of any one of claims 23-32, comprising a code to be performed.

[0230] Although various embodiments of the invention have been described and illustrated herein, those skilled in the art perform the functions described herein and / or the results and / or advantages described herein. Various other means and / or structures for obtaining one or more of these are readily envisioned, each of such variants and / or modifications of the embodiments of the invention described herein. It is considered to be within range. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary and actual parameters, dimensions, materials, and / or configurations. Those skilled in the art will readily appreciate that the teachings of the present invention are determined by one or more specific applications in which they are used. One of ordinary skill in the art will be able to elucidate them simply by recognizing a number of equivalents of the embodiments of the specific inventions described herein or by performing mundane experiments. Accordingly, the embodiments described above are presented merely as examples, and within the scope of the appended claims and their equivalents, the embodiments of the invention are provided in ways other than those specifically described and claimed. It shall be understood that it can be carried out. The embodiments of the invention of the present disclosure are directed to the individual features, systems, products, materials, kits, and / or methods described herein. Moreover, any combination of two or more such features, systems, products, materials, kits, and / or methods must be inconsistent with each other in their features, systems, products, materials, kits, and / or methods. For example, it is included in the scope of the invention of the present disclosure.

[0231] Further, various invention concepts can be embodied as one or more methods, and an example thereof is provided. Acts performed as part of the method may be ordered in any suitable manner. Therefore, it is possible to construct embodiments in which the acts are performed in a different order than those exemplified, and these embodiments are shown as sequential acts in the exemplary embodiments, but how many. It may include performing such acts at the same time.

[0232] Some embodiments and / or methods described herein may be performed by different software, hardware, or a combination thereof (executed on hardware). The hardware module may include, for example, a general purpose processor, a field programmable gate array (FPGA), and / or an application specific integrated circuit (ASIC). Software modules (running on hardware) include C, C ++, Java ™, Ruby, Visual Basic ™, and / or other object-oriented, procedural, or other programming languages and development tools. It can be expressed in various software languages (eg, computer code), including. Examples of computer code include microcode or microinstruction, machine instructions as generated by a compiler, code used to generate web services, and high-level instructions executed by a computer using an interpreter. Contains, but is not limited to, files. For example, embodiments include imperative programming languages (eg, C, Fortran, etc.), functional programming languages (Haskell, Erlang, etc.), logical programming languages (eg, Prolog), and object-oriented programming languages (eg, Java, C ++). Etc.), or can be implemented using other suitable programming languages and / or development tools. Additional examples of computer codes include, but are not limited to, control signals, encryption codes, and compression codes.

[0233] In order to describe various elements, terms such as "first", "second", and "third" may be used herein, but these elements are not included. It will be understood that these terms are not limited. These terms are used only to distinguish one element from another. Accordingly, the first element described herein can be named the second element without departing from the teachings of the present disclosure.

[0234] The terminology used herein is solely for the purpose of describing a particular embodiment and is not intended to be restrictive. As used herein, the singular forms "a," "an," and "the" are intended to include the plural, unless the context clearly indicates other meanings. As used herein, the term "having" ("comprises" and / or "comprising"), or the term "includes" ("includes" and / or "including") is a specified feature, area, or. Specifies the existence of an integer value, step, action, element, and / or component, but the existence of one or more other features, regions, integer values, steps, actions, elements, components, and / or groups of these. Or it will be further understood not to exclude additions.

Claims (18)

Operation elements and
With transport elements configured to move along the surface,
With a set of sensors
Memory and
A processor operably coupled to the memory, the operating element, the transport element, and the set of sensors.
Obtaining environmental information using a subset of the first sensor from the set of sensors.
Choosing a skill model based on the information in the environment
Using the model to generate a plan to perform at least a portion of the skill using the model, wherein the plan is one or more of the operating element or at least one of the transporting elements. To generate, including the behavior of
To move the operating element or at least one of the transporting elements based on the plan.
The environment, the operating element, or said Acquiring information on at least one of the transport elements,
Updating a map of the environment based on said at least one piece of information about the environment, the operating element, or the transporting element.
With a processor configured to do
Has a device. The part of the skill is the first part, and the processor
Determining that the second part of the skill requires specialization for the environment,
Responding to the determination that the second part of the skill requires specialization, and requesting input from the user.
To generate a plan to perform the second part of the skill in response to receiving the input from the user.
To move at least one of the operating element and the transport element based on the plan of executing the second part of the skill.
The apparatus according to claim 1, wherein the device is configured to further perform. The part of the skill is the first part, and the processor
Determining that the second part of the skill requires specialization for the environment,
In response to the determination that the second part of the skill requires specialization, requesting the user to demonstrate and
Obtaining information collected by a subset of a third sensor and relating to at least one of the environment, the operating element, or the transport element for each position from the plurality of locations associated with the demonstration.
Using the information collected by the model and a subset of the third sensor to generate a specialized model for performing the skill in the environment.
The apparatus according to claim 1, wherein the device is configured to further perform. The processor sets one or more success criteria for the operation of the operating element or at least one of the transporting elements based on the environment, the operating element, or at least one said information of the transporting element. Determining what was not met and
Modifying the plan to perform at least a portion of the skill in response to the determination that the action did not meet the one or more success criteria.
To move the operating element or at least one of the transporting elements based on the modification plan.
The apparatus according to claim 1, wherein the device is configured to further perform. The map of the environment comprises at least a semantic layer and a context layer, and the processor.
To store information related to one or more objects in the environment in the semantic layer.
Determining one or more behaviors to be performed near the one or more objects based on the information associated with the one or more objects.
Saving the one or more behaviors in the context layer
The device according to claim 1, wherein the map of the environment is configured to be updated by the device. The processor
To display to the user the plan to perform at least a portion of the skill.
After the display, receiving an input from the user indicating confirmation to start executing the at least a part of the skill.
Is configured to do more,
The device of claim 1, wherein the processor is configured to move at least one of the operating element or the transporting element in response to receiving the input. The processor
To display to the user the plan to perform at least a portion of the skill.
After the display, receiving an input from the user indicating a change to the plan,
Modifying the plan based on the input
Is configured to do more,
The apparatus according to claim 1, wherein the processor is configured to move at least one of the operating element or the transporting element based on the plan after the modification of the plan. The operating element comprises a plurality of movable components connected by a plurality of joints.
The set of sensors
A sensor configured to measure the torque acting on a joint from the plurality of joints, or the engagement of a movable component from the plurality of movable components with a physical object from the set of the physical objects is detected. Sensors configured to
The device of claim 1, comprising at least one of the above. The device of claim 1, wherein the operating element comprises an end effector configured to engage one or more objects in the environment. Operation elements and
With transport elements configured to move along the surface,
With a set of sensors
Memory and
A processor operably coupled to the memory, the operating element, the transport element, and the set of sensors.
Using the set of sensors to obtain environmental information,
In response to receiving the information in the environment, the computing device sends the information in the environment to the computing device so that the information in the environment is presented to the user.
Receiving input from the user from the computing device
To select a skill model based on the information in the environment and the input.
Using the model to generate a plan to perform at least a portion of the skill, based on the information in the environment and the input.
To move the operating element or at least one of the transporting elements based on the plan.
With a processor configured to do
Has a device. The input comprises an instruction to acquire additional information about the environment, the skill is involved in the move from the first place to the second place, and the processor.
Using the set of sensors to obtain said additional information about the environment, including information related to at least one object in the environment.
Sending the additional information of the environment to the computing device and
Is configured to do more,
A claim configured in which the processor is configured to move the operating element or at least one of the transport elements by operating the transport element to move from the first location to the second location. Item 10. The apparatus according to item 10. The input exhibits the characteristics of the object in the environment, the skill is involved in the interaction with the object, and the processor identifies the object in the environment at least partially based on the input. 10. The apparatus of claim 10, further configured as such. The input comprises a command to perform a social behavior, the skill is involved in an interaction with at least one human, and the processor is:
Identifying the at least one person in the environment and
Performing the social behavior in the vicinity of the at least one person,
10. The apparatus of claim 10. The operating element comprises an end effector configured to grab one or more objects in the environment, and the skill is involved in moving the object from a first location to a second location. By moving the operating element so that the processor grabs the object and moving the transport element so as to move from the first location to the second location, the operating element and the transport. 10. The device of claim 10, configured to move at least one of the elements. The input comprises an instruction to modify the model, the processor.
Further configured to modify the model based on the input,
10. The device of claim 10, wherein the processor is configured to use the model after the modification to generate the plan to perform said at least a portion of the skill. 10. The apparatus of claim 10, wherein the input comprises the model, wherein the model is configured by the user in the computing device based on the information in the environment. Obtaining environmental information using the first set of sensors in a robotic device,
To select a skill model stored in the memory of the robot device based on the information in the environment.
Using the model to generate a plan to perform at least a portion of the skill using the model, wherein the plan is at least one of an operating element or a transport element of the robot device. Or to generate, including multiple actions,
To move the operating element or at least one of the transporting elements based on the plan.
When the operating element or at least one of the transport elements is moving according to the plan, the environment, the operating element, or the transport element is used with the second set of sensors of the robot device. To get at least one piece of information
Updating the map of the environment stored in the memory based on the at least one piece of information about the environment, the operating element, or the transport element.
The method. Sending information indicating the update to the map of the environment to the computing device
Receiving from the computing device information indicating an update to the map of the environment received by the computing device from a set of robotic devices other than the robotic device.
Further have a method.

Images