JP6526613B2 - Mobile robot system - Google Patents

Description

[Cross-reference to related applications]
This application is based on US Provisional Application No. 61 / 428,717 filed Dec. 30, 2010, US Provisional Application No. 61 / 428,734 filed Dec. 30, 2010, 2010 U.S. Provisional Patent Application No. 61 / 428,759 filed on December 30, and US Patent Law Section 119 (e) to US Provisional Application No. 61 / 429,863 filed on January 5, 2011 Claim priority under The disclosures of these prior applications are considered to be part of the disclosure of the present application and are incorporated herein by reference in their entirety.

  The present disclosure relates to a mobile robotic system that incorporates cloud computing.

  Robots are generally electromechanical machines that are guided by computer or electronic programming. Mobile robots have the ability to move around in their environment and are not fixed at one physical location. Examples of mobile robots commonly used today are automated guided vehicles or automatic guided vehicles (AGVs). AGVs are mobile robots that generally follow markers or wires in the floor or use a vision system or laser for navigation. Mobile robots can be found in the industrial, military and security environments. They also appear as consumer products that perform specific tasks such as entertainment or vacuum cleaner cleaning and home support.

  One aspect of the present disclosure is to communicate with a mobile robot having a controller that executes a control system for controlling the motion of the robot, a cloud computing service in communication with the controller of the robot, and a cloud computing service A robotic system is provided that includes a remote computing device. The remote computing device communicates with the robot through a cloud computing service.

  Implementations of the present disclosure may include one or more of the following features. In some implementations, the remote computing device executes an application for generating a layout map of a robot operating environment. The remote computing device may store the layout map in the external cloud storage using a cloud computing service. In some embodiments, the robot's controller accesses the layout map through the cloud computing service to issue drive commands to the robot's drive system.

  The remote computing device may execute an application (eg, a software program or routine) that provides remote control of the robot. For example, the application may drive the robot, change the pose of the robot, view the video from the camera of the robot, and manipulate the camera of the robot (eg, move the camera, and / or the camera And providing control for at least one of taking a snapshot or a picture using

  In some implementations, the remote computing device executes an application that provides a video conference between a user of the computing device and a third party within the field of view of the robot's camera. The remote computing device may execute an application to schedule use of the robot. Also, the remote computing device may execute an application to monitor the use and operation of the robot. The remote computing device may optionally comprise a tablet computer having a touch screen.

  Another aspect of the present disclosure is a mobile robot having a controller executing a control system for controlling the motion of the robot, a computing device in communication with the controller, and a cloud in communication with the computing device. A robotic system is provided that includes a computing service and a portal in communication with the cloud computing service.

  Implementations of the present disclosure may include one or more of the following features. In some implementations, the portal comprises a web-based portal that provides access to content. The portal may receive robot information from the robot through the cloud computing service. Also, the robot may receive user information from the portal through the cloud computing service.

  In some embodiments, the computing device includes a touch screen (eg, on a tablet computer). The computing device may run an operating system different from that of the controller. For example, the controller may execute an operating system for robot control while the computing device may execute an enterprise operating system. In some embodiments, the computing device executes at least one application that collects robot information from the robot and transmits the robot information to the cloud computing service.

  The robot may include a base defining a vertical central axis and supporting the controller, and a holonomic drive system supported by the base. The drive system comprises first, second and third drive wheels, each of which is triangularly spaced around the vertical central axis and has drive directions perpendicular to the radial axis with respect to the vertical central axis. The robot may also include an expandable leg extending upwardly from the base and a torso supported by the leg. Actuation of the legs causes a change in height of the torso. The computing device can be removably supported above the torso. In some embodiments, the robot includes a neck supported by the torso and a head supported by the neck. The neck may be able to pivot and tilt the head relative to the torso. The head may releasably support the computing device.

  Another aspect of the present disclosure is a mobile robot having a controller executing a control system for controlling motion of the robot, a computing device in communication with the controller, and a controller and computing device. A robotic system is provided that includes an intermediary security device controlling communication, a cloud computing service in communication with the computing device, and a portal in communication with the cloud computing service.

  In some embodiments, the intermediary security device translates communication between the computing device communication protocol of the computing device and the robot communication protocol of the robot. Additionally, the intermediary security device may include an approval chip for approving communication traffic between computing devices in the robot.

  The computing device may communicate with the robot controller wirelessly. In some embodiments, the computing device is releasably attachable to the robot. An exemplary computing device includes a tablet computer.

  The portal may be a web-based portal that provides access to content (eg, news, weather, robot information, user information, etc.). In some embodiments, the portal receives robot information from the robot through a cloud computing service. In a further embodiment, the robot receives user information from the portal through a cloud computing service. The computing device may access the cloud storage using a cloud computing service. The computing device may execute at least one application that collects robot information from the robot and transmits the robot information to the cloud computing service.

  One aspect of the present disclosure corresponds to receiving a layout map corresponding to the environment of the robot, moving the robot in the environment to a layout map position on the layout map, corresponding to the environment, and generated by the robot Recording the robot map position on the robot map, determining the distortion between the robot map and the layout map using the recorded robot map position and the corresponding layout map position, and a corresponding target A method is provided for operating a mobile robot, including applying the determined strain to a target layout map position to determine a robot map position.

  Implementations of the present disclosure may include one or more of the following features. In some implementations, the method includes receiving a layout map from a cloud computing service. The method may include generating a layout map on an application executed on the computing device and storing the layout map on the remote cloud storage device using a cloud computing service.

  In some embodiments, the method determines the scaling size, starting point mapping, and rotation between the layout map and the robot map using the existing layout map position and the recorded robot map position. Resolving the robot map position corresponding to the target layout map position. The method may further include applying an affine transformation to the determined scaling size, origin mapping, and rotation to resolve the target robot map position.

  In some implementations, the method includes determining triangulation between layout map locations indicating boundaries of target layout map locations. The method further determines the scale, rotation, translation, and skew between the mapped triangle in the layout map and the corresponding triangle mapped in the robot map, and the corresponding target robot Applying the determined scale, rotation, translation, and skew to the target layout map position may be included to determine the map position.

  The method, in some embodiments, determines the distance between all layout map locations and the target layout map location, determines the center of gravity of the layout map locations, and all recorded robot maps. Rotation and length scaling to determine the center of gravity of the position, and convert the vectors that extend from the layout map center of gravity to the target layout position to vectors that extend from the robot map center of gravity to the target robot map position for each layout map position. And determining.

  The method may include generating a robot map using a sensor system of the robot. In some implementations, the method comprises emitting light over the scene of the environment, receiving reflections of radiation from the surface of the scene, determining the distance of each reflective surface, and 3 of the scene. And (d) constructing a dimensional depth map. The method may include emitting a speckle pattern of light over the scene and receiving a reflection of the speckle pattern from the scene. In some embodiments, the method includes storing a reference image of a speckle pattern as reflected from a reference object in a scene, wherein the reference image is captured at various different distances from the reference object . The method further captures at least one target image of a speckle pattern as reflected from a target object in the scene and references the at least one target image to determine the distance of the reflective surface of the target object. It may include comparing with the image. In some embodiments, the method determines a first order speckle pattern on the target object, and at least one of cross-correlation and uncorrelation of each of the first order speckle pattern and the speckle pattern of the reference image. Calculating one. The method may include maneuvering the robot relative to the target object based on the determined distance of the reflective surface of the target object.

  In some implementations, the method includes determining a time of flight between the emission of light and the reception of the reflected light to determine the distance to the reflective surface of the scene. The method may include emitting light onto the scene in intermittent pulses. The method may also include changing the frequency of the emitted light pulse.

  In yet another aspect, a robotic system includes a moveable robot having a controller that executes a control system for controlling the motion of the robot and a sensor system in communication with the controller. The robot system also includes cloud computing services in communication with the controller of the robot. The cloud computing service receives data from the controller, processes the data, and sends the processed result back to the controller.

  In some implementations, the cloud computing service at least temporarily stores the received data in the cloud storage, and optionally discards the stored data after processing the data. The robot is, in some embodiments, in communication with the controller and with a camera capable of acquiring an image of a scene around the robot, and / or with the controller, and the volume around the robot And a volume point cloud imaging device capable of acquiring a point cloud from space. The volume may include a floor in the direction of movement of the robot. The controller communicates image data to the cloud computing service.

  The data may comprise data comprising raw sensor data and / or related information from a sensor system. In some embodiments, the data includes image data having at least one of accelerometer data tracking, distance measurement data, and a time stamp.

  The cloud computing service may receive image data of a scene around the robot from the controller and process the image data into a 3-D map and / or model of the scene. Additionally, the cloud computing service may provide a 2-D height map and / or model to the controller, where the cloud computing service calculates the 2-D height map from the 3-D map. In some embodiments, the cloud computing service periodically receives image data and processes the received image data after accumulating up to a threshold image data set.

  In some implementations, the controller is in communication with the controller, and optionally, cloud computing data wirelessly through the portable computing device (eg, a tablet computer), which is removably attachable to the robot Communicate to service. The controller may buffer data and periodically transmit data to the cloud computing service.

  The sensor system may include at least one of a camera, a 3-D imaging sensor, a sonar sensor, an ultrasonic sensor, a LIDAR, a LADAR, an optical sensor, and an infrared sensor.

  In another aspect of the disclosure, a method of operating a mobile robot includes moving a robot around a scene, receiving sensor data indicative of the scene, processing the received sensor data, and processing the processing result Communicating sensor data to a cloud computing service communicating with the The method further includes moving the robot within the scene based on the received processing results.

  In some implementations, the method includes emitting light onto a scene around the robot and capturing an image of the scene along the driving direction of the robot. The image includes at least one of (a) a three-dimensional depth image, (b) an active illumination image, and (c) an ambient illumination image. The sensor data includes an image, and the processing result includes a map or model of a scene.

  The method comprises emitting a speckle pattern of light onto a scene, receiving a reflection of the speckle pattern from an object in the scene, and a reference image of the speckle pattern as reflected from a reference object in the scene. Storing in a cloud storage of the cloud computing service. The reference image is captured at various different distances from the reference object. The method also includes capturing at least one target image of a speckle pattern as reflected from target objects in the scene and communicating the at least one target image to a cloud computing service. The cloud computing service compares the at least one target image to a reference image to determine the distance of the reflective surface of the target object. In some embodiments, the method determines a first order speckle pattern on the target object, and at least one of cross-correlation and uncorrelation of each of the first order speckle pattern and the speckle pattern of the reference image. Calculating one.

  The cloud computing service may at least temporarily store received sensor data in the cloud storage, and optionally discard the stored sensor data after processing the data. The sensor data may include image data with associated sensor system data that may include at least one of accelerometer data tracking, mileage measurement data, and a time stamp.

  In some implementations, the cloud computing service receives image data from the robot and processes the image data into a 3-D map and / or model of the scene. The cloud computing service may provide a 2-D height map and / or model to the robot. The cloud computing service calculates a 2-D height map from the 3-D map.

  The method may include regularly communicating sensor data to a cloud computing service, which processes image data received after accumulating a threshold sensor data set. In some embodiments, the method is wirelessly clouding sensor data wirelessly through a portable computing device (eg, a tablet computer) in communication with the robot and optionally, removably attachable to the robot. Including communicating to computing services.

  In another aspect, a method of navigating a mobile robot comprises capturing a streaming sequence of a high density image of a scene around a robot along a robot's motion trajectory at a real time capture speed, and high density annotations. Associating with at least some of the images. The method also includes transmitting the high density images and annotations to the remote server at a transmission rate slower than the real time acquisition rate and receiving the data set from the remote server after the processing time interval. The data set originates from and represents at least a portion of the high density image sequence and the corresponding annotation, but excluding the raw image data of the high density image sequence. The method includes moving the robot relative to the scene based on the received data set.

  The method may include transmitting the high density images and annotations to the local server and buffer, and then transmitting the high density images and annotations to the remote server at a transmission rate slower than the real time capture rate. The local server and buffer may be within a relatively short range (e.g., 20 to 100 feet or within wireless communication range) of the robot.

  In some implementations, the annotations include at least one of: a time stamp, such as an absolute time reference, corresponding to at least some of the high density images; and travel distance measurement data, accelerometer data, tilt data, and angular velocity data. And attitude related sensor data, which may include one. The annotations can be associated with a high density image reflecting the hazard event captured within the time interval for the robot's hazard response (eg, avoiding a cliff, exiting a containment situation, etc.). In further embodiments, associating the annotation may include associating a key frame identifier with the subset of high density images. The key frame identifier may enable identification of high density images based on characteristics of the key frame identifier (eg, flag, type, group, etc.).

  The annotations may include a low density set of 3-D points derived from structural and motion restoration of features tracked between high density images in a high density image streaming sequence. The low density set of 3-D points may come from a volumetric point imaging device on the robot. Additionally, the annotations may include camera parameters such as camera pose relative to individual 3-D points of the low density set of 3-D points. The markers of the transmissible and non-crossable areas of the scene may be annotations to the high density image.

  The data set includes one or more texture maps, such as 2-D height maps, extracted from high density images, and / or terrain maps representing features in high density images of the scene. May be. The data set may include trained classifiers to classify features in the captured new dense image of the scene.

  In yet another aspect, a method of abstracting mobile robot environment data includes receiving a sequence of high density images of a robot environment from a mobile robot at a receive rate. High density images are captured along the trajectory of the mobile robot's motion at real time capture speed. The reception rate is slower than the real time acquisition rate. The method further comprises receiving an annotation associated with at least some of the high density images in the sequence of high density images, high density data in at least some of the high density images, Dispatching a batch processing task to reduce to a data set that represents at least a portion of the sequence of images. The method also includes transmitting the data set to the mobile robot. The data set excludes raw image data of high density image sequences.

  In some implementations, batch processing tasks include processing sequences of high density images into high density 3-D models of a robotic environment, processing high density 3-D models into 2-D locations and Making a terrain model for the coordinate system at least one height above the floor surface. In some embodiments, the terrain model is for a coordinate system of 2-D locations and multiple occupied and unoccupied height boundaries from the floor surface. For example, a terrain model that allows the robot to determine if a room with a table can pass under the table, providing data indicating the upper and lower heights of the associated table surface It is.

  Batch processing tasks accumulate high-density image sequences corresponding to multiple robot environments (eg, so that the cloud can build classifiers to identify desired features in any environment) May be included. Thus, batch processing tasks may include training one or more classifiers on a plurality of classifiers on a sequence of high density images and / or a sequence of high density images. For example, the batch processing task may associate an annotation reflecting a dangerous event with a high density image captured within a time interval for the mobile robot's dangerous response, for example, a data set of classifier model parameters Training the hazard related high density image classifier using the related risk event annotation and the corresponding high density image as training data, as provided. The classifier may include at least one support vector machine that builds at least one hyperplane for classification, and model parameters may be trained hyperplanes that can classify the data set into risk related classifications. Define. The model parameters may include sufficient parameters to define the kernel of the support vector machine and soft margin parameters.

  In some embodiments, batch processing tasks include instantiating a plurality of scalable virtual processes that are proportional to the scale of the high density image sequence being processed. At least some of the virtual processes are released after transmission of the data set to the robot. Similarly, batch processing tasks may include instantiating a plurality of scalable virtual memories that are proportional to the scale of the stored high density image sequence. At least some of the virtual memory is released after transmission of the data set to the robot. In addition, the batch processing task may also include distributing scalable virtual servers according to one of geographic proximity to mobile robots and / or network traffic from mobile robots. Good.

  The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

FIG. 1 is a perspective view of an example mobile human interface robot. FIG. 1 is a schematic view of an example mobile human interface robot. FIG. 1 is an elevated perspective view of an example mobile human interface robot. FIG. 1 is a front perspective view of an exemplary base of a mobile human interface robot. FIG. 4B is a rear perspective view of the base shown in FIG. 4A. FIG. 4B is a top view of the base shown in FIG. 4A. FIG. 1 is a front schematic view of an exemplary base of a mobile human interface robot. FIG. 1 is a top schematic view of an example base of a mobile human interface robot. FIG. 1 is a front view of an exemplary holonomic wheel for a mobile human interface robot. FIG. 5C is a side view of the wheel shown in FIG. 5C. FIG. 1 is a front perspective view of an example trunk of a mobile human interface robot. FIG. 1 is a front perspective view of an example barrel with touch sensing capabilities of a mobile human interface robot. It is a bottom perspective view of the trunk | drum shown by FIG. 6B. FIG. 1 is a front perspective view of an exemplary neck of a mobile human interface robot. FIG. 1 is a schematic diagram of an example circuit of a mobile human interface robot. FIG. 1 is a schematic diagram of an example circuit of a mobile human interface robot. FIG. 1 is a schematic diagram of an example circuit of a mobile human interface robot. FIG. 1 is a schematic diagram of an example circuit of a mobile human interface robot. FIG. 1 is a schematic diagram of an example circuit of a mobile human interface robot. FIG. 1 is a schematic diagram of an example circuit of a mobile human interface robot. FIG. 1 is a schematic diagram of an example circuit of a mobile human interface robot. FIG. 1 is a perspective view of an example mobile human interface robot having a removable web pad. FIG. 1 is a perspective view of a person interacting with an example mobile human interface robot. FIG. 1 is a perspective view of a person interacting with an example mobile human interface robot. FIG. 1 is a perspective view of a person interacting with an example mobile human interface robot. FIG. 1 is a perspective view of a person interacting with an example mobile human interface robot. FIG. 1 is a perspective view of a person interacting with an example mobile human interface robot. FIG. 1 is a schematic view of an example mobile human interface robot. FIG. 1 is a perspective view of an example mobile human interface robot having a plurality of sensors directed to the ground. FIG. 1 is a schematic view of an example imaging sensor for sensing an object in a scene. FIG. 6 is a schematic view of an example arrangement of operations for operating an imaging sensor. FIG. 1 is a schematic diagram of an exemplary three-dimensional (3D) speckle camera sensing an object in a scene. FIG. 7 is a schematic diagram of an example arrangement of operations for operating a 3D speckle camera. FIG. 1 is a schematic view of an exemplary 3D time-of-flight (TOF) camera for sensing objects in a scene. FIG. 1 is a schematic view of an example arrangement of operations for operating a 3D TOF camera. FIG. 5 is a schematic diagram of an exemplary control system implemented by a controller of a mobile human interface robot. FIG. 1 is a perspective view of an example mobile human interface robot that receives human touch commands. An exemplary telephony schematic is provided for initiating and communicating with a mobile human interface robot. 1 provides a schematic of an example robotic system architecture. 1 provides a schematic of an example robotic system architecture. 1 provides a schematic of an example robotic system architecture. 1 provides a schematic of an example robotic system architecture. 1 provides a schematic of an example robotic system architecture. An example arrangement of operations for a method of navigating a mobile robot is provided. An example arrangement of operations for a method of abstracting mobile robot environmental data is provided. 1 provides a schematic of an example robotic system architecture. FIG. 2 is a schematic diagram of an example occupancy map. FIG. 1 is a schematic view of a mobile robot with a view of a scene in a work area. FIG. 1 is a schematic diagram of an example layout map. FIG. 18B is a schematic view of an example robot map corresponding to the layout map shown in FIG. 18A. The layout map and the robot map are used to provide an example arrangement of operations for operating the mobile robot to navigate around the environment. FIG. 7 is a schematic diagram of an exemplary layout map with triangulation-type layout points. FIG. 19B is a schematic view of an example robot map corresponding to the layout map shown in FIG. 19A. The layout map and the robot map are used to provide an example arrangement of operations for determining a target robot map position. FIG. 5 is a schematic diagram of an example layout map with a tight layout point centroid. FIG. 20B is a schematic view of an example robot map corresponding to the layout map shown in FIG. 20A. The layout map and the robot map are used to provide an example arrangement of operations for determining a target robot map position. FIG. 6 provides an exemplary schematic of a local perceptual space while the mobile human interface robot is stationary. FIG. 7 provides an exemplary schematic view of a local perceptual space during movement of a mobile human interface robot. FIG. 2 provides a schematic representation of the local perceptual space while the mobile human interface robot is stationary. FIG. 7 provides an exemplary schematic view of a local perceptual space during movement of a mobile human interface robot. An example schematic view of a mobile human interface robot with a corresponding sensor field of view moving closely around a corner is provided. FIG. 7 provides an example schematic view of a mobile human interface robot with a corresponding sensor view that travels widely around a corner.

  Like reference symbols in the various drawings indicate like elements.

  Mobile robots can interact or interact with people to provide numerous services ranging from home support to commercial support and others. In the case of home support, the mobile robot may maintain a dosing plan, move support, communication support (e.g. teleconferencing, telephone communication, internet access etc), residence or site monitoring (indoor and / or outdoors), person monitoring, And / or support the daily life of the elderly, including but not limited to the provision of a personal emergency response system (PERS). In commercial support, the mobile robot can provide video conferencing (eg, in a hospital setting), a point of sale terminal, interactive information / marketing terminal, and the like.

  Referring to FIGS. 1-2, in some implementations, mobile robot 100 includes a robot body 110 (or chassis) that defines a forward drive direction F. The robot 100 also includes a drive system 200, an interface module 300, and a sensor system 400, each supported by the robot body 110 and in communication with a controller 500 that coordinates the movement and movement of the robot 100. A power source 105 (eg, battery (s)) may be carried by the robot body 110, in electrical communication with each of these components, and deliver power thereto as needed. For example, controller 500 may include a computer capable of greater than 1000 MIPS (one million instructions per second), and power supply 1058 provides a sufficient battery to power the computer for more than three hours.

In the illustrated embodiment, the robot body 110 includes a base 120, at least one leg 130 extending upwardly from the base 120, and a torso 140 supported by the at least one leg 130. Base 120 may support at least a portion of drive system 200. Robot body 110 also includes a neck 150 supported by torso 140. The neck 150 supports the head 160, and the head 160 supports at least a portion of the interface module 300. The base 120 is supported to maintain the mechanical stability, comprising a sufficient weight to maintain a low overall center of gravity CG _R of low center of gravity CG _B and robot 100 of the base 120 (e.g., power supply 105 (the battery) By).

  With reference to FIGS. 3 and 4A-4C, in some implementations, the base 120 defines a trigonally symmetrical shape (e.g., a triangular shape in top view). For example, the base 120 supports a base body 124 having first, second and third base body portions 124a, 124b, 124c corresponding to each leg of a triangular base 120 (see, for example, FIG. 4A) The base chassis 122 may be included. Each base body portion 124a, 124b, 124c can be movably supported by the base chassis 122 such that it can move independently relative to the base chassis 122 in response to contacting an object. The triangular symmetry shape of the base 120 enables collision detection of 360 degrees around the robot 100. Each base body portion 124a, 124b, 124c can have an associated contact sensor (eg, a capacitive sensor, reed switch, etc.) that detects movement of the corresponding base body portion 124a, 124b, 124c relative to the base chassis 122.

    In some implementations, drive system 200 provides omnidirectional and / or holonomic motion control of robot 100. As used herein, the term "omnidirectional" refers to the ability to move in virtually any planar direction, ie, the ability to move laterally (laterally), forward / backward, and rotationally. These directions are generally referred to herein as x, y, and θz, respectively. Furthermore, the term "holonomic" substantially corresponds to the use of the term in the literature and refers to the ability to move in a planar direction with three planar degrees of freedom, ie two translations and one rotation. Thus, the holonomic robot continues the ability to move in the planar direction at a velocity comprised of substantially any ratio of three planar velocities (front / rear, lateral, and rotational), as well as substantially three ratios. Have the ability to change

The robot 100 can operate within a human environment (e.g., an environment typically designed for a bipedal walking resident) using wheel mobility. In some implementations, drive system 200 includes first, second, and third drive wheels 210a, 210b, 210c (i.e., spaced 120 degrees apart) evenly spaced about vertical axis Z (i.e., (Triangular symmetry), however, other arrangements are possible as well. Referring to FIGS. 5A and 5B, drive wheels 210a, 210b, 210c can facilitate the maneuverability of holonomic drive system 200, laterally arched rolling surfaces (ie, across rolling direction D _R , or It is also possible to define a contour (curved in a direction perpendicular to it). Each drive wheel 210a, 210b, 210c can drive the drive wheel 210a, 210b, 210c in a forward and / or reverse direction independently of the other drive motors 220a, 220b, 220c, respectively, drive motor 220a, 220b and 220c. Each drive motor 220a-c can have a respective encoder 212 (FIG. 8C) that provides controller 500 with wheel rotation feedback. In some embodiments, each drive wheel 210a, 210b, 210c is mounted on or near one of the three points of the equilateral triangle, relative to the bisector of the corner of the respective triangle end Drive directions (forward and reverse). The drive of the trigonal symmetrical holonomic base 120 with forward drive direction F autonomously escapes from confinement or scatter and then rotates and / or translates to drive along forward drive direction F after escape is resolved Enables the robot 100 to move in the non-forward drive direction.

Referring to FIG. 5C and 5D, in some implementations, each of the drive wheels 210 each have a rotational direction D _R perpendicular rotational direction D _r of the drive wheel 210, the inner and outer rows 232 of rollers 230, Including 234. The rows 232, 234 of the rollers 230 can be alternated (e.g., such that one roller 230 of the inner row 232 is equally positioned between two adjacent rollers 230 of the outer row 234). The roller 230 provides infinite sliding perpendicular to the drive direction of the drive wheel 210. The rollers 230 define an arcuate (e.g., convex) outer surface 235 perpendicular to their rotational direction D _r such that the rollers 230 together define a circular or nearly circular perimeter of the drive wheel 210. The contour of the roller 230 influences the overall contour of the drive wheel 210. For example, the rollers 230 may define an arcuate outer roller surface 235 that together define the corrugated rotational surface of the drive wheel 210 (eg, as a tread for static friction). However, by configuring the roller 230 to have a contour that defines a circular generally rotating surface of the drive wheel 210, the robot 100 moves smoothly on a plane instead of vibrating vertically at the tread of the wheel Make it possible. In order to climb an object as high, or nearly as high, as the wheel radius R of the drive wheel 210, alternating rows 232, 234 of rollers 230 (of radius r) as it approaches the object at an angle It can be used as a tread.

In the embodiment shown in FIGS. 3 to 5B, the first drive wheel 210a is arranged as a tip drive wheel along the forward drive direction F and the remaining two drive wheels 210b, 210c follow from behind . In this arrangement, to move forward, the controller 500 slides the first drive wheel 210a along the forward drive direction F while advancing the second and third drive wheels 210b, 210c at an equal speed. A drive command may be issued to drive in the rolling direction. Furthermore, this drive wheel arrangement enables the robot 100 to stop suddenly (e.g. generate a sudden negative acceleration in the forward drive direction F). This is due to the natural dynamic instability of the three-wheel design. If the forward drive direction F is along the bisector of the angle between the two forward drive wheels, the sudden stop will produce a torque that can cause the robot 100 to tip over, of the two "front" wheels. Pivot up. Instead, forward movement with one drive wheel 210a necessarily supports or prevents the robot 100 from tipping forward, when it needs to stop quickly. However, when accelerating from a stop, the controller 500, the inertia moment I from the total center of gravity CG _R of the robot 100 may be considered.

In some implementations of the drive system 200, each of the drive wheels 210a, 210 b, 210 has a rolling direction D _R is aligned to the vertical axis Z and the radial direction perpendicular to the X and Y axes of the robot 100. The first drive wheel 210a can be arranged as a tip drive wheel along the forward drive direction F, and the remaining two drive wheels 210b, 210c follow behind. In this arrangement, to move forward, the controller 500 drives the first drive wheel 210a in the forward direction, and the second and third drive wheels 210b and 210c have the same speed as the first drive wheel 210a. Drive commands may be issued to drive in the reverse direction.

  In another implementation, drive system 200 is positioned such that the bisector of the angle of the angle between the two drive wheels 210a, 210b is aligned with the forward drive direction F of robot 100. It can be arranged to have a second drive wheel 210a, 210b. In this arrangement, to move forward, the controller 500 drives the first and second drive wheels 210a, 210b in the forward direction at an equal speed, while driving the third drive wheel 210c in the reverse direction. A drive command may be issued that causes the vehicle to idle or to be kept idle and dragged behind the first and second drive wheels 210a, 210b. In order to turn left or right while advancing, the controller 500 may issue a command to drive the corresponding first or second drive wheel 210a, 210b at a relatively faster / slower speed. Other drive system 200 arrangements can be used as well. The drive wheels 210a, 210b, 210c may define a cylindrical, circular, oval or polygonal contour.

  Referring again to FIGS. 1-3, base 120 supports at least one leg 130 extending upwardly from base 120 in the Z direction. The leg (s) 130 may be configured to have a variable height to raise and lower the torso 140 relative to the base 120. In some implementations, each leg 130 includes first and second leg portions 132, 134 that move relative to one another (eg, telescoping, linear, and / or angular movement). In the illustrated embodiment, the second leg portions 134 have continuously smaller diameter extrusions that telescopically move in and out of each other and out of the relatively larger base extrusions. Rather, telescopically moving over the first leg portion 132 and thus placing other components along the second leg portion 134 and potentially the second leg portion Together with 134, it can be moved to the relative proximity of the base 120. The leg 130 may include an actuator assembly 136 (FIG. 8C) for moving the second leg portion 134 relative to the first leg portion 132. The actuator assembly 136 may include a motor driver 138a in communication with the lift motor 138b and the encoder 138c to provide position feedback to the controller 500.

Generally, nested telescopic arranged in order to keep as low as possible the center of gravity CG _L of the entire leg 130, the base 120, expands and contracts moved telescopically upward out of the relatively larger extrusion unit, Includes continuously smaller diameter extrusions. In addition, more robust and / or larger components can be placed at the bottom to account for the greater torque that the base 120 receives when the legs 130 are fully extended. However, this approach presents two problems. First, when relatively smaller components are placed on top of the legs 130, any rain, dust, or other particulates tend to flow or roll down the extrusion, and between the extrusions. Into the void space, thus plugging the extrusions. This results in a very difficult sealing problem while still trying to maintain full mobility / articulation of the leg 130. Second, it may be desirable to mount a load or accessory on the robot 100. One common place for mounting accessories is at the top of the torso 140. If the second leg portion 134 is telescoping out of the first leg portion, the accessories and components may be extended to the entire second leg portion 134 if they need to move with the barrel 140. It can only be mounted above. Otherwise, any component mounted on the second leg portion 134 will limit the telescoping movement of the leg 130.

  By telescopically moving the second leg portion 134 on the first leg portion 132, the second leg portion 134 can be moved vertically relative to the base 120, an additional load attachment Provide points. This type of arrangement is such that water or airborne particulates do not enter the space between the leg portion 132 and the leg portion 134, and the outer side of each leg portion 132, 134 (e.g., an extruded portion) of the torso portion 140. Let it flow down. This makes it much easier to seal any joints of the legs 130. Furthermore, the load / accessory mounting features of the torso 140 and / or the second leg portion 134 are always exposed and available regardless of how far the leg 130 is extended.

  Referring to FIGS. 3 and 6A, the leg (s) 130 support the torso 140, which may have a shoulder 142 extending above and above the base 120. In the illustrated embodiment, the barrel 140 has a downward facing surface or bottom surface 144 (e.g., toward the base) and an opposite upward facing surface or top surface 146 that form at least a portion of the shoulder 142. With a side 148 extending therebetween. The body 140 comprises a central portion 141 supported by the leg (s) 130 and a peripheral free portion 143 extending laterally beyond the extent of the leg (s) 130. It may define various shapes or geometric shapes, such as circular or oval shapes, thus providing an overhanging portion defining the downwardly facing surface 144. In some embodiments, the torso 140 defines a step that provides a flared portion that extends beyond the leg (s) 130 on the base 120, such as a polygon or other complex shape Define

  The robot 100 may include one or more accessory ports 170 (eg, mechanical and / or electrical interconnection points) to receive the load. The accessory port 170 may be positioned such that the received load does not block or obstruct the sensor of the sensor system 400 (eg, such as on the bottom 144 and / or the top 146 of the barrel 140). it can. In some implementations, as shown in FIG. 6A, the torso 140 may, for example, receive a load within the basket 360 and may be on the front portion 147 of the torso 140 or other portions of the robot body 110. One or more accessory ports 170 are included on the rear portion 149 of the barrel 140 so as not to obstruct the sensor.

The outer surface of the torso 140 may be sensitive to touch or touch by the user to receive touch commands from the user. For example, when the user touches the top surface 146 of the body 140, the robot 100, by lowering the height H _T of the body portion relative to the floor (e.g., the height of the leg portion 130 supporting the torso 140 Respond by decreasing H _L ). Similarly, when the user touches the bottom surface 144 of the torso 140, the robot 100 raises the torso 140 relative to the floor (eg, the height H _L of the legs 130 supporting the torso 140) Respond by increasing Further, upon receiving a user touch on the front, back, right, or left portions of the side surface 148 of the torso 140, the robot 100 receives the corresponding direction of the received touch command (eg, back, front, left, and Respond by moving to the right). The outer surface of the torso 140 may include a capacitive sensor in communication with the controller 500 to detect user touch.

  Referring to FIGS. 6B and 6C, in some implementations, the torso body 140 has an upper panel 145t, a bottom panel 145b, a front panel 145f, a back panel 145b, a right panel 145r, and a left panel 145l. Including 145. Each panel 145t, 145b, 145f, 145r, 145r, 145l may move independently with respect to the other panels. Further, each panel 145t, 145b, 145f, 145r, 145r, 145l is associated with an associated movement and / or contact sensor 147t, 147b, in communication with the controller 500, which detects movement and / or contact with the respective panel. It may have 147f, 147r, 147r, 147l.

Referring again to FIGS. 1-3 and 7, the torso 140 supports a neck 150 that provides for pivoting and tilting of the head 160 relative to the torso 140. In the illustrated embodiment, the neck 150 includes a pivot 152 and a ramp 154. The rotating portion 152 may provide a range of angular movement θ _R (eg, centered on the Z-axis) of about 90 ° to about 360 °. Other ranges are possible as well. Further, in some embodiments, the rotating portion 152 maintains the electrical communication between the head 160 and the remainder of the robot 100, and the head 150 relative to the barrel 140 for an infinite number of rotations. Includes electrical connectors or contacts that allow for a continuous 360 ° rotation of the Inclined portion 154 is the same or similar electrical connector or allowing electrical rotation of head 160 relative to barrel 140 while maintaining electrical communication between head 160 and the remainder of robot 100. It may include contact. The rotating portion 152 may include a rotating portion motor 152m coupled to or mated to a ring 153 (eg, a toothed ring rack). The inclined portion 154 may move the head at an angle θ _T with respect to the trunk portion 140 independently of the rotating portion 152 (for example, around the Y axis). In some embodiments, the ramp 154 includes a ramp motor 155 that moves the head 160 at an angle θ _T of ± 90 ° with respect to the Z-axis. Other ranges, such as ± 45 °, are likewise possible. The robot 100 is configured such that the leg (s) 130, the torso 140, the neck 150, and the head 160 remain within the perimeter of the base 120 to maintain stable movement of the robot 100. May be In the exemplary circuit diagram shown in FIG. 8F, the neck 150 includes a pivot and tilt assembly 151 that includes a pivot 152 and a ramp 154 along with corresponding motor drivers 156a, 156b and encoders 158a, 158b.

  The head 160 may be sensitive to touch or touch by the user to receive touch commands from the user. For example, when the user pulls the head 160 forward, the head 160 tilts forward with passive resistance and then holds position. In addition, if the user has pushed / pulled the head 160 vertically downward, the torso 140 may be lowered to lower the head 160 (via a reduction in the length of the leg 130). The head 160 and / or the neck 150 may include strain gauges and / or contact sensors 165 (FIG. 7) that sense user touch or manipulation.

  8A-8G provide exemplary circuit diagrams of the robot 100. FIG. 8A-8C may include a proximity sensor such as sonar proximity sensor 410 and step proximity sensor 420, contact sensor 430, laser scanner 440, sonar scanner 460, and drive system 200, base 120. Provides an example circuit diagram of The base 120 may also house the controller 500, the power supply 105, and the leg actuator assembly 136. The body 140 receives and responds to the user's touch or touch by the microcontroller 140c, the microphone (s) 330, the speaker (s) 340, the scanning 3-D image sensor 450a, and the controller 500. (Eg, by moving the barrel 140 relative to the base 120, pivoting and / or tilting the neck 150, and / or issuing commands to the drive system 200 in response thereto) , And the body touch sensor system 480. The neck 150 may also include a pivot-tilt assembly 151, which may include a pivot motor 152 with a corresponding motor driver 156a and encoder 158a and a tilt motor 154 with a corresponding motor driver 156b and encoder 158b. Good. The head 160 may house one or more web pads 310 and a camera 320.

  Referring to FIGS. 1-3 and 9, in some implementations, the head 160 supports one or more portions of the interface module 300. The head 160 may include a dock 302 for releasably receiving one or more computing tablets 310, also called web pads or tablet PCs, which may each have a touch screen 312. Web pad 310 may be oriented forward, backward, or upward. In some implementations, the web pad 310 includes a touch screen, an optional input / output (eg, a button and / or a connector such as micro USB), a processor, and memory in communication with the processor. An exemplary web pad 310 is available from Apple, Inc. Includes iPad by. In some embodiments, the web pad and 10 function as controller 500 or assist in controlling controller 500 and robot 100. In some embodiments, the dock 302 may be detached from and attached to the first computing tablet 310a fixedly attached thereto (eg, a wired interface for data transfer at relatively high bandwidth such as gigabyte speeds) And a second computing tablet 310b possibly attached. The second web pad 310b may be received over the first web pad 310a, as shown in FIG. 9, or the second web pad 310b may be headed against the first web pad 310a. It may be received on the opposite side or the opposite side of the portion 160. In a further embodiment, the head 160 supports a single web pad 310 which may be either fixed thereto or removably attached. Touch screen 312 may receive user input and detect, monitor, and / or reproduce points of user touch thereon to provide a graphical user interface that is touch interactive. In some embodiments, web pad 310 includes a touch screen call that allows the user to know when it is removed from robot 100.

  In some implementations, the robot 100 includes a plurality of web pad docks 302 on one or more portions of the robot body 110. In the embodiment shown in FIG. 9, the robot 100 optionally includes a web pad dock 302 disposed on the legs 130 and / or the torso 140. This may, for example, adapt to users of different heights, capture a video using cameras of web pad 310 at different viewpoints, and / or receive a plurality of web pads 310 on robot 100. , Allow the web pad 310 to be docked at different heights on the robot 100.

  The interface module 300 may include a camera 320 disposed on the head 160 that can be used to capture video from a high viewpoint of the head 160 (eg, for video conferencing) (eg, See Figure 2). In the embodiment shown in FIG. 3, the camera 320 is located on the neck 150. In some embodiments, the camera 320 is activated only when the web pad 310, 310a is removed from the head 160 or removed from the dock. When web pad 310, 310a is mounted or docked on head 160 in dock 302 (and optionally over camera 320), robot 100 captures web pad 310a to capture video. You may use the camera of. In such a case, the camera 320 may be placed behind the docked web pad 310 and becomes active when the web pad 310 is removed from or docked from the head 160, It becomes inactive when the web pad 310 is mounted or docked on the head 160.

  The robot 100 can provide video conferencing (e.g., at 24 fps) through the interface module 300 (e.g., using the web pad 310, the camera 320, the microphone 320, and / or the speaker 340). Video conferencing may be multi-person. The robot 100 can provide eye contact between both parties of the video conference by maneuvering the head 160 to face the user. Also, the robot 100 can have a gaze angle of less than 5 degrees (e.g., an angle away from an axis perpendicular to the anterior surface of the head 160). The at least one 3-D image sensor 450 and / or the camera 320 on the robot 100 can capture a full-size image, such as body language. Controller 500 may synchronize audio and video (e.g., by less than 50 ms).

  In the embodiment shown in FIGS. 10A-10E, the robot 100 adjusts the height of the web pad 310 above the head 160 and / or the camera 320 (by raising or lowering the torso 140); Alternatively, by turning and / or tilting the head 160, a teleconference can be provided to a standing or sitting person. Camera 320 may be movable within at least one degree of freedom independent of web pad 310. In some embodiments, the camera 320 is positioned more than 3 feet above ground but not more than 10 percent of the height of the web pad from the top edge of the display area of the web pad 310. It has an objective lens. Additionally, the robot 100 can zoom the camera 320 to capture close-up pictures or videos of the robot 100's surroundings. The head 160 may include one or more speakers 340 to emit sound from the head 160 in the vicinity of the web pad 310 displaying a video conference.

  In some embodiments, the robot 100 can receive user input into the web pad 310 (eg, via a touch screen), as shown in FIG. 10E. In some implementations, the web pad 310 is a display or monitor, but in other implementations the web pad 310 is a tablet computer. The web pad 310 can have easy and intuitive control, such as a touch screen, providing high interactivity. The web pad 310 may have a monitor display 312 (eg, a touch screen) having a display area of 150 square inches (about 968 square centimeters) or more movable in at least one degree of freedom.

  The robot 100 can provide EMR integration by providing video conferencing between a physician and a patient and / or other physicians or nurses, in some embodiments. The robot 100 may include a pass-through diagnostic tool. For example, the robot 100 may include a stethoscope that is configured to convey a hearing to a videoconference user (e.g., a physician). In another embodiment, the robot enables direct connection to class II medical devices such as electronic stethoscopes, otoscopes, and ultrasounds to transmit medical data to remote users (doctors). including.

  In the example shown in FIG. 10B, the user is for remote control of the robot 100, video conferencing (eg, using the camera and microphone of the web pad 310), and / or the use of software applications on the web pad 310. The web pad 310 may be removed from the web pad dock 302 on the head 160. The robot 100 is configured to receive the first and second cameras 320a on the head 160 to obtain different viewpoints for video conferencing, navigation, etc. while the web pad 310 is being removed from the web pad dock 302. 320b may be included.

  Interactive applications that are executable on and / or in communication with controller 500 may require more than one display on robot 100. A plurality of web pads 310 associated with the robot 100 can provide different combinations of “FaceTime”, Telestration, HD appearance (eg, for web pads 310 with a built-in camera) to remotely control the robot 100 Can serve as a remote operator control unit (OCU) and / or provide a local user interface pad.

  Referring again to FIG. 6A, the interface module 300 includes a microphone 330 (eg, or a microphone array) for receiving audio input, and one or more speakers 340 disposed on the robot body 110 for delivering audio output. And may be included. Microphone 330 and speaker 340 may each be in communication with controller 500. In some embodiments, interface module 300 includes a basket 360 that may be configured to hold brochures, emergency information, household items, and other items.

  Referring to FIGS. 1 to 4C, and FIGS. 11A and 11B, in order to achieve reliable and robust autonomous movement, the sensor system 400 has intelligence on the actions taken by the robot 100 in its environment. It may include several different types of sensors that can be used in conjunction with each other to create a perception of the robot's environment that is sufficient to make it possible to make a strategic decision. Sensor system 400 may include one or more types of sensors, which may include obstacle detection obstacle avoidance (ODOA) sensors, communication sensors, navigation sensors, etc. supported by robot body 110. For example, these sensors include proximity sensors, contact sensors, three-dimensional (3D) imaging / depth map sensors, cameras (eg, visible and / or infrared cameras), sonars, radars, light detection and ranging (LIDAR) Measurement of the characteristics of the scattered light, which may require optical remote sensing), LADAR (Laser Detection and Ranging), etc., in order to find out the distance of the target and / or other information. It is not limited to these. In some implementations, the sensor system 400 includes a ranging sonar sensor 410 (e.g., nine around base 120), a proximity step detector 420, a contact sensor 430, a laser scanner 440, and one or more A 3-D imaging / depth sensor 450 and an imaging sonar 460 are included.

  There are several issues associated with placing the sensor on the basic framework of the robot. First, the sensors need to be positioned such that they have a maximum coverage area of interest around the robot 100. Second, the sensors may have to position the robot 100 itself so that it only absolutely blocks the sensor, essentially the sensors are "covered and covered" by the robot itself Can not be fixed. Third, the placement and mounting of the sensor should be unobtrusive to the rest of the industrial design of the basic framework. From the aesthetic point of view, robots mounted so that the sensor is unobtrusive can be considered as "attractive" than others. From the practical point of view, the sensor should be mounted so as not to interfere with normal robot operation (such as getting caught on an obstacle).

  In some implementations, sensor system 400 is in communication with controller 500 and disposed within one or more areas or portions of robot 100 to detect any nearby or intruding obstacles. A set of proximity sensors 410, 420 or an array thereof is provided (e.g., disposed on or near the base body portions 124a, 124b, 124c of the robot body 110). The proximity sensors 410, 420 provide signals to the controller 500 when the object is within a given range of the robot 100. Focused infrared (IR) emitter-sensor element, sonar sensor, ultrasound sensor, and / or ultrasonic sensor, and / or It may be an imaging sensor (for example, a 3D depth map image sensor).

  In the embodiment shown in FIGS. 4A-4C, the sonar-type proximity is a robot 100 disposed around (e.g., substantially equidistant) around the base body 120 and with an upward view. An array of sensors 410 is included. The first, second, and third sonar proximity sensors 410a, 410b, 410c are disposed on or near the first (front) base body portion 124a and are radially outermost of the first base body 124a. At the edge 125a of the sensor, at least one of the sonar proximity sensors. The fourth, fifth, and sixth sonar proximity sensors 410d, 410e, 410f are disposed on or near the second (right side) base body portion 124b, and are radially outermost of the second base body 124b. In the vicinity of the edge 125 b of the sensor, at least one of the sonar proximity sensors is provided. Seventh, eighth, and ninth sonar proximity sensors 410g, 410h, 410i are disposed on or near the third (right side) base body portion 124c and are radially outermost of the third base body 124c. There is at least one of the sonar proximity sensors near the edge 125 c of the sensor. This configuration provides at least three detection zones.

  In some embodiments, the set of sonar proximity sensors 410 (e.g., 410a-410i) disposed around the base body 120 are arranged to face upward (e.g., substantially in the Z direction), and are optional Optionally, it is angled outwardly away from the Z-axis, thus producing a detection curtain 412 around the robot 100. A shroud for each sonar proximity sensor 410a-410i to direct sonar emission upwards, or at least not towards other parts of the robot body 110 (e.g. not detecting movement of the robot body 110 relative to itself) Alternatively, it may have a release guide 414. The release guide 414 may define a shell or half shell shape. In the illustrated embodiment, the base body 120 extends laterally beyond the legs 130 and the sonar proximity sensor 410 (eg, 410a-410i) is mounted on the base body 120 (eg, around the legs 130). , Substantially along the periphery of the base body 120). In addition, the upward facing sonar proximity sensors 410 are spaced to create a continuous or substantially continuous sonar detection curtain 412 around the legs 130. The sonar detection curtain 412 can be used to detect obstacles having raised lateral projections such as table tops, shelves and the like.

  Looking up, the sonar proximity sensor 410 provides the ability to see close objects in a horizontal plane, such as the table top. These objects may be missed by the laser scanner 440 or other sensors of the sensor system, such as the imaging sensor 450, due to their aspect ratio, which may cause problems for the robot 100. An upward looking sonar proximity sensor 410 disposed around the periphery of the base 120 provides a means to view or detect these types of objects / obstacles. In addition, the sonar proximity sensor 410 is angled slightly outward around the maximum width point around the base so that it is not blocked or blocked by the torso 140 or head 160 of the robot 100. And thus does not result in false detections by sensing portions of the robot 100 itself. In some implementations, the sonar proximity sensor 410 is arranged to leave a volume around the barrel 140 outside of the field of view of the sonar proximity sensor 410 (upwardly and outwardly), and thus, such as the basket 360 Freely receive the mounted objects or accessories. The sonar proximity sensor 410 can be retracted into the base body 124 to provide visual concealment and eliminate external features that may catch on or collide with obstacles.

  Sensor system 400 may be directed backwards (eg, opposite to forward drive direction F) to detect obstacles during retraction, eg, one or more sonar proximity sensors 410 (eg, rear proximity sensors 410 j May be included. The rear sonar proximity sensor 410 j may include a discharge guide 414 that directs the sonar detection field 412. In addition, the rear sonar proximity sensor 410j can be used to range to determine the distance between the robot 100 and the detected object in the field of view of the rear sonar proximity sensor 4lOj (e.g. As "). In some embodiments, the rear sonar proximity sensor 410j is mounted recessed within the base body 120 so as not to introduce any visual or functional irregularities into the housing configuration.

  Referring to FIGS. 3 and 4B, in some implementations, the robot 100 may detect a step before the drive wheels 210a, 210b, 210c encounter a step (eg, a step) , A step proximity sensor 420 disposed near or around the drive wheels 210a, 210b, 210c. For example, the step proximity sensor 420 may be located at or near each of, or between, the radially outermost edges 125a-c of the base bodies 124a-c. In some cases, step sensing may include an infrared emitter 422 and an infrared detector 424 that are angled towards one another so that the floor has overlapping radiation and detection fields at the location where the floor is to be expected, and thus the detection area. It is realized using infrared (IR) proximity or real range sensing. IR proximity sensing may have a relatively narrow field of view, reliability may depend on the surface albedo, and may have varying distance accuracy between surfaces. As a result, multiple separate sensors can be placed around the perimeter of the robot 100 to properly detect steps from multiple points on the robot 100. Furthermore, IR proximity based sensors can not typically distinguish between steps and safety events, such as immediately after the robot 100 climbs a threshold.

  The step proximity sensor 420 can detect when the robot 100 encounters a low edge of the floor, such as when it encounters a set of stairs. The controller 500 (which executes the control system) may execute a behavior that causes the robot 100 to take action such as changing its moving direction when an edge is detected. In some implementations, sensor system 400 includes one or more secondary step sensors (e.g., other sensors configured for step sensing and optionally other types of sensing). The step detection proximity sensor 420 provides data to distinguish between actual steps and safety events (such as crossing a threshold) so as to provide early detection of steps, and their field of view, the robot body It may be arranged to be positioned at the outer lower side so as to include at least a part of 110 and an area away from the robot body 110. In some implementations, the controller 500 may include an edge of a support work surface (eg, a floor), an increase in distance past the edge of the work surface, and / or a distance between the robot body 110 and the work surface. Perform step detection routine to identify and detect increases. This implementation is 1) early detection of potential steps (which may allow for faster travel speeds in unknown environments), 2) whether step events are not really safe, or safely The autonomous mobility by the controller 500 receiving the step imaging information from the step detection proximity sensor 420 in order to know whether it is possible to cross (for example, climb up a threshold, etc.) 3) to enable increased reliability, 3) reduced step false detection (eg, by edge detection vs. use of a separate IR proximity sensor with multiple narrow views). For redundancy and to detect situations where the distance sensing camera can not reliably detect a particular type of step, additional sensors arranged as "derailing" sensors can be used.

  The sill and stair detection allows the robot 100 to efficiently plan either crossing a sill that can be climbed or avoiding a stair that is too high. This can be the same for indiscriminate objects on the work surface that the robot 100 can or can not safely traverse. Knowing the height of the obstacles or sills that the robot 100 determines can climb is to maximize the smoothness and minimize any instability due to sudden acceleration. In addition, it is possible to decelerate appropriately, if it is considered necessary to enable a smooth transition. In some implementations, threshold and step detection may be performed at the height of the object above the work surface, in addition to geometric shape recognition (eg, to determine threshold or electric cable vs. blobs such as socks) Based on. The threshold may be recognized by edge detection. The controller 500 receives imaging data from the step detection proximity sensor 420 (or another imaging sensor on the robot 100), executes an edge detection routine, and issues a drive command based on the result of the edge detection routine. It is also good. The controller 500 may also use pattern recognition to identify objects. The threshold detection allows the robot 100 to change its orientation relative to the threshold so as to maximize the ability to smoothly climb the stairs.

  Proximity sensors 410, 420 may function alone or, alternatively, may function in combination with one or more contact sensors 430 (eg, a raised switch) for redundancy. For example, one or more touch or bump sensors 430 on the robot body 110 can detect whether the robot 100 physically encounters an obstacle. Such sensors may use physical characteristics such as capacitance or physical movement within the robot 100 to determine when an obstacle is encountered. In some implementations, each base body portion 124a, 124b, 124c of base 120 detects an associated movement of the corresponding base body portion 124a, 124b, 124c relative to base chassis 122 (see, eg, FIG. 4A). It has a sensor 430 (for example, a capacitive sensor, a reed switch, etc.). For example, each base body 124a-c may move radially relative to the Z axis of base chassis 122 to provide three-way clash detection.

  Referring again to FIGS. 1-4C, FIGS. 11A and 11B, in some implementations, the sensor system 400 is mounted on the front portion of the robot body 110 and in communication with the controller 500. A scanner 440 is included. In the illustrated embodiment, the laser scanner 440 is on the base body 120 facing forward on or above the first base body 124a (e.g., to have maximum imaging coverage along the drive direction F of the robot). Mounted (e.g. with a field of view along the forward drive direction F). In addition, placement of the laser scanner on or near the front tip of the triangular base 120 ensures that the outer angle (eg, 300 degrees) of the robot base is greater than the field of view 442 (eg, about 285 degrees) of the laser scanner 440. This means that the base 120 prevents blocking or blocking of the detection field 442 of the laser scanner 440. The laser scanner 440 is configured to minimize any portion of the laser scanner sticking out past the base body 124 (eg, for aesthetics and to minimize snagging on obstacles) ), Without occluding the field of view, it can be retracted and mounted as much as possible in the base body 124.

  The laser scanner 440 scans the surrounding area of the robot 100, and the controller 500 uses the signals received from the laser scanner 440 to create an environmental map or an object map of the scanned area. Controller 500 may use object maps for navigation, obstacle detection, and obstacle avoidance. Further, controller 500 may use sensed input from other sensors of sensor system 400 to create an object map and / or for navigation.

  In some embodiments, the laser scanner 440 may rapidly scan the area in one dimension as the "main" scan line and a phase difference or similar technique for assigning depth to each pixel generated in the line. Is a scanning LIDAR (returns a two-dimensional depth line in the scan plane), which may use time-of-flight imaging elements using. To generate a three dimensional map, the LIDAR can perform an "auxiliary" scan in a second direction (e.g., by "raising and lowering the scanner"). This mechanical scanning technique allows time-of-flight calculations for the full 2-D matrix of pixels, if not compensated, to provide each pixel's depth, or even a series of depths of each pixel. It can be complemented by techniques (using coded illuminators or illumination lasers), techniques using semiconductor stacks, focal plane imaging element sensors of the "Flash" LIDAR / LADAR and "Swiss Ranger" type, etc.

  Sensor system 400 may include one or more three-dimensional (3-D) image sensors 450 in communication with controller 500. If the 3-D image sensor 450 has a limited field of view, the controller 500 or the sensor system 400 may generate a relatively wider field of view to perform a robust ODOA, the 3-D image sensor 450a, It can be operated to scan left and right. Referring to FIGS. 1-3 and 10B, in some implementations, the robot 100 is mounted on the forward portion of the robot body 110 with a view along the forward drive direction F (eg, the robot , The scan 3-D image sensor 450a). The scanning 3-D image sensor 450a can be mainly used for obstacle detection / obstacle avoidance (ODOA). In the illustrated embodiment, the scanning 3-D image sensor 450a is, for example, a step on the barrel 140, as shown in FIG. 3, to prevent the user from contacting the scanning 3-D image sensor 450a. It is mounted on the lower portion of the portion 142 or on the bottom surface 144 and is retracted into the barrel 140 (e.g., flush with or beyond the bottom surface 144). The scanning 3-D image sensor 450 has a downward view 452 in the front of the robot 100 (eg, by the base 120 or other portion of the robot body 110) for obstacle detection and obstacle avoidance (ODOA). It can be arranged to aim substantially downward and away from the robot body 110). The placement of the scanning 3-D image sensor 450a on or near the front edge of the barrel 140 allows the field of view of the 3-D image sensor 450 (e.g. approximately 285 degrees) to be the barrel relative to the 3-D image sensor 450. It can be made smaller than the outer surface angle of 140 (e.g. 300 degrees), thus preventing the barrel 140 from blocking or blocking the detection field 452 of the scanning 3-D image sensor 450a. In addition, the scanning 3-D image sensor 450a (and the associated actuator) as close as possible to its view (e.g. also for aesthetics and also to minimize getting stuck with obstacles) , Embedded in the body portion 140 can be mounted. The distracting scanning motion of the scanning 3-D image sensor 450a results in an interactive experience that is not visible to the user and is not as unpleasant. Unlike protruding sensors or features, the recessed scanning 3-D image sensor 450a is particularly effective in moving or scanning, since there are virtually no moving parts extending beyond the perimeter of the barrel 140. There is a tendency not to have the unintended interaction (person, obstacles etc.).

  In some implementations, sensor system 400 includes an additional 3-D image sensor 450 disposed on base body 120, legs 130, and neck 150 and / or head 160. In the embodiment shown in FIG. 1, the robot 100 includes a 3-D image sensor 450 on the base body 120, the torso 140 and the head 160. In the embodiment shown in FIG. 2, the robot 100 includes a 3-D image sensor 450 on the base body 120, the torso 140 and the head 160. In the embodiment shown in FIG. 11A, the robot 100 includes a 3-D image sensor 450 on the legs 130, the torso 140 and the neck 150. Other configurations are possible as well. One 3-D image sensor 450 (eg, on the neck 150 and above the head 160) can be used for human recognition, gesture recognition, and / or video conferencing while another 3-D image sensor 450 (eg, on base 120 and / or legs 130) can be used for navigation and / or obstacle detection and obstacle avoidance.

  A front facing 3-D image sensor 450 disposed on the neck 150 and / or the head 160 can be used for human recognition, face recognition, and / or gesture recognition of people around the robot 100. For example, using the signal input from the 3-D image sensor 450 on the head 160, the controller 500 creates a three-dimensional map of the viewed / captured user's face and generates the created three-dimensional map The user may be recognized by comparing with a known 3-D image of the face and determining a match with one of the known 3-D face images. Face recognition may be used to authenticate the user as a valid user of the robot 100. Additionally, one or more of the 3-D image sensors 450 may be determined based on the gesture or gestures (e.g., hand, hand gestures, and / or hand signals) in which the robot 100 determines the gesture of the viewer. And can optionally be used to react. For example, controller 500 may issue a drive command in response to manual feeding in a particular direction to be recognized.

  The 3-D image sensor 450 can generate the following types of data: (i) depth map, (ii) reflectance based intensity image, and / or (iii) standard intensity image. The 3-D image sensor 450 may acquire such data by image pattern matching, time of flight and / or phase delay shift measurements of light emitted from the source and reflected from the target.

  In some implementations, the inference or control software executable on the processor (eg, of robot controller 500) uses a combination of algorithms executed using various data types generated by sensor system 400. Do. The inference software processes and outputs data collected from the sensor system 400, for example, to make navigation decisions as to whether the robot 100 can move without colliding with an obstacle. By accumulating imaging data over time of the robot's envelope, the result is that the inference software is selected for sensing the image (s) to improve depth measurement of the 3-D image sensor 450 Efficient methods can be applied to the segments. This may include using appropriate temporal and spatial averaging techniques.

  Confidence in performing collision-free robot movement is (i) confidence levels built by high-level temporal reasoning, and (ii) three main types of data for analysis, (a) depth images , (B) active illumination image, and (c) ambient illumination image may be based on a depth perception sensor. For each of the images acquired by the depth perception imaging sensor 450, an algorithm can be implemented that recognizes different types of data. Aggregated data may improve confidence levels as compared to systems that use only one type of data.

  The 3-D image sensor 450 may obtain an image containing depth and brightness data (eg, a sensor view portion of a room or work area) from a scene 100 around a robot containing one or more objects . The controller 500 may be configured to determine occupancy data of the object based on the reflected light captured from the scene. Further, in some embodiments, the controller 500 issues drive commands to the drive system 200 to bypass obstacles (ie, objects in the scene) based at least in part on occupancy data. The 3-D image sensor 450 repeatedly captures scene depth images to make real-time decisions by the controller 500 to navigate the robot 100 around the scene without colliding with any objects in the scene. It is also good. For example, the speed or frequency at which depth image data is acquired by the 3-D image sensor 450 may be controlled by the shutter speed of the 3-D image sensor 450. In addition, controller 500 may receive an event trigger (eg, from another sensor component of sensor system 400, such as proximity sensor 410, 420, to notify controller 500 of nearby objects or hazards). The controller 500 may, in response to an event trigger, cause the 3-D image sensor 450 to increase the frequency with which the depth image is captured, and occupancy information is obtained.

  Referring to FIG. 12A, in some implementations, the 3-D imaging sensor 450 includes a light source 1172 that emits light to a scene 10, such as a surrounding area (eg, a room) of the robot 100. In addition, the imaging sensor 450 captures reflected light from the scene 10 (for example, as a scene depth image) including reflected light of light emitted from the light source 1172 (for example, as an array of light sensing pixels 1174p) ) May also be included. In some embodiments, imaging sensor 450 includes source lens 1176 and / or detector lens 1178 to manipulate (eg, speckle or focus) the emitted light and the received reflected light, respectively. . A robot controller 500 or a sensor controller (not shown) in communication with the robot controller 500 may monitor the image 12 of the object 12 in the scene 10 based on the image pattern match and / or time of flight characteristics of the reflected light captured by the imaging unit 1174 A light signal is received from an imaging unit 1174 (eg, pixel 1174p) to determine depth information.

  FIG. 12B provides an example arrangement 1200 of operations for operating imaging sensor 450. With additional reference to FIG. 12A, the action is to emit light to a scene 10 around the robot 100 (1202) and to reflect the radiation from the scene 10 onto the imaging portion (eg, an array of photosensitive pixels). And (1204). In operation, the controller 500 receives a light detection signal from the imaging unit (1206), and detects one or more features of the object 12 in the scene 10 using image data obtained from the light detection signal. And (1210) tracking (1210) the position of the detected feature or features of the object 12 within the scene 10 using (1208) and image depth data obtained from the light detection signal. The operation is to emit light (1202), to receive light reflection (1204), to receive light detection signals, to increase the resolution of the image data or image depth data and / or to provide a confidence level. And repeating (1212) the operations of: doing 1206; detecting the object feature (s) (1208); and tracking the position of the object feature (s) (12010) .

  The repeating operation (1212) can be performed fast at a relatively slow speed (eg, slow frame rate), at a medium speed, or with a relatively low resolution, because of relatively high resolution. The frequency of repeating the action 1212 may be adjustable by the robot controller 500. In some implementations, controller 500 may increase or decrease the frequency of repeating 1212 operations upon receiving an event trigger. For example, a sensed item in the scene may increase 1212 the frequency at which it is repeated to sense potentially high objects 12 in the scene 10 (eg, doorways, sills, or steps). Triggering events. In an additional embodiment, an elapsed time event between detected objects 12 may reduce the frequency of repeating the action (1212), or may stop for a period of time (e.g., to become inactive until triggered by another event). Become). In some embodiments, detecting 1208 one or more features of the object 12 in the scene 10 may cause a relatively large number of repetitive operations to increase the speed at which the image depth data is acquired. Trigger a feature detection event that results in 1212). The relatively large acquisition speed of the image depth data can enable relatively more reliable feature tracking within the scene.

  The operation also includes outputting 1214 navigation data to bypass the object 12 in the scene 10. In some implementations, the controller 500 issues drive commands to the drive system 200 and uses the output navigation data to move the robot 100 to avoid collision with the object 12.

  In some implementations, sensor system 400 detects multiple objects 12 in scene 10 around robot 100, and controller 500 tracks the position of each of detected objects 12. The controller 500 may create an occupancy map of the object 12 in an area around the robot 100, such as a bounded area of a room. The controller 500 may use the image depth data of the sensor system 400 to match the scene 10 with a portion of the occupancy map and update the occupancy map with the location of the object 12 being tracked.

  Referring to FIG. 12C, in some implementations, the 3-D image sensor 450 includes a three-dimensional (3D) speckle camera 1300 that enables image mapping through speckle decorrelation. Speckle camera 1300 emits a speckle pattern to scene 10 (as a target area), speckle emitter 1310 (eg, of infrared, ultraviolet, and / or visible light) and object 12 of scene 10. And an imaging unit 1320 that captures an image of the speckle pattern on the surface.

  Speckle emitter 1310 may include a light source 1312, such as a laser, that emits a light beam onto diffuser 1314 and onto reflector 1316 for reflection and thus for projection as a speckle pattern onto scene 10. The imaging unit 1320 may include an objective optical element 1322 that forms an image on an image sensor 1324 having an array of light detectors 1326 such as a CCD or CMOS based image sensor. The optical axes of the speckle radiation unit 1310 and the imaging unit 1320 are shown to be on the same straight line, but in the uncorrelated mode, for example, the optical axes of the speckle radiation unit 1310 and the imaging unit 1320 are also non It may be co-linear, while in cross-correlation mode, for example, the imaging axis may be offset from the emission axis.

The speckle radiation unit 1310 emits a speckle pattern to the scene 10, and the imaging unit 1320 captures a reference image of the speckle pattern in the scene 10 at a wide range of different object distances Z _n from the speckle radiation unit 1310. (For example, the Z-axis may be defined by the optical axis of the imaging unit 1320). In the illustrated embodiment, a reference image of the projected speckle pattern is captured at successive planes of different distances from the origin, such as reference positions labeled Z ₁ , Z ₂ , Z ₃ etc. The distance ΔZ between reference images may be set to a threshold distance (eg, 5 mm) or may be adjustable by controller 500 (eg, in response to a triggered event). Speckle camera 1300 allows for decorrelation of the speckle pattern with the distance from speckle radiation 1310 to perform ranging of the object 12 captured in the subsequent image, the captured reference image Store and index each radial distance. For example, reference distances Z ₁ , Z ₂ , Z ₃ ,. . . , The speckle pattern on the object 12 at position Z _A can be correlated with the reference image of the speckle pattern captured at Z ₂ . On the other hand, for example, the speckle pattern on the object 12 of the ZB can be correlated with the reference image of Z _3. These correlation measurements yield an approximate distance of the object 12 from the origin. In order to map object 12 in three dimensions, controller 500 receiving information from speckle camera 1300 or speckle camera 1300 can use local cross correlation with the reference image that resulted in the closest match.

  For example, other details and features in 3D image mapping using speckle ranging via speckle cross-correlation using triangulation or decorrelation that may be combined with those described herein Can be found in PCT Patent Application No. PCT / IL2006 / 000335, the entire content of which is incorporated herein by reference.

  FIG. 12D provides an example arrangement 1400 of operations for operating speckle camera 1300. The operations include emitting a speckle pattern to scene 10 (1402) and capturing a reference image (e.g., of reference object 12) at different distances from speckle emitting portion 1310 (1404). The operation further includes emitting a speckle pattern to the target object 12 in the scene 10 (1406) and capturing a target image of the speckle pattern on the object 12 (1408). The operation compares the target image (of the speckled object) with a different reference image to identify (1410) a reference pattern that most strongly correlates with the speckle pattern on the target object 12; And determining 1412 an estimated distance range of the target object 12. This may include determining a primary speckle pattern on the object 12 and finding a reference image having a speckle pattern that most strongly correlates with the primary speckle pattern on the object 12. A range of distances can be determined from the corresponding distance of the reference image.

  The motion is optionally 3D of the surface of object 12 by local cross-correlation between the speckle pattern on object 12 and a reference pattern identified, for example to determine the position of object 12 in the scene. Includes building 1414 maps. This is determined by determining a primary speckle pattern on the object 12 and obtaining a primary speckle pattern on multiple regions of the object 12 in the target image to obtain a three-dimensional (3D) map of the object. And V. finding each offset between the primary speckle pattern in the reference image. The use of solid state components for 3D mapping of scenes provides a relatively inexpensive solution for robotic navigation systems.

  Typically, at least some of the different respective distances are axially separated longer than the axial length of the primary speckle pattern at each respective distance. Comparing the target image with the reference image comprises calculating respective cross-correlations between the target image and at least some of each of the reference images and having a maximum respective cross-correlation with the target image Selecting a reference image.

  The actions may include repeating actions 1402-1412 or actions 1406-1412 (1416) to optionally track movement of object 12 in scene 10 (e.g., continuously), and optionally, action 1414. Good. For example, speckle camera 1300 may capture continuous target images while object 12 is moving for comparison with a reference image.

  Other details and features in 3D image mapping using speckle ranging that may be combined with those described herein are incorporated herein by reference in their entirety. Patent No. 7,433,024, U.S. Patent Application Publication No. 2008/0106746 entitled "Depth Mapping Using Projected Patterns", U.S. Patent Application Publication No. 2010/0118123 No. "Depth Mapping Using Projected Patterns", Title US Patent Application Publication No. 2010/0034457 of “Modeling Of Humanoid Forms From Depth Maps”, entitled “Depth Mapping Using M US Patent Application Publication No. 2010/0020078 of ulti-Beam Illumination, US Patent Application Publication No. 2009/0185274 of “Optical Designs For Zero Order Reduction”, US of “Three-Dimensional Sensing Using Speckle Patterns” Patent Application Publication No. 2009/0096783, US Patent Application Publication No. 2008/0240502 entitled “Depth Mapping Using Projected Patterns”, and US Patent Application Publication No. 2008 entitled “Depth-Varying Light Fields For Three Dimensional Sensing” / 0 No. 106,746.

  Referring to FIG. 12E, in some implementations, the 3-D imaging sensor 450 includes a 3D time-of-flight (TOF) camera 1500 for acquiring depth image data. The 3D TOF camera 1500 communicates with a light source 1510, a complementary metal oxide semiconductor (CMOS) sensor 1520 (or charge coupled device (CCD)), a lens 1530, a light source 1510, and a CMOS sensor 1520. Control logic or camera controller 1540 (and / or robot controller 500). The light source 1510 may be a laser or a light emitting diode (LED) with an intensity modulated by a periodic high frequency signal. In some embodiments, light source 1510 includes focusing lens 1512. CMOS sensor 1520 may also include an array of pixel detectors 1522, or other arrangement of pixel detectors 1522, where each pixel detector 1522 can detect the intensity and phase of photon energy impinging thereon Good. In some embodiments, each pixel detector 1522 has a dedicated detector circuit 1524 for processing the detected charge output of the associated pixel detector 1522. Lens 1530 focuses light reflected from scene 10 containing one or more objects of interest 12 onto CMOS sensor 1520. Camera controller 1540 provides a series of operations that format pixel data acquired by CMOS sensor 1520 into depth maps and lightness images. In some embodiments, 3D TOF camera 1500 may also include input / output (IO) 1550 (eg, in communication with robot controller 500), memory 1560, and / or camera controller 1540 and / or pixel detector Also included is a clock 1570 in communication with 1522 (eg, detector circuit 1524).

FIG. 12F provides an example arrangement 1600f of operations for operating the 3D TOF camera 1500. The operation is to emit a light pulse (e.g. infrared, ultraviolet and / or visible light) to the scene 10 (1602f) and to start the time measurement of the time of flight of the light pulse (e.g. And (1604f). The operations include receiving a reflection of light emitted from one or more surfaces of object 12 in scene 10 (1606f). The reflections may be reflected from the surface of object 12 at different distances Z _n from light source 1510. The reflection is received through lens 1530 onto pixel detector 1522 of CMOS sensor 1520. Operation includes receiving time of flight of each light pulse reflection received on each corresponding pixel detector 1522 of CMOS sensor 1520 (1608f). During the round-trip time-of-flight (TOF) of the light pulse, the counter of the detector circuit 1523 of each respective pixel detector 1522 accumulates clock pulses. The accumulation of a larger number of clock pulses represents a longer TOF, and thus a greater distance between the light reflection point on the object 12 to be imaged and the light source 1510. The operation further includes determining the distance between the reflective surfaces of the objects 12 of the received light pulse reflection (1610f) and optionally constructing the three-dimensional object surface (1612f). In some implementations, operations include repeating operations 1602f-1610f (1614f), and optionally 1612f to track movement of object 12 within scene 10.

  Other details and features in 3D time-of-flight imaging, which may be combinable with those described herein, are referred to as “CMOS Compatible 3--3”, which is incorporated herein by reference in its entirety. U.S. Patent No. 6,323,942 of D Image Sensor, U.S. Patent No. 6,515,740 of Methods for CMOS-Compatible Three-Dimensional Image Sensing Using Quantum Efficiency Modulation, and Name: System to Enhance Dynamic Range Conversion Usable with CMOS Three-Dimens It can be found in PCT Patent Application No. PCT / US02 / 16621 of onal Imaging ".

  In some implementations, the 3-D imaging sensor 450 provides three types of information: (1) depth information (eg, from each pixel detector 1522 of the CMOS sensor 1520 to the corresponding location on the scene 12) , (2) provide ambient light intensity at each pixel detector location, and (3) provide active illumination intensity at each pixel detector location. The depth information allows the position of the object 12 to be detected to be tracked over time, particularly with respect to the proximity of the object to the robot deployment site. Active illumination intensity and ambient light intensity are different types of lightness images. Active illumination intensity is captured from reflections of active light (such as that provided by light source 1510) reflected from target object 12. The ambient light image is that of the ambient light reflected from the target object 12. The two images together provide additional robustness, especially when the lighting conditions are poor (eg too dark or excessive ambient lighting).

  Image segmentation and classification algorithms may be used to classify and detect the position of the object 12 in the scene 10. The information provided by these algorithms, as well as the distance measurement information obtained from imaging sensor 450 may be used by robot controller 500 or other processing resources. The imaging sensor 450 is adjusted based on the time-of-flight principle, and more specifically, reflected from the scene 10, including techniques for modulating the sensitivity of the photodiode to filter ambient light. It can operate based on detectable phase delay in the light pattern.

  The robot 100 has 1) mapping, localization & navigation, 2) object detection & object avoidance (ODOA), 3) object search (e.g. for finding people), 4) gesture recognition (e.g. for companion robots) ), 5) human & face detection, 6) human tracking, 7) monitoring the manipulation of objects by the robot 100, and other suitable applications for autonomous operation of the robot 100, using the imaging sensor 450 It is also good.

  In some implementations, at least one of the 3-D image sensors 450 is positioned at a height above one foot (about 30.5 cm) or two feet (about 61 cm) above the ground on the robot 100 Volume point cloud imaging device (speckle or flight, directed so that point clouds can be obtained (via omnidirectional drive system 200) from a space volume that includes the floor surface in the direction of movement of the robot It may be a time camera or the like. In the embodiment shown in FIGS. 1 and 3, the first 3-D image sensor 450a may be located at a height of more than one or two feet above the ground on the base 120 (or about one or two feet above the ground). Aiming along the forward drive direction F (eg obstacle detection and obstacles) so as to capture an image of the volume including the floor (eg volume point cloud) while positioning and driving at height) It can be done). Mounted on the head 160 (e.g., at a height of more than about 3 or 4 feet above the ground) so that skeleton recognition and definition points can be obtained from the space volume adjacent to the robot 100; A second 3-D image sensor 450b is shown. The controller 500 may execute skeletal / digital recognition software to analyze captured volumetric point cloud data.

  Properly sensing the object 12 using the imaging sensor 450 may be important, regardless of ambient light conditions. In many circumstances, the lighting conditions can range from direct sunlight to bright fluorescent lighting, overshadowed shadows, and can result in large changes in the surface texture and basic reflectivity of the object 12. The illumination may vary as well within a given location and from scene 10 to scene 10 as well. In some implementations, in all situations, imaging sensor 450 may be used to identify and resolve person and object 12 with relatively minor effects from ambient light conditions (e.g., ambient light rejection). It can be used.

  In some implementations, the VGA resolution of imaging sensor 450 is 640 horizontal pixels by 480 vertical pixels, however, other resolutions such as 320 by 240 (eg, for short range sensors) are possible as well. is there.

  The imaging sensor 450 may include a pulsed laser and a camera iris that acts as a band pass filter in the time domain to view the object 12 only within a certain range. The variable iris of the imaging sensor 450 can be used to detect objects 12a of different distances. Furthermore, in outdoor applications, higher power laser pulses can be emitted.

  In some implementations, the robot includes a sonar scanner 460 for acoustic imaging of the area surrounding the robot 100. In the embodiment shown in FIGS. 1 and 3, the sonar scanner 460 is disposed on the front portion of the base body 120.

  With reference to FIGS. 1, 3B, and 11B, in some implementations, the robot 100 may use a laser scanner or laser rangefinder 440 for redundant sensing, and a sonar proximity sensor 410j that is directed backward for safety. , Both of which are oriented parallel to the ground G. The robot 100 may include first and second 3-D image sensors 450a, 450b (depth cameras) to provide robust sensing of the environment around the robot 100. The first 3-D image sensor 450 a is mounted on the torso 140 so as to point the ground G downward at a fixed angle. By angling the first 3-D image sensor 450a downwards, the robot 100 receives a high density sensor coverage in the area immediately before or adjacent to the robot 100 that is associated with short term travel of the robot 100 in the forward direction. . Backwards-facing sonar 410 j provides object detection as the robot moves backwards. For the robot 100, where backward movement is typical, the robot 100 includes a third 3D image sensor 450 pointing downward and backward so as to provide high density sensor coverage immediately following or adjacent to the robot 100. May be.

  The second 3-D image sensor 450 b is mounted on the head 160 which can be pivoted and tilted via the neck 150. The second 3-D image sensor 450b may be useful for remote actuation as it allows a human operator to see where the robot 100 is moving. The neck 150 allows the operator to tilt and / or pivot the second 3-D image sensor 450b to look at both near and far objects. Rotating the second 3-D image sensor 450b increases the associated horizontal visibility. During high speed movement, the robot 100 may increase the overall or combined view of both the 3-D image sensors 450a, 450b, and provide sufficient time for the robot 100 to avoid obstacles (more The second 3-D image sensor 450b may be tilted slightly downward because high speed generally means that the time to react to obstacles is shorter). At lower speeds, the robot 100 tilts the second 3-D image sensor 450 b upward, or approximately parallel to the ground G, so as to track the individual that the robot 100 is intended to follow You may Further, while driving at a relatively low speed, the robot 100 can pivot the second 3-D image sensor 450b to increase its field of view around the robot 100. The first 3-D image sensor 450a can remain fixed (eg, moved relative to the base 120) when the robot is driving to expand the robot's perceived range Absent).

  In some implementations, at least one of the 3-D image sensors 450 is at a height greater than or equal to 1 foot or 2 feet above ground (or above ground) The point cloud can be obtained from a space volume that is located on the robot 100 at a height of about 1 or 2 feet and includes the floor surface in the direction of movement of the robot (via the omnidirectional drive system 200) It may be a volume point cloud imaging device (such as a speckle or time-of-flight camera) that is oriented as such. In the embodiment shown in FIGS. 1 and 3, the first 3-D image sensor 450a is positioned on the base 120 at a height of one or more feet above ground and (eg, obstacle detection and failure) While driving (for object avoidance), aiming along the forward driving direction F can be made to capture an image of the volume including the floor (e.g., a group of volume points). (E.g., to be about 3 feet above the ground (at a height greater than about 91 cm or 4 feet) so that it is possible to obtain skeletal recognition and definition points from the space volume adjacent to the robot 100 A second 3-D image sensor 450b is shown mounted on the head 160. The controller 500 executes skeletal / digital recognition software to analyze captured volume point cloud data May be

2 and 4A~4C again, the sensor system 400, to measure and monitor the moment of inertia of the robot 100 for the total center of gravity CG _R of the robot 100, an inertial measurement unit in communication with the controller 500 (IMU) And 470 may be included.

  Controller 500 may monitor any deviation in feedback from IMU 470 from threshold signals corresponding to normal unhindered operation. For example, if the robot starts to tilt away from the upright position, it may be that it was "tackled", or otherwise disturbed or someone suddenly added a heavy load. In these situations, it may be necessary to take emergency action (including, but not limited to, avoidance maneuvers, recalibration, and / or the issue of audio / visual alerts) to ensure the safe operation of the robot 100. .

  The robot 100 may operate in a human environment, so it may interact with humans and operate in a space designed for humans (and without consideration of robot constraints). The robot 100 can limit its drive speed and acceleration when in a crowded, constrained or extremely dynamic environment such as a cocktail party or a crowded hospital. However, while it is safe for the robot 100 to drive at relatively high speeds, such as long corridors without anyone, it may encounter situations where someone can still suddenly decelerate as it traverses the robot's motion path.

When accelerating from a stop, the controller 500, so as to prevent the robot from falling down, it may be considered the moment of inertia of the robot 100 from the total center-of-gravity CG _R. Controller 500 may use a model of its pose, including its current moment of inertia. When the payload is supported, the controller 500 measures the load impact on the overall center of gravity CG _R, may monitor the moment of inertia of the movement of the robot. For example, the torso 140 and / or the neck 150 may include strain gauges that measure strain. If this is not possible, the controller 500 applies a test torque command to the drive wheel 210 to determine the safety limit experimentally, and measures the actual linear and angular acceleration of the robot using the IMU 470 It is also good.

  During sudden deceleration, commanded loads on the second and third drive wheels 210b, 210c (rear wheels) are reduced while the first drive wheels 210a (front wheels) slide in the forward drive direction , Supports the robot 100. If the loads on the second and third drive wheels 210b, 210c (rear wheels) are asymmetrical, the robot 100 may "yaw", which reduces dynamic stability. An IMU 470 (eg, a gyro) may be used to detect this yaw and command the second and third drive wheels 210b, 210c to reorient the robot 100.

  Referring to FIGS. 3-4C and 6A, in some implementations, the robot 100 includes multiple antennas. In the illustrated embodiment (although the antenna may be disposed on any other part of the robot 100 such as the legs 130, the torso 140, the neck 150, and / or the head 160) both A first antenna 490a and a second antenna 490b both disposed on the base 120. The use of multiple antennas provides robust signal reception and transmission. The use of multiple antennas provides robot 100 with multiple inputs and multiple outputs or MIMO, which is the use of multiple antennas for transmitters and / or receivers to improve communication performance. MIMO provides a significant increase in data throughput and link range without additional bandwidth or transmit power. It achieves this with higher spectral efficiency (more bits per second of hertz of bandwidth) and link certainty or diversity (reduced fading). Because of these properties, MIMO is an important part of modern wireless communication standards such as IEEE 802.11n (Wifi), 4G, 3GPP Long Term Evolution, WiMAX, and HSPA +. In addition, the robot 100 can act as a Wi-Fi bridge, hub, or hotspot for other nearby electronic devices. The mobility of the robot 100 and the use of MIMO can allow the robot to become a relatively very secure Wi-Fi bridge.

  MIMO can be subdivided into three main categories: precoding, spatial multiplexing or SM, and diversity coding. Precoding is a type of multi-stream beamforming and is considered to be all spatial processing that occurs at the transmitter. In (single-layer) beamforming, the same signal is emitted from each of the transmit antennas with appropriate phase (and sometimes gain) weighting so that the signal power is maximized at the receiver input. The advantage of beamforming is to increase the received signal gain and reduce multipath fading effects by making the signals emitted from different antennas constructively add. If there is no scattering, beamforming can result in a clear directivity pattern. When the receiver has multiple antennas, transmit beamforming can not simultaneously maximize the signal level at all of the receive antennas and can use precoding with multiple streams. Precoding may require knowledge of channel state information (CSI) at the transmitter.

  Spatial multiplexing requires a MIMO antenna configuration. In spatial multiplexing, a high speed signal is split into multiple low speed streams, each stream being transmitted from different transmit antennas on the same frequency channel. If these signals arrive at a receiver array with sufficiently different spatial evidence, the receiver can separate the streams into (approximately) parallel channels. Spatial multiplexing is a very powerful technique to increase channel capacity with higher signal to noise ratio (SNR). The maximum number of spatial streams is limited by the smaller number of antennas at the transmitter or receiver. Spatial multiplexing can be used with or without knowledge of the transmission channel. Spatial multiplexing can also be used for simultaneous transmission, known as space division multiple access. By scheduling receivers with different spatial evidence, good separability can be ensured.

  Diversity coding techniques can be used when the transmitter does not have knowledge of the channel. In the diversity method, a single stream is transmitted (as opposed to multiple streams in spatial multiplexing), but the signal is encoded using a technique called space-time coding. The signal is emitted from the transmit antenna using full or near orthogonal coding. Diversity coding utilizes independent fading on multiple antenna links to enhance signal diversity. Because there is no knowledge of the channel, there is no beamforming or array gain from diversity coding. Spatial multiplexing may also be combined with precoding when the channel is known at the transmitter, or with diversity coding when decoding certainty is a trade-off.

  In some implementations, the robot 100 includes a third antenna 490c and / or a fourth antenna 490d and a torso 140 and / or a head 160, respectively (see, eg, FIG. 3). In such a case, controller 500 may move antennas 490a-d (eg, by raising or lowering barrel 140 and / or pivoting and / or tilting head 160, etc.) ) Antenna placement can be determined to achieve threshold signal levels for robust communication. For example, the controller 500 can issue a command to raise the third and fourth antennas 490c, 490d by raising the height of the body 140. Also, the controller 500 can issue commands to rotate the head 160 and / or to further orient the fourth antenna 490d with respect to the other antennas 490a-c.

  Referring to FIG. 13, in some implementations, the controller 500 executes a control system 510 that includes a control arbitration system 510a and a behavior system 510b in communication with each other. Control arbitration system 510a allows applications 520 to be dynamically added to and removed from control system 510, each without requiring robot 520 to know about any other application 520. Promote to make it possible to control. In other words, the control arbitration system 510a provides a simple priority control mechanism between the application 520 of the robot 100 and the resource 530. Resources 530 may include drive system 200, sensor system 400, and / or any load or controllable device in communication with controller 500.

  The application 520 may be simultaneously stored on or in communication with the memory of the robot 100 so as to simultaneously execute on the robot 100 (e.g., a processor) and to control the robot 100 simultaneously. Application 520 may access behavior 600 of behavior system 510b. The independently deployed applications 520 dynamically at runtime and share robot resources 530 of the robot 100 (eg, drive system 200, arm (s), head (s), etc.) Can be combined. Low-level policies are implemented to dynamically share robot resources 530 at runtime between applications 520. The policy determines which application 520 has control of the robot resource 530 required by that application 520 (e.g., a priority hierarchy among the applications 520). The applications 520 can be dynamically started and stopped, and run completely independently of each other. Control system 510 also enables complex behaviors 600 that can be combined together to support one another.

  Control arbitration system 510 a includes one or more resource controllers 540, a robot manager 550, and one or more control arbiters 560. These components need not be in a common process or computer, and need not be started in any particular order. The resource controller 540 component provides the application 520 with an interface to the control arbitration system 510a. There are instances of this component for all each application 520. The resource controller 540 removes and encapsulates the complexity of authentication, distributed resource control arbiters, command buffering, etc. Robot manager 550 coordinates application 520's prioritization by controlling which application 520 has any exclusive control of robot resource 530 at any particular time. Since this is a central coordinator of information, there is only one instance of robot manager 550 per robot. Robot manager 550 tracks resource control arbiter 560, which implements a priority policy and has hardware control, with a linearly prioritized order of resource controllers 540. Control arbiter 560 receives commands from all each application 520, generates a single command based on the application's priority, and publishes it to its associated resource 530. Control arbiter 560 also receives state feedback from its associated resources 530 and sends it back to application 520. Robot resources 530 may be a network of functional modules (eg, actuators, drive systems, and groups thereof) having one or more hardware controllers. The control arbiter 560 commands are specific to the resource 530 performing a particular operation.

  A dynamic model 570 executable on controller 500 is configured to calculate the outer products of the center of gravity (CG), the moment of inertia, and the inertia of various parts of robot 100 to evaluate the current state of the robot. be able to. The kinetic model 570 may also model the shape, weight, and / or moment of inertia of these components. In some embodiments, the kinetic model 570 may be located on the robot 100 and in communication with the controller 500 (eg, an accelerometer and / or the like) to calculate various centroids of the robot 100. Or communicates with the moment of inertia unit 470 (IMU) or part thereof. The controller 500 can use the kinetic model 570, along with other programs 520 or behaviors 600, to determine the operating limits of the robot 100 and its components.

  Each application 520 has an operation selection engine 580, a resource controller 540, one or more behaviors 600 connected to the operation selection engine 580, and one or more behavior models 590 connected to the operation selection engine 580 . Behavioral system 510 b provides predictive modeling, allowing behavior 600 to jointly determine robot motion by evaluating the possible consequences of robot motion. In some embodiments, the behavior 600 provides a hierarchical state complete evaluation function that couples sensor feedback from multiple sources, along with a priori limits and information, to evaluation feedback on the robot's allowed motion. It is an in component. As behavior 600 is pluggable into application 520 (e.g., in or out of application 520), removing and adding without modifying application 520 or any other part of control system 510 Can. Each behavior 600 is an independent policy. In order to make the behavior 600 more powerful, it is possible to attach the outputs of multiple behaviors 600 together to another input so that they can have complex combinatorial functions. The behavior 600 is intended to realize a manageable part of the overall recognition of the robot 100.

  The operation selection engine 580 is a coordinated element of the control system 510 and executes a high speed optimum operation selection cycle (prediction / correction cycle) that searches for the optimum operation in consideration of the inputs of all the behaviors 600. The operation selection engine 580 has three phases: nomination, operation selection search, and completion. In the nominating phase, each behavior 600 is notified that a motion selection cycle has been initiated and is provided with cycle start time, current state, and robot actuator space limits. Based on internal policies or external inputs, each behavior 600 determines whether it wants to participate in this operation selection cycle. During this stage, a list of active behavior primitives is generated, this input affecting the choice of commands to be executed on the robot 100.

  In the motion selection search stage, the motion selection engine 580 also generates feasible results from the space of available motions, also referred to as motion space. The behavior selection engine 580 is operable to provide a pool of feasible commands (within limits) and corresponding results as a result of simulating the behavior of each command at different time steps of the future target period. Use The operation selection engine 580 calculates a preferable result based on the result evaluation of the behavior 600, transmits a command corresponding to the control arbitration system 510a, and notifies the operation model 590 of the command selected as feedback.

  At the completion stage, the commands corresponding to the collaborative highest score results are combined together as a general command presented to resource controller 540 for execution on robot resource 530. Optimal results are provided for the active behavior 600 as feedback to be used in future evaluation cycles.

  Sensor signals received from sensor system 400 can cause interaction with one or more behaviors 600 to perform an operation. For example, using control system 510, controller 500 controls the coordinated movement of each robot component to avoid collisions with itself and with any objects around robot 100 that robot 100 is aware of. In order to achieve efficiently, the movement (or movement command) of each robot component (eg motor or actuator), the set of possible movements or movements of the corresponding movement space (eg the component of that particular element) Choose from). As described in US patent application Ser. No. 61 / 305,069, filed Feb. 16, 2010, the entire contents of which are incorporated herein by reference, the controller 500 may be an Ethernet IO network or the like. Coordinated commands can be issued on the robot network of

  Control system 510 may provide adaptive speed / acceleration of drive system 200 to maximize stability of robot 100 at different configurations / positions as robot 100 maneuvers around the area ( For example, via one or more behaviors 600).

  In some implementations, controller 500 issues commands to drive system 200 that propels robot 100 according to the heading and speed settings. One or more behaviors 600 may be selected to perform one (in isolation or in combination with other commands as a general robot command) to handle obstacles, and to predict the results of feasible commands The signals received from sensor system 400 may be used to evaluate. For example, a signal from proximity sensor 410 may cause control system 510 to change the commanded velocity or heading of robot 100. For example, a signal from the proximity sensor 410 by a nearby wall may result in the control system 510 issuing a command to decelerate. In another case, a collision signal from the touch sensor (s) due to encounter with a chair may cause the control system 510 to issue a command to change the heading. In other cases, the speed setting of the robot 100 may not be reduced in response to the contact sensor, and / or the travel direction setting of the robot 100 may not be changed in response to the proximity sensor 410.

  Behavioral system 510 b is configured to change velocity orientation 600 (e.g., a behavior routine executable on a processor) configured to adjust the velocity settings of robot 100, and to change the orientation of robot 100. And directional behavior 600. The velocity and heading behavior 600 may be configured to execute simultaneously and independently of one another. For example, the velocity behavior 600 may be configured to poll one of the sensors (eg, one or more proximity sensors 410, 420), and the heading behavior 600 may be another sensor (eg, another sensor (eg, one). For example, the dynamic bump sensor may be configured to be polled.

  Referring to FIGS. 13 and 14, behavior system 510b is configured to respond to a user 15 touching body 140 for remote control (eg, guiding robot 100). An operation behavior 600a (eg, a behavior routine executable on a processor) may be included. The torso touch teleoperation behavior 600a is activated when the sensor system 400 detects that the torso has received contact (eg, human contact) for at least a threshold period (eg, 0.25 seconds). Can. For example, motion and / or contact in communication with the controller 500 and associated with the corresponding top panel 145t, bottom panel 145b, front panel 145f, back panel 145b, right panel 145r, and left panel 145l of the torso body 145 Sensors 147t, 147b, 147f, 147r, 147r, 147l can detect movement and / or contact with the respective panels, as shown in FIGS. 6B and 6C. Once active, the torso touch teleoperation behavior 600a receives a touch force direction (e.g., as sensed and calculated from the elliptical position of the touch) and local X / Y coordinates (using holonomic mobility) Issue a speed command to the drive system 200. The obstacle detection and obstacle avoidance behavior may be turned off while the torso touch remote control behavior 600a is active. When the sensed touch position, force, or direction changes, the torso touch remote control behavior 600a changes the speed command to correspond to the sensed touch force direction. The torso touch teleoperation behavior 600a may perform a stop routine when the sensor system 400 no longer senses contact with the robot 100 for a threshold period (eg, 2 seconds). The stop routine may cause drive system 200 to stop driving after about 0.5 seconds when sensor system 400 no longer senses contact with robot 100 (e.g., with barrel 140). The torso touch teleoperation behavior 600a can provide a delay in stopping the robot 100 to allow movement of the touch point without having to wait for the trigger period.

  The torso touch remote operation behavior 600a may issue an assistive drive command to the drive system 200 that allows the user to push the robot 100 while receiving drive assistance from the drive system 200 (e.g. 100 can not move, but can support the movement of the robot 100 by the user, partial velocity command).

The torso touch remote control behavior 600a may be partially disposed on the torso 140 (and elsewhere on the robot 100 such as the head 160), a touch sensor system 480 (eg, a button, a capacitive sensor, A sensor signal may be received from a contact sensor or the like. The torso touch remote control behavior 600a is between 3 and 5 feet from the ground G to position at least a portion of the touch sensor system 480 at a height accessible to a typical user. The body 140 may be positioned by H _T.

In some implementations, the torso touch remote control behavior 600a recognizes the robot 100 and a user touch to place a particular pose. For example, when the user 15 depresses the torso 140, the sensor system 400 detects a downward force on the torso 140 and sends a corresponding signal to the controller 500. The torso touch teleoperation behavior 600a receives an indication of a downward force on the torso 140 and causes the control system 510 to reduce the height H _L of the legs 130, thereby causing the torso 140 height H Issue a command to decrease _T. Similarly, when the user 15 pushes / pulls the torso 140, the torso touch remote control behavior 600 a receives an indication of an upward force from the sensor system 400 to the torso 140 and the control system 510 to the leg A command is issued to increase the length H _L of the portion 130, thereby increasing the height H _T of the barrel 140.

  When the user 15 presses, pulls, and / or rotates the head 160, the torso touch remote control behavior 600a instructs the user operation from the sensor system 400 (e.g. And the control system 510 may respond by causing the head 160 to move accordingly and then issue a command to hold the pose.

  In some implementations, robot 100 provides passive resistance and / or active support to user manipulation of robot 100. For example, the legs to the user's manipulation of the robot 100 such that the motors 138b, 152, 154 provide the user with feedback for manipulation as well as assistance for moving relatively heavy components, such as raising the torso 140. Activate passive resistance and / or active support of 130 and neck 150. This allows the user to move the various robot components without having to bear the entire weight of the corresponding component.

  Behavioral system 510b may include tap alert behavior 600b (e.g., a behavioral routine executable on a processor) configured to focus the attention of robot 100 towards the user. In the tap caution behavior 600b, the sensor system 400 detects that the body 140 (or some other part of the robot 100) has received contact (eg, human touch) for at least a threshold period (eg, 0.25 seconds). When you do, you may be active. Furthermore, the tap alert behavior 600b may only be active when the torso touch teleoperation behavior 600a is inactive. For example, a sensed touch on the torso 140 for 0.2 seconds will not trigger the torso touch teleoperation behavior 600a, but will trigger the tap attentional behavior 600b. The tap caution behavior 600b may use the contact position on the barrel 140 to tilt and / or pivot the head 160 (via actuation of the neck 150) to look at the user. When the head 160 reaches a position looking towards the touch position, the stopping criteria of the behavior 600b can be reached.

  In some implementations, behavior system 510b may be configured to stop drive system 200 from driving (eg, stop robot 100), tap stop behavior 600c (eg, executable on a processor) Behavior routine). The tap stop behavior 600 c causes the sensor system 400 to detect that the barrel 140 has received a touch (eg, a human touch) and issues a zero speed drive command to the drive system 200 to cancel any previous drive commands. When you do, you may be active. When the robot is driving and the user desires to stop it, the user can tap on the torso 140 (or some other part of the robot 100) or touch sensor. In some embodiments, the tap stop behavior 600c can be activated only if higher priority behaviors such as the torso touch teleoperation behavior 600a and the tap attention behavior 600b are not active. The tap stop behavior 600c may end when the sensor system 400 no longer detects a touch on the torso 140 (or elsewhere on the robot 100).

  In some implementations, the robot 100 is an intermediary security device 350 (also referred to as a bridge, to enable communication between the web pad 310 and the controller 500 (and / or other components of the robot 100) 9). For example, the bridge 350 may convert the communication of the web pad 310 from a web pad communication protocol to a robotic communication protocol (e.g., Ethernet with gigabyte capacity). The bridge 350 may authenticate the web pad 310 and provide communication conversion between the web pad 310 and the controller 500. In some embodiments, bridge 350 includes an approval chip 352 that approves / provides any communication traffic between web pad 310 and robot 100. The bridge 350 may notify the controller 500 when it checks the approved web pad 310 in an attempt to communicate with the robot 100. Also, after approval, the bridge 350 may notify the web pad 310 of communication approval. The bridge 350 may be placed on the neck 150 or head (as shown in FIGS. 2 and 3) or elsewhere on the robot 100.

  Session Initialization Protocol (SIP) is an IETF-defined signaling protocol that is widely used to control multimedia communication sessions such as voice and video calls over the Internet Protocol (IP). The protocol can be used to create, modify, and terminate two-way (unicast) or multi-party (multicast) sessions, including one or more media streams. The modifications may involve changing the address or port, inviting more participants, and deleting or adding media streams. Other possible application examples include video conferencing, streaming multimedia delivery, instant messaging, presence information, file transfer, etc. Voice over Internet Protocol (Voice over IP, VoIP) is part of a group of methods, communication protocols, and transmission technologies for delivering voice communications and multimedia sessions over Internet Protocol (IP), such as the Internet. Other terms frequently encountered and often used as synonyms for VoIP are IP Telephony, Internet Telephony, Voice over Broadband (VoBB), Broadband Telephony, and Broadband Telephones.

  FIG. 15 provides an example of telephone communication including interaction with the bridge 350 to initiate and conduct communication through the robot 100. The phone A's SIP calls the SIP application server. The SIP calls the VoIP dial function and sends an HTTP Post request to the VoIP web server. The HTTP Post request may behave like a callback function. The SIP application server sends a ringing tone to phone A to indicate that the call has been initiated. The VoIP server initiates a call to the callback number contained in the HTTP Post request via the PSTN. The callback number terminates on the SIP DID provider, which is configured to send the callback to the SIP application server. The SIP application server matches the incoming call with the outgoing call of phone A and answers both calls with an OK response. A media session is established between phone A and a SIP DID provider. Phone A may listen to the artificial ringing tone generated by VoIP. Once VoIP verifies that the callback process has been answered, it initiates a PSTN call to a destination such as robot 100 (via bridge 350). The robot 100 responds to the call, and the VoIP server bridges the media from the SIP DID provider with the media from the robot 100.

  16A-16D illustrate the robot 100 (or a portion of the controller 500 or drive system 200, etc.), the computing device 310 (removable or fixedly attachable to the head 160), and the cloud 1620 (cloud computing) And provides a schematic of an example robotic system architecture 1600, 1600a-d, which may include a portal 1630 and a portal 1630.

  The robot 100 is movable (eg, drive system 200), a secure, safe and stable robot intelligence system such as control system 510 (FIG. 13) implemented on controller 500, power supply 105, sensing system 400 Various core robot features can be provided, which can include, optional manipulation with a manipulator in communication with controller 500. Control system 510 can provide head and speed control, body attitude control, navigation, and core robot applications. The sensing system 400 may include vision (eg, via the camera 320), depth map imaging (eg, via the 3-D imaging sensor 450), collision detection, obstacle detection and obstacle avoidance, and / or ( For example, inertial measurement can be provided via inertial measurement unit 470.

  The computing device 310 may be a tablet computer, a portable electronic device such as a telephone or a personal digital assistant, or a tablet or display without data processing capability (e.g. a tablet serving as a monitor for an atomic scale PC in the robot body 110) It may be In some embodiments, the tablet computer can have a touch screen for displaying a user interface and receiving user input. The computing device 310 may be a software application (eg, stored in memory and executable on a processor) for security, drug compliance, telepresence, behavior instruction, social networking, active alarms, home management, etc. One or more robotic applications 1610 may be executed, which may be included. Computing device 310 may provide communication capabilities (eg, secure wireless connectivity and / or cellular communications), precision application development tools, speech recognition, and personal or object recognition capabilities. The computing device 310 may, in some embodiments, be Google, Inc. Provided by Android, Apple, Inc. Use an interactive / COMS equipped operating system such as an iPad OS, other smart phone operating systems, or a special robot operating system such as RSS A2.

  Cloud 1620 provides cloud computing and / or cloud storage capabilities. Cloud computing may provide Internet-based computing, whereby a sharing server provides resources, software, and data to computers and other devices on demand. For example, cloud 1620 is a cloud computing service that includes at least one server computing device that may include a service abstraction layer and a hypertext transfer protocol wrapper on a server virtual machine instantiated thereon. It is also good. The server computing device may be configured to parse the HTTP request and send an HTTP response. Cloud computing may be a technology that uses the Internet and a central remote server to maintain data and applications. Cloud computing can allow a user to access and use the application 1610 without installation and to access personal files on any computer using Internet access. Cloud computing enables relatively efficient computing by concentrating storage, memory, processing, and bandwidth. Cloud 1620 can provide scalable on-demand computing power, storage, and bandwidth while reducing robot software requirements (eg, by releasing CPU and memory usage). Robot connectivity to the cloud 1620 enables automated data collection of robot motion and usage history without requiring the robot 100 to return to the base station. In addition, continuous data collection over time can yield a wealth of data that can be data mined for marketing, product development, and support.

  Cloud storage 1622 may be a model of network computer data storage in which data is stored on a plurality of virtual servers hosted generally by third parties. By providing communication between the robot 100 and the cloud 1620, the information collected by the robot 100 can be safely viewed by authorized users via a web-based information portal.

  Portal 1630 may be a web-based user portal for collecting and / or providing information such as personal information, housing status information, and robot status information. The information can be integrated with third party information to provide additional functionality and resources to the user and / or the robot 100. Robot system architecture 1600 can facilitate positive data collection. For example, application 1610 running on computing device 310 may collect data and reports about the operations performed by robot 100 and / or the individuals or environments viewed by robot 100 (using sensing system 400) May be This data may be a unique characteristic of the robot 100.

  In some embodiments, portal 1630 is a personal portal website on the World Wide Web. Portal 1630 may provide a personalization ability and a path to other content. Portal 1630 may use distributed applications, different numbers and types of middleware and hardware to provide services from several different sources. In addition, the business portal 1630 can share collaborative activities at work and provide content that can be used on multiple platforms such as personal computers, personal digital assistants (PDAs), and mobile / mobile phones Good. Information, news, and updates are examples of content that may be delivered through portal 1630. The personal portal 1630 may relate to any particular topic, such as providing information to friends on social networks or providing links to external content that may be useful to others.

  "High density data" versus "low density data" and "high density features" versus "low density features" are referred to herein for spatial data sets. Without limiting or reducing the meaning as one skilled in the art would interpret to mean such terms, “high density” versus “low density” generally means many data point pairs per spatial representation It means fewer data points, and in particular it may mean:

  (I) "Dense" image data, in the context of 2-D image data, or 3-D "image", including 2-D data and range, the pixels were almost completely populated, or Or “low density,” including image data that can be rasterized into pixels with substantially no arch factor from the original image capture (including nearly uncompressed, raw, or uncorrupted compressed images) The image is quantized, sampled, degraded compressed, vectorized, segmented (eg, into superpixels, nodes, edges, surfaces, points of interest, voxels) or otherwise original capture. The fidelity of the image is significantly reduced, or must be interpolated when rasterized into pixels to represent the image.

  (Ii) In the context of 2-D or 3-D features, "high density features" can all be detected and recorded, almost unconstrainedly populated features until resolution of the detection approach, and And / or may be features recognized by a detector that is recognized to collect many features (HOGs, wavelets) across sub-images, "low density features" being feature inputs, lateral suppression, and / or features In the number of choices, the number may be intentionally constrained and / or recognized by a detector that is recognized to identify a limited number of separated points (Harris corners, edges, Shi-Tomasi) in the image .

  With regard to the 3-D environment structure, the robot 100 may acquire images such as high density images 1611 of the scene 10 around the robot 100 while moving around the work surface 5. In some implementations, the robot 100 uses a camera 320 and / or an imaging sensor 450 (eg, a volume point cloud imaging device) to acquire high density images 1611. The controller 500 in communication with the camera 320 and / or the sensor 450 may include information 1613 such as accelerometer data tracking, distance measurement data and / or other data from the sensor system 400 along with the time stamp, with the high density image 1611 One may associate (eg, annotate or tag high density image 1611 with data). In some embodiments, the robot 100 captures a streaming sequence 1615 of high density images 1615, annotates the high density image sequences 1615 with annotation data 1613, and provides an annotated high density image sequence 1615a. The robot 100 periodically transmits image data 1601 (eg, via the controller 500 or the web pad 310) to a cloud storage device 1622 that can store potentially very large image data sets 1603 over working time You may Image data 1601 may be used as raw sensor data (eg, point cloud or signal or high density image sequence 1615) or tagged data such as annotated high density image sequence 1615a (eg, Java Script Object Notation (JSON) objects, etc. Data object having characteristics or attributes). Cloud service 1620 processes received image data 1601 (eg, high density image sequence 1615 or high density annotated image sequence 1615 a) and processes processed data set 1617 to robot 100, eg, controller 500 and / or web pad You may reply to 310. The robot 100 may also issue drive commands 1619 to the drive system 200 (eg, via the controller 500 or the web pad 310) based on the received processed data set 1617 to operate around the scene 10. Good.

  After a threshold period, or after a threshold amount of image data 1601, 1603 has been stored in the cloud storage 1622, the cloud service 1620 processes the image data set 1603 to obtain a high density of scene 10 (environment) 3- Make the D map or model 1605 and then simplify this high density 3D map or model 1605 to be a 2D map with height data at each point (eg similar to a 2D terrain map) 2 One of a variety of off-line methods may be implemented to make the D height map 1607. In some embodiments, the 2-D height map 1607 is a terrain map having X and Y coordinates with Z data. Each X, Y coordinate may have one or more Z points (i.e. height data). Unlike high density 3-D maps, which can have multiple Z points (eg, hundreds or thousands of Z points) for each X, Y coordinate, 2-D height map 1607 has 2 to 20 There may be fewer Z points than the threshold for each X, Y coordinate, such as between (eg, 10) points. The 2-D height map 1607 derived from the 3-D map of the table in the room is the first Z point of the bottom of the table surface for each X, Y coordinate along the table and the second of the top surface of the table surface Can indicate the Z point of This information allows the robot 100 to determine if it can pass under the table surface. By reducing the Z points from the high density data set of the continuous range of Z points for each X, Y coordinate to a selected number of low density data points of Z points indicative of the detected object 12, the robot 100 A 2-D height map 1607 can be received, which has a relatively smaller size than the 3-D map used by the service 1620. This, in turn, causes the robot 100 to store the 2-D height map 1607 on local memory having a practical and cost effective size compared to the scalable memory space available to the cloud service 1620. Make it possible to The robot 100 receives a 2-D height map 1607 from the cloud 1620, which provides the robot 100 and associated controller 500 with navigation data for future work in the scene 10.

  Additional methods and features of 3-D map data comparison are incorporated herein by reference in their entirety. Triebel, P .; Pfaff, and W. It is disclosed by Burgard in "Multi-Level Surface Maps for Outdoor Terrain Mapping and Loop Closing", IEEE / RSJ International Conference on Intelligent Robots and Systems, 2006.

  The cloud 1620 provides the robot 100 with on-demand expansion of resources (eg, computations, processing, memory, etc.) that may not be practical or cost effective for the robot 100. For example, cloud 1620 is expanded to a first size for storing and / or processing relatively large amounts of data 1601 that may be used only for a short period of time and then discarded and then reduced again to a second size An extensible cloud storage device 1622 can be provided. In addition, the cloud 1620 can provide computing power to perform relatively complex calculations or "forceful" algorithms that may not otherwise be possible on the robot. By transferring computing power and memory to the scalable cloud 1620, the robot 100 can use the controller 500 with relatively little computing power and memory, thus providing a cost effective solution. . In addition, robot 100 may also perform real-time tasks (such as obstacle avoidance) (on controller 500 or web pad 310) while passing non-real-time and non-time-dependent tasks to cloud 1620 for processing and subsequent retrieval. Good.

  Cloud 1620 may include one or more filters (eg, bundle adjustment, RANSAC, expectation maximization, SAM, or other 3-D structure estimation algorithms) to process image data set 1603 into a 3-D representation. It may be executed. Once processed and the high density 3-D map 1605 created or updated, discard the image data set 1603 from the cloud storage device 1622, free resources, and allow the cloud 1620 to expand accordingly Can. As a result, the robot 100 requires neither built-in storage nor processing to handle storage and processing of the image data set 1603 by use of cloud-based resources. Cloud 1620 may return processed navigation data 1601 or map 1607 (eg, a compressed 2-D height map) to robot 100, which is then used for relatively simpler localization and navigation processing can do.

  Further methods and features of 3-D reconstruction are described in "3D Models From Extended Uncalibrated Video Sequences: Addressing Key-frame Selection and Projective Drift", which is incorporated herein by reference in its entirety. Repko and M. Pollefeys; Fifth International Conference on 3-D Digital Imaging and Modeling, 2005.

  With respect to floor classification, the robot 100 may acquire images of the work surface 5 while maneuvering around the scene 10 thereon. The controller 500 may receive images and perform object detection routines for object detection and obstacle avoidance (ODOA). In some implementations, the controller 500 associates information such as accelerometer data tracking, mileage measurement data, and / or other data from the sensor system 400 with time stamps with the image (e.g. Tag). The images may capture falling objects, tufts of rugs, socks on rugs, etc. As in the previous example, the robot 100 can stream image data 1601 to the cloud storage device 1622 while operating for subsequent batch processing. When the robot encounters an undesirable event (ie an accidental collision), a special "danger" tag is inserted into the data set so that data moments before the danger may be identified for the learning algorithm Be done. Once the annotated image data set and associated tags 1603 are stored (possibly with data sets from many other robots 100), the parallel cloud host 1620 is, for example, a temporally advanced danger tag, An annotated image data set 1603 may be invoked to process using a monitored learning algorithm that calculates a "dangerous image" class from many images of the real environment. Once training of the hazard image category model 1609 is complete, the parameters (small amounts of data) of the model 1609 can be downloaded back to many different robots 100. Therefore, the entire robot team 100 can learn the elements of the environment online. Learning methods applicable to this method include genetic algorithms, neural networks, and support vector machines. All of these can be too complex and may not be able to operate online (ie, on a local robot processor) with low cost robot 100 as it takes up too much memory, but cloud 1620 Provides access to the "fully trained" classifier 1625 to the robotics.

  The “Classifier” 1625 typically employs an iterative training algorithm to, for example, minimize error functions, optimize / optimize cost functions, or otherwise improve performance using training data. Is a machine learning algorithm. Typically, the algorithm has several parameters whose values are learned from the training data. Three types include monitored regression, monitored classification, and unmonitored classification via clustering or dimensionality regression. Examples of classifiers 1625 include support vector machines (SVMs) (using gradient descent or other cost function minimization techniques), simple Bayesian classifiers, logistic circuits, AdaBoost, K-nearest neighbors, of various types and kernels. ("K-NN") and / or K-NN regression, neural networks, random forests, and linear models. A "classifier" may be represented by a vector, matrix, "descriptor" or other data set, and various algorithms such as orientation gradient histogram (HOG), shape context histogram (SCH), color patch, texture Scale, invariant feature descriptors from patches, intensity patches, SIFT, SURF, or equivalent, affine invariant descriptors like MSER, labeled superpixels or segments in an image, identified by Hough transform, or RANSAC line detection Classify the "features" to be obtained, and other features that can be combined with this.

  Classifiers 1625 can be implemented online from photos only, to help robot systems avoid hazards in their environment. Once the model parameters have been determined, the image data set 1603 stored in the cloud storage device 1622 can be discarded. This embodiment can be combined with the previous one, where the determination of the 3-D structure of the environment allows the identification of traversable areas to train the classification algorithm. One learning technique example is described in "Long-Term Learning Using Multiple Models For Outdoor Autonomous Robot Navigation," by Michael J., which is incorporated herein by reference in its entirety. Procopio, Jane Mulligan, and Greg Grudic, 2007 IEEE International Conference on Intelligent Robots and Systems.

  Streaming is generally distinguished from dispatching batch processing tasks (which can occur at very high data rates) or penetrating images in a delayed fashion, with packets being sent to the transmitter and receiver. Means that the image sequence is transmitted, with or without buffering, with some relation to the order and rate at which the image sequence was captured in real time . The term "streaming" dispatches batch processing tasks to distinguish the speed at which different data are moved within and between robot system entities based on real-world bandwidth, processing, and storage constraints. It is distinguished from

  Referring to FIGS. 16D and 16E, the robot 100 at practical cost levels may have limitations in robot bandwidth, calculations, and storage. By annotating the image 1611 with locally processed data 1613 (ie, data from the robot sensor system 400), the robot 100 is very infrequent (eg, every few seconds instead of several times per second) Times) the upstream bandwidth can be dramatically reduced compared to continuous streaming of data by transmitting the image 1611, high density image sequence 1615, 1615a or image data 1601. Additionally, controller 500 and / or web pad 310 can store image 1611 and annotation data 1613 on robot 100 when wireless connectivity is inaccessible, and the wireless connectivity is accessible. It can include a local cache (memory), which can be transmitted. Buffering allows the robot 100, for example, to collect room equivalent data in a room where signal reception is poor, and once wireless connectivity is again available, such as in an adjacent room, complete data set 1601 , 1603 can still be transmitted. Thus, data management on the robot 100 may be necessary to adapt to different communication environments.

  One or more cameras 320 and / or imaging sensors 450 may be coupled to the robot 100 as the mobile robot is guided, directed or automatically navigates through the scene 10 (eg, home, office, etc.). One or more sequences 1615 of images 1611 may be acquired, each corresponding to a field of view acquired in the pose (position and orientation) of the robot 100 along the trajectory of A timer (e.g., of controller 500) provides a reference timestamp for each image 1611, which is associated with annotation 1613 corresponding to position, localization, velocity or acceleration, sensor based orientation, etc. It is also good. At least some images 1611 may be annotated with information, and all images 1611 may be annotated with timestamps or other metadata reflecting robotic state, image state, etc. The image 1611 can be captured at some real-time capture rate, which need not be periodic (eg, image 1611 can be, for example, processing availability, bandwidth availability, time of day, task, purpose within the current image, etc. May be captured at an adaptive rate based on the density of the features being

  The camera 320 and / or the imaging sensor 450 may be a hardware encoder, a high-speed bus available to memory, a built-in processor, and a storage device for storing and storing the relatively fast captured image 1611 (eg, flash memory or It may be in communication with a hard disk drive). An image 1611 in which one or more machine vision algorithms are stored so as to create a model such as a set of parameters of the image classifier 1625 that does not itself consume more resources than can be supported on the robot 100 It can be applied to However, there are some obstacles to overcome.

  (1) Typically, identification of at least some training data (e.g., image 1611) or annotations 1613 representing features to be modeled should be applied prior to significant image compression.

  (2) Many algorithms and heuristics can be extended and adapted to the relatively large distributed computing infrastructure instantiated in cloud services to complete modeling in a short period of time It is necessary to get the image 1611 to reach the target computer infrastructure.

  (3) The bandwidth available to transport relatively large data sets may be (i) a robot 100 that is mobile using radio bandwidth typically occupied by other traffic, and (ii) It is limited by Internet access, which can limit bandwidth in the upstream direction.

  To overcome at least these obstacles, high density images 1611 can be transmitted (eg, wirelessly) from the robot 100 to the local server 1640, which may be at a local transmission rate that is relatively slower than the real time capture rate. It may be a wired network with a storage device. The local server 1640 can buffer the high density image 1611 and annotate it with the annotation 1613. Additionally, the local server 1640 may store the annotated high-density image sequence 1615a for subsequent transmission to the cloud service 1620. The local server 1640 may receive image data 1601 (e.g., high density images 1611, high density image sequences, etc.) at a cloud transmission rate slower than the real time capture rate, or at a rate suitable to service the cloud computing infrastructure at high speed 1615 and / or the annotated high density image sequence 1615a) can be communicated to the cloud service 1620.

  The cloud service 1620 originates from and represents, for example, the annotated high-density image sequence 1615a, for example, to provide a simplified data set 1617 that excludes any raw image data, eg, on the image data 1601. By resiliently dispatching a sufficiently fast, parallel, and / or large image set classifier 1625 to process instances to train classifiers 1625 (eg, on high density image 1611 and related annotations 1613) The received image data 1601 can be processed. The cloud service 1620 may, for example, transmit the simplified data set 1617 to the local server 1640 in communication with the robot 100 or directly to the robot 100 after the processing interval. The elements of data set 1617 may be used by robot 100 to issue commands to drive system 200 to maneuver robot 100 relative to scene / environment 10. For example, a “code” or “sock” or “ingestible debris” classifier 1625 trained on the image data 1601 of one of many robots identifies the image pattern and is away from the hazard or Any direction towards the area may be used with the parameters sent back to the robot 100 to direct the robot 100.

  When the wireless connection or consumer / commercial asymmetric broadband service is bandwidth limited in the transmission direction, it buffers the high density image sequence 1615 into the large storage device, and performs packet switching, reordering, correction, service Quality and the like can be used to upload over a relatively longer period of time than the acquisition time (ie, the time period for acquiring the high density image sequence 1615). For example, the upload may occur overnight, over a period of several hours where the robot's trajectory was in minutes, over a period of several days where the robot's trajectory was in hours, etc.

  A service related to the robot software platform of the robot 100, representing a sequence of field views of the environment along the trajectory of the mobile robot 100 captured on the local server 1640 or cloud service 1620 at real time capture rate It can be received by 1623. The service 1623, also referred to as a cloud gateway, may be an agent of a robotic software platform or may be independently controlled by a third party. An algorithm, a classifier 1625, and an application platform in which the cloud gateway 1623 is instantiated / maintained as the high density image sequence 1615 as one or more virtual servers 1621 in the cloud service 1620 (e.g., depending on the annotation content) The high density image sequence 1615 and any annotations 1613 may be packaged as training data for the classifier 1625, which may be consistent with the instance. Alternatively, the cloud gateway 1623 processes the same information as pre-packaged information from the robot 100, local base station etc, and conforms to adaptive and elastic cloud service application programming interface (API) (eg Amazon EC2) The dispatch of a given virtual server instance 1621 may be initiated simply by resiliently adding such other instances. The local server 1640, the cloud gateway 1623, the cloud service manager, or the initial virtual processor instance 1621 each perform at least some of the high density images 1611 in the received high density image sequence 1615, 1615a in the high density image sequence Dispatch a batch processing task to reduce to the data set 1617 derived from and representing 1615, 1615a. As needed, new virtual processor instances 1621 can be serviced / instantiated (either all at once or as the training task becomes more complex). High density images 1611 and trained models 1609 can be kept in long-term memory instances. The trained classifier 1625 or model 1609 parameters are sent back to the robot 100 (e.g., directly or through the robot 100 agent), excluding the sequence of raw images 1611.

  FIG. 16F provides an example arrangement 1600 f of operations for a method of navigating the robot 100. The method comprises capturing 1602 f a streaming sequence 1615 of a high density image 1611 of the scene 10 around the robot 100 along the trajectory of movement of the robot 100 at real time capture speed 1602 f and an annotation 1613 out of the high density image 1611. Associating with at least some of 1604f. The method also transmits 1606f the high density image 1611 and the annotation 1613 to the remote server 1620 at a transmission rate slower than the real time capture rate and receiving the data sets 1607, 1617 from the remote server 1620 after the processing time interval. And 1608f. Data set 1607, 1617 originates from and represents at least a portion of high density image sequence 1615, 1615a and corresponding annotation 1613, but excludes the raw image data of sequence 1615, 1615a of high density image 1611. The method includes moving 1610 f the robot 100 relative to the scene 10 based on the received data sets 1607, 1617.

  The method sends the high density image 1611 and the annotation 1613 to the local server and buffer 1640 (FIGS. 16D and 16E) and then sends the high density image 1611 and the annotation 1613 to the remote server 1620 at a transmission rate slower than the real time capture rate. May include sending to. The local server and buffer 1640 may be within a relatively short range of the robot 100 (e.g., 20 to 100 feet or within wireless communication range). For example, the local and buffer 1640 may be a personal computer in the home of the user housing the robot 100, or a local server of a building housing the robot 100.

  In some implementations, the annotation 1613 may include a time stamp, such as an absolute time reference, corresponding to at least some of the high density images 1611 and travel distance measurement data, accelerometer data, tilt data, and angular velocity data. And attitude related sensor data, which may include at least one of The annotations 1613 can be associated with a high density image 1611 reflecting the hazard event captured within the time interval for the robot's hazard response 100 (e.g., avoiding a cliff, exiting a containment situation, etc.). In a further example, associating 1604 f the annotation 1613 may include associating a key frame identifier with a subset of the high density image 1611. The key frame identifier may enable identification of the high density image 1611 based on characteristics of the key frame identifier (eg, flag, type, group, motion, still, etc.).

  The annotations 1613 may include a low density set of 3-D points derived from structural and motion recovery of features tracked between the high density images 1611 of the streaming sequence 1615 of high density images 1611. The low density set of 3-D points may originate from the volume point imaging device 450 on the robot 100. In addition, annotations 1613 may include camera parameters such as the pose of the camera with respect to individual 3-D points of a low density set of 3-D points. The indicia of the translatable and non-traversable areas of the scene 10 may be annotations 1613 to the high density image 1611.

  Data set 1607, 1617 may be extracted from high density image 1611, one or more texture maps such as 2-D height map 1607, and / or terrain map 1607 representing features within high density image 1611 of scene 10. May be included. Data set 1607, 1617 may include a trained classifier 1625 for classifying features in captured new high-density image 1611 of scene 10.

  FIG. 16G provides an example arrangement 1600g of operations for a method of abstracting mobile robot environment data. The method includes receiving 1602 g a sequence 1615 of high density images 1611 of the robotic environment 10 from the mobile robot 100 at a receiving speed. The high density image 1611 is captured along the motion trajectory of the mobile robot 100 at real time capture speed. The reception rate is slower than the real time acquisition rate. The method also includes receiving 1604 g an annotation 1613 associated with at least some of the high density images 1611 in the sequence 1615 of high density images 1611 and a high in at least some of the high density images 1611. Dispatching 1606 g a batch processing task to reduce density data to a data set 1607, 1617 representing at least a portion of the sequence 1615 of high density image 1611. The method also includes transmitting 1608 g the data set 1617 to the mobile robot 100. Data sets 1607, 1617 exclude the raw image data of the sequence 1615 of high density images 1611.

  In some implementations, the batch processing task processes the sequence 1615 of high density images 1611 into a high density 3-D model 1609 of the robot environment 10 and processes the high density 3-D model 1609. Making the terrain model 1607 for a coordinate system at 2-D positions and at least one height from the floor surface G. In some embodiments, terrain model 1607 is for a coordinate system of 2-D locations and multiple occupied and unoccupied height boundaries from floor G. For example, a terrain model where a room with a table provides data indicating the upper and lower heights of the associated table surface so that the robot 100 can determine whether it can pass under the table. It is 1607.

  Batch processing tasks may correspond to high density image sequences corresponding to multiple robotic environments 10 (e.g., so that the cloud 1620 can build classifiers 1625 to identify desired features in any environment) 1615, 1615a may be included. Thus, batch processing tasks may include training multiple classifiers 1625 and / or one or more classifiers 1625 on a sequence 1615 of high density images 1611. For example, the batch processing task may associate an annotation 1613 reflecting the hazard event with the high density image 1611 captured within the time interval for the hazard response of the mobile robot 100, eg, model parameters of the classifier 1625. Training the classifier 1625 of the hazard related high density image 1611 using the relevant hazard event annotation 1613 and the corresponding high density image 1611 as training data to provide the data set 603, 1607, 1617 of May be included. The classifier 1625 may include at least one support vector machine that builds at least one hyperplane for classification, and model parameters may classify the data set 1603, 1607, 1617 into a risk related classification. Define a trained hyperplane. The model parameters may include sufficient parameters to define the kernel of the support vector machine and soft margin parameters.

  In some embodiments, batch processing tasks include instantiating a plurality of scalable virtual processes 1621 that are proportional to the scale of the high-density image sequence 1615, 1615a to be processed. At least some of the virtual processes 1621 are released after transmission of the data sets 1607, 1617 to the robot 100. Similarly, batch processing tasks may include instantiating a plurality of scalable virtual memories 1622 that are proportional to the scale of the stored high density image sequences 1615, 1615a. At least some of virtual memory 1622 is released after transmission of data sets 1607, 1617 to robot 100. Also, the batch processing task distributes the scalable virtual servers 1621 according to one of the geographic proximity to the mobile robot 100 and / or the network traffic from the mobile robots 100. May be included.

  US Patent Publication No. 2011/0238857 "Committed Processing Rates for Shared Resources", published on September 29, 2011, to Certain et al., Is incorporated herein by reference in its entirety. The described cloud processing infrastructure of Certain et al. Is one species that can be combined with it, such as the management system 202 or the node manager module 108, which is the seed or part of the cloud gateway 1623, the seed or part of the elastic processing server 1621. One program execution service (PES) and / or virtual machine 110, and archive storage 222 or block data service (BDS) 204 or archive manager 224, which is the seed or part of long-term storage instance 1622.

  FIG. 16H is a schematic diagram of an example mobile human interface robot system architecture 1600d. In the illustrated example, application developer 1602 generates application 1610 executable on web pad 310 or computing device 1604 (eg, desktop computer, tablet computer, mobile device, etc.) in communication with cloud 1620 Application development tools 1640 can be accessed and used as desired. Example application development tools 1640 include integrated development environment 1642, software development kit (SDK) library 1644, development or SDK tools 1646 (eg, modules of software code, simulators, cloud usage monitor and service configurator, and cloud service extension uploader / Developer), and / or source code 1648, but is not limited thereto. The SDK library 1644 may allow enterprise developers 1602 to leverage the mapping, navigation, scheduling, and conferencing technologies of the robot 100 in the application 1610. Example applications 1610 may include, but are not limited to, map builder 1610a, mapping and navigation application 1610b, video conferencing application 1610c, scheduling application 1610d, and usage application 1610e. Applications 1610 may be stored on one or more application servers 1650 (eg, cloud storage 1622) in cloud 1620 and may be accessed through a cloud service application programming interface (API). Cloud 1620 may include one or more databases 1660 and a simulator 1670. The web services API enables communication between the robot 100 and the cloud 1620 (eg, with the application server 1650, the database 1660, and the simulator 1670). External system 1680 may also interact with cloud 1620 to access application 1610, for example.

  In some embodiments, map builder application 1610a may use camera 320 or 3-D imaging sensor 450 using reference coordinates as provided by odometry, global positioning system, and / or waypoint navigation. A map 1700 (FIG. 17A) of the environment around the robot 100 can be constructed by linking together the photos or videos captured by the. The map may provide an indoor or outdoor street or route indication of the environment. For malls and shopping centers, the map can provide a route tour through the mall with each store marked as a reference location with additional linked images or videos and / or promotional information. Maps and / or configuration images or videos can be stored in database 1660.

  The application 1610 may communicate seamlessly with the cloud service, which may be customized and extended based on the needs of each user entity. Enterprise developer 1602 may upload to cloud 1620 a cloud-side extension that fetches data from an external proprietary system for use by application 1610. The simulator 1670 allows developers 1602 to build enterprise-wide applications without the robot 100 or associated robot hardware. Users may use the SDK tool 1646 (eg, usage monitor and service configurator) to add or disable cloud services.

  Referring to FIGS. 17A and 17B, in some situations, the robot 100 receives an occupancy map 1700 of the object 12 in the scene 10 and / or the work area 5, or the robot controller 500 sequentially captures the imaging sensor 450. An occupancy map 1700 is generated (and may be updated) based on image data and / or image depth data received from (eg, second 3-D image sensor 450b). Co-localization and mapping (SLAM) may be used to construct map 1700 in an unknown environment or scene 10 (without using a priori knowledge) or while tracking its current position (while A technique that may be used by the robot 100 to update the map 1700 in a known environment) using a priori knowledge from the map. Map 1700 may be used to delineate the environment for planning and navigation in order to determine its location in the environment. The map 1700 assists in the evaluation of the actual position by recording the information obtained from some form of perception and comparing it to the current set of perceptions. The benefit of map 1700 in aiding in the assessment of position increases as the accuracy and quality of the current perception decreases. Map 1700 generally represents the state when map 1700 is provided or generated. This does not necessarily match the state of the environment when the map 1700 is used. Other localization techniques include monocular visual SLAM (MonoSLAM), and implementations that use an extended Kalman filter (EKF) in the MonoSLAM solution.

  The controller 500 may perform scale invariant feature transform (SIFT) to detect and represent local features in the captured image. For any object 12 in the image, points of interest on the object 12 can be extracted to provide a "feature description" of the object 12. This description, which is then extracted from the training image, can be used to identify object 12 as it seeks to locate object 12 in a test image containing many other objects. For reliable recognition, it is important that features extracted from the training image be detectable even under changes in image scale, noise and illumination. Such points are usually located on high contrast areas of the image, such as the edge of the object. For object recognition and detection, the robot 100 is invariant to position, scale and rotation, and stands out for its key points that are robust to affine transformations (scale, rotation, shear and change of position) and illumination changes. You may use SIFT to find out. In some implementations, robot 100 captures (using camera 320 and / or imaging sensor 450) multiple images of scene 10 or object 12 (eg, under different conditions, from different angles, etc.) , Store the image in the matrix, etc. The robot 100 can access the stored images to identify new images by comparison, filters, etc. For example, SIFT features can be obtained from an input image and matched to a SIFT feature database obtained from training images (previously captured). Feature matching can be done through Euclidean distance based nearest neighbor approaches. The Hough transform may be used to increase object identification by clustering these features that belong to the same object and reject the excluded matches in the clustering process. SURF (Acceleration Robust Feature) may be a robust image detector and descriptor.

  In addition to the localization of the robot 100 in the scene 10 (e.g. the environment around the robot 100), the robot 100 uses the sensor system 400 to other points in the connection space (e.g. the work area 5) You may move. The robot 100 maps nearby areas around the robot 110 and shorts (eg, mounted on the back of the torso 140 as shown in FIGS. 1 and 3) to identify relatively close objects 12 A distance-type imaging sensor 450a and a relatively large area around the robot 100 are mapped to distinguish relatively distant objects 12 (eg, on the head 160 as shown in FIGS. 1 and 3) And the long-distance type imaging sensor 450b may be included. The robot 100 identifies the known object 12 in the scene 10 as well as the occlusion 16 (e.g. where the object 12 should or should not be identified from the current viewpoint). An occupation map 1700 can be used to do this. The robot 100 attempts to bypass the obstruction 16 or new object 12 so as to register the occlusion 16 or new object 12 in the scene 10 and to verify the position of the new object 12 or any object 12 in the occlusion 16 be able to. Alternatively, using the occupancy map 1700, the robot 100 can determine and track the movement of the object 12 in the scene 10. For example, the imaging sensors 450, 450a, 450b may detect the new position 12 'of the object 12 in the scene 10 while not detecting the mapped position of the object 12 in the scene 10. The robot 100 may attempt to detour the obstruction 16 by registering the position of the old object 12 as occlusion 16 so as to verify the position of the object 12. The robot 100 may compare the new image depth data to previous image depth data (eg, map 1700) and assign a confidence level of the position of the object 12 within the scene 10. The position confidence level of object 12 within scene 10 may time out after a threshold period. Sensor system 400 may update the position confidence level of each object 12 after each imaging cycle of sensor system 400. In some embodiments, detected new occlusions 16 (eg, missing objects 12 from the occupancy map 1700) within the occlusion detection period (eg, less than 10 seconds) are “live” objects in the scene 10. Twelve (for example, moving objects 12) may be represented.

In some implementations, the intended second object 12b located behind the detected first object 12a in the scene 10 may not initially be undetected as a block 16 in the scene 10 . The closure 16 may be an area in the scene 10 not easily detectable or visible by the imaging sensor 450, 450a, 450b. In the illustrated embodiment, the sensor system 400 of the robot 100 (e.g., or a portion thereof, such as the imaging sensors 450, 450a, 450b, etc.) views the viewing angle θ _V (between 0 and 360 degrees) to view the scene 10. ) Can have any field of view 452). In some embodiments, imaging sensor 450 includes omnidirectional optics for a 360 degree viewing angle θ _V , while in other embodiments imaging sensors 450, 450a, 450b have less than 360 degrees For example, it has a viewing angle θ _V of between about 45 degrees and 180 degrees. In embodiments where the viewing angle θ _V is less than 360 degrees, the imaging sensors 450, 450a, 450b (or components thereof) rotate relative to the robot body 110 to achieve a 360 degree viewing angle θ _V May be In some implementations, the imaging sensor 450, 450a, 450b or portions thereof can move relative to the robot body 110 and / or the drive system 200. Also, to detect the second object 12b, the robot 100 drives around the scene 10 in one or more directions so as to obtain a viewpoint, which enables detection of the second object 10b. Imaging sensors 450, 450a, 450b (eg, by translating and / or rotating on work surface 5). Robot movement or independent movement of the imaging sensor 450, 450a, 450b or parts thereof may also solve monocular difficulty.

  A confidence level may be assigned to the detected position or tracked movement of the object 12 in the work area 5. For example, upon generating or updating the occupancy map 1700, the controller 500 may assign a confidence level for each object 12 on the map 1700. The confidence level can be directly proportional to the probability that the object 12 is actually located within the work area 5, as shown on the map 1700. The confidence level may be determined by several factors, such as the number and type of sensors used to detect the object 12. For example, touch sensor 430 may provide the highest level of confidence when touch sensor 430 senses actual contact by robot 100 with object 12. Imaging sensor 450 may provide different levels of confidence that may be higher than proximity sensor 430. Data received from more than one sensor of sensor system 400 can be aggregated or stored to provide a relatively higher level of confidence than any one sensor.

  The travel distance measurement uses data from the movement of the actuator to estimate the change in position (travel distance) over time. In some embodiments, an encoder is disposed on drive system 200 to measure wheel rotation and thus travel distance by robot 100. The controller 500 may use odometry to evaluate the confidence level of the object position. In some implementations, sensor system 400 includes an odometer and / or an angular velocity sensor (eg, a gyroscope or IMU 470) to sense the distance traveled by robot 100. A gyroscope is a device for measuring or maintaining orientation based on the principle of angular momentum conservation. The controller 500 uses odometry and / or gyro signals respectively received from the odometer and / or angular velocity sensor to determine the position of the robot 100 in the work area 5 and / or on the occupancy map 1700. May be In some embodiments, controller 500 uses dead reckoning. Dead reckoning is the process of estimating the current position based on a previously determined position and advancing that position over an elapsed time and course based on a known or estimated velocity. Knowing the robot position in the work area 5 (for example via odometry, gyroscope etc) as well as the sensed position of one or more objects 12 in the work area 5 (via the sensor system 400) Allows the controller 500 to evaluate a relatively higher confidence level (as opposed to without distance measurement or gyroscope) of the position or movement of the object 12 on the occupancy map 1700 and in the work area 5 .

  Distance measurement based on wheel motion can be electrically noisy. The controller 500 independently of the wheel-based distance measurement of the drive system 200 calculates image data of the environment or scene 10 around the robot 100 from the imaging sensor 450 in order to calculate robot motion through visual distance measurement. It may be received. Visual mileage measurement may require the use of optical flow to determine the motion of the imaging sensor 450. The controller 500 can use the motion calculated based on the imaging data of the imaging sensor 450 to correct any errors in the wheel based odometry, thus enabling improved mapping and motion control Make it Visual odometry may have limitations due to poor texture or low light scene 10 when imaging sensor 450 can not track features in the captured image.

  Other details and features of the odometry and imaging system, which may be combinable with those described herein, are incorporated herein by reference in their entirety. U.S. Pat. No. 7,158,317 (describing a "depth of field" imaging system), and U.S. Pat. No. 7,115,849 (describing a wavefront coded interface contrast imaging system).

  When a robot is the first building that it will operate, the robot needs to be guided or provided with a map of the building (for example, the location of rooms and corridors) for autonomous navigation. possible. For example, in a hospital, robots may need to know the location of each room, nursing station, etc. In some implementations, the robot 100 can be trained to receive the layout map 1810, such as that shown in FIG. 18A, to learn the layout map 1810. For example, while guiding the building to the robot 100, the robot 100 may record a particular position that corresponds to the position on the layout map 1810. The robot 100 may display a layout map 1810 on the web pad 310, and when the user takes the robot 100 to a particular position, the user may (eg, touch screen or other pointing device on the web pad 310) Can be tagged on the layout map 1810). The user may choose to enter an indicator of a tagged location, such as the name of the room or the number of the room. When tagging, the robot 100 may store the tag with points on the layout map 1810 and corresponding points on the robot map 1820, such as that shown in FIG. 18B.

  Using the sensor system 400, the robot 100 may construct a robot map 1820 as it moves around. For example, sensor system 400 can provide information as to how far robot 100 has moved and the direction of movement. The robot map 1820 may include fixed obstacles in addition to the walls provided in the layout map 1810. Robot 100 may use robot map 1820 to perform autonomous navigation. In the robot map at 1820, the "wall" is perfectly straightened, for example with the detected packing generated along the wall in the corresponding corridor and / or the furniture detected inside the various small rooms May not be visible. Additionally, differences in rotation and resolution may exist between the layout map 1810 and the robot map 1820.

  After map training, when the user wants to send the robot 100 to a position, the user can point to the indicator / tag (eg, enter the indicator or tag into the position text box displayed on the web pad 310 Or the robot 100 can display the layout map 1810 to the user on the web pad 310 and the user can select a position on the layout map 1810. If the user selects a tagged layout map position, the robot 100 can easily determine the position on the robot map 1820 that corresponds to the selected position on the layout map 1810, and the selected position You can proceed to navigate to.

  If the selected location on layout map 1810 is not a tagged location, robot 100 determines a corresponding location on robot map 1820. In some implementations, the robot 100 calculates scaling sizes, starting point mappings, and rotations between the layout map 1810 and the robot map 1820 using existing tagged locations, and then (eg, Apply the calculated parameters to determine the robot map position, using affine transformation or coordinates.

  The robot map 1820 may not be the same orientation and scale as the layout map 1810. Furthermore, the layout map may not be to scale and may have different distortions depending on the map area. For example, layout maps 1810 created by scanning fire evacuation maps typically found in hotels, offices, and hospitals are not usually drawn to scale, but in different areas of the map It may even have different scales. The robot map 1820 may have its own distortion. For example, the position on the robot map 1820 may be calculated by counting the rotation of the wheels as a measure of distance, if the floor was slightly slippery, or cornering caused extra rotation. Accurate rotation calculations may cause the robot 100 to determine the incorrect position of the mapped object.

  The method of mapping a given point 1814 on the layout map 1810 to the corresponding point 1824 on the robot map 1820 is to map the layout map 1810 and the robot in the area containing the layout map points (e.g. within the threshold range) It may involve using the existing tagged 1812 points to calculate the local distortion with the map 1820. The method further includes applying distortion calculations to layout map points 1814 to find corresponding robot map points 1824. Starting from a given point on the robot map 1820, the reverse can be done, for example, if one wants to find a corresponding position on the layout map 1810 to determine the robot's current position.

  FIG. 18C provides an example arrangement 1800 of operations for operating the robot 100 to navigate around the environment using the layout map 1810 and the robot map 1820. 18B and 18C, operations include receiving 1802 c a layout map 1810 corresponding to the environment of the robot 100 and moving 1804 c the robot 100 within the environment to a layout map location 1812 on the layout map 1810. Using the robot map position 1822c, corresponding to the environment, and recording the robot map position 1822 on the robot map 1820 generated by the robot 100, using the recorded robot map position 1822 and the corresponding layout map position 1812 Determining 1808 c between the 1820 and the layout map 1810 and the corresponding target robot map location 1824 are determined, such that the robot is selected 1814 on the layout map 1810. To allow to navigate, and a 1810c applying is determined distortion to the target layout map location 1814. In some implementations, the operation uses an existing tagged position to determine scaling size, origin mapping, and rotation between the layout map and the robot map, and target layout map position. Resolving the selected target robot map position corresponding to 1814. The operations may include applying an affine transformation to the determined scaling size, origin mapping, and rotation to resolve robot map locations.

  Referring to FIGS. 19A-19C, in some implementations, the method causes the entire area of the layout map 1810 to be covered by at least one triangle 1910 located at the layout map point 1912 tagged with the vertices. Using the tagged layout map point 1912 (also called the recorded layout map position) to derive the triangulation of the area inside the boundary shape containing the tagged layout map point 1912 Including. The method further finds a triangle 1910 containing the selected layout map point 1914, and maps the triangle 1910 mapped in the layout map 1810 and the corresponding triangle 1920 mapped in the robot map 1820 (ie, Determining scaling, rotation, translation, and skew between robot map triangles with same tagged vertices. The method includes applying the determined scale, rotation, translation, and skew to selected layout map points 1914 to find a corresponding target robot map point 1924.

  FIG. 19C provides an example arrangement 1900 of operations for determining the target robot map position 1924. The operation determines triangulation between layout map locations indicating boundaries of target layout map locations 1902, and between mapped triangles in the layout map and corresponding triangles mapped in the robot map. Determining 1904 scale, rotation, translation, and skew, and applying the determined scale, rotation, translation, and skew to the target layout map position to determine a corresponding target robot map point And 1906.

  Referring to FIGS. 20A and 20B, in another embodiment, the method determines the distance of all tagged points 1912 in the layout map 1810 to points 1914 in the selected layout map; Determining the centroid 2012 of the tagged point 1912 of the layout map. The method also includes determining the centroid 2022 of all tagged points 1922 on the robot map 1820. For each tagged layout map point 1912, the method converts vector 2014 ranging from layout map centroid 2012 to the selected layout point 1914 into vector 2024 ranging from robot map centroid 2022 to robot map point 1924 Determining the required rotation and length scaling. Using this data, the method further includes determining an average rotation and a scale. For each tagged layout map point 1912, the method further applies the "ideal robot map coordinate" point 1924i by applying the translation of the center of gravity, the average rotation, and the average scale to the selected layout map point 1914. Including determining Further, for each tagged layout map point 1912, the method determines the distance from that layout map point 1912 to the selected layout map point 1914, and from these shortest distances to the longest distances. And categorizing separately tagged layout map points 1912. The method uses the inverse square of the distance between the tagged layout map point 1912 and the selected layout map point 1914 to “tag” the tagged layout map point 1912. Including determining. The “ideal robot map coordinates” point 1924 i and robot map point 1924 are then prorated by using the influence factor of the tagged layout map point 1912 for the tagged layout map point 1912. Determining the vector, which is the difference between The method includes summing the prorated vectors and adding them to the "ideal robot map coordinates" point 1924i for the selected layout map point 1914. The result is the corresponding robot map point 1924 on robot map 1820. In some embodiments, the method / algorithm includes only the closest N tagged layout map points 1912, rather than all tagged layout map points 1912.

  FIG. 20C provides an example arrangement 2000 of operations for determining a target robot map location 1924 using layout map 1810 and robot map 1820. The operation determines the distance between all layout map positions and the target layout map position 2002, determines the center of gravity of the layout map position 2004, and determines the center of gravity of all recorded robot map positions. Determining rotation and length scaling such that, for each layout map position, converting a vector ranging from the layout map centroid to the target layout location into a vector ranging from the robot map centroid to the target robot map location including.

  Referring again to FIG. 16C, although the robot 100 may operate autonomously, it may be desirable for the user to control or manage the robot 100 through the application 1610. For example, the user may desire to control or define the movement of the robot 100 within the environment or scene 10, such as by providing navigation points on a map. In some implementations, the map builder application 1610a may allow the user to capture an occupancy map 1700 (FIG. 17A) of the scene 10 based on sensor data generated by the sensor system 400 of one or more robots 100 in the scene 10. Or allow the layout map 1810 (FIG. 18A) to be created. In some embodiments, robot map 1810 can be post-processed to generate one or more vector based layout maps 1810. The map builder application 1610a may allow the user to customize the maps 1700, 1810 (e.g., lining up walls that should appear parallel). Also, annotations that can be recognized by the robot 100 and by the application 1610a (and / or other applications 1610) can be added to the maps 1700, 1810. In some implementations, the map builder application 1610a uses cloud services to ensure robot map data, layout map data, user-defined objects, and annotations in the cloud storage 1622 on the cloud 1620. May be stored. Relevant data sets may be push delivered from the cloud service to the appropriate robot 100 and application 1610.

  In some implementations, the mapping and navigation application 1610b (FIG. 16C) allows the user to locate the destination location on the layout map 1810 and request the robot 100 to drive to the destination location. Do. For example, the mapping and navigation application 1610b may be run on a computing device, such as a computer, tablet computer, mobile device, etc., wherein a user is in communication with the cloud 1620. The user accesses the layout map 1810 of the environment around the robot 100, marks or otherwise sets the destination location on the layout map 1810, and requests the robot 100 to move to the destination location. Can. The robot 100 may then navigate autonomously to the destination location using the layout map 1810 and / or the corresponding robot map 1820. In order to navigate to the destination location, the robot 100 determines its local perceptual space (i.e. the space around the robot 100 as perceived through the sensor system 400) and an object detection obstacle avoidance (ODOA) strategy Depending on its ability to perform.

  Referring to FIGS. 11B and 21A-21D, in some implementations, the robot 100 (eg, control system 510 shown in FIG. 13) has its local perceptual space an obstacle (black) 2102, unknown (gray) It is divided into three categories 2104 and known free space (white) 2106. Obstacle 2102 is an observed (ie, sensed) point above ground G that is below the height of robot 100, and an observed point below ground (eg, a hole, gradual decrease, etc.) ). The known free area 2106 corresponds to the area where the 3-D image sensor 450 can see the ground G. Data from all sensors in sensor system 400 can be incorporated into a discretized 3-D voxel grid. The 3-D grid can then be analyzed and transformed into a 2-D grid 2100 using three local perceptual spaces. FIG. 21A provides an example schematic of a local perceptual space while the robot 100 is at rest. The information in the 3-D voxel grid is persistent but degrades over time if not augmented. When the robot 100 is moving, it has more known free space 2106 to navigate for durability.

  The object detection obstacle avoidance (ODOA) navigation strategy of control system 510 may include either accepting or rejecting robot positions that may be due to commands. Different commands can be used to generate potential robot paths 2110 of many levels of depth, resulting in robot position at each level. FIG. 21B provides an example schematic view of a local perceptual space while the robot 100 is moving. The ODOA behavior 600b (FIG. 13) can evaluate each predicted robot path 2110. These evaluations can be used by the motion selection engine 580 to determine the desired results and the corresponding robot commands. For example, for each robot position 2120 in robot path 2110, ODOA behavior 600b identifies each cell in grid 2100 that is in the bounding box around the corresponding position of robot 100, and classifies each cell. Methods for object detection and obstacle avoidance may be implemented, including receiving. For each cell classified as an obstacle or unknown, a collision check is performed by taking the grid points corresponding to the cell and determining if the grid point is within a collision circle around the position of the robot 100. If the grid points are in a collision circle, then the method further performs a three-point comparison test of whether the grid points are in a collision triangle (eg, the robot 100 can be modeled as a triangle). including. If the grid point is in a collision triangle, the method includes rejecting the grid point. If the robot position is inside the sensor system view of the parent grid point on robot path 2110, then the "unknown" grid point will be known by the time the robot 100 reaches those grid points It is ignored because it is assumed to

  The method may include a robot path region (eg, a rectangular shape) between successive robot positions 2120 within robot path 2110 to prevent robots from colliding during a transition from one robot position 2120 to the next robot position. In other embodiments, it may include determining whether any obstacle collisions exist.

  FIG. 21C illustrates the local perceptual space of the robot 100 and the sensor system view 405 (the control system 510 uses only certain sensors, such as the first and second 3-D image sensors 450a, 450b, for robot path determination. Provides a schematic diagram of Using the holonomic mobility of the drive system 200, the robot 100 uses the known ground G persistence to enable the sensor system field of view 405 to actively drive out of coverage. be able to. For example, if the robot 100 is sitting still with the first and second 3-D image sensors 450a, 450b pointing forward, the robot 100 can be driven laterally but the robot As illustrated in the example shown in FIG. 21C, which shows regions that are classified as unknown to the side of 100, the control system 510 is proposed because the robot 100 does not know what is on its side. Reject the move. When the robot 100 drives forward with the first and second 3-D image sensors 450a and 450b pointing forward, the first and second 3-D images when the robot 100 drives forward The ground G next to the robot 100 may be classified as a known empty 2106, as both sensors 450a, 450b can see the ground G as empty and the persistence of classification is still not degraded ( For example, see FIG. 21B). In such situations, the robot 100 can be driven laterally.

  Referring to FIG. 21D, in some embodiments, ODOA behavior 600b causes the robot to see where it is headed (not now), given the large number of possible trajectories due to holonomic mobility ) But may select a locus. For example, the robot 100 can expect sensor field of view orientation, which allows the control system 510 to detect an object. The robot can rotate while translating, so the robot can increase the sensor field of view 405 while driving.

  By understanding the field of view 405 of the sensor system 400 and what it will look at in different positions, the robot 100 can select a movement trajectory that will help to see where it is going. For example, when turning a corner, as shown in FIG. 21E, the robot 100 may eventually go to the robot position 2120 without current knowledge, rather than the sensor system view 405 of the parent robot position 2120. May refuse a trajectory that makes a sharp turn in the corner. Instead, the robot 100 selects a moving trajectory that turns to the desired motion direction quickly, as shown in FIG. 21F, moves sideways, and then drives system 200 to move straight around the corner. Holonomic mobility may be used.

In some embodiments, the mapping and navigation application 1610b (FIG. 16C) provides remote control functionality. For example, a user can drive the robot 100 using video waypoint actuation (e.g., using one or more of the cameras or imaging sensors 320, 450). The user changes the height H _L of the leg 130 to raise / lower the height H _T of the torso 140 (FIG. 14) to change the field of view of one of the imaging sensors 450, and / or Alternatively, the robot head 160 may be pivoted and / or tilted to change the field of view of the supported camera 320 or imaging sensors 450, 450b (see, eg, FIGS. 11A and 11B). In addition, the user can use the drive system 200 to rotate the robot around its Z-axis to obtain a camera or other view of the imaging sensor 320, 450. In some embodiments, the mapping and navigation application 1610 b may allow the user to switch between multiple layout maps 1810 (eg, for different environments or different robots 100) and / or multiple robots on one layout map 1810. Allows to manage 100. The mapping and navigation application 1610b may communicate with the cloud service API to enforce the policies for proper robot usage described by the robot owner or organization.

  Referring again to FIG. 16C, in some implementations, the video conferencing application 1610 c allows users to initiate and / or participate in video conferencing sessions with other users. In some embodiments, video conferencing application 1610 c may be connected to the user of robot 100, a remote user on a computing device connected to cloud 1620, and / or the Internet using a mobile handheld device. Allows to initiate and / or join a video conferencing session with another remote user. Video conferencing application 1610 c may provide an electronic whiteboard, an image viewer, and / or a PDF viewer for sharing information.

  The scheduling application 1610d allows the user to schedule the use of one or more robots 100. When there are fewer robots 100 than humans would like to use, the robots 100 may become insufficient resources and scheduling may be required. Scheduling resolves conflicts in resource allocation and allows higher resource utilization. The scheduling application 1610d may be robot centric and may integrate with a third party calendaring system such as Microsoft Outlook or Google Calendar. In some embodiments, scheduling application 1610 d communicates with cloud 1620 through one or more cloud services to dispatch robot 100 at a pre-scheduled time. The scheduling application 1610d may select time-related data (eg, maintenance schedules, etc.) to other robot data (eg, robot position, health, etc.) to enable selection of the robot 100 by the cloud service for the task identified by the user. Integrated with the state).

  In one scenario, scheduling on a computing device (e.g., a portable tablet computer or handheld device) in communication with the cloud 1620 to schedule a doctor to schedule a round-trip at a remote hospital later in the week Applications 1610d may be accessed. The scheduling application 1610d can schedule the robot 100 in the same manner as allocating a conference room on an electronic calendar. Cloud service manages the schedule. If, at midnight, the doctor receives a call that he needs to see a heavy patient in a remote hospital, the doctor uses the scheduling application 1610d to request the robot 100 and / or the mapping and navigation application 1610b. The robot 100 can be used to send it to a room. The physician may access the medical record and the patient's video or images on his computing device (eg, by accessing the cloud storage device 1622) using the video conferencing application 1610c. Cloud services may be integrated with robot management, electronic health record systems, and medical imaging systems. The physician may control the movement of the robot 100 to interact with the patient remotely. If the patient speaks only Portuguese, the video conferencing application 1610c may automatically translate the language, or a third party translator is communicating with the cloud 1620 (eg, via the Internet) Another computing device may be used to join the video conference. Cloud services can be used to request, perform, record, and bill translation services.

  Usage / statistics application 1610 e may be a general purpose application for the user to monitor robot usage, generate robot usage reports, and / or manage robot fleet 100. This application 1610e may also provide general operation and troubleshooting information for the robot 100. In some embodiments, usage context / statistics application 1610e registers a usage of one or more simulators 1670 that allows the user to add / disable services associated with usage of the robot 100 usage policy for the robot Make it possible to correct etc.

  In another scenario, a business may have a team of robots 100 for at least one telepresence application. The status (e.g., location, Usage and maintenance schedules, battery information, location history, etc.) may be monitored. In some embodiments, the location manager may assist users with robotic problems by sharing user sessions. The location manager navigates the corresponding robot 100, speaks through the robot 100 (i.e. telepresence), goes into a power saving mode (e.g. reduces functionality), finds the charger, etc. A command can be issued to any of the robots 100, using. The location manager or user can use the application 1610 to manage the user, security, layout map 1810, manage video view, add / remove robots from the formation, etc. The remote operator of robot 100 schedules / schedules / cancels robot reservations (eg, using scheduling application 1610 d) and simulated locations (eg, using simulator 1670 executed on a cloud server) You can take part in the training course using a simulated robot that roams around.

表１ SDK library 1644 may include one or more source code libraries for use by developers 1602 of application 1610. For example, a visual component library can provide a visual component having a graphical user interface, or an interface for accessing encapsulation functionality. Example visual components include code for laying out layout map tiles and robots, making video conferences, viewing images and documents, and / or displaying calendars or schedules. The robot communication library (for example, web service API) can provide RESTful (Representational State Transfer), JSON (JavaScript Object Notation) based API for directly communicating with the robot 100. The robot communication library can provide Objective-C binding (for example, for iOS development) and Java (registered trademark) binding (for example, for Android development). These object oriented object APIs allow applications 1610 to communicate with the robot 100 while encapsulating the underlying data transfer protocol of the robot 100 from the developer 1602. The individual following the routine of the robotic communication library may return video screen coordinates corresponding to the individual tracked by the robot 100. The face recognition routine of the robot communication library returns the coordinates of the face on the camera view of the camera 320 and, optionally, the name of the tracked individual being recognized. Table 1 provides a typical list of robotic communication services.

Table 1

  The cloud services communication library allows applications 1610 to communicate with the robot 100 in communication with the cloud 1620 (eg, with the cloud storage 1622, the application server 1650, the database 1660, and the simulator 1670) and / or the cloud 1620. May include an API. Cloud service communication library can be provided by both Objective-C and Java (registered trademark) binding. Examples of cloud services APIs include navigation APIs (eg, for retrieving locations, setting destinations, etc.), map storage and retrieval PAI, camera feed APIs, remote control APIs, usage statistics APIs, and others .

  The cloud service extensibility interface may allow the cloud service to interact with web services from external sources. For example, the cloud service may define a set of enhanced interfaces that allow enterprise developers 1602 to implement interfaces for external proprietary systems. Extensions can be uploaded to the cloud infrastructure and deployed. In some embodiments, cloud services can employ standard extensibility interfaces defined by various industrial communities.

  Simulator 1670 may enable debugging and testing of application 1610 without connectivity to robot 100. The simulator 1670 can model or simulate the motion of the robot 100 without actually communicating with the robot 100 (eg, to plan paths and access map databases). To perform the simulation, in some implementations, simulator 1670 generates a map database (eg, from layout map 1810) without using robot 100. This may involve image processing (eg, edge detection) so that features (such as walls, corners, columns, etc.) are automatically identified. Simulator 1670 can use the map database to simulate path planning within the environment determined by layout map 1810.

  Cloud service extension uploader / developer connects to an external third party user authentication system where users upload extensions to the cloud 1620, to an external database or storage device (eg, patient information for pre- and post-examination) It may be possible to access, for example, access to an illustrative image in a video conference session. The cloud service extension interface may enable integration of proprietary systems with the cloud 1620.

  Various implementations of the systems and techniques described herein may be implemented in digital electronic circuits, integrated circuits, ASICs (application-specific integrated circuits) designed specifically for, application specific integrated circuits, computer hardware, firmware, software, and / or the like. It can be realized by the combination of These various implementations may be either dedicated or generic, coupled to receive data and instructions from the storage system, and to transmit data and instructions thereto. Include one or more computer program implementations executable and / or interpretable on a programmable system, including a programmable processor, at least one input device, and at least one output device it can.

  These computer programs (also known as programs, software, software applications or codes) comprise machine instructions for a programmable processor, a high level procedural and / or object oriented programming language, and / or an assembly / It can be realized in machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" provide machine instructions and / or data to a programmable processor, including a machine readable medium that receives machine instructions as a machine readable signal. Computer program product, apparatus, and / or device (eg, magnetic disk, optical disk, memory, programmable logic device (PLD)) used to The term "machine readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

  Implementations and functional operations of the subject matter described herein may be digital electronic circuitry, or computer software, firmware, or hardware, or the like, including the structures disclosed herein and their structural equivalents. It can be realized by one or more combinations of the above. An embodiment of the subject matter described herein is one or more computer program products, ie computer programs encoded on a computer readable medium, for execution by or to control the operation of a data processing apparatus It can be implemented as one or more modules of instructions. The computer readable medium can be a machine readable storage device, a machine readable storage circuit board, a memory device, a composition that affects the machine readable propagation signal, or a combination of one or more thereof. The term "data processing apparatus" encompasses, by way of example, all devices, devices and machines for processing data, including programmable processors, computers or multiple processors or computers. The device, in addition to the hardware, creates code that creates the execution environment of the computer program in question, eg, processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. Can be included. The propagated signal is an artificially generated signal, eg, a machine-generated electrical, optical or electromagnetic signal, generated to encode information for transmission to a suitable receiving device.

  A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including languages compiled or interpreted, and as a single program Or may be deployed in any form, including deployment as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program does not necessarily have to correspond to the files in the file system. The program is tailored to a portion of the file or files that hold other programs or data (eg, one or more scripts stored in a markup language document) in a single file dedicated to the program in question It can be stored in a file (eg, a file that stores one or more modules, subprograms, or portions of code). The computer program can be deployed to run on multiple computers, located on a single computer, or located at a single location, or distributed across multiple locations, interconnected by a communications network .

  The processes and logic flows described herein are implemented by one or more programmable processors executing one or more computer programs that operate on input data and perform functions by generating outputs. can do. Also, the processes and logic flows may be implemented by dedicated logic circuitry, eg, an FPGA or ASIC, or the apparatus may be implemented as them.

  Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, the computer also includes one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical disks, or optical disks, or for receiving data therefrom, or It is operatively linked to it, for forwarding, or both. However, the computer does not have to have such a device. Furthermore, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a portable audio player, a global positioning system (GPS) receiver, to name but a few. Computer readable media suitable for storing computer program instructions and data include, by way of example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices, magnetic disks such as internal hard or removable disks, magneto-optical disks And all forms of non-volatile memory, media, and memory devices, including CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

  Implementations of the subject matter described herein may include, for example, middleware components including back-end components as data servers, front-end components including, eg, application servers, eg, through which the user is directed A computing system, or one or more such back end, middleware, or front end, including a client computer, having a graphical user interface or web browser, capable of interacting with implementations of the subject matter described in It can be realized by any combination of components. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), such as the Internet.

  The computing system can include clients and servers. Clients and servers are generally remote from each other and typically communicate through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship to each other.

  Although the specification contains many details, these should not be construed as limiting the invention or its claimed scope, but rather as interpreting specific features of particular implementations of the invention. It should be. Also, certain features that are described in the context of separate implementations herein can be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable subcombination. Further, while features are described above to work in a particular combination, and may even be initially claimed as such, in some cases, one or more features of the claimed combination are clipped from the combination The claimed combinations may be directed to subcombinations or variants of subcombinations.

  Similarly, although the operations are depicted in the drawings in a particular order, it may be performed that such operations are performed in the particular order shown, or in the order in which they occur, in order to achieve the desired result. It should not be understood that, or all illustrated operations need to be performed. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the program components and systems described will generally be It should be understood that they can be integrated together into one software product or packaged into multiple software products.

  A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results.

Embodiment 1-1
A method of operating a mobile robot (100), comprising:
Moving the robot (100) around the scene (10);
Receiving sensor data (1601) indicative of the scene (10);
Transmitting the sensor data (1601) to a cloud computing service (1620) that processes the received sensor data (1601) and transmits a processing result (1607, 1609) to the robot (100);
Moving the robot (100) within the scene (10) based on the received processing result (1607, 1609);
Equipped with
The cloud computing service (1620) receives image data (1601) from the robot (100) and processes the image data (1601) to obtain a 3-D map (1605) and / or the like of the scene (10). Or model (1609),
Furthermore,
Emitting light onto the scene (10) around the robot (100);
The image of the scene (10) along the driving direction (F) of the robot (100), of the (a) three-dimensional depth image, (b) active illumination image, and (c) ambient illumination image Capturing an image, including at least one;
Including
The sensor data (1601) comprises the image, and the processing result (1607, 1609) comprises a map (1607) or a model (1609) of the scene (10),
The step of processing the image data (1601) comprises: connecting together photos or videos captured by a camera (320) or a volume point cloud imaging device (450) located on the robot (100); Further comprising constructing a map (1605) of the scene (10) around the robot (100) by using reference coordinates provided by the measurement, a global positioning system, and / or waypoint navigation ,
Marking a reference position on the map (1605) and planning a path through the reference position marked on the map (1605);
The map (1605) includes an indoor or outdoor street view or route view of the scene (10), and each reference position marked on the map (1605) may be linked image or video or promotional information The way, including.
(Embodiment 1-2)
Emitting a speckle pattern of light onto the scene (10);
Receiving the reflection of the speckle pattern from an object (12) in the scene (10);
A reference image of the speckle pattern reflected from a reference object (12) in the scene (10), the reference image captured at various different distances (Z _n ) from the reference object (12), Storing in the cloud storage device (1622) of the cloud computing service (1620);
Capturing at least one target image of the speckle pattern reflected from target objects (12) in the scene (10), and communicating the at least one target image to the cloud computing service (1620); ,
Further include
Embodiment 1-1, wherein the cloud computing service (1620) compares the at least one target image to the reference image to determine the distance (ΔZ) of the reflective surface of the target object (12). Method described.
Embodiment 1-3
Determining a primary speckle pattern on the target object (12), and at least one of cross-correlation and decorrelation of each of the primary speckle pattern and the speckle pattern of the reference image The method of embodiment 1-2, further comprising the step of calculating.
Embodiment 1-4
The cloud computing service (1620) at least temporarily stores the received sensor data (1601) in a cloud storage (1622), optionally optionally after processing the data (1601) The method of any of embodiments 1-1 to 1-3, wherein the sensor data (1601) is discarded.
(Embodiment 1-5)
The sensor data (1601) comprises image data having associated sensor system data, the sensor system data comprising at least one of accelerometer data tracking, mileage measurement data, and a time stamp. The method in any one of -1 to 1-4.
Embodiment 1-6
The cloud computing service (1620) provides the robot (100) with a 2-D height map (1607) and / or the model (1609), and the cloud computing service (1620) The method according to embodiment 1-1, wherein the 2-D height map (1607) is calculated from the D map (1605).
Embodiment 1-7
The method may further include periodically transmitting the sensor data (1601) to the cloud computing service (1620), the cloud computing service (1620) storing the threshold sensor data set (1603), The method according to any of the embodiments 1-1 to 1-6, wherein the received image data (1601) is processed.
Embodiment 1-8
The sensor data (1601) may be wirelessly transmitted to the cloud computing service through a portable computing device (310) in communication with the robot (100) and removably attachable to the robot (100). The method according to any of embodiments 1-1-1-7, further comprising the step of communicating to 1620).
(Embodiment 1-9)
The method according to embodiment 1-1, wherein the scene (10) is a store mall and the map (1605) comprises a route view with each store marked as a reference position in the map (1605) .
Embodiment 1-10
The method according to embodiments 1-9, further comprising the step of providing a route tour through the mapped stores of the mall.

Embodiment 2-1
A method of navigating a mobile robot (100), comprising:
A streaming sequence (1615) of a high density image (1611) of a scene (10) around the robot (100) along a trajectory of movement of the robot (100) at real time capture speed by a volumetric point cloud imaging device Step of capturing;
Associating an annotation (1613) with at least some of the high density images (1611);
Sending the high density image (1611) and annotations (1613) to a remote server (1620) at a transmission rate that is slower than the real time capture rate;
Receiving a data set (1607, 1617) from the remote server (1620) after a processing time interval for a data set generation process by the remote server (1620), the data set (1607, 1617) comprising , Excluding original image data from at least a portion of the sequence (1615) and the corresponding annotation (1613) of the high density image (1611) and corresponding to the sequence (1615) of the high density image (1611) 2D map comprising height data at each point generated by the high density 3D map or model obtained by processing the sequence (1615) of the high density image (1611) The receiving step being data including (1607);
Moving the robot (100) relative to the scene (10) based on the received data set (1607, 1617);
Method, including.
(Embodiment 2-2)
Send the high density image (1611) and annotations (1613) to the local server and buffer (1640), then send the high density image (1611) and annotations (1613) at a transmission rate slower than the real-time capture rate The method of embodiment 2-1, further comprising transmitting to a remote server (1620).
Embodiment 2-3
The annotation (1613) is
A timestamp, an absolute time-based timestamp corresponding to at least some of the high density images (1611);
Embodiment 2-1 or 2-2, comprising attitude related sensor data, the attitude related sensor data including at least one of travel distance measurement data, accelerometer data, tilt data, and angular velocity data. The method described in.
(Embodiment 2-4)
The step of associating annotations (1613) includes capturing annotations (1613) reflecting hazard events that are events that pose hazards to the robot (100) within a time interval to the hazard response of the robot (100) The method of any of embodiments 2-1 to 2-3, comprising the step of associating with a high density image (1611).
Embodiment 2-5
Associating an annotation (1613) includes associating a key frame identifier for identifying key frames in the sequence of high density images (1611) with a subset of the high density image (1611). The method in any one of 1-2-4.
Embodiment 2-6
The annotation (1613) is a set of low density 3- to restore the structure and motion of features tracked between the high density images (1611) of the streaming sequence (1615) of high density images (1611). The method according to any of the embodiments 2-1 to 2-5, comprising a point D.
Embodiment 2-7
The method according to embodiments 2-6, wherein the annotation (1613) comprises camera parameters, wherein the camera parameters comprise an orientation of the camera relative to an individual 3-D point of the low density set of 3-D points. .
(Embodiment 2-8)
The embodiment (1) according to any of the embodiments 2-1 to 2-7, wherein the annotation (1613) comprises a sign indicating the area that the robot (100) can traverse and the area that can not be traversed in the scene (10). Method.
Embodiment 2-9
The data set (1607, 1617) is trained using at least the high density image (1611) as training data to classify features in the captured new high density image (1611) of the scene (10) The method according to any of the embodiments 2-1 to 2-8, comprising a labeled classifier (1625).
Embodiment 2-10
A method of abstracting mobile robot environment data, comprising:
Receiving a sequence (1615) of a high density image (1611) of the robot environment (10) from the mobile robot (100) at a receiving speed, said high density image (1611) comprising a volume point cloud Said receiving step being captured by an imaging device at a real time capture rate along a trajectory of movement of said mobile robot (100), said receive rate being slower than said real time capture rate;
Receiving an annotation (1613) associated with at least some of the high density images (1611) in the sequence (1615) of the high density images (1611);
To reduce high density data in at least some of the high density images (1611) to a data set (1607, 1617) representing at least a portion of the sequence (1615) of the high density images (1611) Dispatching batch processing tasks for
Transmitting the data set (1617) to the mobile robot (100), wherein the data set (1607, 1617) is an original corresponding to the sequence (1615) of the high density image (1611). Data excluding image data, height at each point generated by high density 3-D model (1609) obtained by processing the sequence (1615) of the high density image (1611) Transmitting, which is data comprising a two-dimensional map (1607) comprising data;
Method, including.
Embodiment 2-11
The batch processing task is
Processing the sequence (1615) of the high density image (1611) into the high density 3-D model (1609) of the robot environment (10);
Processing the high density 3-D model (1609) into a 2-D map (1607) for a coordinate system at 2-D locations and at least one height from floor surface (G);
The method according to embodiment 2-10, comprising
Embodiment 2-12
The two-dimensional map (1607) has a 2-D position and a height from the floor surface (G) of a portion blocked by the object and a floor surface (G) of a portion not blocked by the object The method according to any of embodiments 2-11, which is for a plurality of bounding coordinate systems corresponding to each of the heights of
Embodiment 2-13
The method according to any of embodiments 2-10 to 2-12, wherein the batch processing task comprises the step of accumulating high density image sequences (1615, 1615a) corresponding to a plurality of robot environments (10).
(Embodiment 2-14)
Embodiment 2-10. The batch processing task comprises training one or more classifiers (1625) on the sequence of multiple classifiers (1625) and / or high density images (1611). The method according to any one of -13.
Embodiment 2-15
The batch processing task is
An annotation (1613) reflecting a dangerous event, which is an event that poses a risk to the mobile robot (100), is captured in a time interval relative to the dangerous response of the mobile robot (100) (1611) Associating with),
The classifier (1625) of the hazard related high density image (1611) is trained and the classifier (1625) using the associated hazard event annotation (1613) and the corresponding high density image (1611) as training data. Providing a data set (1603, 1607, 1617) of model parameters of
The method according to any of embodiments 2-10 to 2-14, comprising
Embodiment 2-16
The classifier (1625) includes at least one support vector machine that builds at least one hyperplane for classification, and the model parameters classify the data set (1603, 1607, 1617) into a risk related classification. The method of embodiments 2-15, which defines a trained hyperplane that can.
Embodiment 2-17
17. The method of embodiments 2-16, wherein the model parameters include sufficient parameters to define a kernel of a support vector machine and soft margin parameters.
Embodiment 2-18
The batch processing task comprises the steps of instantiating a plurality of scalable virtual processes (1621) proportional to the scale of the high density image sequence (1615, 1615a) to be processed, the virtual processes (1621) The method according to any of embodiments 2-10 to 2-17, wherein at least some of are released after transmission of said data set (1607, 1617).
Embodiment 2-19
The batch processing task comprises the steps of instantiating a plurality of scalable virtual memories (1622) proportional to the scale of the high density image sequence (1615, 1615a) to be stored, the virtual memories (1622) The method according to any of embodiments 2-10 to 2-18, wherein at least some of are released after transmission of said data set (1607, 1617).
(Embodiment 2-20)
The batch processing task may comprise a plurality of scalable virtual servers according to one of geographical proximity to the mobile robot (100) and / or network traffic from a plurality of mobile robots (100) The method according to any of embodiments 2-10 to 2-19, including the step of dispersing 1621).

Embodiment 3-1
A method of operating a mobile robot (100), comprising:
Receiving a layout map (1810) corresponding to the environment (10) of the robot (100);
Moving the robot (100) in the environment (10) to a layout map position (1812) on the layout map (1810);
Recording a robot map position (1822) corresponding to the layout map position (1812) corresponding to the environment (10) and on a robot map (1820) generated by the robot (100);
Determining distortion between the robot map (1820) and the layout map (1810) using the recorded robot map position (1822) and the corresponding layout map position (1812);
The determined distortion is applied to the target layout map position (1814) selected on the layout map (1810) for the purpose of navigating the robot, and the target on the robot map (1820) Determining a target robot map position (1824) corresponding to the layout map position (1814);
Method, including.
Embodiment 3-2
The method according to embodiment 3-1, further comprising receiving the layout map (1810) from a cloud computing service (1620).
Embodiment 3-3
A remote cloud storage device (1622) is generated using the cloud computing service (1620) and generating the layout map (1810) on an application (1610) running on the remote computing device (310, 1604). And 3.) storing the layout map (1810) on top).
Embodiment 3-4
Determine scaling size, starting point mapping, and rotation between the layout map (1810) and the robot map (1820) using the existing layout map position (1812) and the recorded robot map position (1822) Step and
Resolving a target robot map position (1824) corresponding to the target layout map position (1814);
The method according to any of embodiments 3-1 to 3-3, further comprising
Embodiment 3-5
The method of embodiments 3-4, further comprising: applying an affine transformation to the determined scaling size, start point mapping, and rotation to resolve the target robot map location (1824).
Embodiment 3-6
In any of the embodiments 3-1 to 3-5, further including a step of determining triangulation (1910) between layout map positions (1812, 1912) indicating boundaries of the target layout map positions (1814, 1914). Method described.
Embodiment 3-7
Determine the scale, rotation, translation, and skew between the mapped triangle (1910) in the layout map (1810) and the corresponding triangle (1920) mapped in the robot map (1820) And applying the determined scale, rotation, translation and skew to the target layout map position (1814, 1914) to determine the corresponding target robot map position (1824, 1924). Methods according to embodiments 3-6.
Embodiment 3-8
Determining the distance between all layout map locations (1812, 1912) and the target layout map locations (1814, 1914);
Determining a center of gravity (2012) of the layout map position (1812, 1912);
Determining the center of gravity (2022) of all recorded robot map positions (1822, 1922);
Rotation and length scaling are determined for each layout map position (1812, 1912), and a first vector (2014) ranging from the layout map centroid (2012) to the target layout position (1814, 1914) Converting to a second vector (2024) ranging from the center of gravity (2022) to the target robot map position (1824, 1924);
The method according to any of embodiments 3-1 to 3-7, further comprising
Embodiment 3-9
The method according to any of the embodiments 3-1 to 3-8, further comprising generating the robot map (1810) using a sensor system (400) of the robot (100).
Embodiment 3-10
Emitting light onto the environmental scene (10);
Receiving the reflection of the radiation from the surface of the scene (10);
Determining the distance of each reflective surface;
Constructing a three-dimensional depth map (1700) of the scene (10);
The method according to embodiments 3-9, further comprising
Embodiment 3-11
Emitting a speckle pattern of light onto the scene (10) and receiving a reflection of the speckle pattern from the scene (10), preferably reflected from a reference object (12) in the scene (10) The method according to embodiments 3-10, further comprising storing a reference image of the speckle pattern, wherein the reference image is captured at various different distances (Z _n ) from the reference object (12). .
Embodiment 3-12
Capturing at least one target image of the speckle pattern reflected from a target object (12) in the scene (10), and determining the distance of the reflective surface of the target object (12) The method according to embodiment 3-11, further comprising the step of comparing one target image to the reference image.
Embodiment 3-13
Determining a first order speckle pattern on the target object (12) and calculating at least one of cross-correlation and non-correlation of each of the first order speckle pattern and the speckle pattern of the reference image The method according to embodiment 3-12, further comprising the step of
Embodiment 3-14
Embodiment 13 as described in embodiment 3-12 or 13, further comprising moving the robot (100) relative to the target object (12) based on the determined distance of the reflecting surface of the target object (12). the method of.
Embodiment 3-15
The method according to embodiments 3-10, further comprising: determining a time of flight between the light emission and the reception of the reflected light; and determining a distance to the reflective surface of the scene (10).
Embodiment 3-16
Embodiment 17. The method according to any of the embodiments 3-10-3-15, further comprising emitting the light onto the scene (10) with intermittent pulses, preferably changing the frequency of the emitted light pulses. .

Claims (12)

A method of operating a mobile robot (100), comprising:
Moving the robot (100) in an environment (10);
Capturing an image of the environment (10) along a driving direction (F) of the robot (100);
Receiving the sensor data (1601) indicating the environment (10) and comprising the image;
Transferring the sensor data (1601) from the robot (100) to a cloud computing service (1620);
2-D height calculated from the 3-D map (1605) of the environment (10) based on the sensor data (1601) and including height data of each of the positions in the 3-D map (1605) Receiving, at the robot (100), the cloud map (1607) from the cloud computing service (1620);
Moving the robot (100) within the environment (10) based on the 2-D height map (1607);
Including
The 3-D map (1605) of the environment (10) is a concatenation of photos or videos captured with a camera (320) or volume point cloud imaging device (450) located on the robot (100); Based on the use of reference coordinates provided by distance measurement, global positioning system, and / or waypoint navigation,
The method wherein the 3-D map (1605) comprises an indoor or outdoor street view or route view of the environment (10).   The method of claim 1, further comprising rotating the camera or the imaging device within a range of less than 360 degrees.   The environment (10) received by the camera or the imaging device for assembly or combination of the sensor data (1601) to construct a 360 degree field of view or calculation of the 2-D height map (1607) 3. The method of claim 2, further comprising the step of providing said sensor data (1601) indicative of, wherein said 2-D height map (1607) comprises height data for one or more Z points for each X, Y coordinate. The method described in.   The method according to claim 2 or 3, wherein the robot (100) is moved based on a field of view from the camera or the imaging device to navigate and / or change the field of view of the camera or the imaging device.   The method according to any of the preceding claims, wherein the height data of the 2-D height map (1607) is indicative of an obstacle or danger to the robot (100). The number of one or more Z points for each X, Y coordinate in the 2-D height map is smaller than the number of Z points for each X, Y coordinate in the 3-D map,
Locally storing the 2-D height map in an internal memory of the robot;
Moving the robot based on the 2-D height map stored locally in the internal memory;
The method according to any one of claims 3 to 5, further comprising The 2-D height map includes a terrain map showing occupied and unoccupied height boundaries,
The method according to any of claims 3 to 6, wherein moving the robot further comprises determining whether the robot can pass below based on the occupied and unoccupied height boundaries.   The step of moving the robot specifies, from a computing device associated with a remote user, navigating the robot based on a view from the camera or the imaging device and / or changing a view of the camera or the imaging device 5. The method of claim 4, further comprising the step of: receiving a user's command, responsive to the user's command.   The method according to claim 8, wherein moving the robot includes raising or lowering a field of view of the camera or the imaging device to change height in response to a command of the user. Receiving a user request from the computing device to specify one or more navigation points on the layout map;
Autonomously navigating the robot based on the specified one or more navigation points in response to the user's request;
The method of claim 1, further comprising The step of autonomously navigating the robot based on the navigation point comprises:
Generating a robot map that includes one or more detected obstacles in addition to the information contained in the layout map;
Map one or more points on the layout map to corresponding one or more points on the robot map based on local distortion calculations and / or one or more points tagged on the layout map Step and
Navigating the robot autonomously using the robot map;
11. The method of claim 10, comprising: Transmitting to the computing device associated with the remote user a signal comprising an image of said environment captured in real time;
Receiving a signal comprising voice of the user from the computing device associated with the remote user;
Outputting the audio from one or more speakers of the robot to provide a telepresence function;
The method of claim 1, further comprising

Images