JP2016528483A - System and method for detecting an object - Google Patents

Abstract

A haptic detection and integrated vision system that solves the problems of existing systems is disclosed. The tactile sensing skin can be formed in any shape, size, or form factor, including large areas. A computer implemented algorithm can detect position-orientation and force-torque at landmark points of a given set of objects. The result is a modular sensing system that is highly scalable in terms of price, quantity, size, and application. Such skin technology and associated software can include a sensing package that integrates haptic and visual data into accompanying software for machine state estimation, situational recognition, and automatic control. The addition of haptic data can serve to limit and / or extend the visual pose estimation method, and can provide pose estimation for visually occluded objects.

Description

Cross-reference This application is a benefit of US Provisional Patent Application No. 61 / 833,457 filed on June 11, 2013 and US Provisional Patent Application No. 61 / 950,761 filed March 10, 2014. Each of which is incorporated herein by reference in its entirety.

BACKGROUND The field of robotics deals with computer systems for robot design, structure, operation, and application as well as robot control, sensory feedback, and information processing. These techniques deal with automated machines that can replace humans in hazardous environments or manufacturing processes, or that can resemble humans in appearance, behavior, and / or cognition.

  Some effort has been made to create machines that support or extend human functions. The field of robotics has developed many techniques and methods for detecting and manipulating external objects. This has proved useful in many areas, including augmenting or replacing humans who perform dangerous, difficult, precise, or repetitive tasks. Currently widely used with robotics, computer vision, high energy density battery system, small robot high performance calculation, advanced wireless communication link, pressure, magnetism, orientation and acceleration microsensors, and input display and multiple sensors There are technologies from possible communication devices.

  Haptic, touch, and pressure sensing are human sensory features that are difficult to imitate accurately and efficiently with machines. Tactile sensations can be incorporated into feedback loops to robotic manipulators and actuators and integrated with other sensors to provide situational awareness and the ability to monitor, identify, grasp and manipulate physical objects. For example, data from a visual sensor, an acceleration sensor, and a tactile sensor can be fused in real time to guide the robot arm in grasping and moving delicate parts. However, currently available haptic techniques do not provide the level of sensing performance required to enable at least the applications described above.

  Several approaches to these detection and data fusion problems have been attempted. The potential usefulness of current approaches can be examined in terms of basic characteristics and implications for sensitivity, dynamic range, and robustness. Currently, no commercial vendor claims the robustness of sensor packages across different environments. In some cases, the operating temperature range is usually the only robust feature of these products.

Overview Various deficiencies and limitations associated with current robotics and sensing systems are now recognized. For example, current detection systems may not provide sufficient detection resolution to detect and manipulate objects at various settings, such as consumer and industrial applications. As another example, current sensing systems may not be easily integrated into the system for use in various settings such as industrial applications. Clearly, advances in tactile sensing are highly sought to improve the robot's ability to identify and manipulate objects and interact better with humans and unorganized environments.

  The present disclosure provides sensing materials, devices, systems, and methods. Some embodiments provide a conformal elastic material. The devices and systems of the present disclosure can be used for automatic machine detection and manipulation of physical objects.

  Disclosed herein is an inexpensive tactile sensing and integrated vision system that can overcome the problems of existing systems. This touch sensor and vision system can result in lower purchasing costs compared to existing vision-based systems and is easily calibrated with a shorter set-up time for new production processes.

  In an aspect, the present disclosure provides a conductive skin formed of a polymeric material (eg, rubber) that can be doped with a chemically inert material such as a carbon-containing material. In some examples, the carbon-containing material is carbon (eg, carbon powder) and / or carbon nanostructures. The polymeric material can include an elastomer. Inert materials cannot interfere with the elastomer curing process. When cured, the polymeric material can produce a skin that can be flexible. The skin can be manufactured in any form factor. The skin can be wrapped around the housing and in some cases can be fastened with a non-conductive material (eg, a plastic zipper). Skins, for example, at least 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17 , 18, 19, 20, 30, 40, 50, or 100 electrode arrays, such as an array of electrodes. The electrode can function as a boundary-based tomography unit.

  The tactile sensing skin can be formed in any shape, size, or form factor, including large areas. The algorithm detects position-orientation and force-torque at the landmark points of a given set of objects. The result is a modular sensing system that is highly scalable in terms of price, quantity, size, and application. The skin technology and associated software of the present disclosure can include a sensing package that integrates haptic and visual data into accompanying software for machine state estimation, situational recognition, and automatic control.

  In some embodiments, the sensing system includes an advanced multimodal “fingerpad” sensor suite that can sense the triaxial force, torque, position, and orientation of a pressed object, and any form factor over a large area. And a robust and inexpensive robot “skin” that can recognize normal pressure distribution that is easy to shape.

  In some embodiments, the sensing system includes a haptic component that estimates posture and force. The system can include a conductive material that detects contact events and static changes in resistance to compressive and tensile stresses. The conductive material can be an elastic skin with electrodes around it, resulting in an easily replaceable part with adjustable mechanical properties in the absence of wires or electronic devices in the workpiece. The elastic skin can be formed of a polymeric material such as polysiloxane (silicone rubber), polyurethane, or other elastic compound. The skin can further include foam and carbon black.

  In some examples, the sensing device can include an embedded integrated rubber electrode. The sensing device may include a skin that includes a dopant and a foaming agent and produces a material that can simultaneously change mechanical, thermal, and electrical properties.

  The sensing system can further include an optical device that can collect visual information. The optical device can be a movement detection system, which can detect the spatial orientation or placement of an object in two or three dimensions.

  The sensing system can further include a computer processor programmed to generate an initial estimate of posture based on visual data, or otherwise configured, and when tactile data is introduced, The estimation can be further improved. Each measurement (both visual and tactile) can include a set of position and surface normal vectors (6D), or data points. The measurement can be a point cloud, each point having a corresponding surface normal vector direction. For each data point, the computer processor can calculate the nearest point on the known model and subsequently calculate the attitude difference of the data points.

  Aspects of the present disclosure provide a system for manipulating an object and / or detecting the presence of an object, the system being at least one polymeric substrate that includes a plurality of non-metallic sensing electrodes, the non-metallic sensing The electrode is flexible and the non-metallic sensing electrode includes at least one polymeric substrate configured to sense a change in electrical impedance when placed in or near the object. The system is electrically coupled to a plurality of sensing electrodes, (i) measures a signal indicative of a change in impedance of at least a subset of the non-metallic sensing electrodes, and (ii) executes an electrical impedance tomography algorithm, And (iii) a computer programmed to identify one or more features of the object based on the one or more forces A processor is further included. In an embodiment, the one or more features are selected from the group consisting of the presence of an object, the shape of the object, and the proximity of the object to the polymeric substrate. In another embodiment, the system further includes an operating member disposed adjacent to the polymeric substrate, wherein the operating member is configured to operate the object. In another embodiment, the polymeric substrate is wound on an operating member. In another embodiment, the operating member is a robot gripper. In another embodiment, the manipulation member is configured to apply or apply a magnetic field to grasp or grasp an object. In another embodiment, the manipulating member is configured to identify one or more characteristics of the object by passing an electric current through the object.

  In embodiments, the polymeric substrate includes a polymeric material and a fabric. In another embodiment, the polymeric substrate includes a first component volume and a second component volume, the first component volume includes a plurality of non-metallic sensing electrodes, and the second component volume. Includes a plurality of conductive paths each in electrical contact with the non-metallic sensing electrode of the plurality of non-metallic sensing electrodes. In another embodiment, the conductive path through the second component volume is a metal wire. In another embodiment, the conductive path through the second component volume is a plurality of tunnels, each tunnel being filled with a polymeric material.

  In the embodiment, each of the plurality of non-metallic detection electrodes includes a pair of conductive paths. In another embodiment, the computer processor is programmed to apply an excitation voltage to the pair of conductive paths. In another embodiment, the computer processor is programmed to measure the voltage of the conductive path following application of the excitation voltage.

  In embodiments, the polymeric substrate has a hemispherical shape, a cylindrical shape, or a box-like shape. In another embodiment, the non-metallic sensing electrode is formed of a polymeric material. In another embodiment, the polymeric material has a higher conductivity than the polymeric substrate. In another embodiment, the non-metallic sensing electrode includes a carbon-containing material. In another embodiment, the carbon-containing material is selected from the group consisting of carbon powder or carbon nanostructures. In another embodiment, the non-metallic sensing electrode includes a foaming agent.

  Another aspect of the present disclosure provides a papermaking system that includes any of the above systems or systems elsewhere herein.

  Another aspect of the present disclosure provides a method of manipulating an object and / or detecting the presence of an object, the method comprising providing a detection system comprising at least one polymeric substrate, comprising: The molecular substrate includes a plurality of flexible non-metallic sensing electrodes, wherein the non-metallic sensing electrodes are configured to sense a change in electrical impedance when disposed at or near the object. including. Next, a signal indicative of a change in impedance of at least a subset of the non-metallic sensing electrodes is measured. The signal is measured when the object is placed at or near the detection system. Using a computer processor electrically coupled to the sensing system, an electrical impedance tomography algorithm is executed to identify one or more forces applied to the polymeric substrate from the measured signal. One or more characteristics of the object are identified based on the identified force or forces.

  In an embodiment, the one or more features are selected from the group consisting of the presence of an object, the shape of the object, and the proximity of the object to the polymeric substrate. In another embodiment, the method further comprises applying an excitation voltage to the subset of non-metallic sensing electrodes. In another embodiment, the method further includes measuring a subset of voltages following application of the excitation voltage. In another embodiment, the method further includes manipulating the object using the manipulation member, wherein the manipulation member is part of or is electrically coupled to the detection system. Once the object, one or more features of the object are identified, the object can be manipulated.

  Another aspect of the present disclosure provides a method for detecting and / or manipulating an object, the method providing an operating system including an operating member and a movement input sensing device, the operating member comprising: When the object is located at or near the sensing electrode, the mobile input sensing device includes providing at least one sensor having a sensing electrode that measures a change in impedance, and identifying the spatial configuration of the object. Next, the object moves to or near the operating system. Using the mobile input sensing device, a first data set is collected from the object, the first data set indicating a spatial configuration of the object. Next, using the sensing electrodes of the sensor, a second data set is collected under boundary conditions identified from the first data set, the second data set indicating an impedance change. Next, when it is determined that the object is at or near the sensor based on the impedance change, the object is operated using the operation member.

  In an embodiment, the boundary situation is identified from one or more boundaries of the object from the first data set. In another embodiment, the first data set has a lower spatial resolution than the first data set. In another embodiment, the method further comprises identifying one or more features of the object from the first data set. In another embodiment, the method further comprises refining one or more features of the object based on the second data set. In another embodiment, the method further comprises the step of combining the first data set and the second data set. In another embodiment, the method further includes fitting the combined data to one or more predetermined objects having known characteristics. In another embodiment, the method further includes identifying a posture estimate following the fit.

  Another aspect of the present disclosure provides a computer-readable medium comprising machine-executable code that, when executed by one or more computer processors, implements any of the above methods or methods elsewhere herein. .

  Another aspect of the present disclosure provides a system that includes one or more computer processors and a memory coupled to the computer processors. The memory includes machine-executable code that, when executed by one or more computer processors, implements any of the above methods or methods elsewhere herein.

  Additional aspects and advantages of the present disclosure will be readily apparent to those skilled in the art from the following detailed description, wherein merely exemplary embodiments of the present disclosure are shown and described. As will be realized, the disclosure is capable of other and different embodiments, and its several details are capable of modification in various obvious respects, all without departing from the disclosure. Accordingly, the drawings and descriptions are to be regarded as illustrative rather than as restrictive.

INCORPORATION BY REFERENCE All publications, patents, and patent applications mentioned herein are as if each individual publication, patent, or patent application was specifically indicated to be specifically incorporated by reference. Incorporated herein by reference to the same extent.

BRIEF DESCRIPTION OF THE DRAWINGS The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (referred to herein as "figure and FIG.) Is also obtained by referring to).

1 is a schematic view of a physical arrangement of a detection system. 1 is a schematic view of a physical arrangement of a detection system. 1 is a schematic view of a physical arrangement of a detection system. 1 is a schematic view of a physical arrangement of a detection system. It is the schematic of a robot gripper. FIG. 3 is a schematic diagram of an arrangement of components that can be used to implement a sensor of the present disclosure. 2 illustrates a technique for providing an electrical connection of a sensor of the present disclosure. 2 illustrates a technique for providing an electrical connection of a sensor of the present disclosure. 2 illustrates a technique for providing an electrical connection of a sensor of the present disclosure. 2 illustrates a method of assembling a sensor of the present disclosure. Fig. 3 shows the components of the sensing assembly. 1 is a schematic diagram of an electronic device that can be associated with a sensing assembly of the present disclosure. FIG. FIG. 3 is an illustration of an electronic device that can be used with the sensing assembly of the present disclosure. Fig. 3 schematically illustrates an example of a series of operations that can be used to collect data from electrodes of a sensing assembly. It is a flowchart which shows a series of operation | movement which collects data from the electrode of a detection assembly. 1 schematically shows a wireless connection to a rotating machine. FIG. 6 is a schematic diagram of components configured to perform wireless detection and communication. Fig. 2 schematically shows a sensing elastomer and electrical connection points. Fig. 2 shows an array of sensing elements or "texels" superimposed on a sensing elastomer. 1 schematically illustrates an example data processing workflow. 1 schematically illustrates a sensing system including various integrated sensors. 1 schematically illustrates an example of a data process workflow. It is the schematic of the structure of the component of a detection assembly. It is the schematic of the structure of the component of a detection assembly. 1 schematically shows a vision system and a robot gripper. 1 schematically shows a layout of a sensing pad having two sensors. 1 schematically shows a layout of a sensing pad having 16 sensors. 1 schematically shows an example of a sensor that can be used in various applications such as object detection. 1 schematically illustrates a computer system that can be programmed or otherwise configured to implement the various devices, systems, and methods provided herein. 1 schematically illustrates a gripper that uses magnetic force to grip an object. 1 schematically illustrates a gripper that uses magnetic force to grip an object. 1 schematically shows a robot gripping mechanism.

DETAILED DESCRIPTION While various embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many variations, modifications, and substitutions may occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be utilized.

  The term “object” as used herein generally refers to any three-dimensional tangible object. Examples of objects include, but are not limited to, parts, wood products (eg, paper), electronic devices, components of electronic devices, and food products.

  The term “impedance” as used herein generally refers to electrical impedance, which is a measure of the resistance a circuit exhibits to current when a voltage is applied. The current can be an alternating current (AC).

  The following detailed description is presented in part with respect to algorithms and symbolic representations of operations on data bits within a computer memory that represent alphanumeric characters or other information. These descriptions and representations can be used to efficiently convey the gist of the work of those skilled in the data processing arts to other persons skilled in the art.

  An algorithm can be a self-consistent series of operations that yields a desired or predetermined result that can be implemented when executed by one or more computer processors. These operations are those requiring physical manipulation of physical quantities. In some cases, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, or otherwise manipulated. Occasionally, it has proven convenient to refer to these signals as bits, values, symbols, characters, display data, terms, numbers, etc. mainly due to common practice. Such terms may be associated with appropriate physical quantities, and such terms are merely used herein as convenient labels applied to these quantities.

  Some algorithms may use data structures to collect and store information and generate desired results. Data structures greatly facilitate data management by the data processing system and can only be accessed through advanced software systems. A data structure is not the information content of a memory device, but rather represents a specific electronic structural element that gives a physical organization to the information stored in memory. Beyond just abstraction, a data structure is a specific electrical or magnetic structural element in memory that accurately represents complex data at the same time and provides increased efficiency in computer operation.

  The operations of the present disclosure can be machine operations, which can be performed using a machine including a computer control system or with the assistance of a machine. Useful machines for performing the operations of this disclosure include, but are not limited to, general purpose digital computers or other similar devices. In all cases, a distinction should be recognized between the method operation in operating a computer and the calculation method itself. The present disclosure provides devices, systems, and methods related to the operation of a computer that processes electrical signals or other (eg, mechanical, chemical) physical signals to generate other physical signals. The present disclosure also provides devices, systems, and methods associated with apparatuses that perform these operations. The apparatus may be specially constructed for a given purpose in a required or other manner, or may include a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. The apparatus can also include a “cluster,” where a plurality of computers having interconnected data networks are configured to work together for the required purpose. The algorithms presented herein are not inherently related to any particular computer or other apparatus. In particular, it is more convenient for various general purpose machines to be used in conjunction with programs written in accordance with the teachings herein, or to build a more specialized device that performs the required method operations. May be proved. The required structure for a variety of these machines will appear from the description below.

  In the following description, some frequently used terms have special meaning in this context. The terms “window environment”, “run in window”, and “object oriented operating system” refer to computer user interfaces in which information is manipulated and displayed on a video display, such as in a delimited area on a raster scanned video display. Used to indicate The terms “network”, “local area network”, “LAN”, “wide area network”, or “WAN” mean two or more computers connected so that messages can be transmitted between the computers. In such a computer network, one or more computers are usually “servers” that are computers having a large-capacity storage device such as a hard disk drive and communication hardware that operates peripheral devices such as a printer or a modem. Operate. Other computers, referred to as “workstations” or “clients”, provide a user interface that allows users of a computer network to access network resources such as shared data files, common peripherals, and inter-workstation communications. A user activates a computer program or network resource to create a “process” that includes both general-purpose operation of the computer program and specific operating characteristics determined by the input variables and its environment. Similar to a process is an agent (sometimes called an intelligent agent), which is a process that collects information on some regular schedule or performs some other service without user intervention . Typically, the agent typically uses parameters provided by the user to locate a location on the host machine or some other point on the network, collect information related to the agent's purpose, and periodically To the user.

  The term “window” and related terms such as “window environment” or “run in window” as defined above are computer users exemplified by several window systems available from Microsoft Corporation located in Redmond, Washington. Refers to the interface. Other window computer interfaces are available, for example, from Apple Computers Incorporated, located in Cupertino, California, and as a component of the Linux operating environment. In particular, it should be understood that the use of these terms in the description herein does not imply limitations to any particular computing environment or operating system.

  The term “desktop”, as used herein, generally refers to a user interface (UI) that presents a user associated with the desktop with a display of a menu or object with associated settings. The UI can be a graphical user interface (GUI) or a web-based user interface. If the desktop accesses network resources, the application program may need to run on a remote server, and the desktop calls the application program interface (“API”) to provide the user with commands to the network resource and any output Can be observed.

  The term “browser” as used herein generally refers to a program that is responsible for fetching and rendering information, although not necessarily obvious to the user. Browsers are designed to retrieve information, such as text information, graphical information, and format information, using communication protocols. This information is accessed using a computer network, often the “worldwide web” or simply “web”. Examples of browsers that support the present invention include the Internet Explorer program sold by Microsoft Corporation (Internet Explorer is a trademark of Microsoft Corporation), Opera Browser program made by Opera Software ASA, or Firefox browser program distributed by Mozilla Foundation (Firefox is Is a registered trademark of the Mozilla Foundation). Although the following description details such operation with respect to a browser graphical user interface, the present invention may be a text-based interface or audio or visually activated with many of the features of a graphics-based browser. It is also possible to implement using an interface.

  The browser displays information formatted in a general markup language (“SGML”) or hypertext markup language (“HTML”), both of which are non-visual in text documents through the use of text codes. A scripting language that embeds code. Files in these formats can be easily transmitted across computer networks, including global information networks such as the Internet, allowing browsers to display text, images, and play audio and video recordings. The web utilizes these data file formats along with communication protocols to transmit such information between the server and the workstation. The browser can also be programmed to display the provided information in an extensible markup language (“XML”) file, which can be used with several document type definitions (“DTD”). Therefore, it is more versatile than SGML or HTML. An XML file can be similar to an object because it contains separate data and stylesheet formats (the format can be thought of as a method for displaying information, and thus the XML file has data and associated methods).

  The term “personal information terminal” (“PDA”), as used herein, generally refers to any handheld mobile device that combines computing functions, telephone functions, fax functions, e-mail functions, and network functions. Point to.

  The term “wireless wide area network” (“WWAN”), as used herein, generally refers to a wireless network that functions as a medium for transmitting data between a handheld device and a computer.

  The term “synchronization” as used herein generally refers to the exchange of information between a handheld device and a desktop computer via wired or wireless. Synchronization ensures that the data on both the handheld device and the desktop computer is the same.

  In a wireless wide area network, communication can occur primarily through the transmission of radio signals over an analog network, a digital cellular network, or a personal communication service (“PCS”) network. The signal can also be transmitted through the microwave using various techniques that modulate the properties of the electromagnetic wave. The electromagnetic waves used for communication may include “light” waves at visible or near visible frequencies that are transmitted through free space or that use “optical fibers” as waveguides. At present, most wireless data communications include code division multiple access (“CDMA”), time division multiple access (“TDMA”), global system for mobile communications (“GSM”), personal digital cellular (“PDC”), etc. Over the cellular system using advanced technology or through packet data technology via analog systems such as cellular digital packet data ("CDPD") used in advanced mobile telephone services ("AMPS").

  The terms “real-time” (also “realtime” and “real time”) or “near-real-time” as used herein generally refer to timing for primary design purposes. Refers to a system design method that uses. In particular, a real-time system completes one or more operations within a time interval that meets predetermined criteria. The term may also be used to refer to the action being performed, eg, “real time update”. The time interval criteria may be a specific amount of time or may be defined relative to another non-real time system, sometimes referred to as a “batch” or “offline” system.

  The time interval criteria for real-time systems can be determined by different requirements between systems. For example, a high performance aircraft real-time control system may be required to respond within a few microseconds, while for a real-time reservoir, a level adjuster update interval of several hours may be acceptable. In human user interaction, systems that provide “real-time responses” typically receive responses to input fast enough to allow the user to use the system interactively or “live” without an unpleasant delay. Refers to that.

  In real-time transaction processing, the system can be designed to quickly complete operations that affect system data. The resulting modified data can be provided to other system components as quickly as possible, in some cases without the need for an offline synchronization process. The exact timing of such a system can depend on several factors such as processing time and the propagation of data across the network, but an important feature is the fast provision of data that has changed as a result of a transaction or event It is.

  As used herein, the term “elastomer” refers to a material that changes properties in response to an applied force. Elastomers respond to normal drag, compression, torque, or shear stress or force in a variety of formulations. Some elastomers are also referred to as “rubbers”, “polymers”, or “silicones”. Usually, but not always, elastomers physically deform in response to application of force. Further, elastomers can be designed to change various properties, such as impedance, in response to application of force, stress, or torque. Elastomers can be configured to change properties when stressed in one or more dimensions.

  Elastomers have various properties that may be desirable for a given application, such as desired flexibility, stiffness (ie, dimensional change in response to spring constant or pressure), and conformability (ie, curved or complex contours). Ability to follow), thickness, color, or conductivity or thermal conductivity. Another property of elastomers is the “durometer”, which is hardness, ie resistance to permanent deformation.

  FIG. 1A is a schematic diagram of the physical arrangement of a sensor (or sensing assembly) according to some embodiments. High sensitivity elastomer 102 contacts object 104. Elastomer 102 can change properties (eg, resistance or impedance) in response to the presence of object 104 and observe these changed properties to provide data about object 104. For example, the position of the object 104 relative to the edge of the elastomer 102 is specified. The force applied to the elastomer by the object 104 can be specified, and the force can be normal, shear, torque, or a combination thereof. The properties of elastomer 102 that change in response to contact with object 104 can be used to detect movement of object 104, including position, velocity, acceleration, and other derivations. Although a single object 104 is shown, multiple objects contacting the elastomer 102 can be detected simultaneously, separately, or sequentially. In some cases, a continuous map of pressure, force, or impedance distribution on the surface of elastomer 102 can be identified.

  FIG. 1B is another schematic diagram of a physical arrangement of sensors according to some embodiments. The system 110 incorporates a sensing elastomeric skin 112 mounted on an optional curved surface 114, showing an example where the elastomeric skin 112 is flexible and shaped to fit the curved surface. Although a simple two-dimensional curvature is shown, in various examples, the elastomeric skin 112 can be formed into a complex three-dimensional shape to achieve three-dimensional force and torque sensing. For example, the skin 112 can be formed on a sensing fingertip or glove or other surface. The skin 112 can be formed or applied to a housing such as a housing that is part of a system for manipulating objects.

  In the example, the skin 112 is applied to the fingertip of the robot operation system. Application to the fingertip may allow for higher resolution and higher sensitivity robot skins. Such a device can resolve forces and torques by placing a skin and electrode system, and the placement can allow for 6 degrees of freedom force and torque using the electrodes in the skin. . The electrodes can be arranged such that sensing over a three-dimensional (3D) space is possible. Machine learning techniques can then be used to convert skin deformations into forces and torques in three dimensions, respectively.

  The electrodes can be distributed within the skin, such as the periphery of the skin or various other configurations. The electrode can be located at a major location under the conductive skin by routing wires through the housing to provide electrical contact. A signal can be received from the electrode, and the signal can correspond to an impedance measurement made by the electrode. This can increase the spatial resolution of the device by providing additional boundary situation definitions for electrical impedance tomography (EIT).

  FIG. 1C illustrates another physical arrangement of sensors according to some embodiments. In system 120, two counter rotating rollers 122 and 128 function to supply material 126 through the gap between rollers 122 and 128. This arrangement is found in various industrial processes such as paper and cardboard manufacturing production. These systems often require monitoring for a variety of purposes, including wear, failure, and maintenance of tolerances. In the system 120, the roller 122 is covered with an elastomeric skin 124. Elastomeric skin 124 may be configured to sense the distribution of forces that cause roller 122 to contact material 126 and roller 128. The resulting force distribution is useful for machine and process monitoring. In some cases, data is processed in real time, displayed to an operator, or presented to an automated monitoring system. Data can be collected and stored for later use in maintenance and process control or improvement. In some situations, a failure of system 120 is detected by observing skin 124 and provides a signal to stop movement or shut down the machine.

  FIG. 1D shows another schematic diagram of the physical arrangement of the sensors, according to some embodiments. System 130 includes a mounting substrate 132 and an elastic sensing finger 134, thus forming a probe. Rather than moving the object into contact with the fixed position sensing elastomer, the probe generated by attaching the sensing finger 134 to the substrate 132 can be moved. The base material 132 can be formed of a polymer material. In some cases, the substrate includes other materials such as polymeric materials and fabrics. The woven fabric can be formed of a material that can be bonded to a polymeric material. In some examples, the fabric is one or more of cotton, silk, and polyester.

  The arrangement of FIGS. 1A-1D is illustrative and not limiting. It will be appreciated that there are other possible physical arrangements that incorporate the sensing elastomer.

  The sensor of the present disclosure can be attached to an operation member (or device) such as a robot gripper. FIG. 2 schematically shows a robot gripper system 200. The gripper 200 includes rigid gripper fingers 204 and 206 that are joined at the rotary joint 202 so that the fingers 204 and 206 can open and close to grip the object 208. Robots that can be configured similarly to the gripper 200 are available from FANUC America, located in Rochester Hills, Michigan, and Kawasaki Robotics (USA), located in Wixom, Michigan.

  A sensing elastomeric pad 210 can be applied to the gripper fingers 204 of the gripper 200 to observe the applied force over time and, in particular, to provide near real-time observation of the force at the pad 210. it can. Elastomeric pad 210, in combination with the sensing electronics described herein, provides information about the contact between finger 204 and object 208. For example, the information may include information about the applied normal force, shear force, or slip force and the orientation of the object 208. This information is useful in gripping delicate objects where it is critical to control the gripping force and confirm proper gripping before attempting to move the object 208.

  Still referring to FIG. 2, a high sensitivity elastomer 212 can be applied to the gripper finger 204. Elastomer 212 is attached to the outer surface of finger 204 and is used to detect contact between finger 204 and the outer surface. For example, this is useful when the gripper 200 is inserted into a box or bin of parts, and observation of the force applied to the elastomer 212 detects contact between the gripper fingers 204 and the wall of the box or bin. be able to. Therefore, a collision with an object is detected.

  Finger 206 can be attached to conformal elastomer skin 214. Conformal skin 214 provides the same function as sensitive elastomers 210 and 212. In some cases, skin 214 can be a single elastomer, providing a sensitive surface around the entire tip of finger 206. The elastomer 214 may be formed in the form of a glove finger that conforms to the shape and surrounds the surface of the finger 206. Various techniques described herein identify the location of the force applied to the skin 214, thereby making contact with the object 208 and with other entities such as walls, bins, and other structures. Contact can be detected separately.

  The gripper can include a sensor provided herein. The gripping system or mechanism can include one or more grippers. The gripper can be configured to detect various characteristics of the object, such as the electrical resistance of the object. The gripper mechanism can include a plurality of grippers that can characterize various properties of the object, such as gripping properties, by passing current (AC or DC) through the object. The gripper mechanism can grip an object using magnetic force or electrostatic force.

  FIG. 3 is a schematic diagram of a configuration of components that can be used to implement the devices and systems of the present disclosure that can incorporate multiple sensors. System 300 represents an automated iterative industrial process in which a series of objects 322, 324, 326, and 328 are moved on conveyor 330 and are gripped and manipulated by robot 318. It can be seen that the objects 322, 324, 326, and 328 are in any different orientation, which is the robot required to manipulate the object without causing damage due to excessive force or dropping. Complicate 318 movement sequences and forces. The robot 318 includes a sensing elastomer skin 320 that identifies the force applied to the grasped object. A skin sensing electronic device 310 is connected to the skin 320. This connection may be through any suitable data communication interface, including, for example, a wire, fiber optic, or wireless data link.

  Imaging system 316 provides visual data about the orientation and position of objects 322, 324, 326, and 328. An exemplary vision system includes KINECT available from Microsoft Corporation. The imaging system 316 is connected to the imaging detection electronic device and process 308. Data from the imaging sensing electronic device 308 and the skin sensing electronic device 310 can be merged (eg, aggregated) in the seno fusion 304. Sensor fusion 304 provides more complete situational and orientation recognition than is available from visual sensing alone or tactile sensing alone.

  The system controller 302 can provide closed loop control. The sensing input from sensor fusion 304 provides visual information, tactile information, and integrated information about the situation on conveyor 330. The controller 302 transmits a command to the robot controller 312 to control the robot 318. The system controller 302 also sends commands to the lighting controller 306 to adjust the light provided by the light source 314. Illumination is critical to the vision system, and adjustments in illumination angle, intensity, or type provide additional functionality. For example, in some situations, bright diffuse illumination in the human visible spectrum is best. In other situations, laser scanning or infrared illumination is useful, and in some situations, a combination of illumination applied simultaneously or sequentially can provide more useful visual information. Accordingly, the system controller 302 can control the light source 314 to adjust the illumination detected by the imaging system 316.

  4A, 4B, and 4C illustrate techniques for making electrical connections. FIG. 4A shows the electrical connections applied to the various points of the sensing elastomers 402 and 412. The elastomer 402 is a bottom view and the elastomer 412 is a side view. Four contacts 404, 406, 408, and 410 are shown. Four contacts are shown for illustration. In other cases, at least about 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 30, 40, 50, 60, 70, 80, More contacts are included, such as 90 or 100 contacts.

  Referring to FIG. 4A and with reference to elastomer 412, representative contacts 404 are connected to electrical wires 416 that connect to sensing electronics and excitation circuitry. It can be seen that this arrangement stresses the contacts 414 and wires 416, which can cause damage and failure.

  FIG. 4B shows a sensor having an elastic skin 420 with peripheral contacts 422, 426, 430, and 434. Contacts 422, 426, 430, and 434 are electrically connected to wires 424, 428, 432, and 436, respectively. By moving the contacts to the periphery of the elastic skin 420, robustness and resistance to breakage from the force applied to the elastic skin 420 can be provided.

  However, the peripheral contacts 422, 426, 430, and 434 may require more sophisticated electronics and processing algorithms to obtain the desired sensing and resolution of the force applied to the elastic skin 420. In the example, electrical impedance tomography (EIT) is used to detect the pressure distribution across the conductive conform skin. EIT is a non-invasive technique that measures the internal impedance of a material through the distribution of electrodes at the boundary. In general, EIT involves measuring an impedance set from various electrode combinations and then combining the measurements through the application of an inverse problem solving technique to yield an impedance distribution. This distribution is then related to various properties of the elastomer, such as density or pressure, depending on the application. Materials that can undergo EIT techniques include conductive yarn fabrics, carbon or metal encapsulated rubber, and each material uses a variety of algorithms to provide a tomographic map. The EIT can be performed when an object is in contact with one or more of the contacts 422, 426, 430, and 434, and when no object is in contact with the contact.

  In some cases, the electrical impedance and diffusion optical tomography reconstruction software (EIDORS) software package at Matlab® can be used to evaluate the pressure profile across the entire area of the sensor. Each 14 × 16 matrix of voltage data is transformed into a pressure distribution across the virtual electrode or texel mesh. EIDORS is a software suite for reconstructing images with electrical impedance tomography (EIT) and diffuse optical tomography (DOT).

  Resistance evaluated over a set of powers can be used to interpret whether the object is in contact with one or more contacts and the shape of the object. For example, a small or narrow object creates a local deformation of the elastomeric skin that produces a large resistance change only for elements close to the point of contact or a small population of texels.

  FIG. 4C shows a sensor having an elastic skin 440 that includes a sensing area 442 where force is detected and a peripheral area outer area 442. The area of skin 440 in outer area 442 is used to form conductive paths 444, 446, 448, and 450, which can be similar to electrical wires. The conductive path 444 is formed by manufacturing the skin 440 with a highly conductive elastomer volume that has been treated to have a much higher conductivity than the surrounding material. Such highly conductive elastomers can have a resistance of about 0.0001 Ω-cm to 100 Ω-cm or 0.001 Ω-cm to 10 Ω-cm. The elastic skin 440 can be manufactured as a complex matrix that includes a volume of low-conductivity elastomer for structural support, a volume of high-conductivity elastomer that provides electrical connections for voltage and current, and a sensing region 442. it can. Such low conductive elastomers can have a resistance of about 10 Ω-cm to 100 kΩ-cm or 100 Ω-cm to 10 kΩ-cm.

  This provides several advantages. The connection points do not require attachment with wires at points on the periphery. All external connections can be made with a connector 452, which can be arranged to provide a robust and convenient connection to the electronic device.

  The sensors of the present disclosure can include any number, arrangement, and distribution of contacts. The contact can be an electrode. For example, the sensor is at least about 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 30, 40, 50, 60. , 70, 80, 90, or 100 contacts. A subset of such contacts can be reference (or ground) electrodes during measurement. For example, a given sensor may include eight electrodes that provide power and eight electrodes that are reference (or ground) electrodes.

  The sensor can have contacts in various configurations and arrangements such as an arrangement having a circular, triangular, square, rectangular, pentagonal, hexagonal, heptagonal, or octagonal arrangement, or a partial shape or combination thereof. The contacts can have various filling arrangements, such as closest packing (eg, hexagonal closest packing). In some examples, 16 contacts are used.

  The contacts (or electrodes) of the present disclosure can be independently addressable. This allows the control system to address and obtain the signal from each contact independently from another contact, and use the signal to generate a matrix of signals from all contacts. it can.

  FIG. 5A illustrates an example method that can be used to manufacture the sensor of the present disclosure. The sensor of the present disclosure may require a complex structure in which various elastomers with different properties are combined in a three-dimensional volume. Several techniques can be applied to achieve this combination, and the description herein is illustrative of one such technique. In FIG. 5A, a mold 502 is formed into a desired or predetermined shape from a material suitable for casting an elastic object such as, for example, polysiloxane, polyurethane, or other compatible elastomer. The creation of an elastic body, component, or layer can begin with a raw elastomer solution in liquid form, which can then be shaped and cured.

  In FIG. 5A, three layers of liquid elastomer are sequentially poured into the mold and each is cured. The substrate layer 504 is first poured and cured, followed by the intermediate layer 506, followed by the top layer 508. Each layer can have different properties. However, alternatively, more or fewer layers can be used. Examples of elastomers can be obtained from Nusil Silicone Technology, located in Carpentaria, California. Various materials are added to the liquid elastomer and then cured to obtain the desired properties. For example, carbon is added to change the conductivity and the blowing agent changes the density.

  The elastomeric material can be a mixture of polymeric material, foam, and carbon black (or other conductive agent). The polymeric material can be, for example, polysiloxane (silicone rubber) or polyurethane. Foam and rubber can be supplied with a two-part liquid component that is mixed to the desired mechanical properties. Foam and rubber are obtained with a two-part liquid component that is mixed towards the desired mechanical properties.

  FIG. 5B shows the components of the sensing assembly. Sensing assembly 510 includes a hard elastomeric hemisphere 512 and an elastic cup 536. The hemisphere 512 incorporates various connections and electronic devices. For example, the hemisphere 512 includes a conductive sensing point (or electrode) 514 wired to the embedded electronic bus 538 through a conductor 524. The bus 538 can be a printed circuit board (PCB). Similarly, sensing points (or electrodes) 516, 518, 520, and 522 are connected to bus 538 through conductors 526, 528, 530, and 532, respectively. Each of the conductors 524, 526, 528, 530, and 532 can include a metal wire embedded in the hemisphere 528, a conductive elastic tunnel can be formed, or other techniques for forming a conductive path. Can be used.

  The sensing assembly of FIG. 5B can include any number of sensing points. For example, the sensing assembly is at least 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 30, 40, 50, 60, 70, 80, 90, or 100 sensing points can be included. The detection points can have various distributions over the hemisphere 528. For example, the detection points can be distributed across the rows along the surface of the hemisphere 528.

  FIG. 6 is a schematic diagram of an electronic device that can be associated with a sensor of the present disclosure. Terminal 602 is an electrical connection point from a set of sensing connection points on a highly sensitive elastomer. In order to form a sensing device from an elastomer, a plurality of connection points or terminals similar to terminal 602 are required and spaced apart to help apply a particular sensing algorithm to the collected data. For example, the EIT algorithm requires several terminals spaced apart to provide a plurality of separate conductive paths through the elastic sensing area.

  The arrangement example uses 16 terminals (or electrodes) arranged at equal intervals in a circular pattern so that each terminal is spaced 22.5 degrees from the adjacent terminal. It is desirable, but not essential, that the terminals be arranged so that each of the plurality of conductive paths excited in the sensing process has the same path length through the elastomer. It will be appreciated that many terminals and arrangements are possible.

  Referring again to FIG. 6, terminal 602 is conductively connected to switch 610 through conductor 604. The switch 610 is a three-position switch. Switch 610 may be an electrical connection between terminal 602 and exactly one of excitation electronics 608 through conductor 606, an electrical connection between terminal 602 and sensing electronics 614 through conductor 612, or a reference or ground. Establish electrical connection to potential node 616.

  In some examples, 16 terminals similar to terminal 602 are attached to the sensing elastic skin. The sensing procedure applies excitation 608 to one of the 16 terminals, connects the reference potential 616 to another terminal at the same time through another set of electronic devices, and uses the other electronic devices to detect the sensing electrons at the same time. Device 614 is connected to the remaining 14 terminals. All excitation and sensing electronics are connected to a common reference 616, so that each measurement of potential voltage uses the same zero volt reference so that the voltage from several terminals can be compared and the potential difference between the terminals can be calculated. To do.

  The excitation signal is applied to terminal 602 by conductor 606, switch 610, and excitation 608 through conductor 604. The excitation signal may be a predetermined voltage potential with respect to the reference 616 or a predetermined current between the excitation 608 and the reference 616. The excitation signal is in some cases direct current (DC) or voltage. Alternatively, alternating current (AC) or voltages having various waveforms are generated by excitation 608. In some cases, the excitation signal is a predetermined direct current of at least about 1 milliamp (mA), 2 mA, 3 mA, 4 mA, 5 mA, 6 mA, 7 mA, 8 mA, 9 mA, 10 mA, 15 mA, 20 mA, 30 mA, 40 mA, or 50 mA. It is.

  The sensing electronics 608 measures the voltage or electromotive voltage at the connection terminal 602 relative to the reference 610 when the switch 604 is appropriately configured. In some cases, sensing electronics 608 measures instantaneous voltage, average voltage, square root of voltage, peak voltage, derivative of voltage, or a combination of these measurements at terminal 602.

  As will be discussed further herein, each of the 16 terminals corresponding to an instance of terminal 602 provides a different function at different times according to the configuration of switch 604. Each of the 16 switches that are instances of switch 604 is automatically controlled by a processor executing a data collection algorithm.

  FIG. 7 is another view of an electronic device that can be used in conjunction with the sensors described herein. The elastic sensing system 700 includes a sensitive elastomer 703 connected to sensing electronics components through a matrix connection 702 in accordance with the drawing convention that intersecting lines are electrically connected only if they are marked with black circles. Elastomer 703 is shown as having eight connection points for the sake of brevity, but the description of FIG. 7 may be applied to any number of connection points to elastomer 703 in accordance with the invention described herein. Those skilled in the art will understand that scaling is possible.

  Each of the three multiplexers 704, 706, and 708 functions to establish a conductive connection between one of the eight connection points and each multiplexer or mux output terminal. Accordingly, voltage source 710, ground 720, and filter 714 may each connect to one of the connection points to elastomer 703. Control of the selection of which connection points are connected is achieved by processor 712 through control connection 722 that connects to the selection inputs of multiplexers 704, 706, and 708.

  Voltage source 710 provides voltage or current excitation to elastomer 710. In some cases, source 710 provides a fixed predetermined voltage. The processor 712 can configure the source 710 to provide a desired or predetermined voltage, change the voltage, and execute a sensing strategy or algorithm. In some situations, the voltage source 710 also includes a current sensor that can be used to change the voltage to produce the desired current flow in the elastomer 703 or to monitor the current. If the voltage and current are known, Ohm's law can be applied to calculate the impedance.

  In FIG. 7, multiplexer 708 can connect one of the terminals of elastomer 703 to filter 714. Filter 714 can be an anti-alias filter, low pass filter, or other filter, either an analog (continuous) or digital (discrete) filter. For example, the filter 714 is a filter configured to reject high frequency signals and reduce noise applied to subsequent processing through low frequency signals.

  From filter 714, the signal flows to time derivative operation 716 and gain 726. Time derivative operation 716 determines the rate of change of the input signal with respect to time. The output of derivative operation 716 is applied to gain stage 718. The gain stages 718 and 726 change the amplitude of the signal before transmitting it to the data acquisition module 724. The amplitude change achieved in gain stages 718 and 726 may apply a gain greater than 1 to increase the signal amplitude, or apply a gain less than 1 to attenuate the signal amplitude. Or may apply a strictly unity gain and function to buffer the signal.

  The data acquisition module 724 can convert the signal into digital data suitable for the processor 712. In some cases, the data acquisition module 724 includes a sampler that captures and holds the input signal voltage and an analog / digital converter that converts the sampled signal into a numerical representation.

  In some situations, the system 700 of FIG. 7 can measure data at only one terminal of elastomer 703 at any given time, and to collect data from several terminals, multiplexer 708 Requires sequential selection and conversion controlled by processor 712. Alternatively, two or more points are sampled simultaneously. In the example, multiplexer 708 is replaced with a multi-channel data acquisition module that can sample all applied input signals simultaneously.

  FIG. 8A schematically shows a series of operations for collecting data. An elastomer with six terminals is shown for reference and each of six different excitation and measurement situations performed in sequence are shown at excitations 802, 804, 806, 808, 810, and 812, respectively.

  FIG. 8B is a flowchart showing a series of operations for collecting data. At 850, frame or data set collection begins. At 852, excitation is applied across a pair of terminals. At 854, a delay can be applied to settle any transient current before measurement. At 856, terminal measurements are addressed. In the example, measurement 856 includes measuring the voltage at every terminal. In another example, only the unexcited terminals are measured.

  At 858, the measurement data is stored in a data structure. At 860, it is determined whether all data for the current frame has been collected or if there are additional terminal pairs that can be excited to complete the detection algorithm. If there are additional terminal pairs to be excited, execution continues to 852. If there are no more terminal pairs to be excited, execution continues to 862 and the collected data is processed.

  FIG. 9A schematically illustrates a system that includes a wireless connection to a rotating machine. A wireless connection to a moving machine can impose certain problems on both power application and data collection. In FIG. 9A, the system 900 can address this problem by eliminating the elastic sensor and the wires connecting the data processing and storage. The curved outer surface of the rotating machine 902 is covered with an elastic skin sensor 904. The elastic skin sensor 904 is connected to the rotating electronic module 906. The rotation module 906 communicates with the fixed electronic module 910 using the wireless communication link 910.

  FIG. 9B is a schematic diagram of components arranged to perform wireless detection and communication. For example, system 918 in FIG. 9B corresponds to rotating electronic module 906 in FIG. 9A. Elastomeric sensor 920 is connected to excitation 924 and sensing 926 through multiplexed electronics 922 and is controlled by data acquisition module 928. Module 928 performs a sequence to apply the excitation and collect the data necessary to detect the force applied to skin 920. The collected data is then transmitted from wireless communication 930 using antenna 932. It should be understood that various communication schemes can be used depending on the environment and performance. For example, short-range communication methods such as Bluetooth, line-of-sight optical communication, or various radio frequency modulation methods are useful. In some examples, near field communication technology is used.

  Another problem that arises in collecting data from elastic skins attached to rotating machines is the problem of powering excitation and sensing electronics. In FIG. 9B, the power of sensor 918 is sent through power control module 942. The power control module 942 obtains power from one of the two sources. The first source is an energy storage device 944, which can be a replaceable or rechargeable energy storage device that utilizes technologies such as supercapacitors, lithium ion batteries, nickel metal hydride batteries, or NiCd batteries. Other energy storage technologies currently available or discovered in the future are also applicable without departing from the invention disclosed herein.

  Another energy source is the energy harvesting module 940. The energy harvesting module uses vibration, heat, or motion to generate power that operates the sensor system 918. The energy from module 940 is sent to the electronic device or sent to recharge the energy storage device 944.

  FIG. 10A schematically shows the sensing elastomer and electrical connection points. The sensing skin 1000 is shown having six connection points 1002, 1004, 1006, 1008, 1010, and 1012.

  FIG. 10B shows an array of sensing elements or “texels” overlaid on the sensing elastomer. The detection skin 1020 corresponds to the detection skin 1000 of FIG. 10A. A grid or map 1022 is superimposed on the skin 1020 to help locate the detected force. It should be understood that the grid 1022 is not a physical grid but a model used to refer to points on the grid. Each element of the grid 1022 is called a “texel” and means a tactile sensing element. Although grid 1022 is shown as a square texel tiling, various shapes of texels can be used, including hexagons and rectangles.

  FIG. 11 schematically illustrates a workflow for processing sensor data, according to some embodiments. Data collected from the elastic skin is transmitted to the data storage 1102. The data storage 1102 is also used for storing application results of various data processing algorithms. A data processing algorithm is applied to identify the pressure distribution, force distribution, and torque distribution from the measured values of the elastic sensor characteristics. In FIG. 11A, each processing algorithm is shown for illustration. The algorithm 1106 is a Kalman filter, the algorithm 1108 is a neural network, the algorithm 1110 is a tomographic algorithm, and the algorithm 1112 is a point group that can use a specific filter. In some cases, these and other algorithms may be applied individually or in combination. The result is displayed on the real time display 1104.

  FIG. 12 schematically illustrates integration of other sensors, according to some embodiments. In the system 1200, an inertial measurement unit (IMU) 1202 includes an acceleration sensor (accelerometer), a rotation sensor (gyro), and a magnetic field sensor (magnetometer) on three axes. This is representative of the small and inexpensive IMU package currently found in smartphones or other electronic devices. The acceleration sensor can be configured to detect skin acceleration. Such a sensor can detect slip and texture by detecting vibrations on the contact surface with the accelerometer. Alternatively or additionally, piezoelectric materials that can generate a voltage when deformed can be used.

  Elastic skin 1204 and associated electronic device 1206 identify the force and pressure distribution at elastic skin 1204. The processor 1208 combines the force and pressure data from the skin sensor electronics 1206 with the rotation and translation rate data from the IMU 1202. In some cases, data from different sensors are combined to yield results that are not available from any individual sensor. Alternatively, data from sensors that are noisy or less accurate are combined and processed to produce data with higher fidelity or accuracy. The results can optionally be displayed on display 1210.

  FIG. 13 schematically illustrates a workflow for processing sensor data, according to some embodiments. Visual data is collected by the imaging system 1302 and stored as visual point cloud data 1306. Tactile and gripper data is collected by the gripper and robot skin 1304 and stored as tactile point cloud data 1308. Typically, these data sets correspond to physical objects that exist in the environment. In many cases, it is desirable to automatically locate or manipulate an object using a robot gripper or arm without human intervention. The cloud data can be stored in an electronic storage medium such as a memory, and the electronic storage medium can be located locally or remotely (eg, in the cloud) from a sensor used for data collection.

  Various object models that may be encountered in the environment are stored as CAD models 1312 which are converted to point cloud data for transfer to the particle filter 1310. The particle filter 1310 fuses the data from the point cloud data 1306 with the point cloud data 1308 and further determines a CAD model from the data 1312 that matches the fused sensor data. The result of the particle filter 1310 is a robust resolution object orientation that is independent of the gripper's visual occlusion. This is useful for automatic operation of an object by a robot such as pick and place work.

  Although the above description teaches the detection and manipulation of a single object at a single point in time, the invention described herein is capable of detecting multiple objects, moving and determining the position and orientation of the object. It should be understood that changes can be dealt with in real time.

  It is important to define some terms used in connection with computer vision and machine vision algorithms. A “point cloud” is a set of points in a three-dimensional space corresponding to the surface of a physical object. Point clouds can be obtained by scanning an object, such as a computer vision system, or from a model that represents an object in a computer aided design (CAD) system.

  The “posture” of a physical or virtual object is a combination of the position and orientation of the object in the coordinate system. A “particle filter” forms an estimate based on the probability of the future position of a set of objects or points given some past history.

  “Mahalanobis distance” is a measure of the distance between two points in a multidimensional space taking into account the mean and covariance in each dimension of all data points. Basically, data points are mapped to a new coordinate system, and the transformation mapping depends on the distribution of data points.

In an example, the particle filter makes an initial estimate of posture based on visual data and further improves the estimate by including tactile (eg, pressure or impedance) data. The visual data can be associated with a first data set, the haptic data can be associated with a second data set, and the first set can be larger than the second data set. The particle filter architecture is used in stereoscopic vision and uses a “proximity measurement model” known as “likelihood field” in mobile robotics. Each measurement (both visual and tactile) consists of a set of position vectors and surface normal vectors in six-dimensional space, denoted as “data points”. That is, each measured value is a point group having a surface normal vector direction to which each point corresponds. For each data point (M), the nearest tuft (O) on the known CAD model is found. The attitude difference (D) of the data points is calculated based on the following formula.

In this equation, σ is the Gaussian noise variance, and the CAD model is in an a priori pose in a six-dimensional (6D) space containing three positions and three normal vectors, respectively. D is the Mahalanobis distance between O and M. D is calculated for each data point in the measurement. The sum of all D is the “total distance” of the measurement from the CAD model. The particle filter represents a probability distribution of the CAD model pose in 6D, and each particle is a point in the 6D CAD model pose space. Thus, the particle filter is used as a search algorithm for the best CAD model pose that matches the measured data from the visual and tactile sensors.

  14A and 14B are schematic views of the arrangement of the components of sensing assembly 1400. FIG. In FIG. 14A, an end view of assembly 1400 is shown. The assembly 1400 includes a plurality of volumes arranged as concentric layers. The substrate 1402 occupies the innermost volume. The base material 1404 is a volume that is coaxial and overlaps the base material 1402. The substrate 1406 surrounds the substrate 1404. The substrates 1402, 1404, and 1406 are shown as cylinders or hollow cylinders that share a common axis for illustrative purposes, but other shapes and arrangements of layers are possible. For example, the volume can have a square, oval, elliptical, or asymmetric boundary shape.

  The arrangement according to assembly 1400 can be useful in creating an elastic volume of a specific composition with the desired overall properties. For example, the volume of the substrate 1404 is a highly conductive material, and the volume of the substrate 1406 is a material that is much less conductive and conductive between end points, but is insulated on the outer surface but has an assembly 1400 that is insulated. create. The assembly 1400 may also be a component included in other elastic compositions or subsequent manufacturing processes in some embodiments.

  Both the assembly 1400 and the techniques for manufacturing the assembly 1400 and creating the hierarchical structure provide various advantages over other currently available system techniques. In one embodiment, the substrate 1402 is a high tensile flexible filament, such as a thread or wire. Next, the substrate 1402 is coated with the material used to form the substrate 1404. In one embodiment, the substrate 1402 is a cotton yarn that is immersed one or more times in an uncured liquid elastomer, and then the elastomer is cured to form a solid elastomeric layer. Subsequently, the combination of substrates 1402 and 1404 is immersed or dipped in the uncured liquid elastomer desired for substrate 1406 to form an external volume. Other techniques, including but not limited to dip coating and injection molding, can be applied to create the volume layer of the assembly 1400.

  Referring to FIG. 14B, a side view of assembly 1450 is shown, which is an assembly of multiple concentric volumes of material. The central volume 1452 is surrounded by a second material layer 1454, and the second material layer 1454 is further surrounded by a third material layer 1456. Although three volume regions are shown for illustration, any number of layers may be included in the assembly 1450. In one embodiment, similar to a multi-conductor electrical cable, multiple inner layers are surrounded by outer layers.

Computer Control System FIG. 18 illustrates a computer system 1801 that is programmed or otherwise configured to implement the devices, systems, and methods of the present disclosure. Computer system 1801 includes a central processing unit (CPU, also referred to herein as “processor” and “computer processor”), which may be a single-core or multi-core processor, or a plurality of processors for parallel processing. It may be a processor. The computer system 1801 is a communication interface for communicating with a memory or memory location 1810 (eg, random access memory, read only memory, flash memory), an electronic storage unit 1815 (eg, hard disk), and one or more other systems. 1820 (eg, a network adapter) and peripheral devices 1825 such as caches, other memory, data storage devices, and / or electronic display adapters. The memory 1810, the storage unit 1815, the interface 1820, and the peripheral device 1825 communicate with the CPU 1805 through a communication bus (solid line) such as a motherboard. The storage unit 1815 can be a data storage unit (or data repository) that stores data. Computer system 1801 can be operatively coupled to a computer network (“network”) 1830 using a communication interface 1820. Network 1830 can be the Internet, the Internet and / or an extranet, or an extranet in communication with an intranet and / or the Internet. In some cases, network 1830 is a telecommunications network and / or a data network. The network 1830 can include one or more computer servers that can enable distributed computing, such as cloud computing. Network 1830 may in some cases use computer system 1801 to implement a peer-to-peer network, which may allow devices coupled to computer system 1801 to behave as clients or servers.

  The CPU 1805 can execute machine-readable instruction sequences, which can be implemented in programs or software. The instructions may be stored in a memory location such as memory 1810. Examples of operations performed by the CPU 1805 can include fetch, decode, execute, and write back.

  The CPU 1805 can be part of a circuit such as an integrated circuit. One or more other components of the system 1801 can be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC).

  The storage unit 1815 can store files such as drivers, libraries, and saved programs. The storage unit 1815 can store user data, such as preferences and user programs. The computer system 1801 may include one or more additional data storage units that are external to the computer system 1801, such as, in some cases, located on a remote server that communicates with the computer system 1801 over an intranet or the Internet. .

  Computer system 1801 can communicate with one or more remote computer systems over network 1830. For example, the computer system 1801 can communicate with a user (eg, operator) remote computer system. Examples of remote computer systems include personal computers (eg, portable PCs), slate or tablet PCs (eg, Apple® iPad, Samsung® Galaxy Tab), phones, smartphones (eg, Apple® ) IPhone, Android compatible device, Blackberry®, or personal information terminal, etc. A user can access the computer system 1801 via the network 1830.

  The methods described herein may be implemented by machine (eg, computer processor) executable code stored in an electronic storage location of computer system 1801, such as memory 1810 or electronic storage unit 1815, for example. Machine-executable or machine-readable code may be provided in the form of software. In use, the code can be executed by the processor 1805. In some cases, the code can be retrieved from storage unit 1815 and stored in memory 1810 for easy access by processor 1805. In some situations, electronic storage unit 1815 can be eliminated and machine-executable instructions are stored in memory 1810.

  The code may be precompiled and configured for use with a machine having a processor configured to execute the code, or it may be compiled during runtime. The code can be provided in a programming language that can be selected to allow the code to be precompiled or run as compiled.

  Aspects of the systems and methods provided herein, such as computer system 1801, can be implemented in programming. Various aspects of the present technology typically are in the form of machine (or processor) executable code and / or associated data carried on or implemented within a type of machine-readable medium. It can be considered as “product” or “manufactured product”. Machine-executable code can be stored in such memory (eg, read-only memory, random access memory, flash memory) electronic storage units or hard disks. A “storage” type medium may include any and all tangible memories such as computers, processors, etc. or various semiconductor memories, modules associated with it such as tape drives, disk drives, etc., which may be used for non-transitory storage from time to time for software programming Can be provided. All or part of the software may occasionally communicate through the Internet or various other telecommunications networks. Such communication may allow, for example, software to be loaded from one computer or processor to another computer or processor, eg, from a management server or host computer to the application server's computer platform. Thus, other types of media that can carry software elements include light waves, radio waves, such as used across physical interfaces between local devices, through wired and optical land networks, and through various air links. And electromagnetic waves. Physical elements that carry such waves, such as wired or wireless links, optical links, etc. may also be considered as media carrying software. As used herein, unless limited to non-transitory tangible “storage” media, terms such as computer or machine “readable medium” may participate in providing instructions to a processor for execution. Refers to the medium.

  Accordingly, machine-readable media such as computer-executable code may take many forms, including but not limited to, tangible storage media, carrier wave media, or physical transmission media. Non-volatile storage media include, for example, optical disks or magnetic disks, such as any storage device in any computer that can be used to implement a database or the like shown in the drawings. Volatile storage media include dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include coaxial cables, copper wire, and optical fibers, including the wires that make up the bus in a computer system. Carrier-wave transmission media can take the form of electrical, electromagnetic, acoustic, or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, general forms of computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROM, DVD, or DVD-ROM, any other optical media, Punch card, paper tape, any other physical storage medium with hole pattern, RAM, ROM, PROM, EPROM, flash EPROM, any other memory chip or cartridge, carrier wave carrying data or instructions, such carrier wave Or any other medium from which a computer can read programming code and / or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

Example 1
In an example, FIG. 15 shows an inexpensive tactile and visual system that eliminates the environmental problems of existing visual systems. The system includes a robot having a gripper having a skin with a touch sensor. The system further includes a motion sensing input device (eg, a Kinect vision system) that can be calibrated with a short setup for new production processes compared to existing vision-based systems. The robot can be configured for an operation with 6 degrees of freedom. Skins can be shaped to any form factor, including large areas. The algorithm detects position-orientation and force-torque at landmark points for a given set of objects. The result is a modular sensing system that is highly scalable (in terms of price, quantity, size, and application).

  Still referring to FIG. 15, when the object passes by the conveyor, the guidance information and the initial posture are passed from the movement detection input device to the robot. The gripper uses skins to break down the posture for pick and place movement.

Example 2
In another example, FIG. 16A shows a layout of sensing pads (or sensors). The sensing pad includes a low conductive rubber substrate and three high conductive rubber wires in electrical communication with the two electrodes. The rubber wire can be a highly conductive silicone elastomer (eg, from NuSil Silicone Technology). The substrate can be a less conductive silicone elastomer. The sensing pad is designed as a mold with the top open and can be manufactured with hard wax (eg, rolled at a 2.54 mm pitch). Elastomeric electrodes and wires can be poured into the mold. More substrate can be vacuum injected to complete the elastomer sensing pad. Standard connectors can be mechanically clamped to the sensing pad. Additional electrodes and lines can be created to produce the sensing pad shown in FIG. 16B, which shows a sensing pad with 16 electrodes (smaller circles). Each electrode can include three lines, the first line is for power, the second line is for data, and the third line is for ground. The 16 electrodes can be part of the sensing skin of the sensing pad.

  Referring to FIG. 16B, three 16-channel multiplexers (MUX) can be used to control power, data, and ground between each electrode. A printed circuit board (PCB) can be used to filter the data signal and split it into raw and differential voltages. The voltage measurement can be extracted using a voltage divider. The exported data can be distributed in a 15 × 16 matrix. A given data frame may include 14 voltage measurements (each measurement from the sensing electrode) based on the resistance of the material sensed by the sensing electrode. An additional measurement is the sum of squares of the differential signal to identify contact-slip. Power and ground can be assigned to the counter electrode position when measurements are made using the remaining 14 electrodes. Power and ground are incremented around the sensor until a complete circle is created and 16 data frames are captured. As each measurement frame is constructed, the 15 × 1 vector can be concatenated into a 15 × 16 matrix.

  The electrodes can be embedded throughout the conductive substrate or under the electrokinetic substrate, and object measurements can be extracted directly from the voltage changes measured by the electrodes. However, the electrode can be moved to the periphery and a tomographic algorithm can be used to construct an image of the pressure distribution of the object in contact with the sensing pad. This can be used to generate a much denser texel pressure map compared to direct electrode measurements.

  To evaluate the pressure profile across the entire area of the sensor, electrical impedance and diffuse optical tomography reconstruction algorithms are used to span each 14 × 16 matrix of voltage data across a virtual electrode mesh that corresponds to the actual electrode of the sensor. It can be converted into a pressure distribution. The algorithm can be part of a software suite for image reconstruction in electrical impedance tomography and diffuse optical tomography. The resistance (or impedance) evaluated across the electrodes can be used to interpret the shape of the object adjacent to the sensor. For example, a small or narrow object can produce a local deformation of the sensor skin that can produce large impedance changes for only a small population of elements close to the contact point.

Example 3
In another example, FIG. 17 shows a sensor that includes an elastic molding 1701 and elastic skin pins 1702 on the surface of the molding 1701. The sensor further includes a printed circuit board having a conditional electronic device 1703. The enlarged display on the left side of the figure shows one of the hemispherical electrodes. The sensor also includes an electrode in the form of a wire that contacts the surface of the housing and thereby directly contacts the skin.

  The hemispherical electrode may allow the skin to be suspended from the surface of the device or the like. This can increase the sensitivity to deformation, which can have at least two effects. First, only hemispherical electrodes can be in direct contact with the elastic molding 1701, while the other surface electrodes can only be in capacitive contact. If contact with an object causes deformation and contact of the electrode surface, this abrupt transition can be evident in the signal read from the target electrode because a resistive element called conductive rubber is introduced. Second, full contact with the device housing can require much less force to bend the elastomer suspended between the two points (ie, two different spring constants) than distorting the elastomer. Can be).

Example 4
In another example, FIGS. 19A and 19B show a gripper that can use a magnetic force to grip an object. In FIG. 19A, gripper 1902 includes a magnetic field source configured to emit a magnetic field that can be switched on and off under control as desired to grip or release object 1904. The strength of the magnetic field can be adjusted by controlling the power to the magnetic field source. The sensing surface 1906 is conformal with the surface of the gripper 1902 and can be used to detect the characteristics of the object 1904 or to identify aspects of the instantaneous relationship between the object 1904 and the gripper 1902, thereby For example, the quality of the attachment is specified before the object 1904 is lifted or moved. The sensing surface 1906 can have a sensor disclosed elsewhere herein.

  In FIG. 19B, the gripper 1922 includes a magnetic field source that generates a magnetic field. The object 1924 can include a ferromagnetic material, and the magnetic field uses force to attract the object 1924 to the gripper 1922. Sensing surface 1926 is between gripper 1922 and object 1924. The sensing surface 1926 can have sensors disclosed elsewhere herein. In some cases, sensing surface 1926 is a sensing elastomer that incorporates a plurality of conductive points and electrical properties are measured to identify the instantaneous aspects of the force and orientation of object 1924 relative to gripper 1922. The material attracted by the magnetic force is conductive. For example, measurements using impedance and electrical impedance tomography (EIT) techniques can be utilized.

Example 5
In another example, FIG. 20 shows a schematic representation of a robot gripping mechanism. The gripping mechanism can be used to characterize an object by passing an AC or DC electrical signal through the object. The gripping mechanism includes two opposing fingers 2002 and 2004. Finger 2002 is configured with an elastic sensing surface 2006, which may have features described elsewhere herein. Finger 2004 is configured with an elastic sensing surface 2008, which may have features described elsewhere herein. In some embodiments, only one of sensing surfaces 2006 and 2008 is used for measurement, while in other embodiments both surfaces 2006 and 2008 are active and used for measurement. In one embodiment, the gripper fingers 2002 and 2004 are closed by magnetic force, which can be controlled to open, close, or modulate the gripping force applied to the object 2010.

  The object 2010 may or may not be conductive or ferromagnetic. In some cases, sensing surfaces 2006 and 2008 are used to apply electrical excitation to the object 2010 to measure the properties of the object 2010 and to identify the quality and properties of the gripping fingers 2002 and 2004 that the object 2010 has. . The surfaces of 2006 and 2008 can be textured with mesoscale or micro-to-nanoscale structures such as those used in fibril gripping mechanisms (eg gecko skin). When these structures are pressed against an object, the contact area between the sensors (2006, 2008) can increase and the amount of DC or AC current through the object can increase. The properties of the object can be inferred by examining the increase in measured current through the sensor electrodes.

  For example, when the object 2010 is conductive, the various measurement techniques described above can be applied. For example, one-dimensional, two-dimensional, or three-dimensional conductivity can be measured. These measurements can use a DC or AC voltage or current as a stimulus. If the object 2010 is non-conductive, a capacitive assembly is created, where the sensing pads 2006 and 2008 are two sides of a capacitor that includes the object 2010 as a dielectric material. In this case, an alternating voltage or current of known frequency is applied and the capacitance measurement provides information about the quality of the object 2010 and the grip.

  The methods and systems of the present disclosure can be utilized for use in various settings, such computers and industrial settings. In some examples, the methods, devices, and systems of the present disclosure can be utilized for use in healthcare (eg, surgery), industrial settings (eg, device manufacturing). For example, the methods, devices, and systems of the present disclosure can be utilized for papermaking and cardboard manufacture.

  While preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. It is not intended that the present invention be limited by the specific examples provided herein. Although the invention has been described with reference to the above specification, the descriptions and illustrations of the embodiments herein are not intended to be construed in a limiting sense. Many variations, modifications, and substitutions will now occur to those skilled in the art without departing from the invention. Furthermore, it is to be understood that all aspects of the invention are not limited to the specific diagrams, configurations, or relative proportions described herein that depend on various conditions and variables. It should be understood that various alternatives to the embodiments of the invention described herein can be utilized in the practice of the invention. Accordingly, the present invention is intended to embrace any such alternatives, modifications, variations, or equivalents. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered by the claims.

Claims (34)

A system for manipulating an object and / or detecting the presence of an object,
At least one polymer substrate including a plurality of non-metallic sensing electrodes, wherein the non-metallic sensing electrodes are flexible, and the non-metallic sensing electrodes are electrically At least one polymeric substrate configured to sense a change in impedance;
Electrically coupled to the plurality of sensing electrodes; (i) measuring a signal indicative of a change in impedance of at least a subset of the non-metallic sensing electrodes; and (ii) performing an electrical impedance tomography algorithm; Programmed to identify one or more forces applied to the polymeric substrate, and (iii) to identify one or more features of the object based on the one or more forces A computer processor.   The system of claim 1, wherein the one or more features are selected from the group consisting of the presence of the object, the shape of the object, and the proximity of the object to the polymeric substrate.   The system of claim 1, further comprising an operating member disposed adjacent to the polymeric substrate, wherein the operating member is configured to operate the object.   The system according to claim 3, wherein the polymer substrate is wound around the operation member.   The system according to claim 3, wherein the operation member is a robot gripper.   The system of claim 3, wherein the operating member is configured to apply a magnetic field to grip or grasp the object.   The system of claim 3, wherein the manipulating member is configured to pass current through the object to identify one or more characteristics of the object.   The system of claim 1, wherein the polymeric substrate comprises a polymeric material and a fabric.   The polymer substrate includes a first component volume and a second component volume, the first component volume includes the plurality of non-metallic sensing electrodes, and the second component volume includes: The system of claim 1, comprising a plurality of conductive paths each in electrical contact with a non-metallic sensing electrode in the plurality of non-metallic sensing electrodes.   The system of claim 9, wherein the conductive path through the second component volume is a metal wire.   The system of claim 9, wherein the conductive path through the second component volume is a plurality of tunnels, each tunnel being filled with a polymeric material.   The system of claim 1, wherein each of the plurality of non-metallic sensing electrodes includes a pair of conductive paths.   The system of claim 12, wherein the computer processor is programmed to apply an excitation voltage to the pair of conductive paths.   The system of claim 13, wherein the computer processor is programmed to measure a voltage of the conductive path following application of the excitation voltage.   The system of claim 1, wherein the polymeric substrate has a hemispherical shape, a cylindrical shape, or a box-like shape.   The system of claim 1, wherein the non-metallic sensing electrode is formed of a polymeric material.   The system of claim 16, wherein the polymeric material has a higher conductivity than the polymeric substrate.   The system of claim 1, wherein the non-metallic sensing electrode comprises a carbon-containing material.   The system of claim 18, wherein the carbon-containing material is selected from the group consisting of carbon powder or carbon nanostructures.   The system of claim 1, wherein the non-metallic sensing electrode comprises a foaming agent.   A papermaking system comprising the system according to claim 1. A method of manipulating an object and / or detecting the presence of an object,
(A) providing a sensing system comprising at least one polymeric substrate, the polymeric substrate comprising a plurality of flexible non-metallic sensing electrodes, the non-metallic sensing electrodes comprising: Providing, when arranged at or near an object, configured to sense a change in electrical impedance;
(B) measuring a signal indicative of a change in impedance of at least a subset of the non-metallic sensing electrodes, wherein the signal is measured when the object is located at or near the sensing system. And steps to
(C) Using a computer processor that is electrically coupled to the sensing system, performing an electrical impedance tomography algorithm, 1 applied to the polymer substrate from the signal measured in (b) Identifying one or more forces;
(D) identifying one or more features of the object based on the one or more forces identified in (c).   23. The method of claim 22, wherein the one or more features are selected from the group consisting of the presence of the object, the shape of the object, and the proximity of the object to the polymeric substrate.   23. The method of claim 22, further comprising applying an excitation voltage to the subset of non-metallic sensing electrodes.   25. The method of claim 24, further comprising measuring the subset of voltages following application of the excitation voltage.   Subsequent to (d), further comprising the step of manipulating the object using a manipulating member, wherein the manipulating member is part of the sensing system or is electrically coupled to the sensing system. 23. The method according to 22. A method for detecting and / or manipulating an object comprising:
(A) A step of providing an operation system including an operation member and a movement input detection device, wherein the operation member detects a change in impedance when an object is disposed at or near the detection electrode. Including at least one sensor having electrodes, wherein the mobile input sensing device identifies and provides a spatial configuration of the object;
(B) moving the object to or near the operating system;
(C) collecting a first data set from the object using the mobile input sensing device, wherein the first data set indicates the spatial configuration of the object; Collecting steps,
(D) using the sensing electrode of the sensor to collect a second data set under boundary conditions identified from the first data set, the second data set being an impedance change Collecting a second data set indicative of:
And (e) operating the object using the operating member when it is determined that the object is at or near the sensor based on the impedance change.   28. The method of claim 27, wherein the boundary situation is identified from one or more boundaries of the object from the first data set.   28. The method of claim 27, wherein the first data set has a lower spatial resolution than the first data set.   28. The method of claim 27, further comprising identifying one or more features of the object from the first data set prior to (d).   31. The method of claim 30, further comprising, following (d), refining the one or more features of the object based on the second data set.   28. The method of claim 27, further comprising combining the first data set and the second data set.   35. The method of claim 32, further comprising, following (d), fitting the combined data to one or more predetermined objects having known characteristics.   34. The method of claim 33, further comprising identifying a posture estimate following the fitting.