WO2023094488A1

WO2023094488A1 - An article for use with a robotic manipulator

Info

Publication number: WO2023094488A1
Application number: PCT/EP2022/083048
Authority: WO
Inventors: Jelizaveta KONSTANTINOVA
Original assignee: Ocado Innovation Limited
Priority date: 2021-11-24
Filing date: 2022-11-23
Publication date: 2023-06-01
Also published as: GB2613157A; GB2613157B; GB202116928D0

Abstract

The present disclosure relates to an article for use with a robotic manipulator with a view to understanding forces exerted on the article during its manipulation by the robotic manipulator. The article comprises a plurality of sensors, configured to be communicably coupled to a processing unit, positioned around a core and a pliable outer shell surrounding the core. The sensors are configured to output signals indicative of forces applied to the outer shell by the robotic manipulator during the manipulation of the article.

Description

AN ARTICLE FOR USE WITH A ROBOTIC MANIPULATOR

The present disclosure relates to an article for use with a robotic manipulator.

BACKGROUND

The manipulation of a product using a robotic manipulator can result in damage to the product. This is particularly true if the product is relatively soft and easily damaged like some grocery products, such as fruit. It is important therefore to understand better how such products are damaged during their manipulation, and which movements executed by the robotic manipulator are most likely to cause damage.

It is against this background that the invention was devised.

SUMMARY

Accordingly, there is provided an article for use with a robotic manipulator, the article comprising a core, a plurality of sensors arranged around the core, the plurality of sensors being configured to be communicably coupled to a processing unit and, a pliable outer shell surrounding the core, wherein the plurality of sensors are configured to output signals indicative of forces applied to the outer shell by the robotic manipulator during the manipulation of the article.

Optionally, the core is substantially rigid.

Optionally, the core is hollow.

Optionally, the core comprises at least two parts that are releasably securable to form a whole.

Optionally, the article further comprises a MARG (Magnetic, Angular Rate, and Gravity) sensor housed within the core.

Optionally, the article further comprises an access port providing access to the interior of the core. Optionally, the article further comprises a plug for the access port. Preferably, the plug is configured to form part of the outer shell.

Optionally, the core and the outer shell are substantially the same shape, ensuring that the thickness of the outer shell around a substantial part of the article is practically uniform. This arrangement encourages the uniformity of any compression experienced by the outer shell as a result of an applied force regardless of where on the article the force is applied.

Optionally, the plurality of sensors are positioned at regular intervals around the core, so as to provide a generally uniform coverage of the forces applied to the outer shell.

Optionally, the outer shell is made of silicon. This means that it can be easily tailored during its manufacture to reflect general characteristics, such as the weight and pliability, of a product that the article is intended to imitate.

Optionally, the outer shell comprises at least two parts that are releasably securable to form a whole. This arrangement provides easy access to the sensors around the core.

Optionally, the article further comprises a sleeve arranged to cover at least partially the outer shell. The use of the sleeve means that the friction coefficient of the surface of the article can be easily conveniently changed.

Optionally, the article is shaped like a grocery product.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

Figure 1 is a schematic depiction of a picking system for use with the invention;

Figure 2 is an isometric view of an article according to an embodiment of the invention;

Figure 3a is an isometric view of a core of the article of Figure 2; Figure 3b is an end view of a core of the article of Figure 2;

Figure 4 is a cross-sectional view of the article of Figure 2 along the line A-A; and,

Figure 5 is a schematic view of the telemetry module shown in Figure 4.

In the drawings, like features are denoted by like reference signs where appropriate.

DETAILED DESCRIPTION

In the following description, some specific details are included to provide a thorough understanding of the disclosed embodiment. One skilled in the relevant art, however, will recognise that other embodiments may be practiced without one or more of these specific details, or with other components, materials, etc. In some instances, well-known features or systems, such as processors, sensors, storage devices, network interfaces, fasteners, electrical connectors, and the like are not shown or described in detail to avoid unnecessarily obscuring descriptions of the disclosed embodiment.

Unless the context requires otherwise, throughout the specification and the appended claims, the word “comprise” and variations thereof, such as, “comprises” and “comprising” are to be construed in an open, inclusive sense that is as “including, but not limited to.”

Reference throughout this specification to “one”, “an”, or “another” applied to “embodiment”, “example”, means that a particular referent feature, structure, or characteristic described in connection with the embodiment, example, or implementation is included in at least one embodiment, example, or implementation. Thus, the appearances of the phrase “in one embodiment” or the like in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments, examples, or implementations.

It should be noted that, as used in this specification and the appended claims, the users forms “a”, “an”, and “the” include plural referents unless the content clearly dictates otherwise. It should also be noted that the term “or” is generally employed in its sense including “and/or” unless the content clearly dictates otherwise.

With reference to Figure 1 , there is illustrated an example of a picking system 100 of the sort that is appropriate for use with the present invention. The picking system 100 may form part of an online retail operation, such as an online grocery retail operation, but may also be applied to any other operation requiring the picking and/or sorting of items or articles. In this example, the system 100 includes a manipulator apparatus 102 comprising a robotic manipulator 121 configured to pick an article 132 from a first location and place the article 132 in a second location. The manipulator apparatus 102 is communicably coupled via a communication interface 104 to other components of the system 100, such as to one or more optional operator interfaces 106, from which an observer may observe or monitor the operation of the system 100 and the manipulator apparatus 102. The operator interfaces 106 may include a WIMP interface and an output display of explanatory text or a dynamic representation of the manipulator apparatus 102 in a context or scenario. For example, the dynamic representation of the manipulator apparatus 102 may include video and audio feed, for instance a computer-generated animation. Examples of suitable communication interface 104 include a wire based network or communication interface, optical based network or communication interface, wireless network or communication interface, or a combination of wired, optical, and/or wireless networks or communication interfaces.

The system 100 further comprises a control system 108 including at least one controller 1 10 communicably coupled to the manipulator apparatus 102 and the other components of the system 100 via the communication interface 104. The controller 110 comprises a control unit or computational device having one or more electronic processors, within which is embedded a set of control instructions provided as processor-executable data that, when executed, cause the controller 1 10 to issue actuation commands or control signals to the manipulator system 102, causing the manipulator 121 to carry out various actions, e.g., identify and manipulate articles 132. The one or more electronic processors may include at least one logic processing unit, such as one or more microprocessors, central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), programmable gate arrays (PGAs), programmed logic units (PLUs), or the like. In some implementations, the controller 1 10 is a smaller processor-based device like a mobile phone, single board computer, embedded computer, or the like, which may be termed or referred to interchangeably as a computer, server, or an analyser. The set of control instructions may also be provided as processorexecutable data associated with the operation of the system 100 and manipulator apparatus 102 included in a non-transitory processor-readable storage device 112, which forms part of the system 100 and is accessible to the controller 110 via the communication interface 104. In some implementations, storage device 112 includes two or more distinct devices. The storage device 112 can, for example, include one or more volatile storage devices, for instance random access memory (RAM), and one or more non-volatile storage devices, for instance read only memory (ROM), flash memory, magnetic hard disk (HDD), optical disk, solid state disk (SSD), or the like. A person of skill in the art will appreciate storage may be implemented in a variety of ways such as a read only memory (ROM), random access memory (RAM), hard disk drive (HDD), network drive, flash memory, digital versatile disk (DVD), any other forms of computer- and processor-readable memory or storage medium, and/or a combination thereof. Storage can be read only or read-write as needed.

The system 100 includes a sensor subsystem 114 comprising one or more sensors that detect, sense, or measure conditions or states of manipulator apparatus 102 and/or conditions in the environment or workspace in which the manipulator 121 operates, and produce or provide corresponding sensor data or information. Sensor information includes environmental sensor information, representative of environmental conditions within the workspace of the manipulator 121 , as well as information representative of condition or state of the manipulator apparatus 102, including the various subsystems and components thereof, and characteristics of the articles 132 to be manipulated. The acquired data may be transmitted via the communication interface 104 to the controller 110 for directing the manipulator 121 accordingly. Such information can, for example, include diagnostic sensor information that is useful in diagnosing a condition or state of the manipulator apparatus 102 or the environment in which manipulator 121 operates. For example, such sensors may include contact sensors, force sensors, strain gages, vibration sensors, position sensors, attitude sensors, accelerometers, and the like. Such sensors may include one or more of cameras or imagers 116 (e.g., responsive in visible and/or nonvisible ranges of the electromagnetic spectrum including for instance infrared and ultraviolet), radars, sonars, touch sensors, pressure sensors, load cells, microphones 118, meteorological sensors, chemical sensors, or the like. In some implementations, the diagnostic sensors include sensors to monitor a condition and/or health of an on-board power source within the manipulator apparatus 102 (e.g., battery array, ultra-capacitor array, fuel cell array). In some implementations, the one or more sensors comprise receivers to receive position and/or orientation information concerning the manipulator 121. For example, a global position system (GPS) receiver to receive GPS data, two more time signals for the controller 110 to create a position measurement based on data in the signals, such as, time of flight, signal strength, or other data to effect a position measurement. Also, for example, one or more accelerometers, which also form part of the manipulator apparatus 102, could be provided on the manipulator 121 to acquire inertial or directional data, in one, two, or three axes, regarding the movement thereof.

The manipulator 121 may be piloted by a human operator at the operator interface 106. In human operator controlled or piloted mode, the human operator observes representations of sensor data, for example, video, audio, or haptic data received from one or more sensors of the sensor subsystem 114. The human operator then acts, conditioned by a perception of the representation of the data, and creates information or executable control instructions to direct the manipulator 121 accordingly. In piloted mode, the manipulator apparatus 102 may execute control instructions in real-time (e.g., without added delay) as received from the operator interface 106 without taking into account other control instructions based on sensed information.

In some implementations, the manipulator apparatus 102 operates autonomously. That is, without a human operator creating control instructions at the operator interface 106 for directing the manipulator 121 . The manipulator apparatus 102 may operate in an autonomous control mode by executing autonomous control instructions. For example, the controller 1 10 can use sensor data from one or more sensors of the sensor subsystem 114, the sensor data being associated with operator generated control instructions from one or more times the manipulator apparatus 102 was in piloted mode to generate autonomous control instructions for subsequent use. For example, by using deep learning techniques to extract features from the sensor data such that in autonomous mode the manipulator apparatus 102 autonomously recognize features or conditions in its environment and the article 132 to be manipulated, and in response perform a defined act, set of acts, a task, or a pipeline or sequence of tasks. In some implementations, the controller 1 10 autonomously recognises features and/or conditions in the environment surrounding the manipulator 121 , as represented by a sensor data from the sensor subsystem 114 and one or more virtual articles composited into the environment, and in response to being presented with the representation, issue control signals to the manipulator apparatus 102 to perform one or more actions or tasks. In some instances, the manipulator apparatus 102 may be controlled autonomously at one time, while being piloted, operated, or controlled by a human operator at another time. That is, operate under an autonomous control mode and change to operate under a piloted mode (i.e., non-autonomous). In another mode of operation, the manipulator apparatus 102 can replay or execute control instructions previously carried out in a human operator controlled (or piloted) mode. That is, the manipulator apparatus 102 can operate without sensor data based on replayed pilot data.

The manipulator apparatus 102 further includes a communication interface subsystem 124, e.g., a network interface device, that is communicably coupled to a bus 126 and provides bidirectional communication with other components of the system 100 (e.g., the controller 110) via the communication interface 104. The communication interface subsystem 124 may be any circuitry affecting bidirectional communication of processor-readable data, and processor-executable instructions, for instance radios (e.g., radio or microwave frequency transmitters, receivers, transceivers), communications ports and/or associated controllers. Suitable communication protocols include FTP, HTTP, Web Services, SOAP with XML, WIFI™ compliant, BLUETOOTH™ compliant, cellular (e.g., GSM, CDMA), and the like.

The manipulator 121 is an electro-mechanical machine comprising one or more appendages, such as a robotic arm 120, and a gripper assembly or end-effector 122 mounted on an end of the robotic arm 120. The gripper assembly 122 is a device of complex design configured to interact with the environment in order to perform a number of tasks, including, for example, gripping, grasping, releasably engaging or otherwise interacting with the article 132. The manipulator apparatus 102 further includes a motion subsystem 130, communicatively coupled to the robotic arm 120 and gripper assembly 122, comprising one or more motors, solenoids, other actuators, linkages, drive-belts, and the like operable to cause the robotic arm 120 and/or gripper assembly 122 to move within a range of motions in accordance with the actuation commands or control signals issued by the controller 1 10. The motion subsystem 130 is communicatively coupled to the controller 1 10 via the bus 126.

The manipulator apparatus 102 also includes an output subsystem 128 comprising one or more output devices, such as speakers, lights, and displays that enable the manipulator apparatus 102 to send signals into the workspace in order to communicate with, for example, an operator and/or another manipulator apparatus 102.

A person of ordinary skill in the art will appreciate the components in manipulator apparatus 102 may be varied, combined, split, omitted, or the like. In some examples one or more of the communication interface subsystem 124, the output subsystem 128, and/or the motion subsystem 130 may be combined. In other examples, one or more of the subsystems (e.g., the motion subsystem 130) are split into further subsystems.

Figure 2 is an isometric view of the article 132 according to an embodiment of the invention. In this embodiment, the article 132 is shaped like a grocery product, specifically a mango. The skilled reader will understand, however, that the shape of the article 132 is not a limitation of the invention, and that other embodiments may have an altogether different shape. The article 132 comprises a pliable outer shell 134 typically made from silicon, meaning that it can be easily tailored during its manufacture to reflect general characteristics, such as the weight and pliability, of the product that the article 132 is intended to imitate. The surface of the outer shell 134 can be patterned or textured in order to replicate the appearance and frictional characteristics of the imitated product.

Alternatively, the outer shell 134 can be held in a sleeve arranged to cover, at least partially, the outer shell 134, the sleeve having a coefficient of friction similar to that of the imitated product. With reference to Figures 3a and 3b, the article 132 further comprises a core 136 that is substantially surrounded by the pliable outer shell 134, together with a plurality of sensors 138 arranged around the outer surface of the core 136, positioning them between the outer shell 134 and the core 136. The plurality of sensors 138 are communicably coupled to a processing unit 142 and are configured to output signals indicative of forces applied to the outer shell 134 by the manipulator 121 during the manipulation of the article 132. The output signals are then analysed to determine if the applied forces are likely to result in damage to the imitated product and, if necessary, used in establishing a set of control instructions for directing the manipulator 121 with a view to preventing such damage.

Preferably, the plurality of sensors 138 are positioned at regular intervals around the core 136 so as to provide a generally uniform coverage of the forces applied to the outer shell 134. In the example shown, the plurality of sensors 138 comprises four sensors positioned radially about a central belt of the core 136. In other embodiments, the plurality of sensors 138 might include fewer or more sensors, which might also be ordered differently, such as in an arrangement about the longitudinal axis of the core 136.

With reference to Figure 4, the core 136 and the outer shell 134 are substantially the same shape for the purpose of ensuring that the thickness of the outer shell 134 around a substantial part of the article 132 is practically uniform. This arrangement encourages the uniformity of any compression experienced by the outer shell 134 as a result of an applied force regardless of where on the article 132 the force is applied. In order that the article 132 can be physically isolated, the core 136 is hollow and, in this embodiment, houses a telemetry module, generally designated by 139, that is operable to monitor the output signals from the plurality of sensors 138, along with those signals of any other sensors incorporated into the module 139 or article 132, and, if necessary, transmit data acquired from the signals, via the communication interface 104, to the controller 1 10 to achieve a set of control instructions for directing the manipulator 121. The article 132 further comprises an access port 140, formed within the outer shell 134, for providing access to the interior of the core 136, together with a plug 143 configured to close the access port 140, preventing the ingress of material into the core 136. The plug 143 is preferably made of the same material as the outer shell 134, such that it forms part of the outer shell 134 when it is positioned within the access port 140.

With reference to Figure 5, the telemetry module 139 comprises the processing unit 142 that is configured to execute instructions needed for monitoring the output signals from the plurality of sensors 138 and transmitting data acquired from those signals. The instructions may be stored in and read from a memory unit 144, which also forms part of the telemetry module 139 or may, alternatively, be stored in and retrieved from the storage device 1 12 of the system 100. In order to access the storage device 112 and controller 110, the telemetry module 139 is provided with a communication interface unit 150, providing wireless bidirectional communication of processor-readable data to the system 100 via the communication interface 104. Suitable communication protocols include FTP, HTTP, Web Services, SOAP with XML, WI-FI™ compliant, BLUETOOTH™ compliant, cellular (e.g., GSM, CDMA), and the like. Data associated with the output signals, or any derivatives thereof, may also be written to the memory unit 144 and recalled in order to, for example, execute processes according to the instructions being carried out by the processing unit 142. Similar to the storage device 1 12, the memory unit can include one or more volatile storage devices. In order that the article 132 be self-contained, untethered from external systems or components that might be required for its functioning, the telemetry module 139 further comprises a rechargeable power supply 146 for supplying power to the its various units and components and an access port 148 for charging the power supply 146 and data retrieval. The telemetry module 139 may also include a suite of other sensors, generally indicated by 152, such as a temperature sensor or a MARG (Magnetic, Angular Rate, and Gravity) sensor, for providing further telemetry information that can be used to inform control instructions for directing the manipulator 121 . In this representation, a CAN-bus 154 forms a central connection between each of the units (according to a suitable protocol), allowing relevant commands and data to be exchanged between each of the units accordingly.

It will be appreciated by those skilled in the art that the present invention has been described by way of example only, and that a variety of alternative approaches may be adopted without departing from the scope of the invention as defined by the appended claims.

Claims

1 . An article for use with a robotic manipulator, the article comprising: a core; a plurality of sensors arranged around the core, the plurality of sensors being configured to be communicably coupled to a processing unit; and, a pliable outer shell surrounding the core, wherein the plurality of sensors are configured to output signals indicative of forces applied to the outer shell by the robotic manipulator during the manipulation of the article.

2. An article according to claim 1 , wherein the core is substantially rigid.

3. An article according to claim 1 or 2, wherein the core is hollow.

4. An article according to claim 3, wherein the core comprises at least two parts that are releasably securable to form a whole.

5. An article according to claim 3 or 4, further comprising a MARG (Magnetic, Angular Rate, and Gravity) sensor housed within the core.

6. An article according to any one of claim 3 to 5, further comprising an access port providing access to the interior of the core.

7. An article according to claim 6, further comprising a plug for the access port.

8. An article according to claim 7, wherein the plug is configured to form part of the outer shell.

9. An article according to any preceding claim, wherein the core and the outer shell are substantially the same shape.

10. An article according to any preceding claim, wherein the plurality of sensors are positioned at regular intervals around the core.

1 1 . An article according to any preceding claim, wherein the outer shell is made of silicon.

12. An article according to any preceding claim, wherein the outer shell comprises at least two parts that are releasably securable to form a whole.

13. An article according to any preceding claim, further comprising a sleeve arranged to cover at least partially the outer shell.

14. An article according to any preceding claim, wherein the article is shaped like a grocery product.