WO2025008055A1

WO2025008055A1 - Method and system for warehouse management

Info

Publication number: WO2025008055A1
Application number: PCT/EP2023/068400
Authority: WO
Inventors: Aleksandar KRNJAIC; Andrew To; Susanna Felker; Peter Börsting
Original assignee: Dematic Pty Ltd; Dematic GmbH
Current assignee: Dematic Pty Ltd; Dematic GmbH
Priority date: 2023-07-04
Filing date: 2023-07-04
Publication date: 2025-01-09
Anticipated expiration: 2026-01-04

Abstract

Method of picking for order fulfilment in a warehouse, wherein an item is picked by a robotic picker and displaced to a target, all under the control of a control that receives input from at least one sensor to allow the control to identify the item and/or load unit carrying the item and to locate the position of the item and/or said load unit so as to control the robotic picker for picking the item and displacing the item to the target, wherein the item and/or said load unit and robotic picker are in movement relative to each other, wherein the at least one sensor is an event camera and the item and/or said load unit and the event camera are in movement relative to each other, and respective system.

Description

METHOD AND SYSTEM FOR WAREHOUSE MANAGEMENT

The invention relates to a method and system for order fulfilment in a warehouse.

Order fulfilment in a warehouse is generally performed by picking articles needed for the respective order being fulfilled from product load units containing those articles and placing the articles into order load units.

Article picking and placing plays a major role in warehousing fulfillment and manufacturing activities.

Load units (e. g. inventory totes, receptacles, shipping cartons, pallets, load carriers etc.) containing articles are commonly tracked within a warehouse or order fulfilment facility.

It is known to use labels, bar codes and even chemical tags or RFID tags on the load units to identify these and allow the tracking thereof.

Accordingly, it is also known to use similar ways of tracking the articles themselves.

Picking is very labor intensive, and most existing automated approaches that aim to automate the process cannot reach the speeds or accuracies that a human picker can. One of the main causes of delay compared to human pickers is typically the scanning and image capturing, item recognition inference time and grasping time.

Typically, these steps require the load unit or robot to come to be stationary to be executed effectively.

Therefore, warehouse picking robots (whether fixed or mobile in nature) with visionbased guidance often require a non-moving picking target. This creates throughput performance losses due to deceleration, settling and acceleration steps in each pick operation.

Robotic warehouse picking requires scanning of load unit, visual article detection, localization of objects in the load unit and robot target pose calculation that are all versatile enough for a very large spectrum of objects (many 10k). Although systems with 3D cameras and Deep Neural Networks (DNN)-based computer vision approaches may work, they are computationally intensive, have a high latency, are not robust to motion (due to motion blur, shutter configurations, scanning mechanisms and scanning speeds) and may then become the bottleneck in the picking pipeline.

This in turn means that the load unit must be stopped to be statically positioned under the camera and for performing the scanning.

US 11,548,739 B1 describes robotic grasping of items in movement on a conveyor for placing these items onto target conveyors or into contains etc. The disclosed system uses a conventional scanner to scan barcodes on the items and in parallel a 3D camera system to generate a 3D image or point cloud of the items to be handled to allow identification and positioning thereof and to allow calculation of a pick point of the robotic arm for picking the respective item.

An object of the invention is therefore to provide means to allow an improved robotic picking for order fulfilment in a warehouse.

This object is achieved by the method presented in claim 1 and the system of claim 7.

In accordance with the invention, it has been recognized that it is possible to speed up robotic picking operations by performing the picking operation during a movement of either the item and/or the event camera, which in turn assists the use of the event camera as the main sensor, as an event camera captures changes. Therefore, such event cameras can very effectively perform on-the-fly scanning of the item.

Event cameras are vision sensors which transmit binary events per pixel, indicating an increase or decrease of the observed brightness. In other words, each pixel inside an event camera operates independently and asynchronously, reporting changes in brightness as they occur, and staying silent otherwise. Event cameras output an asynchronous stream of events triggered by changes in scene illumination.

Since event-based cameras do not capture full images but continuously changing pixels, they have extremely low latency and lower data bandwidth than classical imaging cameras. Processing pixel changes on-the-fly then has the dual advantage of starting visual scene analysis immediately rather than waiting for image frames and keeping the amount of data to be processed to the relevant minimum, namely the changes. This means not only that item detection can be done so quickly that the transport and/or assembly line does not have to be stopped, but also that the movement of items is used to generate the desired pixel changes (also called "events").

A common problem of conventional vision systems used in dynamic picking in motion is motion blur, which is caused by fast motion and high exposure times. During the exposure time interval, each pixel of the moving camera integrates light from different points in the scene, resulting in a mixture of color values. Hence, motion blur is essentially a loss of information that limits further processing and makes identification and positioning of items to be picked difficult if not impossible. Simply reducing the exposure time is often not feasible since this also reduces the amount of light received on the sensor and increases the amount of noise. Another problem of conventional cameras is their limited dynamic range, which can cause bright regions of a scene to appear fully white and dark regions to appear fully black. Both issues can be solved by the invention in picking operations by introducing the use of event cameras.

The event camera used in the invention may use a pixel-based brightness sensor, especially a dynamic vision sensor (DVS), retinomorphic sensor, neuromorphic sensor or dynamic and active pixel vision sensor (DAVIS).

The data captured by the event camera can also be optionally combined with data from other sensors, e. g. other scanner data, such as from a line profile sensor, or traditional 2D/3D cameras.

Data captured by the event camera may be beneficially processed using neuromorphic computing.

Improved image recognition, visual item detection, localization and robot target picking pose calculation may be further improved by utilizing hardware acceleration, especially artificial intelligence-based hardware acceleration. This allows for instance to speed up steps of item detection, localization and picking pose calculation significantly, for example by using neuromorphic chips, enabling robotic movement and gripping as well as placing operations to be performed while the item is still in motion.

Neuromorphic Computing uses a new type of neural network (so called spiking neural network) that is capable of processing image data faster and with higher energy efficiency. To perform the calculations in spiking neural networks as efficiently as possible, ideally these are run on specialized neuromorphic chips. These then include physical artificial neurons to do computations.

The algorithms used in the invention, possibly running on accelerated hardware, can be configured to output one or more of the following: identification of an item, position and/or orientation of an item in up to 6 degrees of freedom, classification of detected items, image segmentation maps, image feature detection, robot picking poses for the identified item.

Beneficially, it is possible to also combine use of both event-based cameras and neuromorphic computing to achieve lowest possible latencies in both image capturing and image recognition and thus enable robotic picking while in movement (on-the-fly) from AGVs, AM Rs, conveyors or any other transport system conveying items, e. g. to a desired (e. g. moving) target.

One main advantage of the invention by not stopping for pick operations is improvements in throughput, which will allow throughput to exceed that of human pickers. This type of on-the-fly (in movement) picking is challenging in many pick stations for manual picking approaches due to fatigues and safety constraints.

In addition, avoiding the transport systems to decelerate and accelerate for every pick operation should save substantial amounts of energy, reduce wear and tear of the equipment, and reduces noise generation and emissions in the warehouse. This is especially important for transport systems like AGVs, AM Rs, and the like that are not continuously connected to the power grid and are battery powered. Here, energy efficiency is paramount for keeping up operating cycles as long as possible. It is possible that either the item is moving, the event camera is moving or both the item and the event camera are moving, even with different velocities.

It is also possible that the item is moving, the robotic picker is moving or both the item and the robotic picker are moving, even with different velocities.

The invention also relates to a system for picking for order fulfilment in a warehouse, with a robotic picker for picking an item and displacing it to a target, with at least one sensor to identify the item and to locate the position of the item and with a control that receives input from the sensor so as to control the robotic picker for griping and displacing the item to the target, wherein the item and robotic picker are implemented to be in movement relative to each other, characterized in that the at least one sensor is an event camera and the item and the event camera are implemented to be in movement relative to each other.

Use of an event camera for performing item identification and location while the item is in relative movement to the event camera allowing to pick the item and place it for order fulfilment in a warehouse.

Further features and details of the invention are apparent from the description hereinafter of the drawing, in which

Figure 1 shows a schematic view of an embodiment of the invention with an item conveyor system and a stationary robotic picker using a fixed event camera;

Figure 2 shows a schematic view of an embodiment of the invention with an item conveyor system and a moving robotic picker using a fixed event camera;

Figure 3 shows a schematic view of an embodiment of the invention with an item conveyor system and a moving robotic picker using a moving event camera;

Figure 4 shows a schematic view of an embodiment of the invention with a moving item transport device and a stationary robotic picker using a fixed event camera; Figure 5 shows a schematic view of an embodiment of the invention with stationary item presentation and a moving robotic picker using a moving event camera, and

Figure 6 shows a schematic view of an embodiment of the invention with a moving item transport device and a moving robotic picker using a moving event camera.

In figure 1 a picking area 2 in a warehouse 1 is schematically shown, that includes a picking station 4 which is sourced from a storage 3 by an item conveyor system 5 leading into and through picking station 4. Items 10 are transported and presented in containers 9.

The picking station 4 includes a stationary robotic picker 6 and a fixed event camera 7 with a field of view covering the item conveyor system 5 in the picking station as indicated by the gray shading. The event camera 7 is located above the item conveyor system 5 in the picking station. The robotic picker 6 may be a multi axis articulated robot with a gripper head. The gripper head may include any suitable gripping means, such as suction cups, pincer etc.

The robotic picker 6, the event camera 7 and the item conveyor system 5 are all operated under the control of control 8. Control 8 is additionally interfaced with the main warehouse management system (WMS) to exchange information on picking orders and the items involved.

Control 8 includes a computation device 11 which in turn includes a neuromorphic chip 12. Neuromorphic chip 12 has artificial neuron structures which are configured to the problem of item and/or load unit identification and classification, position and/or orientation identification of items and/or load units in up to 6 degrees of freedom, segmentation maps, detection of features, and/or robotic picking poses for the items.

An item 10 is picked by the robotic picker 6 and displaced to a target (not shown) under the control of control 8 that receives input from the event camera 7 as the container 9 including the item passes through the field of vision. This allows the control 8 to identify the item 10 using the information on the item provided by the WMS and to also locate the position and orientation of the item 10 in the container 9. This enables the control 8 to calculate the pose of the robotic picker 6 necessary for picking the item 10 and displacing the item 10 to the target.

The container 9 is carrying the item 10 and several other varying items simultaneously and is moving with a velocity V0 on the conveyor system 5. It does not stop while being “scanned” by the stationary event camera 7 nor while the fixed robotic picker 6 grips the moving item 10 and displaces it. The conveyor system 5 maintains the velocity V0.

Therefore, the control 8 must identify the different items, for example a coke can, a dog toy and a banana, that are all within the same container 9 and have different positions and orientations leading to differing calculations of poses for the robotic picker 6 to pick each item and possibly a certain picking sequence of items, if e. g. a certain item is underneath another item.

Figure 2 schematically shows an embodiment with a picking station 40 similar to the previous one with the difference that the robotic picker 6 is not fixed or stationary but located on an AMR 50 that moves with a velocity V1 differing from velocity V0 of the conveyor system. AMR 50 can communicate wirelessly through a respective interface 51 with the other elements of the warehouse 101 and in the picking area 102.

A further difference is that event camera 70 now directly includes computational hardware 71 including a neuromorphic chip 72 and a wifi interface for exchange of information with AMR 50 and the overall warehouse management system.

Therefore, in this embodiment the control (computational hardware 71) must not only identify the different items and their different positions and orientations but also the movement of the AMR 50 to calculate the poses necessary for the robotic picker 6 to pick each item while both the item 10 and the robotic picker 6 are in movement.

Figure 3 schematically shows an embodiment with a picking station 400 similar to the previous ones with the difference that the event camera 700 is not fixed or stationary but located on AMR 500 hanging over the possible gripping zone 103 by support arm 502 that moves with a velocity V1 differing from velocity VO of the conveyor system. AMR 500 can communicate wirelessly through a respective interface 501 with the other elements of the warehouse 1001 and in the picking area 1002.

Therefore, in this embodiment the control (computational hardware 701) must also take the movement of the event camera 700 moving with the AMR 500 into account to calculate the poses necessary for the robotic picker 6 to pick each item while both the item and the robotic picker 6 are in movement.

Figure 4 schematically shows an embodiment with a picking station 4000 similar to that of figure 1 with the difference that item conveyor system is exchanged by AMR 5000 that moves with a velocity V0 and carries or transports the container 9 with item(s) 10 to the fixed robotic picker 6. AMR 5000 can communicate wirelessly through a respective interface 5001 with the other elements of the warehouse 10001 and in the picking area 10002.

Therefore, in this embodiment the control (computational hardware 701) of event camera must take the movement of the AMR 5000 into account to calculate the poses necessary for the robotic picker 6 to pick each item while both the item and the robotic picker 6 are in movement, similar as on item conveyor system 5 but with the difference that the velocity and path are more complex.

Figure 5 schematically shows an embodiment with a picking station 40000 similar to that of figure 3 with the difference that the containers 10 are presented in a fixed or static manner.

Therefore, in this embodiment the control (computational hardware 701) must only take the movement of the event camera 700 moving with the AMR 500 into account to calculate the poses necessary for the robotic picker 6 to pick each item.

Figure 6 schematically shows an embodiment with a picking station 400000 similar to that of figure 3 with the difference that the item conveyor system is replaced by an AMR 5000 as in figure 4. AMR 5000 can communicate wirelessly through a respective interface 501 with the other elements of the warehouse 1000001 and in the picking area 1000002. Therefore, in this embodiment the control (computational hardware 701) must take both the movement of the event camera 700 moving with the AMR 500 and the AMR 5000 into account to calculate the poses necessary for the robotic picker 6 to pick each item while both the item on AMR 5000 and the robotic picker 6 are in movement.

Claims

1. Method of picking for order fulfilment in a warehouse, wherein an item is picked by a robotic picker and displaced to a target, all under the control of a control that receives input from at least one sensor to allow the control to identify the item and/or load unit carrying the item and to locate the position and/or orientation of the item and/or said load unit so as to control the robotic picker for picking the item and displacing the item to the target, wherein the item and robotic picker are in movement relative to each other, characterized in that the at least one sensor is an event camera and the item and/or said load unit and the event camera are in movement relative to each other.

2. Method according to claim 1, characterized in that the event camera uses a pixel based brightness sensor, especially a dynamic vision sensor (DVS), retinomorphic sensor, neuromorphic sensor or dynamic and active pixel vision sensor (DAVIS).

3. Method according to claim 1 or 2, characterized in that data captured by the event camera is processed using neuromorphic computing.

4. Method according to a preceding claim, characterized in that additional data from scanner sensors, profile sensors, 2D- or 3D- conventional cameras is used for identification and/or depth sensor(s) or camera(s).

5. Method according to a preceding claim, characterized in that either the item and/or said load unit is moving, the event camera is moving or both the item and/or said load unit and the event camera are moving with different velocities.

6. Method according to a preceding claim, characterized in that the item and/or load unit is moving, the robotic picker is moving or both the item and/or load unit and the robotic picker are moving.

7. System for picking for order fulfilment in a warehouse, with a robotic picker for picking an item and displacing it to a target, with at least one sensor to identify the item and/or load unit and to locate the position of the item and/or load unit and with a control that receives input from the sensor so as to control the robotic picker for griping and displacing the item to the target, wherein the item and/or said load unit and robotic picker are implemented to be in movement relative to each other, characterized in that the at least one sensor is an event camera and the item and/or said load unit and the event camera are implemented to be in movement relative to each other.

8. Use of an event camera for performing item and/or said load unit identification and localization while the item is in relative movement to the event camera allowing to pick the item and place it for order fulfilment in a warehouse.