WO2024236173A1

WO2024236173A1 - Floor projected user interface for mobile vehicle

Info

Publication number: WO2024236173A1
Application number: PCT/EP2024/063692
Authority: WO
Inventors: Christopher Cacioppo; Hector Padilla
Original assignee: Ocado Innovation Ltd
Current assignee: Ocado Innovation Ltd
Priority date: 2023-05-17
Filing date: 2024-05-17
Publication date: 2024-11-21
Anticipated expiration: 2025-11-17

Abstract

An autonomous vehicle is provided which can display to a user a visual indication of either, or both, the location on the autonomous vehicle for items to be loaded to, or from, and a storage location within a storage facility for items to be loaded to, or from. The visual indication may be projected over the autonomous vehicle location and/or the storage location. The visual indication may convey information using colour, pulses of colour, graphical symbols, etc.

Description

FLOOR PROJECTED USER INTERFACE FOR MOBILE VEHICLE

This application claims priority from US Patent Application No. 63/502,757 filed 17th May 2023, the content of all this application hereby being incorporated by reference.

Technical Field

The present disclosure relates generally to the field of autonomous vehicles and more specifically to autonomous vehicles and methods of operating autonomous vehicles.

Background

Robotic devices are increasingly being used in environments such as factories and warehouses. Various use cases exist, and oftentimes these use cases require that the robotic devices work together with humans to perform certain tasks. As with all work environments, there is a desire to maximize efficiency. In this regard, there is a need to improve the ability and efficiency at which robotic devices and humans are able to work together to complete tasks.

Summary

According to the present disclosure there is provided a computer-implemented method for a robotic vehicle, the method comprising: controlling the robotic vehicle to navigate the robotic vehicle to a location at which one or more items are to be loaded onto or unloaded from the robotic vehicle; and while the robotic vehicle is at the location at which the one or more items are to be loaded onto or unloaded from the robotic vehicle, controlling one or more visual indication devices to project a visual indication, wherein the visual indication indicates: a) a position on the robotic vehicle at which the one or more items are to be loaded or unloaded; and/or b) a storage location from which the one or more items are to be loaded or to which the one or more items are to be unloaded.

The storage location may comprises an area of the floor in proximity to the robotic vehicle. Alternatively, the storage location may comprise a shelf location. The robotic vehicle may further comprise a load carrying mechanism and the visual indication may indicate the load carrying mechanism.

The visual indication may comprise a pattern of light projected by the one more visual indication devices. The visual indication may comprise a number of pulses of a colour of light. The visual indication may comprise a visual graphic. Such a visual graphic may display some or all of the information that is presented to an operator using other visual displays. The visual indication may be updated in accordance with the progress of the loading or unloading of the robotic vehicle.

The orientation of the robotic vehicle may be controlled relative to the storage location from which the one or more items are to be loaded or to which the one or more items are to be unloaded, such that the position on the robotic vehicle to which the one or more items are to be loaded or from which the one or more items are to be unloaded is adjacent to the storage location.

According to a further example of the present disclosure there is provided a robotic vehicle, comprising: a body; one or more visual indication devices; and a control system configured to, in use: control the robotic vehicle to navigate the robotic vehicle to a storage location at which one or more items are to be loaded onto or unloaded from the robotic vehicle; and while the robotic vehicle is at the storage location, control the one or more visual indication devices to project a visual indication onto a) a position on the robotic vehicle at which the one or more items are to be loaded or unloaded; and/or b) a storage location from which the one or more items are to be loaded or to which the one or more items are to be unloaded.

The robotic vehicle may further comprises a load carrying mechanism configured to receive one or more items. The one or more visual indication devices may comprise one or more light emitting devices, and the visual indication may comprises a colour of light projected onto the position on the robotic vehicle at which the one or more items are to be loaded or unloaded and/or the storage location from which the one or more items are to be loaded or to which the one or more items are to be unloaded. The visual indication may comprise a number of pulses of a colour of light.

The one or more visual indication devices may comprise a visual projection device, and the visual indication may comprise a visual graphic. The visual graphic indicates the position on the robotic vehicle at which the one or more items are to be loaded or unloaded and/or the storage location from which the one or more items are to be loaded or to which the one or more items are to be unloaded. The control system may be further configured to control the orientation of the robotic vehicle relative to the storage location, such that the position on the robotic vehicle to which the one or more items are to be loaded or from which the one or more items are to be unloaded is adjacent to the storage location.

According to a further aspect of the present disclosure there is provided non-transitory computer- readable medium comprising instructions executable by processing circuitry of a robotic vehicle whereby the processing circuitry is configured to perform a method as described above.

Brief Description of the Drawings

Examples of the disclosure will now be described by way of example only with reference to the accompanying drawings, in which like reference numbers designate the same or corresponding parts, and in which:

Figure 1 shows a schematic depiction of a robotic vehicle according to an example of the present disclosure;

Figures 2 & 3 show schematic depictions of a robotic vehicle according a further example of the present disclosure;

Figures 4 to 6 show schematic depictions of robotic vehicles generating visual indicators; and Figure 7 shows a schematic depiction of a flowchart describing an example method according to the present disclosure.

Detailed Description of Examples

Robotic vehicles are increasingly being used in environments such as warehouses to transport items throughout the warehouse. Humans work to load items onto the robotic vehicles and/or unload items from the robotic vehicles. For instance, the robotic vehicles may be instructed to collect a number of items located at various locations within the warehouse. A robotic vehicle will navigate to the location of an item(s) to be loaded and then a human will load the item(s) onto the robotic vehicle (e.g., onto a pallet carried by the robotic vehicle or into a basket(s) carried by the robotic vehicle). The robotic vehicle will then proceed to the location of the next item(s) to be loaded, and then a human loads that item(s) onto the robotic vehicle. The process then continues until all of the desired items have been loaded. Unloading works in a similar manner. Within this context, there is a need for the robotic vehicle to instruct the human to load/unload the item(s) to/from particular positions on the load carrying mechanism (e.g., position on the pallet, a particular basket in a set of baskets, etc.).

One way to give such instruction to the human is to include a screen that indicates the item(s) to be loaded/unloaded and to also include lights at the different positions on the load carrying mechanism to indicate the position on the load carrying mechanism from/to which the item(s) is to be loaded/unloaded (i.e. , a light on the side of load carrying mechanism that corresponds to the position on the load carrying mechanism from/to which the item(s) is to be loaded/unloaded is illuminated). This works well in cases in which the load carrying mechanism is relatively high off the floor since the lights can be located on the sides of the load carrying mechanism where the human can easily see them. However, this does not work as well for, e.g., cases when the robotic vehicle carries a pallet low to the ground. In such a case, lights along the side of the robotic vehicle may be infeasible and would be very difficult for the human to see. Further, even if such lights were feasible, the human would likely need to bend over to see the lights, which is not desirable for efficiency or safety. Further, mounting such lights low to the ground would likely result in the lights being damaged during loading/unloading. When the pallet is low to the ground, lights positioned on the top of the pallet would also not be desirable because, e.g., visibility of these lights would be impaired by other items on the pallet.

Figure 1 shows a schematic depiction of one example of a robotic vehicle 100 in which examples of the present disclosure may be implemented. As illustrated, the robotic vehicle 100 includes a body 102, one or more visual indication devices 104, and a control system 106. A load carrying mechanism 108 (e.g., a platform, a pallet, shelves, baskets, etc.) is either an integrated part of the robotic vehicle 100 or carried by the robotic vehicle 100. The visual indication devices 104 may be, for example, lights (e.g., LEDs), a projector(s), or the like. In one example, the control system 106 includes one or more processors (not shown) and memory (not shown) storing software instructions that are executable by the one or more processors whereby the robotic vehicle 100 is caused to operate in accordance with any of the examples described herein.

Systems and methods are disclosed that address the aforementioned and/or other challenges. In one example, one or more visual indication devices (e.g., lights such as, e.g., Light Emitting Diodes (LEDs), a projector or similar project device, or the like) are integrated with or affixed to a robotic vehicle and used to present a visual indication onto an area of a floor in proximity to the robotic vehicle (in addition to floor, light could be projected on walls, shelves or other nearby objects). In one example, the visual indication indicates a position on an associated load carrying mechanism at which one or more items are to be loaded or unloaded. The load carrying mechanism may be an integrated part of the robotic vehicle (e.g., a platform or set of baskets integrated into the robotic vehicle) or a separate mechanism that is carried by the robotic vehicle (e.g., a pallet or set of baskets carried by the robotic vehicle).

In one example, a computer-implemented method for controlling a robotic vehicle includes the following. In one example, the computer-implemented method is performed by a control system (e.g., including one or more processors and memory storing corresponding software instructs executable by the one or more processors) within the robotic vehicle.

• Step 1 : The robotic vehicle is navigated to a location at which one or more items are to be loaded onto or unloaded from a load carrying mechanism that is an integrated part of the robotic vehicle or carried by the robotic vehicle. o In one example, the robotic vehicle is further controlled to move to a position at which there is unobstructed floor space around the robotic vehicle to present the visual indication (see Step 2). The unobstructed floor space is, in one example, detected by the robotic vehicle (e.g., via one or more cameras and associated video analysis) or may be known to the robotic vehicle from predefined or preconfigured information about the environment (e.g., shelf locations, etc.).

■ The location of the unobstructed floor space is adjacent to the location on the load carrying mechanism to/from which the item(s) is(are) to be loaded/unloaded.

• Step 2: While the robotic vehicle is at the location at which the one or more items are to be loaded onto or unloaded from the load carrying mechanism, one or more visual indication devices are controlled to project a visual indication onto an area of a floor in proximity to the robotic vehicle, wherein the visual indication indicates a position on the load carrying mechanism at which the one or more items are to be loaded or unloaded. o In one example, the visual indication further indicates the location from which the one or more items are to be loaded or to which the one or more items are to be unloaded, relative to the robotic vehicle.

■ For example, in one example, the visual indication further includes or conveys loading or unloading information onto the area of the floor (e.g., arrows or other information indicating where to load the item(s) from or where to unload the item(s) to). o In one example, the one or more visual indication devices comprise one or more light emitting devices, and the visual indication comprises a colour of light projected onto the area of the floor in proximity to the robotic vehicle, and a location at which the colour of light is project onto the area of the floor indicates the position on the load carrying mechanism to which the one or more items are to be loaded or from which the one or more items are to be unloaded. o In one example, the one or more visual indication devices comprise one or more light emitting devices, the visual indication comprises a certain colour of light projected onto the area of the floor in proximity to the robotic vehicle, and a location at which the certain colour of light is projected onto the area of the floor and the certain colour of light together indicate the position on the load carrying mechanism to which the one or more items are to be loaded or from which the one or more items are to be unloaded.

Figures 2 & 3 show a schematic depiction of a further example of a robotic vehicle 300 according to the present disclosure. In this case, robotic vehicle 300 has the form of a pallet truck which is capable of lifting and carrying pallets. Figure 2 shows the robotic vehicle 300 comprising a body 302, one or more visual indication devices 304, control system 306 (not shown in Figure 2) , handle 310 and display screen 312. The body 302 of the robotic vehicle further comprises a pallet support platform 308 which is configured to receive a pallet thereupon. The robotic vehicle 300 further comprises one or more forks (not shown) which can be used to lift a pallet from a first storage location such that the pallet can be deposited onto the pallet support platform 308. The pallet may then be transferred from the pallet support platform, for example once the robotic vehicle has moved to a pre-determined location within a storage facility, to a second storage location. The robotic vehicle 300 may operate in an autonomous mode, in which it navigates through the storage facility without any input from a human operator. In another example, an operator may use the handle to move the robotic vehicle, either unaided in a manual mode of operation or on a semi-autonomous mode of operation in which the robotic vehicle interprets the inputs of the operator on the handle of the robotic vehicle and uses the drive means of the robotic vehicle to assist the operator inputs.

The display screen 312 may be used to display useful information to an operator regarding the status of the robotic vehicle. For example, it may indicate whether the robotic vehicle is operating in the autonomous mode, the semi-autonomous mode or the manual mode. If the robotic vehicle is operating in the autonomous mode then the display screen 312 may indicate the current direction of the robotic vehicle, its destination within the storage facility, the next item (or items) to be picked etc. The display screen 312 may provide instructions for an operator, for example requesting that the operator follow the robotic vehicle, specifying the next items to be removed from or loaded onto the robotic vehicle, instructing an operator to move to a further location within the storage facility to interact with a further robotic vehicle or operator, etc.

Figure 3 shows a schematic depiction of the robotic vehicle 300, with the body 302 of the robotic vehicle comprising control system 306, one or more sensors 320, drive means 322, power supply 324, communications interface 326 and forklift drive means 328. The control system 306 comprises non-volatile data storage 306A, processor circuitry 306B and volatile data storage 306C. The non-volatile data storage holds one or more computer programs which are executed by the processor circuitry to control the operation of the robotic vehicle. For example, a management system of the storage facility may send a message to the robotic vehicle which is received by the communications interface 326. The message may comprise a destination within the storage facility that the robotic vehicle is to navigate to. The processor circuitry may determine a route to the specified destination using, for example, map data held within the non-volatile data storage. The one or more sensors may be used to determine that the robotic vehicle is following the route, avoiding collisions with other robotic vehicles, human operators, material that may have been left within the storage facility, etc. The drive means 322 comprises the motor(s) used to move the robotic vehicle, along with any ancillary control circuitry. The power supply 324 may comprise one or more batteries which are used to power the drive means and the other functions of the robotic vehicle. The one or more batteries may be replaced or re-charged when their charge levels are depleted. In an alternative, the power supply may comprise one or more fuel cells or equivalent devices.

The forklift drive means 328 can be actuated by the control means such that the forks of the robotic vehicle can be extended from within the body of the robotic vehicle and manoeuvred such that they are received within the interior of a pallet. The forks can then lift the pallet, along with any material that is received on the pallet, and the pallet can be manipulated such that the pallet is received on the pallet support platform of the robotic vehicle body. The robotic vehicle may then move to a further location where the pallet can be off-loaded from the robotic vehicle or one or more boxes or containers of products may be removed from the robotic vehicle. An example of such a forklift mechanism is disclosed in the Applicant’s co-pending application GB2316875.0, the contents of which are hereby incorporated by reference. Figure 2 shows that the robotic vehicle comprises one or more visual indication devices 304, which may be located on the exterior of the body 302 of the robotic vehicle. In one example, a visual indication device may be provided on the top of the display screen 312. Figure 2 shows a robotic vehicle which comprises visual indication devices in several different locations. It should be understood that such a configuration is merely exemplary and that a robotic vehicle need not have all of the visual indication devices shown in Figure 2. A robotic vehicle may have a greater or lesser number of visual indication devices and the one or more visual indication devices may be provided in different locations on the robotic vehicle. In a further example, one or more visual indication devices 304 may be provided on the underside of the robotic vehicle (not shown in Figure 2) such that those visual indication devices 304 can be used to illuminate regions of the ground.

In operation, the robotic vehicle will received an instruction to navigate to a location within the storage facility such that items (boxes containing one or more products, totes or containers containing one or more product eaches, etc) can be unloaded from the robotic vehicle and/or loaded on to the robotic vehicle. The one or more visual indication devices may be activated to provide a visual indicator to an operator to perform the necessary action, that is the loading and/or unloading of the items. Figure 4 shows a schematic depiction of an example of such a visual indicator in which an arrow 400 is provided to indicate the area of the pallet support platform from which the item(s) are to be unloaded and the area of the floor of the storage facility in which the item(s) are to be placed. In addition, or as an alternative, visual indicator may comprise the perimeter of the floor area 410 into which the item(s) are to be placed, it should be understood that in the scenario shown in Figure 4, the robotic vehicle will have navigated to a position adjacent to a region of floor space which is known to be unoccupied. The robotic vehicle can use its sensors to ensure that the floor space is empty before generating the visual indication(s). The robotic vehicle may re-orient itself before generating the one or more visual indications to facilitate efficient unloading of the item(s). If a heavy item is to be loaded (or unloaded) then the robotic vehicle may re-orient and/or re-position itself to minimise the lifting and handling required.

The robotic vehicle may generate one or more additional visual indications to act as a warning to operatives that items are to be transferred from the robotic vehicle to the unoccupied floor space. If the robotic vehicle detects a further operator (that is, an operator who has not been assigned to the task of unloading the item(s) from the robotic vehicle then the visual indication(s) used for the unloading process may be terminated. Figure 5 shows a schematic depiction of a further example in which the robotic vehicle has navigated to a position near to shelving 500 which comprises multiple shelf locations. Visual indicator 420 comprises an arrow which indicates the area of the pallet support platform from which the item(s) are to be unloaded and the shelf location 510 in which the item(s) are to be stored. If the shelf location 510 stores a number of boxes, totes, etc then the visual indicator may indicate the specific box(es) etc. into which the item(s) are to be stored. Figure 6 shows a schematic depiction of a yet further example in which the visual indicator 430 comprises an arrow which indicates the location of one or more items to be loaded onto the robotic vehicle and the location of the pallet support platform onto which the item(s) are to be loaded. It should be understood that the item(s) to be loaded may be stored on the floor of the storage location, on a shelf location (such as shelf location 510 of Figure 5), in a container, tote, etc.

In addition, or as an alternative, to the simple geometric shapes described above with reference to Figures 4 to 6, the visual indication may comprise more complex graphics to convey information to operators working alongside the robotic vehicles. It should be understood that the display screen of the robotic vehicle may be used to provide information to an operator, for example which item(s) need to loaded to (or unloaded from) the robotic vehicle. In some examples, the visual indicator(s) may comprise some or all of the information displayed via the display screen. If the operator has to unload a number of items then the sensors of the robotic vehicles may determine the number of items which have been loaded (or unloaded) in accordance with the change of the weight of the items received on the robotic vehicle. A count of further items to be loaded (or unloaded) may be provided in the visual indicator(s), and the count may be updated in accordance with the change in weight.

The visual indication(s) may comprise a number of pulses of a colour of light projected onto the area of the floor in proximity to the robotic vehicle, and a location at which the number of pulses of the colour of light are projected onto the area of the floor and the number of pulses together may indicate the position on the load carrying mechanism to which the one or more items are to be loaded or from which the one or more items are to be unloaded.

Figure 7 shows a schematic depiction of a flowchart describing an example method according to the present disclosure. The method starts at S700 and at S701 the robotic vehicle navigates to a pre-determined location in a storage facility. In one example, the robotic vehicle navigates to the storage facility location and is in a suitable position and orientation at the location for the loading (or unloading) process to take place. Then, at S703 the robotic vehicle activates one or more of the visual indicators to generate one or more visual indications, which assist the operator(s) in the loading (or unloading) of the robotic vehicle, as described above. Once the loading (or unloading) of the robotic vehicle is complete then the visual indications can be terminated and the method ends at step S705.

In one alternative, the method comprises the robotic vehicle re-orienting and/or re-positioning itself (step S702) after it has navigated to the storage facility location, in order to facilitate the loading or unloading) of the robotic vehicle, for example to reduce the distance that heavy items need to be moved. Once the re-orienting and/or re-positioning has been completed then the method continues to step S703, as described above.

In a further alternative, the robotic vehicle may update one or more of the visual indications, for example to indicate a number of items which are still to be loaded (or unloaded) whilst the loading (or unloading) process is being executed. Once the loading (or unloading) process is complete then the method ends at step S705.

It should be understood that the control means may comprise a general purpose computing device which executes software to control the operation of the robotic means such as, for example, receive data from the sensors, communicate with the management system of the storage facility, actuate the drive means, etc. In one example, the processor circuitry may comprise one or more CPUs but it should be understood that other forms of processor may be used. The volatile data storage may comprise any conventional form of random access memory (RAM). The non-volatile data storage may comprise a hard drive, flash memory or other conventional form of non-volatile data storage. In an alternative, the control means be implemented with an FPGA or equivalent. Any software may be updated from the management system of the storage facility via the communications interface or may be provided on physical media, for example, DVD, CD-ROM, USB memory stick, etc.

In one respect there is provided an autonomous vehicle is provided which can display to a user a visual indication of either, or both, the location on the autonomous vehicle for items to be loaded to, or from, and a storage location within a storage facility for items to be loaded to, or from. The visual indication may be projected over the autonomous vehicle location and/or the storage location. The visual indication may convey information using colour, pulses of colour, graphical symbols, etc.

The foregoing description of examples of the disclosure has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Modifications and variations can be made without departing from the spirit and scope of the present disclosure.

Claims

1 . A computer-implemented method for a robotic vehicle, the method comprising: controlling the robotic vehicle to navigate the robotic vehicle to a location at which one or more items are to be loaded onto or unloaded from the robotic vehicle; and while the robotic vehicle is at the location at which the one or more items are to be loaded onto or unloaded from the robotic vehicle, controlling one or more visual indication devices to project a visual indication, wherein the visual indication indicates: a) a position on the robotic vehicle at which the one or more items are to be loaded or unloaded; and/or b) a storage location from which the one or more items are to be loaded or to which the one or more items are to be unloaded.

2. The method of claim 1 wherein the storage location comprises an area of the floor in proximity to the robotic vehicle.

3. The method of claim 1 wherein the storage location comprises a shelf location.

4. The method of any of claim 1 to claim 3, wherein the robotic vehicle further comprises a load carrying mechanism and the visual indication indicates the load carrying mechanism.

5. The method any of claim 1 to claim 4, wherein the visual indication comprises a pattern of light projected by the one more visual indication devices.

6. The method of any of claim 1 to claim 5, wherein the visual indication comprises a number of pulses of a colour of light.

7. The method of any of claim 1 to claim 6, wherein the visual indication comprises a visual graphic.

8. The method of any of claim 1 to claim 7, further comprising controlling an orientation of the robotic vehicle relative to the storage location from which the one or more items are to be loaded or to which the one or more items are to be unloaded, such that the position on the robotic vehicle to which the one or more items are to be loaded or from which the one or more items are to be unloaded is adjacent to the storage location.

9. The method of any of claim 1 to claim 8, wherein the visual indication is updated in accordance with the progress of the loading or unloading of the robotic vehicle.

10. A robotic vehicle, comprising: a body; one or more visual indication devices; and a control system configured to, in use: control the robotic vehicle to navigate the robotic vehicle to a storage location at which one or more items are to be loaded onto or unloaded from the robotic vehicle; and while the robotic vehicle is at the storage location, control the one or more visual indication devices to project a visual indication onto a) a position on the robotic vehicle at which the one or more items are to be loaded or unloaded; and/or b) a storage location from which the one or more items are to be loaded or to which the one or more items are to be unloaded.

11. The robotic vehicle of claim 10, wherein the robotic vehicle further comprises a load carrying mechanism configured to receive one or more items.

12. The robotic vehicle of claim 10 or claim 11 , wherein the one or more visual indication devices comprise one or more light emitting devices, and the visual indication comprises a colour of light projected onto the position on the robotic vehicle at which the one or more items are to be loaded or unloaded and/or the storage location from which the one or more items are to be loaded or to which the one or more items are to be unloaded.

13. The robotic vehicle of any of claim 9 to claim 12, wherein the one or more visual indication devices comprise one or more light emitting devices, the visual indication comprises a number of pulses of a colour of light projected onto the position on the robotic vehicle at which the one or more items are to be loaded or unloaded and/or the storage location from which the one or more items are to be loaded or to which the one or more items are to be unloaded.

14. The robotic vehicle any of claim 9 to claim 13, wherein the one or more visual indication devices comprise a visual projection device, and the visual indication comprises a visual graphic.

15. The robotic vehicle of claim 14, wherein the a visual graphic indicates the position on the robotic vehicle at which the one or more items are to be loaded or unloaded and/or the storage location from which the one or more items are to be loaded or to which the one or more items are to be unloaded

16. The robotic vehicle of any of claim 9 to claim 15, wherein the control system is further configured to control the orientation of the robotic vehicle relative to the storage location, such that the position on the robotic vehicle to which the one or more items are to be loaded or from which the one or more items are to be unloaded is adjacent to the storage location.

17. A non-transitory computer-readable medium comprising instructions executable by processing circuitry of a robotic vehicle whereby the processing circuitry is configured to perform a method according to any of claim 1 to 9.