KR20240046119A - Autonomous transport vehicle with vision system - Google Patents

Abstract

The autonomous guided vehicle consists of a frame, drive section, payload handler, sensor system, and auxiliary sensor system. The sensor system has electromagnetic sensors each responsive to the interaction or interface of an emitted or generated electromagnetic beam or electromagnetic field with physical properties, wherein the electromagnetic beam or electromagnetic field is disturbed by the interaction or interface with physical properties, wherein the disturbance is Detected and executed by physical property detection. The sensor system generates sensor data implementing at least one of vehicle navigation attitude or location information and payload attitude or location information. The auxiliary sensor system is a vision system that complements the sensor system and has cameras positioned to capture image data that is indicative, at least in part, of the vehicle navigation attitude or position and the payload attitude or position to supplement that information.

Description

Autonomous transport vehicle with vision system

This application is related to U.S. Provisional Patent Application No. 63/232,546, filed on Aug. 12, 2021, U.S. Provisional Patent Application No. 63/232,531, filed on Aug. 12, 2021, and U.S. Provisional Patent Application No. 63/232,531, filed on Aug. 12, 2021. Claims the benefit of the non-provisional application of Patent Application Serial No. 63/232,531, the disclosure of which is incorporated herein by reference in its entirety.

The disclosed embodiments relate generally to material handling systems, and in particular to transportation vehicles for automated storage and retrieval systems.

Typically, automated storage and retrieval systems employ autonomous vehicles to transport items within the automated storage and retrieval system. These autonomous vehicles are guided throughout the automated storage and retrieval system through location beacons, capacitive or inductive proximity sensors, line tracking sensors, reflected beam sensors, and other narrowly focused beam type sensors. These sensors may provide limited information that impacts the navigation of autonomous vehicles through the storage and retrieval system or may provide limited information related to the identification and identification of hazards that may be present throughout the automated storage and retrieval system.

Additionally, autonomous transport vehicles in logistics/warehousing facilities are typically manufactured to have a predetermined form factor for assigned tasks in a given environment. These autonomous transport vehicles are constructed from custom cast or machined chassis/frames. Other components, some of which may be custom assemblies/components (e.g., wheels, transfer arms, etc.), are mounted on the frame and are carried with the frame as the autonomous transport vehicle traverses along the crossing surface. The transfer arms and payload bays of these autonomous transport vehicles contain numerous components (sensors, encoders, etc.) and motor assemblies for transferring payloads to and from autonomous transport vehicles and justifying the payload within the payload bay. can do. The motors and sensors may be connected substantially directly and continuously to the power supply of the autonomous transport vehicle, such as via an electric bus bar.

The purpose of the present invention is to provide an autonomous transportation vehicle with a vision system that solves the problems of the prior art.

According to one or more aspects of an embodiment of the present invention, an autonomous guided vehicle:

Frame with payload hold;

a drive section coupled to a frame having running wheels that support a vehicle on the crossing surface, the driving wheels effecting vehicle crossing on the crossing surface to move an automatically guided vehicle over the crossing surface within the facility;

In a storage array, a payload handler coupled to a frame configured to transmit a payload, the payload handler having a flat non-deterministic seating surface seated on the payload hold and a storage location for the payload;

A physical property sensor system connected to a frame having electromagnetic sensors, each responsive to the interaction or interface of the sensor to emit or generate an electromagnetic beam or field having a physical property, and to cause the electromagnetic beam or field to be emitted or generated by the interaction or interface with the physical property. The field is disturbed, and the disturbance is sensed and affected by electromagnetic sensors of a physical nature, and the physical sensor system detects and affects the vehicle's navigation pose or location information and a physical characteristic sensor system configured to generate sensor data including at least one of load posture or position information; and

Coupled to the frame, complementary to the physical characteristic sensor system, and arranged to capture image data that is indicative, at least in part, of the vehicle's navigation attitude, position, and payload attitude or position complementary to information from the physical characteristic sensor system. It includes an auxiliary sensor system, which is a vision system equipped with a camera.

According to one or more aspects of the disclosed embodiments, the autonomous guided vehicle further includes a controller coupled to the frame, operably coupled to the drive section or payload handler, and communicatively coupled to the physical characteristic sensor system, wherein the controller It is configured to determine, from information from the characteristic sensor system, a vehicle attitude and position that affects independent guidance of an autonomous guided vehicle traversing the facility.

In accordance with one or more aspects of the disclosed embodiments, a controller may be configured to determine independent underpick and place of a payload relative to a storage location and an independent underpick and place of a payload within a payload hold from information of a physical characteristic sensor system payload pose. It is structured to determine the location of influence.

According to one or more aspects of the disclosed embodiments, the controller is programmed with a reference representation of predetermined features that define at least a portion of the facility to be traversed by the autonomous guided vehicle.

In accordance with one or more aspects of the disclosed embodiment, a controller is configured to register captured image data and generate therefrom at least one image of one or more predetermined features, wherein the at least one image is a predetermined image of the reference representation. Formatted as a virtual representation of the one or more predetermined features to provide comparison with one or more corresponding references to the features.

In accordance with one or more aspects of the disclosed embodiment, a controller is configured to register captured image data and generate therefrom at least one image of one or more predetermined features, wherein the at least one image is a predetermined image of the reference representation. Formatted as a virtual representation of the one or more predetermined features to provide comparison with one or more corresponding references to the features.

According to one or more aspects of the disclosed embodiments, the controller is configured to verify autonomous guided vehicle attitude and position information registered by the controller from a physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

In accordance with one or more aspects of the disclosed embodiments, the controller identifies a variance of the autonomous guided vehicle attitude and position based on a comparison between the virtual representation and the reference representation, and determines the autonomous guided vehicle attitude or position from the physical characteristic sensor system based on the variance. It is configured to update or complete information.

In accordance with one or more aspects of the disclosed embodiments, the controller may be configured to provide information from the physical property sensor system and fidelity of the autonomously guided vehicle attitude and position information from the physical property sensor system based on at least one of the distribution and analysis of the identified at least one image. and determine a posture error of and assign a reliability value based on at least one of posture error and fidelity.

In accordance with one or more aspects of the disclosed embodiments, the controller determines that, with a confidence value below a predetermined threshold, the controller determines the position of an autonomous guided vehicle based on posture and position information generated from a virtual representation on behalf of the posture and position information from a physical property sensor system. It is configured to switch navigation.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue self-guided vehicle navigation to your destination, or

Select a safe path for the autonomous guided vehicle and a trajectory to bring the autonomous guided vehicle from its switching position to a safe position for stopping, or

Initiates communication with the operator, through a user interface device, identifying the destination and autonomous guided vehicle kinematic data for operator selection of autonomous guided vehicle control from automatic operation to semi-autonomous operation or manual operation.

According to one or more aspects of the disclosed embodiments, the controller is configured to verify payload attitude and position information registered by the controller from a physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

In accordance with one or more aspects of the disclosed embodiments, a controller identifies a variance in payload pose and position based on a comparison between a virtual representation and a reference representation and updates payload pose or position information from a physical characteristic sensor system based on the variance. or configured to complete.

In accordance with one or more aspects of the disclosed embodiments, the controller may determine the fidelity of the payload pose and position information from the physical property sensor system based on at least one of the identified distribution and analysis of the at least one image. It is configured to determine a posture error and assign a reliability value based on at least one of posture error and fidelity.

In accordance with one or more aspects of the disclosed embodiments, with a confidence value below a predetermined threshold, the controller determines the autonomous guided vehicle payload based on location information generated from a virtual representation on behalf of the posture and location information from a physical property sensor system. It is configured to switch handling.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue handling the automated guided vehicle to your destination, or

The user interface device initiates communication with the operator identifying payload data along with operator selection of autonomous guided vehicle control from automatic payload processing operations to semi-autonomous payload processing operations or manual payload processing operations.

In accordance with one or more aspects of the disclosed embodiments, a controller may include a virtual representation of one or more corresponding reference presentations of the reference presentation to present augmented reality images in real time to the operator via a wireless communication system communicatively coupling the controller and the operator interface; and transmit a simulated image combining one or more corresponding reference predetermined features.

In accordance with one or more aspects of the disclosed embodiments, a controller is configured to receive real-time operator commands to the autonomous guided vehicle responsive to the real-time augmented reality image, the commands being changes to the real-time augmented reality image transmitted to the operator.

According to one or more aspects of the disclosed embodiments, the auxiliary sensor system at least partially affects immediate justification and/or sorting of case devices mounted on an autonomous guided vehicle.

In accordance with one or more aspects of the disclosed embodiments, one or more of the assistive vehicle navigation attitude or position, and the vehicle navigation attitude or position may be provided via assistive information from an assistive sensor system, one or more of an assistive vehicle navigation attitude or position, and an assistive payload attitude or position. One or more auxiliary payload attitude or position information and one or more of the payload attitude or position information are coated on the reference model to improve the resolution of one or more of the auxiliary payload attitude or position, peripheral facility features and interface facility features.

According to one or more aspects of the disclosed embodiments, an autonomous guided vehicle:

Frame with payload hold;

a drive section coupled to a frame having running wheels that support a vehicle on the crossing surface, the driving wheels effecting vehicle crossing on the crossing surface to move an automatically guided vehicle over the crossing surface within the facility;

In a storage array, a payload handler coupled to a frame configured to transmit a payload, the payload handler having a flat, non-deterministic seating surface seated on the payload hold and a storage location for the payload;

A physical property sensor system connected to a frame having electromagnetic sensors, each responsive to the interaction or interface of the sensor to emit or generate an electromagnetic beam or field having a physical property, and to cause the electromagnetic beam or field to be emitted or generated by the interaction or interface with the physical property. The field is disturbed, and the disturbance is sensed and affected by electromagnetic sensors of a physical nature, and the physical sensor system detects and affects the vehicle's navigation pose or location information and a physical characteristic sensor system configured to generate sensor data including at least one of load posture or position information; and

Coupled to the frame, complementary to the physical characteristic sensor system, and arranged to capture image data that is indicative, at least in part, of the vehicle's navigation attitude, position, and payload attitude or position complementary to information from the physical characteristic sensor system. It includes an auxiliary sensor system, which is a vision system equipped with a camera.

According to one or more aspects of the disclosed embodiments, the autonomous guided vehicle further includes a controller coupled to the frame, operably coupled to the drive section or payload handler, and communicatively coupled to the physical characteristic sensor system, wherein the controller It is configured to determine, from information from the characteristic sensor system, a vehicle attitude and position that affects independent guidance of an autonomous guided vehicle traversing the facility.

In accordance with one or more aspects of the disclosed embodiments, a controller may be configured to determine independent underpick and place of a payload relative to a storage location and an independent underpick and place of a payload within a payload hold from information of a physical characteristic sensor system payload pose. It is structured to determine the location of influence.

According to one or more aspects of the disclosed embodiments, the controller is programmed with a reference representation of predetermined features that define at least a portion of the facility to be traversed by the autonomous guided vehicle.

In accordance with one or more aspects of the disclosed embodiment, a controller is configured to register captured image data and generate therefrom at least one image of one or more predetermined features, wherein the at least one image is a predetermined image of the reference representation. Formatted as a virtual representation of the one or more predetermined features to provide comparison with one or more corresponding references to the features.

In accordance with one or more aspects of the disclosed embodiments, the controller is configured to cause a virtual representation of one or more imaged features to reside on the autonomous guided vehicle, and between the virtual representations of the one or more imaged certain features and one or more corresponding reference certain features. The comparison of is configured to reside on the autonomous guided vehicle.

According to one or more aspects of the disclosed embodiments, the controller is configured to verify automated guided vehicle attitude and position information registered by the controller from a physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

In accordance with one or more aspects of the disclosed embodiments, the controller identifies a variance of the autonomous guided vehicle attitude and position based on a comparison between the virtual representation and the reference representation, and determines the autonomous guided vehicle attitude or position from the physical characteristic sensor system based on the variance. It is configured to update or complete information.

In accordance with one or more aspects of the disclosed embodiments, the controller may determine the fidelity of the autonomous guided vehicle attitude and the position information from the physical property sensor system and the information from the physical property sensor system based on at least one of the distribution and analysis of the identified at least one image. and determine a posture error of and assign a reliability value based on at least one of posture error and fidelity.

In accordance with one or more aspects of the disclosed embodiments, the controller determines that, with a confidence value below a predetermined threshold, the controller determines the position of an autonomous guided vehicle based on posture and position information generated from a virtual representation on behalf of the posture and position information from a physical property sensor system. It is configured to switch navigation.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue navigation of the autonomously guided vehicle to the destination, or select a safe path for the autonomously guided vehicle and a trajectory that will bring the autonomously guided vehicle from the transition position to a safe position for stopping, or

Initiates communication with the operator through a user interface device identifying a destination and autonomous guided vehicle kinematic data for operator selection of autonomous guided vehicle control from automatic operation to semi-autonomous operation or manual operation.

According to one or more aspects of the disclosed embodiments, the controller is configured to verify payload attitude and position information registered by the controller from a physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

In accordance with one or more aspects of the disclosed embodiments, a controller identifies a variance in payload pose and position based on a comparison between a virtual representation and a reference representation and updates payload pose or position information from a physical characteristic sensor system based on the variance. or configured to complete.

In accordance with one or more aspects of the disclosed embodiments, the controller may be configured to determine the fidelity of the payload pose and location information from the physical property sensor system based on at least one of the identified distribution and analysis of the at least one image. It is configured to determine a posture error and assign a reliability value based on at least one of posture error and fidelity.

In accordance with one or more aspects of the disclosed embodiments, with a confidence value below a predetermined threshold, the controller determines the autonomous guided vehicle payload based on location information generated from a virtual representation on behalf of the posture and location information from a physical property sensor system. It is configured to switch handling.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue handling the self-guided vehicle to your destination, or

The user interface device initiates communication with the operator identifying payload data along with operator selection of autonomous guided vehicle control from automatic payload processing operations to semi-autonomous payload processing operations or manual payload processing operations.

In accordance with one or more aspects of the disclosed embodiments, a controller may include a virtual representation of one or more corresponding reference presentations of the reference presentation to present augmented reality images in real time to the operator via a wireless communication system communicatively coupling the controller and the operator interface; and transmit a simulated image combining one or more corresponding reference predetermined features.

In accordance with one or more aspects of the disclosed embodiments, a controller is configured to receive real-time operator commands to the autonomous guided vehicle responsive to the real-time augmented reality image, the commands resulting in changes to the real-time augmented reality image transmitted to the operator. .

According to one or more aspects of the disclosed embodiments, the auxiliary sensor system at least partially effects immediate justification and/or classification of case devices mounted on an autonomous guided vehicle.

According to one or more aspects of the disclosed embodiments, an auxiliary vehicle navigation attitude or position, and assist, from an auxiliary sensor system, via one or more of an auxiliary vehicle navigation attitude or position, an auxiliary vehicle navigation attitude or position, and an auxiliary vehicle navigation attitude or position. One or more of the vehicle navigation attitude or position, and one or more of the auxiliary vehicle navigation attitude or position information, to improve the resolution of the payload attitude or position of the auxiliary vehicle, one or more criteria of peripheral facility features and interface facility features It is characterized by being applied to the model.

According to one or more aspects of the disclosed embodiments, the method includes:

We provide self-guided vehicles equipped with:

A frame with a payload hold,

a drive section coupled to a frame having running wheels that support a vehicle on the crossing surface, the driving wheels effecting vehicle crossing on the crossing surface to move an automatically guided vehicle over the crossing surface within the facility;

In a storage array, a payload handler coupled to a frame configured to transmit a payload, the payload handler having a flat, non-deterministic seating surface seated on the payload hold and a storage location for the payload;

A physical property sensor system connected to a frame having electromagnetic sensors, each responsive to the interaction or interface of the sensor to emit or generate an electromagnetic beam or field having a physical property, and to cause the electromagnetic beam or field to be emitted or generated by the interaction or interface with the physical property. The field is disturbed, and the disturbance is sensed and affected by electromagnetic sensors of a physical nature, and the physical sensor system detects and affects the vehicle's navigation pose or location information and a physical characteristic sensor system configured to generate sensor data including at least one of load posture or position information; and

Coupled to the frame, complementary to the physical characteristic sensor system, and arranged to capture image data that is indicative, at least in part, of the vehicle's navigation attitude, position, and payload attitude or position complementary to information from the physical characteristic sensor system. It includes an auxiliary sensor system, which is a vision system equipped with a camera.

A method according to one or more aspects of the disclosed embodiments further comprises the step of determining a controller from information of the physical characteristic sensor system's vehicle attitude and position affecting independent guidance of the autonomous guided vehicle traversing the facility, wherein the controller determines a frame and operably connected to a drive section or payload handler and communicatively connected to a physical property sensor system.

A method according to one or more aspects of the disclosed embodiments includes determining from information of a physical characteristic sensor system payload pose, independent underpick and location of the payload relative to a storage location and independent underpick and location of the payload within the storage location and payload hold. It further includes positions that influence picks and positions.

According to one or more aspects of the disclosed embodiments, the controller is programmed with a reference representation of predetermined features that define at least a portion of the facility to be traversed by the autonomous guided vehicle.

According to one or more aspects of the disclosed embodiment, the method further comprises registering the captured image data with the controller and generating from at least one image of one or more of the predetermined features, the at least one The image is formatted as a virtual representation of the one or more predetermined features to provide comparison with one or more corresponding references of the predetermined features of the reference representation.

In accordance with one or more aspects of the disclosed embodiments, the controller is configured to cause a virtual representation of one or more imaged features to reside on the autonomous guided vehicle, wherein the virtual representation of the one or more imaged features and the one or more corresponding reference features are configured to reside on the autonomous guided vehicle. The comparison of is configured to reside on the autonomous guided vehicle.

A method according to one or more aspects of the disclosed embodiments further includes confirming with the controller autonomous guided vehicle attitude and position information registered by the controller from the physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

A method according to one or more aspects of the disclosed embodiments includes, with a controller, identifying a variance of an autonomous guided vehicle attitude and position based on a comparison between a virtual representation and a reference representation, and determining the autonomous guided vehicle posture from a physical property sensor system based on the variance. Or, it further includes updating or completing location information.

In accordance with one or more aspects of the disclosed embodiments, the controller determines the fidelity of the autonomous guided vehicle attitude and position information and information from the physical property sensor system based on at least one of the distribution and analysis of the identified at least one image, and determines the attitude error. and assigning a reliability value according to at least one of fidelity.

According to one or more aspects of the disclosed embodiments, based on a confidence value below a predetermined threshold, the controller determines the autonomous guided vehicle It is characterized by switching navigation.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue navigation of the autonomous guided vehicle to the destination, or select a safe path for the autonomous guided vehicle and a trajectory that will bring the autonomous guided vehicle from the transition location to a safe location for termination, or

Initiates communication with the operator, through a user interface device, identifying the destination and autonomous guided vehicle kinematic data for operator selection of autonomous guided vehicle control from automatic operation to semi-autonomous operation or manual operation.

According to one or more aspects of the disclosed embodiments, the controller verifies payload attitude and position information registered by the controller from a physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

In accordance with one or more aspects of the disclosed embodiments, a controller identifies a variance in payload pose and position based on a comparison between a virtual representation and a reference representation and updates payload pose or position information from a physical characteristic sensor system based on the variance. or is characterized by completion.

In accordance with one or more aspects of the disclosed embodiments, the controller may determine the fidelity of the information and payload attitude from the physical property sensor system and the position information from the physical property sensor system based on at least one of the distribution and analysis of the at least one image identified. Determining the posture error and assigning a reliability value based on at least one of the posture error and fidelity.

According to one or more aspects of the disclosed embodiments, based on a confidence value below a predetermined threshold, the controller determines the autonomous guided vehicle Switch payload handling.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue handling the self-guided vehicle to your destination, or

The user interface device initiates communication with the operator identifying payload data along with operator selection of autonomous guided vehicle control from automatic payload processing operations to semi-autonomous payload processing operations or manual payload processing operations.

In accordance with one or more aspects of the disclosed embodiments, a controller provides a virtual representation of one or more imaged features and a reference presentation together with the augmented reality image to the operator in real time via a wireless communication system communicatively coupling the controller and the operator interface. Transmit a simulated image combining certain features with one or more corresponding references.

In accordance with one or more aspects of the disclosed embodiments, a controller receives real-time operator commands to the autonomous guided vehicle responsive to the real-time augmented reality image, and the commands cause changes in the real-time augmented reality image transmitted to the operator to the controller. Received by.

According to one or more aspects of the disclosed embodiments, a controller effect with at least an auxiliary sensor system provides alignment and/or sorting of case devices mounted on an autonomous guided vehicle.

In accordance with one or more aspects of the disclosed embodiments, one or more of the assistive vehicle navigation attitude or position, and the vehicle navigation attitude or position may be provided via assistive information from an assistive sensor system, one or more of an assistive vehicle navigation attitude or position, and an assistive payload attitude or position. One or more auxiliary payload attitude or position information and one or more of the payload attitude or position information are coated on the reference model to improve the resolution of one or more of the auxiliary payload attitude or position, peripheral facility features and interface facility features.

According to one or more aspects of the disclosed embodiments, an autonomous guided vehicle:

Frame with payload hold;

a drive section coupled to a frame having running wheels that support a vehicle on the crossing surface, the driving wheels effecting vehicle crossing on the crossing surface to move an automatically guided vehicle over the crossing surface within the facility;

a payload handler coupled to the frame configured to hold the payload of the vehicle within the storage array and transfer the payload to a storage location of the payload;

An auxiliary sensor system connected to the frame for collaboration between vehicle and operator assists a vehicle autonomous navigation/manipulation sensor system configured to collect at least sensory data embodying vehicle attitude and location information for automatic navigation by vehicles of the installation,

The auxiliary sensor system may, at least in part, be configured such that the at least one camera is arranged to capture image data indicative of objects and/or spatial features within at least a portion of the facility along with vehicles at other locations of the facility. It is a vision system equipped with one camera,

coupled to the frame and communicatively coupled to the auxiliary sensor system to register the information from image data of the at least one camera, wherein the controller determines a predetermined physical characteristic of the at least one object or spatial feature from the information. determining the presence of and, in response, selectively reconfiguring the vehicle into a cooperative vehicle state arranged to receive operator commands for the vehicle to continue vehicle operation.

In accordance with one or more aspects of the disclosed embodiments, the predetermined physical characteristics are such that at least one object or spatial feature is characterized by a lateral surface, a vehicle crossing path across the lateral surface, or through the space of the vehicle, or that of another vehicle crossing the traversing surface. extends over at least part of it.

According to one or more aspects of the disclosed embodiments, a controller is programmed with a reference representation of predetermined characteristics that at least partially define a facility through which a vehicle passes.

According to one or more aspects of the disclosed embodiments, the controller is configured to register captured image data and generate therefrom at least one image of at least one object or spatial feature representing predetermined physical characteristics.

In accordance with one or more aspects of the disclosed embodiments, the at least one image is formatted as a virtual representation of at least one object or spatial feature to provide comparison with one or more reference features of predetermined features of the reference representation.

According to one or more aspects of the disclosed embodiments, the controller may identify the presence of a predetermined physical characteristic of an object or spatial feature based on a comparison between the virtual representation and the reference representation, and based on the preset physical characteristic, determine the presence of a predetermined physical characteristic of the object or spatial feature based on the comparison between the virtual representation and the reference representation. The vehicle is commanded to stop on a preset trajectory based on the location or spatial characteristics of the object.

According to one or more aspects of the disclosed embodiments, the stopping position on the predetermined trajectory maintains an object or spatial reference within the field of view of the at least one camera and performs continuous imaging of the predetermined physical characteristics, such as traffic obstacles, areas to avoid, etc. or initiate a signal to one or more other vehicles in the detour area.

According to one or more aspects of the disclosed embodiments, the predetermined physical characteristics are calibrated and by determining a position of the object within a frame of reference of at least one camera having a predetermined relationship to the vehicle and from an object pose within the frame of reference of the at least one camera. , which determines the presence of predetermined physical properties, which are determined by the controller.

In accordance with one or more aspects of the disclosed embodiments, the controller may be configured to: identify the presence and transition from an autonomous state to a collaborative vehicle state, where the controller transmits communication images to an operator interface for operator collaborative operation of the vehicle, the presence of predetermined physical characteristics; It is configured to initiate identification.

In accordance with one or more aspects of the disclosed embodiments, the controller can autonomously create a trajectory that brings the autonomous guided vehicle to zero speed within a predetermined time period in which the movement of the autonomous guided vehicle along the trajectory is coordinated with the positions of objects and/or spatial features. It is configured to be applied to guided vehicles.

According to one or more aspects of the disclosed embodiments, the capture of image data indicative of object and/or spatial characteristics is an opportunity during payload hold in a vehicle or while transferring the payload to/from a storage location in a storage array.

According to one or more aspects of the disclosed embodiments, a controller is programmed to command a vehicle to another location in a facility associated with the vehicle to perform an autonomous transfer of one or more predetermined payloads, wherein the controller performs an autonomous transfer of one or more predetermined payloads. A task is a separate task that captures image data viewed from different locations by at least one camera.

In accordance with one or more aspects of the disclosed embodiments, the controller determines the presence of a predetermined physical characteristic of at least one object or spatial feature, at least partially consistent with the vehicle, affecting each of one or more predetermined payload automatic transfer tasks. It is structured to be complementary and peripheral to the actions.

In accordance with one or more aspects of the disclosed embodiments, the controller may determine that the presence of a predetermined physical characteristic of at least one object or spatial feature opportunistically determines vehicle actions that affect each of the one or more predetermined payload automatic transfer tasks. It is configured to do this.

According to one or more aspects of the disclosed embodiments, at least one of the one or more predetermined automatic payload transfer operations is performed at at least one of the different locations.

According to one or more aspects of the disclosed embodiments, the cooperative vehicle state assists the autonomous state of the vehicle in influencing each of one or more predetermined autonomous payload transfer tasks.

According to one or more aspects of the disclosed embodiments, the method includes:

We provide self-guided vehicles equipped with:

Frame with payload hold;

a drive section coupled to a frame having running wheels that support a vehicle on the crossing surface, the driving wheels effecting vehicle crossing on the crossing surface to move the vehicle over the crossing surface within the facility;

a payload handler coupled to the frame configured to transfer the payload between the vehicle's payload hold and a storage location for the payload in the storage array;

Using an auxiliary sensor system coupled to the frame for vehicle and operator collaboration, at least one camera generates image data informing objects and/or spatial features within at least a portion of the facility as viewed with vehicles at different locations within the facility. steps; and the auxiliary sensor system is, at least in part, a vision system comprising at least one camera arranged to capture image data, and the auxiliary sensor system is at least in part a vehicle attitude and A vehicle autonomous navigation/actuation sensor system configured to collect detection data embodying location information;

registering with a controller coupled to the frame and communicatively coupled to the auxiliary sensor system, and registering information from image data of the at least one camera; and

Using the controller, determine from the information the presence of a predetermined physical characteristic of at least one object or spatial feature, and in response thereto, move the vehicle from an autonomous state to a collaborative vehicle state arranged to receive operator commands for the vehicle. and selectively reconfiguring the vehicle to continue to operate effectively.

In accordance with one or more aspects of the disclosed embodiments, the predetermined physical characteristic is at least one object or spatial feature that determines whether at least one object or spatial feature is present in a vehicle crossing path across the crossing surface, across the crossing surface, or through the space of the vehicle, or at least a portion of another vehicle crossing the crossing surface. extends across.

According to one or more aspects of the disclosed embodiments, a controller is programmed with a reference representation of predetermined characteristics that at least partially define a facility through which a vehicle passes.

According to one or more aspects of the disclosed embodiments, the method further includes generating at least one image of at least one object or spatial feature showing predetermined physical characteristics from registered captured image data.

In accordance with one or more aspects of the disclosed embodiments, the at least one image is formatted as a virtual representation of the at least one object or spatial feature, comparing the virtual representation to one or more reference features of the predetermined feature of the reference representation. Includes more steps.

A method according to one or more aspects of the disclosed embodiments includes, based on a comparison between a virtual representation and a reference representation, identifying with a controller the presence of a predetermined physical feature of an object or spatial feature, determining dimensions of the predetermined physical feature, and making the comparison. and commanding the vehicle to stop on a predetermined trajectory based on the location of the object or spatial feature determined from.

A method according to one or more aspects of the disclosed embodiments includes maintaining an object or spatial reference within the field of view of at least one camera and performing continued imaging of predetermined physical characteristics, with the vehicle at a resting position within a predetermined trajectory, It further includes initiating a signal to another vehicle in at least one of an obstacle, an area to be avoided, or a detour area.

According to one or more aspects of the disclosed embodiments, the predetermined physical characteristics are calibrated and determine a position of the object within a frame of reference of at least one camera having a predetermined relationship to the vehicle, and determine a position of the object within the frame of reference of the at least one camera. The decision is made by the controller by determining the presence of predetermined physical characteristics of the object.

In accordance with one or more aspects of the disclosed embodiments, a controller may be configured to: identify the presence of a predetermined physical characteristic of at least one object or spatial feature and transfer information from an autonomous state to a collaborative vehicle state to an operator interface for operator collaborative operation of the vehicle; The transmitted communication image is configured to initiate identification of the presence of predetermined physical characteristics.

A method according to one or more aspects of the disclosed embodiments further includes applying, with a controller, a trajectory to the autonomous guided vehicle to bring the autonomous guided vehicle to zero speed within a predetermined time period, wherein the autonomous guided vehicle follows the trajectory. The movement of is coordinated with the positions of objects and/or spatial features.

According to one or more aspects of the disclosed embodiments, the capture of image data indicative of object and/or spatial features is an opportunity during payload hold of the vehicle or transfer of the payload to/from a storage location in a storage array.

According to one or more aspects of the disclosed embodiments, an autonomous guided vehicle:

The vehicle chassis on which the power supply is mounted and the power section connected to the chassis and each powered by the power supply include:

a drive section supporting the vehicle chassis and having motorized running wheels arranged to traverse the autonomously guided vehicle on the transverse surface of the autonomously guided facility;

a payload handling section having at least one payload handling actuator configured such that operation of the at least one payload handling actuator affects payload delivery to the payload bed, vehicle chassis, and storage within the facility;

At least one of an autonomous attitude and navigation sensor, a payload handling sensor, and at least one peripheral motor, wherein the at least one peripheral motor is separate and distinct from each of the motors of the drive section and each actuator of the payload handling section. , peripheral electronic components; and

a controller communicatively coupled to the drive section, payload handling section, and peripheral section, respectively, to effectively perform each autonomous operation of the autonomous guided vehicle; and a comprehensive power management section, wherein the controller is communicatively coupled to the power source to monitor the charge level of the power source,

The comprehensive power management section is connected to each branch circuit of the driving section, the payload handling section, and the peripheral electronic device section from the power supply, and is connected to the power supply to the driving section, the payload handling section, and the peripheral electronic device section. It is characterized in that power is supplied to each branch circuit of the section.

A comprehensive power management section according to one or more aspects of the disclosed embodiments includes each branch switching each branch circuit on or off based on the demand charge level of each branch circuit with respect to the other branch circuits and the charge level available from the power supply. It is configured to manage the demand charge level of the circuit in a predetermined pattern.

In accordance with one or more aspects of the disclosed embodiments, a pattern is arranged to switch off branch circuits with a reduction in the level of charge available from the power supply, thereby maximizing the level of charge available from the power supply to the controller.

In accordance with one or more aspects of the disclosed embodiments, the predetermined pattern is to change the available charge from the power supply such that the available charge level indicated to the controller equals or exceeds the controller's desired charge level for a maximum period of time based on the available charge level of the power supply. It is arranged to switch off the branch circuits with a decrease in level.

According to one or more aspects of the disclosed embodiments, the controller has at least one of the following:

an autonomous navigation control section configured to register and maintain past and present autonomous guided vehicle state and attitude navigation information in volatile memory to determine and describe the current and predicted state, attitude and position of the autonomous guided vehicle; and

an automatic payload processing control section configured to register and maintain current payload identity, state, and attitude information (historical and current) in volatile memory;

The controller, in accordance with an indication from a comprehensive power management section of an impending decrease in the available charge level indicated by the power supply to the controller, below the required level of the controller, the controller determines the state and attitude of the autonomous guided vehicle. At least one of navigation information and payload identity, status, and attitude information held in each registry and memory of the corresponding controller section is configured as an initialization file that can be used when rebooting the controller.

In accordance with one or more aspects of the disclosed embodiments, the controller is configured to cause the controller to cease operation and enter hibernation upon instructions from the comprehensive power management section of an impending decrease in available charge level, as indicated by a power supply to the controller. .

According to one or more aspects of the disclosed embodiments, when the controller indicates from the comprehensive power management section an impending decrease in the available charge level from the power supply to the branch circuit of the drive section, the controller determines the predetermined secondary path and secondary trajectory. It is configured to command the drive section to drive the autonomous guided vehicle along (to a predetermined autonomous guided vehicle assistance stopping location within the facility).

In accordance with one or more aspects of the disclosed embodiments, the controller is configured to: upon an indication from the comprehensive power management section of an impending decrease in the available charge level directed from a power supply to a branch circuit of the payload handling section, the controller configures the payload handling actuator to: , and instruct the payload handling section to move any payload thereon to a predetermined safe payload location within the payload bed.

According to one or more aspects of the disclosed embodiments, the controller includes at least one of the following:

vehicle health monitor,

drive section health monitor,

payload processing section status monitor, and

Peripheral electronics section health monitor; and

State registration section; and

The controller, upon reboot of the controller, monitors the vehicle status, the driving section, upon indication from the comprehensive power management section of an impending decrease in the available charge level indicated by the power supply to the controller below the demand of the controller. The stored status information from at least one of a status monitor, the payload handling section status monitor, and the peripheral electronic section status monitor is configured as an initialization file of the controller.

According to one or more aspects of the disclosed embodiments, the power supply is an ultra-capacitor or the charge level is a voltage level.

According to one or more aspects of the disclosed embodiments, a method for autonomous guided vehicle power management is provided. This method includes:

It provides an autonomous guided vehicle having a vehicle chassis equipped with a power supply and a power section connected to the chassis and each powered by a power supply, the power section comprising:

a drive section supporting the vehicle chassis and having motorized running wheels arranged to traverse the autonomously guided vehicle on the transverse surface of the autonomously guided facility;

a payload handling section having at least one payload handling actuator configured such that operation of the at least one payload handling actuator affects payload delivery to the payload bed, vehicle chassis, and storage within the facility;

At least one of an autonomous attitude and navigation sensor, a payload handling sensor, and at least one peripheral motor, wherein the at least one peripheral motor is separate and distinct from each of the motors of the drive section and each actuator of the payload handling section. , peripheral electronic components; and

performing respective autonomous operations of the autonomous guided vehicle using a controller communicatively coupled to each of the drive section, payload handling section, and peripheral section; and

The charge level of the power supply is monitored through a comprehensive power management section of the controller, said comprehensive power management section being connected to each branch circuit of the drive section, payload handling section and peripheral electronics section, each of which is connected to a power supply section. Provides power to the drive section, payload handling section, and peripheral electronics section from the supply unit, and the comprehensive power management section provides power to the branch circuits based on the level of charge available from the power supply depending on the demand level of each branch circuit. It includes steps to manage consumption.

A comprehensive power management section according to one or more aspects of the disclosed embodiments includes each branch switching each branch circuit on or off based on the demand charge level of each branch circuit with respect to the other branch circuits and the charge level available from the power supply. The demand charge level of the circuit is managed in a predetermined pattern.

In accordance with one or more aspects of the disclosed embodiments, a pattern is arranged to switch off branch circuits with a reduction in the level of charge available from the power supply, thereby maximizing the level of charge available from the power supply to the controller.

According to one or more aspects of the disclosed embodiments, the predetermined pattern is arranged to turn off the branch circuit as the available charge level from the power supply decreases such that the available charge level to the controller is equal to or greater than the available charge level from the power supply. Based on the charge level, the demand charge level of the controller is exceeded for a maximum period of time.

According to one or more aspects of the disclosed embodiments, the method further includes at least one of the following:

Using the autonomous navigation control portion of the controller, registering and maintaining volatile memory autonomous guided vehicle state and attitude navigation information (historical and current) and conclusively describing the current and predicted state, attitude, and position of the autonomous guided vehicle; and

registering and maintaining current payload identity, state and attitude information (historical and current) in volatile memory using the automatic payload processing control section of the controller;

When the controller receives an indication from the comprehensive power management section of an impending decrease in the available charge level indicated by the power supply to the controller below the demand level of the controller, the controller receives the autonomous guided vehicle status and attitude navigation information and At least one of the payload identity, status, and posture information held in each registry and memory of the corresponding controller section is configured as an initialization file that can be used when the controller is rebooted.

According to one or more aspects of the disclosed embodiments, when the comprehensive power management section indicates an impending decrease in the level of available charge from the power supply to the controller, to a level lower than the controller's required level, the controller suspends operation and hibernates. Go in.

In accordance with one or more aspects of the disclosed embodiments, upon indication from the power supply to the branch circuit of the drive section, from the comprehensive power management section of the impending decrease in the available charge level, the controller, in accordance with one or more aspects of the disclosed embodiment, directs the autonomous guided vehicle to predetermined auxiliary paths and auxiliary trajectories. Commands the drive section to navigate to a predetermined autonomous guided vehicle secondary stop location within the facility.

In accordance with one or more aspects of the disclosed embodiments, the controller, in accordance with instructions from the comprehensive power management section of the impending decrease in available charge level, from the power supply to the branch circuit of the payload handling section, controls the payload handling actuator and the same. Commands the payload handling section to move any of the above payloads to a predetermined safe payload location within the payload bed.

According to one or more aspects of the disclosed embodiment method of claim 11,

Provide at least one of the following to the controller:

vehicle health monitor,

drive section health monitor,

payload processing section status monitor, and

Peripheral electronics section health monitor; and

State registration section; and

When the controller receives an indication from the comprehensive power management section of an imminent decrease in the available charge level indicated by the power supply to the controller below the controller's required level, the controller monitors the vehicle health. , the stored status information from at least one of the driving section status monitor, the payload handling section status monitor, and the peripheral electronic product section status monitor is configured into an initialization file that can be used when the controller is rebooted.

According to one or more aspects of the disclosed embodiments, the power supply is an ultra-capacitor or the charge level is a voltage level.

The foregoing aspects and other features of the disclosed embodiments are described in the following description in conjunction with the accompanying drawings.
1A is a schematic block diagram of an exemplary storage and retrieval system facility incorporating aspects of the disclosed embodiments.
FIG. 1B is a top view of the exemplary storage and retrieval system facility of FIG. 1A incorporating aspects of the disclosed embodiments.
FIG. 2 is an exemplary perspective view of an autonomous guided vehicle of the exemplary storage and retrieval system facility of FIG. 1A in accordance with aspects of the disclosed embodiments.
3A and 3B are exemplary perspective views of portions of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 4A is an example plan view of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 4B is an example perspective view of a portion of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 4C is an example perspective view of a portion of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 4D is an example plan view of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
5A, 5A, and 5C are illustrative examples of attitude and position estimation according to aspects of the disclosed embodiments.
FIG. 6 is an example plan view of a portion of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
7A and 7B are top and perspective views, respectively, of a case unit illustrating shelf invariant front detection according to aspects of the disclosed embodiments.
FIG. 8 is an illustrative illustration of data captured by the auxiliary sensor system of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 9A is an example stereo vision image captured by the assistive sensor system of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 9B is an example augmented stereo vision image captured by the assistive sensor system of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 10A is an example augmented image captured by the assistive sensor system of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 10B is an example augmented stereo vision image captured by the assistive sensor system of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 11 is an example block diagram illustrating sensor selection according to operating mode of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
12 is an example flow diagram according to aspects of the disclosed embodiments.
FIG. 13 is an example flow diagram of vision analysis performed by the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
14 is an example flow diagram according to aspects of the disclosed embodiments.
FIG. 15 is an example image captured by the assistive sensor system of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 16 is an example flow diagram of image analysis performed by the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 17 is an example flow diagram of image analysis performed jointly with an operator and an auxiliary sensor system of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 18 is an example flow diagram of image analysis performed jointly with an operator and an auxiliary sensor system of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 19 is an example schematic block diagram of a portion of the autonomous transportation vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 20 is an example schematic block diagram of a portion of the autonomous guided vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 21 is an example schematic charging logic block diagram for the autonomous transportation vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
FIG. 22 is an example protection circuit for the autonomous transportation vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
23 is an example flow diagram according to aspects of the disclosed embodiments.
Figure 24 is an example flow diagram according to aspects of the disclosed embodiments.
25A, 25B, and 25C collectively illustrate example schematics of the control system of the autonomous transportation vehicle of FIG. 2 in accordance with aspects of the disclosed embodiments.
Figure 26 is an example schematic diagram of a portion of the control system of Figures 25A, 25B, and 25C in accordance with aspects of the disclosed embodiments.
Figure 27 is an example flow diagram according to aspects of the disclosed embodiments.
Figure 28 is an example flow diagram according to aspects of the disclosed embodiments.

1A and 1B illustrate an example automated storage and retrieval system 100 in accordance with aspects of the disclosed embodiments. Although aspects of the disclosed embodiments will be described with reference to the drawings, it should be understood that aspects of the disclosed embodiments may be implemented in various forms. Additionally, any suitable size, shape or type of element or material may be used.

Aspects of the disclosed embodiments include an autonomous transport vehicle 110 (herein referred to as: (also known as autonomous guided vehicles). The autonomous transport vehicle 110 may receive supplementary information from the physical characteristic sensor system 276 to verify and increase the accuracy of the vehicle navigation attitude or position and the payload attitude or position. Includes an auxiliary navigation sensor system 288.

In accordance with aspects of the disclosed embodiments, the secondary navigation sensor system 288 provides for a reduction in case unit transport errors (e.g., compared to automatically transporting a case unit in a conventional vehicle lacking the secondary sensor system described herein). ) and a vision system 400 that effects an increase in operational efficiency of the storage and retrieval system 100.

Aspects of the disclosed embodiments also provide an autonomous transport vehicle 110 having an autonomous navigation/actuation sensor system 270 that influences at least in part the determination of at least one of a vehicle navigation attitude or position and a payload attitude or position. Autonomous transport vehicle 110 may detect a preview of at least one object or spatial feature 299 (e.g., see FIGS. 4D and 15 ) within at least a portion of facility 100 on which autonomous transport vehicle 110 is operating. Further comprising an auxiliary or redundant hazard sensor system 290 that supplements information from the automatic navigation/motion sensor system 270 to opportunistically determine or identify the presence of the determined physical feature (i.e., a controller ( 122) is programmed to command the autonomous delivery vehicle to different locations within the facility associated with executing one or more predetermined payload autonomous delivery operations). The vehicle operates using the navigation system to move to another location and perform a predetermined transmission task at another location that is separate and distinct from the image data captured by the auxiliary hazard sensor system 290 at the other location. Opportunistic determination/identification of the presence of predetermined physical characteristics of objects or spatial features 299 that are incidental or peripheral to the vehicle 110 performing the operation and transfer allows the controller 122 to autonomously control the autonomous transportation vehicle 110. selective reconfiguration from the state to the cooperative vehicle state, finalizing the identification of the object or spatial feature 299 as a hazard and identifying the vehicle's mitigation actions for the hazard (i.e., the cooperative state is one of the vehicle's autonomous states). Assisted (reserve), in the autonomous state the vehicle automatically executes each of one or more predetermined autonomous payload transfer tasks and in the reserve/cooperative state the vehicle cooperates with the operator to identify and mitigate hazards as described herein. ).

Note that secondary navigation sensor system 288 and secondary hazard sensor system 290 may be used together or separately from each other and may form a common vision system 400 or separate vision systems. In another aspect, secondary hazard sensor system 290 may include sensors from secondary navigation sensor system 288 or vice versa (i.e., secondary navigation sensor system 288 and secondary hazard sensor system 290 ) shares a common sensor between the two sensor systems).

According to aspects of the disclosed embodiments, autonomous delivery vehicle 110 may include at least stereo vision to focus on at least a payload bed (or bay or area) 210B of autonomous delivery vehicle 110 and a controller (storage and retrieval system). A human operator of storage and retrieval system 100 (such as one or more of the control server 120 of 100, the controller 122 of autonomous transport vehicle 110, or any other suitable controller) may control the payload bed ( Monitor the movement of the case unit (CU) to or from 210B). Autonomous transport vehicle 110 includes one or more imaging radar systems that independently measure the size and center point of the front surface of a case unit (CU) disposed in storage space 130S on a storage shelf of storage level structure 130L. As described herein, the autonomous transport vehicle may use one or more other navigation tools to perform case unit delivery to payload bed 210B and navigation of autonomous transport vehicle 110 throughout each storage structure level 130L. and/or may include a vision sensor. As described below, from an auxiliary sensor system, an imaged or displayed object described by one or more of the auxiliary information, auxiliary vehicle navigation attitude or position, and auxiliary payload attitude or position is one of the auxiliary information, auxiliary vehicle navigation attitude or position. Through one or more of the peripheral facility features and interface facility features, one or more of the reference models (or maps) are applied (fitted/combined) to the model (400VM), such as an auxiliary vehicle navigation attitude or location information, and a payload attitude. or one or more auxiliary payloads of location information to improve attitude or location resolution.

For example, referring to FIG. 4A, the autonomous transport vehicle 110 may store a storage level floor map and storage structure information (e.g., a virtual model 400VM of the location of the storage structure level, a storage structure level (e.g., Detection of storage structure level virtual model (400VM), storage shelves, storage buffers, virtual model of storage structure level locations (400VM), storage structure level locations (e.g. virtual model of storage structure level (400VM)) a forward-looking stereo (e.g., relative to the autonomous transport vehicle 110) and a rear-looking stereo (e.g., direction of movement) configured to affect the localization of the autonomous transport vehicle 110 within the storage structure level 130L. For) may include a vision system. Here, the storage level map (or model) and storage structure information implement the location(s) of the navigation marker, so that upon recognition of the marker by the vision system 400, the autonomous transportation vehicle 110 ) determines a localized location within storage and retrieval system 100. Autonomous transport vehicle 110 detects any suitable navigation marker or fiducial located on the ceiling of storage structure level 130L and creates a storage level floor map. and one or more cameras facing upward to use the storage structure information to determine the location of autonomous transport vehicle 110. Autonomous transport vehicle 110 may be positioned at transfer deck 130B. and transfer deck ( It may include at least one side-facing traffic monitoring camera configured to monitor autonomous transport vehicle traffic along 130B).

Referring to FIG. 4D , autonomous delivery vehicle 110 may also monitor areas or spaces along the autonomous path of autonomous delivery vehicle 110 (autonomous sensor system 270), for example, within storage structure level 130L. assistance) and forward gaze (e.g., of an autonomous transport vehicle 110) configured for imaging (available continuously or periodically) to detect any objects/hazards that may encroach on the bot's path of movement. may include an omnidirectional (x, y, z, θ) vision system (e.g., with respect to the direction of movement) and/or a backward-looking (e.g., with respect to the direction of movement) vision system. In one aspect, as described below, vision system 400 resides on (e.g., onboard) autonomous transport vehicle 110 to enable monitoring and detection (of objects/hazards) by controller 122. ) can effectively perform imaging for auxiliary monitoring and detection, which can be achieved, for example, by virtualizing the location of reference storage level floor maps and storage structure information (e.g. columns, storage shelves, storage buffers, floor joints, etc.) Model (400VM); and the cooperation of a remote operator remotely accessing the vision system 299 to effect cooperative monitoring/detection/verification/identification/mitigation from indication by the controller 122 of such detections and with the vehicle 110 in a cooperative state. It can be performed by (see FIG. 15). If the vision system 400 of the autonomous transport vehicle 110 detects or detects the presence of an object/hazard that is not present in the reference storage level map and storage structure information, the object(s)/hazard(s) type(s) The decision is executed according to the indication of the controller by the remote operator who receives the image/video of the object/hazard transmitted to the user interface UI from/by the autonomous transport vehicle 110.

As described herein, autonomous transport vehicle 110 includes a vision system controller 122VC mounted on the autonomous transport vehicle and communicatively coupled to a vision system 400 of autonomous transport vehicle 110. Vision system controller 122VC may simulate/model storage and retrieval system 100 (e.g., computer-aided drafting (CAD) data of storage and retrieval system structures or storage and retrieval system 100). data acquired with the vision system 400 (based on any suitable information, such as other suitable data stored in memory or accessible by the vision system controller 122VC) to effect modeling/simulation of the vision system 100. It consists of model-based vision, characterized by performing one or more bot localization and imaging of objects/hazards compared to a model/simulation of the storage and retrieval system structure. wherein the autonomous transport vehicle 110 provides a substantially direct comparison between what the autonomous transport vehicle 110 “sees” with the vision system 400 and what the autonomous transport vehicle 110 expects to “see” based on a simulation of the (reference) storage and retrieval system structure. It is composed.

The auxiliary sensor system also provides remote control of the autonomous delivery vehicle 110 described herein as well as augmented reality operator inspection of the storage and retrieval system environment.

According to aspects of the disclosed embodiments, the secondary navigation sensor system 288 and/or the secondary hazard sensor system 290 are present within the facility 100 as being “unknown” (i.e., unidentifiable) by the autonomous transport vehicle 110. Transmission of images/videos (e.g. live) to remote operators for identification of objects/hazards (e.g. objects extending across the bot's path of movement, blocking the bot, closing the bot within a predetermined distance) and a vision system 400 that effectively performs video streaming, time-stamped images, or any other suitable transmission method. According to aspects of the disclosed embodiments, a controller (e.g., control server 120 of storage and retrieval system 100, controller 122 of autonomous transport vehicle 110, vision system controller 122VC, or other suitable controller) Alternatively, the human operator of the storage and retrieval system 100 may use the vision system 400 to determine the bot's movement path as the autonomous transport vehicle 110 navigates the facility to perform automated storage and retrieval operations in accordance with controller 122 commands. monitor. Additionally, incidental to influencing autonomous operations, vehicle 110 may also detect bot operations and/or traffic from other bots within facility 100 (based on predetermined initially identified criteria programmed in controller 122). opportunistically discovers objects/hazards that could interfere with the ), but is configured to match secondary and peripheral actions).

1A, 1B, and 19, aspects of the disclosed embodiment provide an automated storage and retrieval system 100 with an autonomous transport vehicle 110. Each autonomous transport vehicle 110 is comprised of a comprehensive power management section 444 (also referred to herein as a power distribution unit - see FIG. 19). Power distribution unit 444 is configured to manage the power demands of autonomous transport vehicle 110 to preserve higher level functions/operations of autonomous transport vehicle 110, where higher level functions are maintained by autonomous transport vehicle (110). 110) is preserved depending on the power supply level. For example, control and driving operations may be preserved to allow autonomous transport vehicle 110 to travel to a charging station or maintenance location, while other lower-level functions of the autonomous transport vehicle (e.g., transport to a charging station or maintenance site) may be preserved. (not required for location) stops. Managing the low level systems of the autonomous transport vehicle 110 preserves the charge of the onboard vehicle power source, improving the operating time of the autonomous transport vehicle 110 between charging operations and preserving autonomous transport vehicle controller functionality.

The power distribution unit 444 may also be configured to control the charging mode of the autonomous transport vehicle's power supply 481 to maximize the number of charging cycles of the power supply 481. Power distribution unit 444 monitors current draw for components of autonomous transport vehicle 110 (e.g., motors, sensors, controllers, etc. communicatively coupled to power source 481 in a “branch circuit”) and Manages (e.g., switches on and off) the power supply 481 by turning the power supply 481 on or off to each component to save on costs (e.g., energy usage).

Power distribution unit 444 is a loop device or loop power device that provides electrical circuit fault protection (e.g., short circuit protection, overvoltage protection) for components of autonomous transport vehicle 110 that are communicatively connected to power supply 481. , overcurrent protection, etc.). Here, a loop power device refers to an electronic device that is connected to a transmitter loop, such as a current loop, and uses power from the current flowing in the loop for operation without the need for a separate or independent power source.

According to aspects of the disclosed embodiments, the automated storage and retrieval system 100 of FIGS. 1A and 1B may be used to store, for example, a retail distribution center to fulfill orders received from retailers for replenishment merchandise shipped in cases, packages, and/or parcels. Or it can be placed in a warehouse. The terms case, package and parcel are used interchangeably herein and, as previously mentioned, can be any container that can be used for shipping and can be filled by the producer with a case or more units of product. As used herein, case refers to a case, package or parcel unit that is not stored (e.g., contained) in a tray, tote, etc. Case units (CUs) (also referred to herein as mixed cases, cases, and shipping units) are individual units of items/units that can be removed from or placed on a case (e.g., soup cans, cereal boxes, etc.) or pallet. Note that it may include cases of items/units. According to an exemplary embodiment, a shipping case or case unit (e.g., carton, barrel, box, crate, jug, shrink-wrapped trays or groups or any other suitable mechanism for retaining case units) can have variable sizes and can be used to retain case units during shipping and can be configured so that they can be palletized for shipping. Case units are unpacked from their totes, boxes and/or original packaging or disposed of (commonly referred to as breakpack items) and mixed or placed in totes, boxes and/or containers with one or more other individual items of a common type at an order fulfillment station. It may include a container of one or more individual items contained in a. For example, when an incoming bundle or pallet (e.g., from a manufacturer or supplier of case units) arrives at the storage and retrieval system for replenishment of the automated storage and retrieval system 100, the contents of each pallet may be uniform. (e.g., each pallet holds a predetermined number of identical items - one pallet holds soup and another pallet holds cereal). As may be realized, cases of such pallet loads may be substantially similar or, in other words, may be uniform cases (e.g., similar dimensions) and may have the same SKU (otherwise, as previously noted). As noted, the pallets may be “rainbow” pallets with layers formed into uniform cases). When a case or tote leaves the storage and retrieval system with a pallet filling a replenishment order, the pallet may contain any suitable number and combination of different case units (e.g., each pallet may contain a different type of case unit). (The pallet can accommodate a combination of canned soup, cereal, beverage packs, cosmetics and household cleaners). Cases combined on a single pallet may have different sizes and/or SKUs.

The automated storage and retrieval system 100 can generally be described as a storage and retrieval engine 190 connected to a palletizer 162. 1A and 1B, storage and retrieval system 100 may be configured to be installed in, for example, an existing warehouse structure or may be applied to a new warehouse structure. As previously mentioned, the automated storage and retrieval system 100 shown in FIGS. 1A and 1B is representative, e.g., in-feed and out terminating on respective transfer stations 170 and 160. -May include a feed conveyor, lift module(s) 150A, 150B, a storage structure 130, and a number of autonomous transportation vehicles 110 (also referred to herein as “bots”). It should be noted that the storage and retrieval engine 190 is formed by at least a storage structure 130 and an autonomous transport vehicle 110 (and in some embodiments lift modules 150A, 150B; in other embodiments the entirety of which is described herein). Lift module 150A in addition to storage and retrieval engine 190 disclosed in U.S. Patent Application Serial No. 17/091,265, filed June 2020 and entitled “Pallet Building System with Flexible Sequencing,” incorporated by reference. , 150B) can form a vertical sequencer). In an alternative aspect, storage and retrieval system 100 also includes a robot or bot transfer station (not shown) that can provide an interface between autonomous transport vehicle 110 and lift module(s) 150A, 150B. It can be included. The storage structure 130 includes a plurality of levels of storage rack modules, each storage structure level 130L of the storage structure 130 including a respective picking passage 130A, and a storage area of the storage structure 130. It may include transfer decks 130B for transferring the case unit between any of the shelves of the lift module(s) 150A, 150B. In one aspect, picking aisle 130A is configured to provide guided movement (e.g., along rails 130AR) of autonomous transport vehicle 110, while in another aspect, picking aisle 130A is configured to provide guided movement of autonomous transport vehicle 110. It is configured to provide unrestricted movement (e.g., the picking aisle is open and non-deterministic with respect to the guidance/movement of the autonomous transport vehicle 110). Transport deck 130B has an open, indeterministic bot-supported moving surface over which autonomous transport vehicle 110 moves under guidance and control provided by bot steering (described herein). In one or more aspects, the transfer deck has multiple lanes through which autonomous transport vehicles 110 freely switch to access pickup aisle 130A and/or lift modules 150A, 150B. As used herein, “open and non-deterministic” means that the moving surface of the picking aisle and/or the moving deck are mechanical restraints (guides) that limit the movement of autonomous transport vehicle 110 to any given path along the running surface. This means that it does not have rails, etc.). Note that although aspects of the disclosed embodiments have been described with respect to multi-level storage arrays, aspects of the disclosed embodiments are equally applicable to single level storage arrays positioned on or raised above the facility floor.

Pick aisle 130A and transfer deck 130B may also allow autonomous transport vehicle 110 to place case units (CUs) in picking inventory and retrieve ordered case units (CUs) (and one or more other locations). can be any number of locations within the storage structure, but defines other locations where the bot performs autonomous tasks). In an alternative aspect, each level may also include a respective bot transfer station 140. Autonomous transport vehicle 110 may place case units, such as retail merchandise described above, into picking inventory within one or more storage structure levels 130L of storage structure 130 and then place the ordered case units, e.g., in a store or other appropriate location. It may be configured for selective retrieval for delivery to a location. In-feed transfer station 170 and out-feed transfer station 160 are used to transfer case units (CU) to one or more storage structure levels 130L of storage structure 130 in each direction. )(150A, 150B). While lift modules 150A, 150B may be described as being dedicated inbound lift module 150A and outbound lift module 150B, in alternative embodiments, lift modules 150A, 150B each have a storage and retrieval system ( Note that it can be used for both inbound and outbound transfer of case units from 100).

As may be realized, storage and retrieval system 100 may include, for example, a plurality of in-feed and out-feed lift modules 150A accessible by autonomous transport vehicle 110 of storage and retrieval system 100. , 150B), including one or more case units (e.g., case unit(s) not held in a tray) or received (in a tray or tote) at each level from lift modules 150A, 150B. It can be delivered to the storage space of and from each storage space to any one of the lift modules 150A and 150B at each level. Autonomous transport vehicle 110 is positioned in a storage space 130S (e.g., in a pickup aisle 130A or another suitable storage space/case unit buffer disposed along the transfer deck 130B) and a lift module 150A. 150B) can be configured to transfer the case unit. Generally, the lift modules 150A, 150B have at least one device capable of moving the case unit(s) between the in-feed and out-feed transfer stations 160, 170 and each level of the storage space where the case units are stored and retrieved. Contains one movable payload support. The lift module(s) may have any suitable configuration, such as a reciprocating lift or any other suitable configuration. Lift module(s) 150A, 150B may be connected to any suitable controller (control server 120 or other connected to control server 120, warehouse management system 2500, and/or palletizer controllers 164, 164'). and a suitable controller) and capable of forming a sequencer or sorter in a manner similar to that described in U.S. Patent Application Serial No. 16/444,592, filed June 18, 2019, entitled “Vertical Sequencer for Product Order Fulfillment.” there is.

The automated storage and retrieval system may include, for example, one or more control servers communicatively connected via a suitable communication and control network 180 to the in-feed and out-feed conveyors and transfer stations 170, 160, and lift module 150A. It may include a control system including (120). Communication and control network 180 may provide various programmable functions, such as commanding the operation of in-feed and out-feed conveyors and transfer stations 170, 160, lift modules 150A, 150B, and other suitable system automation. It can have any suitable architecture capable of incorporating a logic controller (PLC). Control server 120 may include advanced programming that impacts a case management system (CMS) that manages the case flow system. Network 180 may further include suitable communications to implement a two-way interface with autonomous transport vehicle 110 . For example, autonomous delivery vehicle 110 may include an onboard processor/controller 122 (configured to perform at least the control and safety functions of autonomous delivery vehicle 110 - see also FIGS. 10A-10C). Network 180 may enable autonomous transport vehicle controller 122 to request or receive commands from control server 120 to perform desired transport (e.g., placing into or retrieving from a storage location) a case unit; Contains a suitable two-way communications suite enabling transmission of desired autonomous transport vehicle 110 information and data, including autonomous transport vehicle 110 ephemeirs, status, and other desired data, to the control server 120. can do. 1A and 1B, control server 120 further supports warehouse management system 2500, for example, to provide inventory management and customer order fulfillment information to CMS level programs in control server 120. can be connected As previously mentioned, control server 120 and/or warehouse management system 2500 allows at least some degree of cooperative control of bots 110 via a user interface UI, as further described below. A suitable example of an automated storage and retrieval system arranged to retain and store case units is described in U.S. Patent No. 9,096,375, issued August 4, 2015, the disclosure of which is incorporated herein by reference in its entirety. do.

1A, 1B, and 2, an autonomous transport vehicle 110 (also referred to herein as an autonomous guided vehicle or bot) includes a vehicle frame or chassis 200 equipped with a power supply 481 and an integrated payload. It includes a rod support or bed 210B (referred to herein as a frame). Frame 200 has a front end 200E1 and a rear end 200E2 that define the longitudinal axis LAX of autonomous transport vehicle 110 . Frame 200 may be constructed of any suitable material (eg, steel, aluminum, composite, etc.). As described herein, the power supply sections are coupled to frame 200, where each power supply section is powered by a power supply 481. The power supply section includes a drive section 261D, a payload handling section 210 (also referred to herein as case handling assembly 210), and a peripheral electronics section 778.

Payload handling section or case handling assembly 210 is configured to handle cases/payloads transported by autonomous transport vehicle 110 . Case handling assembly 210 includes at least one payload handling actuator (e.g., transfer arm 210A) such that actuation of the payload handling actuator is directed to and from the payload bed 210B of the frame. Payload from 210B), and is configured to effect transfer to storage within the facility (e.g., storage space 130S of a storage shelf). In some aspects, case handling assembly 210 includes a payload bed 210B (also referred to herein as a payload bay or payload hold) and is configured to move the payload bed in direction VER; In another aspect where the payload bed 210B is formed by the frame 200, the payload bed may be fixed/stationary in the direction VER. As realized, the payload is placed in payload bed 210B.

The transfer arm 201A, when a flat amorphous seating surface is seated on the payload bed 210B, is positioned to store the payload CU in the storage array SA (e.g., the storage shelf shown in Figure 5A). Payload (e.g., storage space 130S on top, shelves, buffers, transfer stations and/or other suitable storage locations on lift modules 150A, 150B) and payload bed 210B on autonomous guided vehicle 110 case unit (CU)), and the storage location 130S in the storage array SA is separate and distinct from the transfer arm 210A and the payload bed 210B. Transfer arm 210A is configured to extend laterally in direction LAT and/or vertically in direction VER to transfer payload to and from payload bed 210B. It consists of extension motors 667A-667C and lift motor(s) 669. The payload bed 210B includes front and rear alignment modules 210ARJ, 210AFJ configured to align case units along the longitudinal axis LAX and laterally in the direction LAT at any location within the payload bed 210B. Includes. For example, the payload bed has an alignment arm (JAR) driven along a longitudinal axis by respective alignment motors 668B and 668E to align case unit(s) (CU) along the longitudinal axis (LAX) (FIG. 10A and Figure 10C). A pusher (JPS) and a puller (JPP) (FIGS. 10A and 10C) are attached to the alignment arm to be driven by respective motors 668A, 668C, 668D, and 668F to align the case unit(s) (CU) in the direction LAT. Can be movably mounted. One or more of the motors 668A-668F may also be operated to clamp or grip the case unit(s) CU secured to the payload bed 210B during transport of the case unit by vehicle 110.

Examples of suitable payload bed 210B and transfer arm 210A and/or autonomous delivery vehicles to which aspects of the disclosed embodiments may be applied are those described in Attorney Docket No. 1127P015753-US (-), which is incorporated herein by reference in its entirety. #3) and U.S. Provisional Patent Application No. 63/236,591, filed August 24, 2021, titled “Autonomous Transport Vehicle,” and issued July 26, 2012, titled “Autonomous Transport Vehicle with Transfer Arm.” U.S. Publication No. 2012/0189416 (U.S. Patent Application No. 13/326,952, filed Dec. 15, 2011); U.S. Patent No. 7591630, titled “Material Handling System Utilizing Autonomous Transfer and Transport Vehicle,” issued September 22, 2009; U.S. Patent No. 7991505, titled “Material Handling System Utilizing Autonomous Transfer and Transport Vehicle,” issued August 2, 2011; U.S. Patent No. 9561905, entitled “Autonomous Transport Vehicle,” issued February 7, 2017; U.S. Patent No. 9082112, entitled “Autonomous Transport Vehicle,” issued July 14, 2015; U.S. Patent No. 9850079, entitled “Autonomous Transport Vehicle Charging System,” issued December 26, 2017; U.S. Patent No. 9187244, entitled “Bot Payload Alignment and Detection,” issued November 17, 2015; U.S. Patent No. 9499338, entitled “Automated Bot Transfer Arm Activation System,” issued November 22, 2016; U.S. Patent No. 8965619, entitled “Bot with High-Speed Stability,” issued February 24, 2015; U.S. Patent No. 9008884, entitled “Bot Location Detection,” issued April 14, 2015; U.S. Patent No. 8425173, entitled “Autonomous Transport for Storage and Retrieval System,” issued April 23, 2013; The invention can be found in U.S. Patent No. 8696010, entitled “Suspension System for Autonomous Transport,” issued April 15, 2014.

Frame 200 includes one or more idler wheels or casters 250 disposed adjacent front end 200E1. Frame 200 also includes one or more drive wheels 260 disposed adjacent to rear end 200E2. In other aspects, the positions of the casters 250 and drive wheels 260 may be reversed (e.g., drive wheels 260 are disposed at the front end 200E1 and casters 250 are disposed at the rear end 200E2 ). Note that in some aspects, autonomous transport vehicle 110 is configured to move with front end 200E1 leading the direction of movement or rear end 200E2 leading the direction of movement. In one aspect, the front corners of frame 200 at front end 200E1 allow autonomous transport vehicle 110 to stably traverse transfer deck 130B and picking aisle 130A of storage structure 130. Casters 250A, 250B (substantially similar to casters 250 described herein) are located on each, and drive wheels 260A, 260B are located at each of the rear corners of frame 200 at rear end 200E2 ( (substantially similar to drive wheels 260 described herein) are positioned (eg, support wheels are positioned at each of the four corners of frame 200).

Autonomous transport vehicle 110 is connected to frame 200 and has a motor 261M that powers (or drives) drive wheels 260 (supporting autonomous transport vehicle 110 on traversing/rolling surface 284). ), wherein the drive wheels 260 move the autonomous transport vehicle 110 over the transversal surface 284 of the facility (e.g., warehouse, store, etc.) according to autonomous guidance. The command performs a vehicle crossing on the crossing surface 284. Drive section 261D has at least one pair of traction drive wheels 260 (also referred to as drive wheels 260 - see drive wheels 260A, 260B) spanning drive section 261D. The drive wheel 260 has a fully independent suspension 280 that couples each drive wheel 260A, 260B of the at least one pair of drive wheels 260 to the frame 200, and has at least one drive wheel ( At least one pivot link (described herein) interposed between 260A, 260B and frame 200 is connected to at least one drive wheel 260A, 260B and a rolling/moving surface 284 - e.g., Figure 4D. , see FIGS. 9A, 9B and 15) and is configured to maintain the traction contact patch in a substantially steady state. Suitable examples of fully independent suspension 280 include U.S. Provisional Patent Application No. 63/213,589, entitled “Autonomous Transportation Vehicle with Synergistic Vehicle Dynamic Response,” filed June 22, 2021 (attorney general). Document No. 1127P015753-US (having -#2), the disclosure of which is hereby incorporated by reference in its entirety.

As described above, and with reference to FIGS. 3A and 3B, frame 200 includes one or more casters 250 disposed adjacent front end 200E1. In one aspect, the casters 250 are positioned adjacent each front corner of the frame 200 and engage a drive wheel 260 disposed at each rear corner of the frame 200 to allow the frame 200 to move to the storage structure 130. ) stably traverses the transfer deck (130B) and the picking passage (130A). 2, 3A and 3B, in one aspect, each caster 250 includes a powered (e.g., active/electric steer) caster 600M; In another aspect, the caster 250 may be a passive (eg, non-powered) caster. In one aspect, powered caster 600M includes a caster wheel 610 coupled to a fixed configuration wheel fork 640 (Figure 3A); In another aspect the caster wheel 610 is coupled to a variable geometry or articulated (e.g. suspension) fork 740. Each powered caster 600M actively pivots its respective caster wheel 610 in a direction 690 about the caster pivot axis 691 (independent of the pivot of the other wheels of the other powered casters) for at least autonomous transport. It is configured to assist in changing the direction of movement of the vehicle 110. Suitable examples of casters have U.S. Provisional Patent Application No. 63/213,589 (previously incorporated herein by reference in its entirety), filed June 22, 2021, and Attorney Docket No. 1127P015753-US (-#5), filed June 22, 2021. The invention can be found in U.S. Provisional Patent Application No. 63/193,188, entitled “Autonomous Transportation Vehicle with Steering,” filed May 26, the disclosure of which is incorporated herein by reference in its entirety.

Autonomous transport vehicle 110 includes a physical characteristic sensor system 270 (also referred to as an autonomous sensor system) coupled to frame 200. Physical property sensor system 270 has electromagnetic sensors. Each of the electromagnetic sensors responds to the interaction or interface of the sensor, wherein the emitted or generated electromagnetic beam or field has physical properties (e.g., of a transient object such as a storage structure or case unit (CU), debris, etc.); Here the electromagnetic beam or field is disturbed by interaction or interface with physical properties. Disturbances in the electromagnetic beam are detected by electrical sensors of a physical nature and affect the detection, and the physical characteristic sensor system 270 provides information (about the storage and retrieval system or facility on which the autonomous transport vehicle 110 operates) and and generate sensor data embodying at least one of information (about storage location 130S or payload bed 210B).

Physical property sensor system 270 includes, for illustrative purposes only, laser sensor(s) 271, ultrasonic sensor(s) 272, barcode scanner(s) 273, position sensor 274, and line sensor. 275 , autonomous attitude and navigation sensors including vehicle proximity sensor 278 , or any other suitable sensor for detecting the position of vehicle 110 . For illustrative purposes, at least one payload handling sensor may be a case sensor 278 (e.g., a case unit within payload bed 210B mounted on vehicle 110 or off-board on vehicle 110). for detecting the case unit on a storage shelf, arm proximity sensor(s) 277, or a payload (e.g., case unit CU) and its position/position during autonomous transport vehicle handling of the payload (CU). and any other suitable sensor for detecting. In some aspects, secondary navigation sensor system 288 may form part of physical characteristic sensor system 270. Suitable examples of sensors that may be included in physical property sensor system 270 include U.S. Provisional Patent Application No. 63/236,591 and U.S. Patent No. 8,425,173, entitled “Automated Transport for Storage and Retrieval System,” issued April 23, 2013, entitled “Bot Location Detection,” issued April 14, 2015 Nos. 9,008,884 and 9,946,265, entitled “Bot with High-Speed Stability,” issued April 17, 2018, the disclosures of which are incorporated herein by reference in their entirety.

25A, 25B, and 25C, the sensors of physical property sensor system 270 may, for example, sense their environment (up to six degrees of freedom X, Y, Z, Rx, Ry, Rz - see FIG. 2). and an autonomous transport vehicle 110 having recognition of external objects and monitoring and control of internal subsystems. For example, the sensor has Attorney Docket No. 1127P015753-US(-#3), entitled “Autonomous Transport Vehicle,” and is described herein and/or the disclosure of which is hereby incorporated by reference in its entirety; It may provide guidance information, payload information, or other suitable information that can be used in the operation of autonomous transport vehicle 110, such as that disclosed in U.S. Provisional Patent Application No. 63/236,591, filed August 24, 2021.

With continued reference to FIG. 2 , barcode scanner(s) 273 may be mounted at any suitable location on autonomous transport vehicle 110 . Barcode scanner(s) 273 may be configured to provide an absolute location of autonomous transport vehicle 110 within storage structure 130. Barcode scanner(s) 273 may, for example, read barcodes located on the transfer deck and select aisle and transfer station floors to determine the location of the autonomous transport vehicle 110, thereby determining the location of the autonomous transport vehicle 110. It may be configured to determine location. Barcode scanner(s) 273 may also be configured to read barcodes located on items stored on shelf 555.

Position sensor 274 may be mounted at any suitable location on autonomous transport vehicle 110. Position sensor 274 may, for example, determine the position of vehicle 110 in relation to the shelving of picking aisle 130A (or transfer deck 130B or buffer/transfer station located adjacent to lift 150). may be configured to detect reference data characteristics (or slats, 520L of the storage shelf 555) for determining (e.g., see FIG. 5A). The reference data information is, for example, the control unit 122 It can be used to calibrate the vehicle's odomerty and to stop the support tines 210AT of the transfer arm 210A positioned for insertion into the space between the slats 520L (e.g. , see Figure 5A). In one exemplary embodiment, vehicle 110 includes position sensors 274 on driven (rear) end 200E2 and driven (front) end 200E1 of autonomous transport vehicle 110. This may allow reference data detection regardless of which end of the autonomous transport vehicle 110 is facing the direction in which the vehicle is moving.

Line sensor 275 may be positioned on frame 200 adjacent drive (rear) and driven (front) ends 200E2, 200E1 of autonomous transport vehicle 110, and may be positioned at any location, such as for illustrative purposes only. There may be any suitable sensors mounted on autonomous transport vehicle 110 at appropriate locations. For illustrative purposes only, line sensor 275 may be a diffuse infrared sensor. Line sensor 275 may be configured to sense, for example, a guide line 900 (see FIGS. 9A and 15 ) provided on the floor of transfer deck 130B. Autonomous transport vehicle 110 may be configured to follow guide lines when moving on transfer deck 130B and define the end of a turn when the vehicle transitions on or off transfer deck 130B. Line sensor 275 may also allow vehicle 110 to detect index references created by guide lines intersected by index references to determine absolute position (see FIGS. 9A and 15).

Case sensors 276 may include case overhang sensors and/or other suitable sensors configured to detect the position/orientation of case units (CUs) within payload bed 210B. Case sensor 276 may be any suitable sensor positioned on the vehicle such that the sensor(s) field of view(s) spans payload bed 210B adjacent the upper surface of support tines 210AT (FIGS. 4A and see Figure 4b). Case sensor 276 is positioned at an edge of payload bed 210B (e.g., payload bed 210B) to detect any case unit (CU) that extends at least partially outside of payload bed 210B. (adjacent to the transfer opening 1199 of 210B).

Arm proximity sensor 277 may be mounted on autonomous transport vehicle 110 at any suitable location, such as transfer arm 210A. Arm proximity sensor 277 detects transfer arm 210A and/or support tines 210AT of transfer arm 210A as transfer arm 210A is raised/lowered and/or support tines 210AT extend/retract. ) can be configured to detect surrounding objects.

The laser sensor 271 and the ultrasonic sensor 272 may be configured to autonomously transport the case unit before it is selected from, for example, a storage shelf 555 and/or a lift 150 (or any other location suitable for retrieving the payload). Autonomous transport vehicle 110 may be configured to position itself relative to each case unit that forms the load that vehicle 110 carries. The laser sensor 271 and ultrasonic sensor 272 may also allow the vehicle to locate itself relative to the empty storage location 130S to place the case unit in the empty storage location 130S. Laser sensor 271 and ultrasonic sensor 272 may also indicate that the storage space (or other load loading location) is empty before the payload carried by autonomous transport vehicle 110 is placed in storage space 130S, for example. The autonomous transport vehicle 110 can confirm. In one example, laser sensor 271 may be mounted on autonomous transport vehicle 110 at an appropriate location to detect the edges of items to be transferred to (or from) autonomous transport vehicle 110. The laser sensor 271 may be connected to, for example, a retro-reflective tape (or other suitable reflective tape) positioned to the rear of the storage shelf 555 to enable the sensor to "see" all the way to the rear of the storage shelf 555. surface, coating or material). Reflective tape located on the back of the storage shelf allows the laser sensor 1715 to be substantially unaffected by the color, reflectivity, circularity or other suitable characteristics of the items located on the shelf 555. Ultrasonic sensor 272 allows autonomous transport vehicle 110 to determine picking depth (e.g., the distance to travel within shelf 555 to pick item(s) from shelf 555). It may be configured to measure the distance from 110) to the first item. One or more of the laser sensor 271 and the ultrasonic sensor 272 may detect, for example, the autonomous transport vehicle 110 and the case unit to be picked as the autonomous transport vehicle 110 reaches a stop adjacent to the case unit to be picked. Measuring the distance between the front surfaces may allow detection of case orientation (e.g., skewing of cases within storage shelves 555). A case sensor may enable confirmation of the placement of a case unit on a storage shelf 555, for example, by scanning the case unit after it has been placed on the shelf.

Vehicle proximity sensor 278 may also be disposed on frame 200 to determine the position of autonomous transport vehicle 110 relative to pickup aisle 130A and/or lift 150. Vehicle proximity sensor 278 is positioned on rail 130AR along which vehicle 110 moves through picking aisle 130A (and/or on the walls of transfer area 195 and/or lift 150 access locations). positioned on the autonomous transport vehicle 110 to sense positioned targets or positioning characteristics. The positions of the targets on rail 130AR are known to form an incremental or absolute encoder along rail 130AR. Vehicle proximity sensor 278 detects targets and provides sensor data to controller 122 so that controller 122 determines the position of autonomous transport vehicle 110 along pickup aisle 130A based on the detected targets. do.

Sensors of physical characteristic detection system 270 are communicatively coupled to controller 122 of autonomous transportation vehicle 110 . As described herein, controller 122 is operably coupled to drive section 261D and/or transfer arm 210A. Controller 122 may monitor vehicle attitude and position of physical property sensor systems 270 (e.g., up to 6 It is configured to determine from the information of the degrees of freedom (X, Y, Z, Rx, Ry, Rz). Controller 122 may also determine from information about the attitude and position (on or off-board autonomous transport vehicle 110) of the payload (e.g., case unit (CU)) of physical characteristic sensor system 270. Configured, from the location of the payload CU to and from the storage location (e.g., lifting the case unit CU from the bottom of the case unit CU) and the storage location 130S within the payload bed 210B. Determines independent underpicks and positions.

As previously discussed and with reference to FIGS. 1A, 1B, 2, 4A and 4B, autonomous transport vehicle 110 includes an auxiliary or redundant navigation sensor system 288 coupled to frame 200. Auxiliary navigation sensor system 288 assists physical characteristic sensor system 270. The auxiliary navigation sensor system 288 may, at least in part, assist with information from the physical characteristics sensor system 270 (relative to the storage and retrieval system structure or facility on which the vehicle 110 operates) and payload attitude or position (storage A vision system 400 having a camera positioned to capture image data indicative of at least one of a position or (relative to) a payload bed 210B. As used herein, a “camera” refers to a two-dimensional camera, a two-dimensional camera with RGB (red, green, blue) pixels, a three-dimensional camera with the definition XYZ+A, where XYZ is the three-dimensional reference frame of the camera and A is the radar. a still imaging or video imaging device, including one or more of a return strength, time-of-flight stamp, or other range determination stamp/indicator) and an RGB/XYZ camera containing both RGB and three-dimensional coordinate system information, Note that limited examples are provided here. In another aspect, the vision system 400 is described herein with respect to an autonomous transportation vehicle 110, and in another aspect the vision system 400 includes a load handling device 150LHD of a vertical lift 150 (FIG. 1 - autonomous transportation vehicle 110). It should be understood that it may be applied to a payload bed 210B of a vehicle 110) or a pallet builder of infeed transfer stations 170. A suitable example of a load handling device for a lift in which the vision system 400 may be integrated is described in U.S. Patent No. 10,947,060, entitled “Vertical Sequencer for Product Order Fulfillment,” issued March 16, 2021, The disclosure is hereby incorporated by reference in its entirety.

2, 4A, 4B, 25A, 25B, and 25C, vision system 400 includes one or more of the following: case unit monitoring cameras 410A, 410B (collectively, monitoring cameras ( 410), front navigation cameras 420A, 420B and rear navigation cameras 430A, 430B (collectively referred to herein as navigation cameras 430), one or more three-dimensional imaging systems 440A, 440B, one One or more case edge detection sensors 450A, 450B, one or more traffic monitoring cameras 460A, 460B (collectively referred to herein as traffic monitoring cameras 460), and one or more out-of-plane (e.g., upward-facing or downward-facing) Positioning cameras 477A, 477B (collectively referred to herein as positioning cameras 477) (the downward facing cameras assist line tracking sensors 275 of physical property sensor system 270); , if the vehicle path deviates from the guidance line 900 so as to remove the guidance line 900 from the field of view of the line following sensor 275, guiding/traversing the vehicle 110 into the field of view of the line following sensor 275. Provides a wider field of view than the line following sensor 275 for repositioning). The out-of-plane positioning camera 477 may be employed in conjunction with the line-following sensor 275 and provides a wider field of view than the line-following sensor 275 so that the autonomous transport vehicle 110 can be positioned within the detection area of the line-following sensor 275. If it deviates to a point outside of , the autonomous transport vehicle 110 can be placed back in the follow line. Images (static images and/or dynamic video images) from other vision system 400 cameras are requested from vision system controller 122VC by controller 122 as desired for any given autonomous transport vehicle 110 task. . For example, to perform navigation of the autonomous transport vehicle along the transfer deck 130B and pickup aisle 130A, the controller 122 is configured to receive signals from at least one of the front and rear navigation cameras 420A, 420B, 430A, 430B. The image is acquired by

As another example, controller 122 may acquire images from one or more of case edge detection sensors 450A, 450B and case unit monitoring cameras 410A, 410B, wherein case edge detection sensors 450A, 450B and Case unit monitoring cameras 410A and 410B are employed to effectively perform case handling by vehicle 110. Case handling may be performed from a case unit holding location (e.g., case unit localization, verification of the case unit, verification of the position of the case unit at a case unit holding location, such as payload bed 210B and/or a storage shelf or buffer location). Includes picking and placing units.

Images from the out-of-plane localization cameras 477A, 477B may influence navigation of the autonomous transport vehicle and/or supplement localization/navigation data from one or more of the front and rear navigation cameras 420A, 420B, 430A, 430B. may be obtained by the controller 122 to provide data (e.g., image data). Images from one or more traffic monitoring cameras 460A, 460B may be acquired by controller 122, where traffic monitoring cameras 460 are positioned on autonomous transport vehicle 110 from pickup aisle 130A to transfer deck 130B. ) is employed to perform movement transitions (e.g., entry into transfer deck 130B and joining of autonomous transport vehicle 110 with another autonomous transport vehicle moving along transfer deck 130B).

Case unit monitoring cameras 410A, 410B are any suitable high resolution or low resolution video cameras (video images containing more than about 480 vertical scan lines and captured at more than about 50 frames/second are considered high resolution). Case unit monitoring cameras 410A, 410B are arranged relative to each other to form a stereo vision camera system configured to monitor case unit (CU) entry and exit into payload bed 210B. Case unit monitoring cameras 410A, 410B are coupled to frame 200 in any suitable manner and are focused on at least payload bed 210B. In one or more aspects, the case unit monitoring cameras 410A, 410B are coupled to the transfer arm 210A to move in a direction LAT with the transfer arm 210A (e.g., when picking and placing the case unit CU). , the focus was on the payload bed 210B and the support tines 210AT of the transfer arm 210A.

5A, case unit monitoring cameras 410A, 410B perform case unit determination, case unit localization, case unit location verification, and case unit alignment characteristics (e.g., alignment blade 471 and pusher 470), and Affects case transport characteristics (e.g., at least partially one or more of tines 210AT, puller 472, and payload bed floor 473. For example, case unit monitoring cameras 410A, 410B) Detect one or more of case unit length (CL, CL1, CL2, CL3), case unit height (CH1, CH2, CH3), and case unit yaw (YW) (e.g., of the transfer arm 210A). (relative to stretch direction LAT). Data from case handling sensors (e.g., mentioned above) may also indicate where payload bed 210B is empty (e.g., not holding a case unit) such as a pusher (e.g., not holding a case unit). 470), the position/positions of the puller 472 and the alignment blade 471 may be provided.

The case unit monitoring cameras 410A, 410B also monitor the front case center point (FFCP) relative to the reference position of the autonomous transport vehicle 110 (e.g., with the case unit placed on a shelf in the X, Y, Z directions, or It is effectively configured with a vision system controller 122VC to determine other maintenance areas outside the vehicle 110. The reference position of autonomous transport vehicle 110 may be defined by one or more alignment surfaces of payload bed 210B or the center line (CLPB) of payload bed 210B. For example, the front case center point (FFCP) may be determined along the longitudinal axis (LAX) (e.g., in the Y direction) with respect to the centerline (CLPB) of the payload bed 210B (FIG. 4A). The front case center point (FFCP) may be determined along the vertical axis (VER) (e.g., Z direction) relative to the case unit support surface (PSP) of the payload bed 210B (FIG. 4B - tines of transfer arm 210A) (210AT) and payload bed floor 473). The front case center point (FFCP) may be determined along the lateral axis (LAT) (e.g., in the X direction) relative to the alignment plane (JPP) of the pusher 470 (FIG. 4A). Determination of the front case center point (FFCP) of a case unit (CU) located on a storage shelf (555) or other case unit holding positions determines the ratio for localization of the autonomous transport vehicle (110) with respect to the case unit (CU) to be selected. As a limiting example, it provides for mapping the location of case units within a storage structure (e.g., in U.S. Patent No. 9,242,800, entitled “Storage and Retrieval System Case Unit Detection,” issued January 26, 2016). In a manner similar to that described, the disclosure of which is incorporated herein by reference in its entirety, the disclosure provides selection and placement accuracy for other case units on storage shelf 555 (e.g., predetermined differences between case units). to maintain gap size). The determination of the front case center point (FFCP) also influences comparison of the “real world” environment in which the autonomous transport vehicle 110 is operating with the virtual model (400VM), allowing the controller 122 of the autonomous transport vehicle 110 to store and Based on a simulation of the recovery system architecture, it allows for a virtually direct comparison of what the vision system 400 "sees" and what the autonomous transport vehicle 110 "sees". Moreover, in one aspect, as illustrated in FIG. 5A , the object (case by case) and characteristics determined by the vision system controller 122VC are enhanced resolution virtual models (400VM) of the object pose in up to 6 degrees of freedom relative to the facility reference frame. It is joined (combined, overlaid) to. As may be realized, registration of the cameras of the vision system 400 with a fixture reference frame can determine the vehicle 110 pose with respect to both a global reference (fixture features rendered in the virtual model 400VM) and the imaged object. and/or allow for improved resolution of location. More specifically, when combining the object image and the virtual model, any apparent and identifiable object position inconsistencies or anomalies (e.g., edges between case unit reference edges to rack slats (520L) in the virtual model (400VM) spacing or case unit inclination or indication), if greater than a predetermined nominal threshold, describes a malposition of one or more of the cases, racks and/or vehicles 110. Identification of whether there is an error in the attitude/position of the case, rack, or vehicle 110 is determined through comparison with attitude data from sensors 270 and auxiliary navigation sensor system 288.

As an example of the improved resolution described above, one case unit placed on a shelf imaged by the vision system 400 is compared to a case unit juxtaposed on the same shelf (also imaged by the vision system) to produce a virtual model (400VM). When turned, vision system 400 determines that one case is biased and manipulates transfer arm 210A and positions transfer arm 210A to select one case based on improved resolution of case posture and position. In order to do this, improved case location information can be provided to the controller 122. As another example, if the edge of the case is offset more than a predetermined threshold from the edge of the slat 520L (see FIGS. 5A-5C), the vision system 400 may generate a position error for the case; If the offset is within a threshold, the auxiliary information from the auxiliary navigation sensor system 288 may provide attitude/position resolution (e.g., salt 520L and vehicle 110 payload bed 210B) transfer arm 210A. Note that this improves the offset (substantially equal to the determined pose/position of the case relative to the frame). It is further noted that if only one case is biased/offset relative to the slat 520L edge, the vision system may generate case position errors; However, if two or more juxtaposed cases are determined to be biased toward the slat 520L edge, the vision system may detect vehicle 110 attitude errors and vehicle 110 position relocation effects (e.g., auxiliary navigation sensor system (e.g., auxiliary navigation sensor system ( 288) correct the position of the vehicle 110 based on an offset determined from supplemental information) or a service message to the operator (e.g., an image conveyed from the vision system 400 to effect remote control operation) A “dash cam” cooperative mode (as described herein) may be created that provides remote control of vehicle 110 by an operator (in still and/or real-time video). Vehicle 110 may be operated by an operator. Vehicle 110 may be stopped (e.g., not traversing picking aisle 130A or transfer deck 130B) until remote control of vehicle 110 begins.

Case unit monitoring cameras 410A, 410B may also be configured to monitor case units prior to and/or after selection/placement of case units, e.g., from a storage shelf or other holding location, e.g., to transfer arm 210A without disturbing the transfer arm. Provide feedback regarding the position of the case unit alignment characteristics and case transport characteristics of the autonomous transport vehicle 110 (to determine the location/position of the alignment characteristics and case transport characteristics, in order to effectively select/place case units). there is. For example, as described above, case unit monitoring cameras 410A, 410B have a field of view surrounding payload bed 210B. Vision system controller 122VC receives sensor data from case unit monitoring cameras 410A, 410B, and any suitable image recognition algorithms stored in or accessible by the memory of vision system controller 122VC, pusher 470. Determine the position of the alignment blades 471, pullers 472, tines 210AT, and/or any other characteristics of the payload bed 210B in engagement with case units held on the payload bed 210B. It is configured to do so. The positions of pusher 470, alignment blade 471, puller 472, tines 210AT, and/or any other features of payload bed 210B are determined by a motor encoder or other respective position sensor. 470, alignment blade 471, puller 472, tines 210AT, and/or any other characteristics of payload bed 210B. However, in some aspects the positions determined by the vision system controller 122VC may be employed as redundancy in the event of an encoder/position sensor malfunction.

The alignment position of the case unit (CU) within the payload bed 21B may be verified by case unit monitoring cameras 410A and 410B. For example, also referring to Figure 4C, vision system controller 122VC receives sensor data from case unit monitoring cameras 410A, 410B and any suitable data stored in memory or accessible by vision system controller 122VC. Image recognition algorithms, e.g., the position of the case unit within the centerline (CLPB) of the payload bed 210B, the reference/home position of the alignment plane (JPP) of the pusher 470, and the case unit support plane (PSP) and configured to determine any suitable image recognition algorithm. Here, the positioning of a case unit (CU) within the payload bed 210B at least affects the positional accuracy relative to other case units on the storage shelf 555 (e.g., a predetermined spacing size between case units). to maintain).

2, 4A, 4B, 6, 7A, 7A, 8, one or more three-dimensional imaging systems 440A, 440B may include, for example, a time-of-flight camera, an imaging radar system, or light detection. and LIDAR, etc., any suitable three-dimensional image(s). One or more three-dimensional imaging systems 440A, 440B, together with a vision system controller 122VC, monitor the size (e.g., X, Y, Z directions) of the front surface (i.e., front surface) of the case unit (CU) and the case. Invariance of the shelf supporting the unit CU (e.g., one or more three-dimensional imaging systems 440A, 440B may affect the position of the case unit CU without reference to the shelf supporting the case unit CU). Determination of the shelf-invariant properties of the case unit may influence the decision as to whether the case unit is supported on a shelf, wherein the determination of the front surface and case center point (FFCP) determines whether the autonomous transport vehicle 110 It also affects the comparison of the virtual model (400VM) with the “real world” environment in which it operates, such that the controller 122 of the autonomous transport vehicle 110 operates the vision system 400 and “ Allows for a substantially direct comparison of what the autonomous transport vehicle 110 "sees". Image data acquired from one or more three-dimensional imaging systems 440A, 440B may be used when event data from cameras 410A, 410B is incomplete or If missing, image data from the cameras 410A and 410B can be supplemented.

As illustrated in FIG. 6 , one or more three-dimensional imaging systems 440A, 440B have their respective fields of view extending past payload bed 210B in the approximate direction LAT, such that each three-dimensional imaging system 440A, 440B ) adjacent to, but external to, the payload bed 210B (e.g., a case unit (CU) or picking aisle (e.g., a case unit (CU) arranged to extend in one or more rows along the length of the picking aisle (130A, see Figure 5A)) Substrate buffer/transfer stations disposed along 130A (similar in configuration to storage racks 599 and their shelves 555 disposed along picking aisle 130A) are arranged to sense substrate buffer/transfer stations disposed along 130A. Each three-dimensional imaging system The field of view (440AF, 440BF) of (440A, 440BF) includes a volume of space (440AV, 440BV) extending the height (670) of the pick range of autonomous transport vehicle (110) (e.g., arm (210A) This case unit can be stacked accessible from a shelf 555 or from a common rolling surface 284 on which autonomous transport vehicle 110 rides (e.g., on transfer deck 130B or picking aisle 130A - see FIG. 2 ). Range/height of the direction VER can move to pick/place a case on the shelf - see FIG. 8), where, as can be seen in FIG. 8, one or more three-dimensional imaging systems 440A, 440B are connected to at least one case unit. Provides sensor data to the vision system controller (122VC), which implements the front surfaces (800A, 800B, 800C) of (CU1, CU2, CU3), where detection of these front surfaces is a reference to the presence of a shelf supporting the case unit. The vision system controller (122VC) determines whether the detected case unit (CU) is placed on a shelf with other case units by determining shelf-invariant characteristics common to each case unit placed on the same shelf. where, for a case unit (CU) with a substantially vertically oriented face, determine the front surface normal vector (e.g., by plane fitting) and the bottom edge of the front surface (e.g., by area edge detection). The extraction of provides the plane equations for the shelf in the autonomous guided vehicle coordinate system X, Y, Z.

7A and 7B, the case unit sitting/seated on shelf 555 is equipped with one or more three-dimensional imaging systems 440A, 440B (and case unit monitoring cameras 410A, 410B) has a front surface or front surface 800. From the detected front surface 800, the vision system controller 122VC determines a front normal vector N that is normal to the front surface 800. Additionally, the detected front surface From 800, vision system controller 122VC (using any suitable image processing algorithm) determines bottom edge 777 (and its vector B) of front surface 800, where case unit (CU) The shelf invariant property of is derived from the front normal vector N and the bottom edge 777. For example, the UP or Z axis vector U can be determined from the cross product of vectors N and B as follows:

U = N x B [Formula 1]

The center point P of the bottom edge 777 is determined by the vision system controller 122VC (along with any suitable image processing algorithm thereof) (representing the bottom surface of the case unit CU seated on the shelf 555). The scalar equation of the plane can be written as:

d = U*P [Equation 2]

where (U, d) is a shelf-invariant property common to any case unit seated on the same shelf 555 (e.g., any case unit seated on the same shelf has the same shelf-invariant property within a predetermined tolerance) has a vector). Here, the vision system controller 122VC scans the case units CU1, CU2, and CU3 having at least one 3D imaging system 440A and 440B, and determines shelf-invariant characteristics to determine the case units CU1, CU2, and It is possible to determine whether CU3 (see FIG. 8) is placed on the same shelf. Determination of shelf invariant properties is based on simulations of storage and retrieval system structures between what the autonomous transport vehicle 110's 110 vision system 400 "sees" substantially and directly and what the autonomous transport vehicle 110 "sees". This may at least partially influence the comparison. Determination of the shelf invariant properties may also affect the placement of case units in the plane of the shelf 555 as determined from the shelf invariant properties.

2 and 4A, front navigation cameras 420A, 420B are any suitable cameras configured to provide object detection and ranging. The front navigation cameras 420A, 420B are positioned opposite the longitudinal centerline (LAXCL) of the autonomous transport vehicle 110, such that the forward-facing fields of view 420AF, 420BF provide stereo vision to the autonomous transport vehicle 110. They may be spaced at any suitable distance. Forward navigation cameras 420A, 420B are any suitable high resolution or low resolution video cameras (wherein video images containing more than about 480 vertical scan lines and captured at more than about 50 frames/sec are considered high resolution), time of flight. a camera, a laser ranging camera, or any other suitable camera configured to provide object detection and ranging to effectively provide for autonomous transport vehicles to traverse along transfer deck 130B and move along picking aisle 130A. . The rear navigation cameras 430A and 430B may be substantially similar to the front navigation cameras. The front navigation cameras 420A, 420B and the rear navigation cameras 430A, 430B are used for obstacle detection and avoidance as well as localization of the autonomous transport vehicle within the storage and retrieval system 100 ( 200E1) provides navigation of an autonomous transportation vehicle 110 having a direction of movement (leading or following a direction of movement). Localization of the autonomous transport vehicle 110 may be accomplished by detecting lines 900 on the running/rolling surface 284 and/or detecting suitable storage structures, including, but not limited to, storage racks 999. It may be affected by one or more of the navigation cameras (420A, 420B) and rear navigation cameras (430A, 430B). The line detection and/or storage structure detection may be compared to floor maps and structure information (e.g., stored in or accessible in memory) of the vision system controller 122VC. The front navigation cameras (420A, 420B) and rear navigation cameras (430A, 430B) also detect the autonomous transport vehicle (110) when an object approaches the autonomous transport vehicle (110) (while the autonomous transport vehicle (110) is stationary or moving). 110) (e.g., on an indeterminate rolling surface of transfer deck 130B) or within picking aisle 130A (e.g., another autonomous transport vehicle within storage and retrieval system 100, case Autonomous transport vehicle 110 may be manipulated (including, for example, another autonomous transport vehicle, case unit, or other transient object) to avoid the unit or other transient object.

Front navigation cameras 420A, 420B and rear navigation cameras 430A, 430B may also provide convoy of vehicles 110 along pickup aisle 130A or transfer deck 130B, where one vehicle 110 follows another vehicle 110A at a fixed, predetermined distance. As an example, Figure 1B shows a convoy of three vehicles 110 with one vehicle closely following the other at a fixed, predetermined distance.

Still referring to FIGS. 2 and 4A , one or more case edge detection sensors 450A, 450B scan the shelves of the storage and retrieval system to determine if the shelves are clear for placing case units (CUs), or to determine if the shelves are clear for placing case units (CUs). Any suitable sensors configured to verify case unit size and location prior to selecting the CU). One case edge detection sensor 450A, 450B is shown on each side of the center line (CLPB, see FIG. 4A) of payload bed 210B, while vehicle 110 has a leading edge 200E1 leading the direction of vehicle movement. ) or two or more case edge detection sensors may be placed at any suitable location on the autonomous transport vehicle 110 to traverse and scan the case unit CU along with the rear end 200E2.

One or more traffic monitoring cameras 460A, 460B are disposed in frame 200 with their respective fields of view 460AF, 460BF facing laterally at lateral LAT1. One or more traffic monitoring cameras 460A, 460B are illustrated as being adjacent to the transfer opening 1199 of transfer bed 210B (e.g., a pick from which arm 210A of autonomous transport vehicle 110 extends). (pick side), on the other side, a traffic surveillance camera may be placed on the non-pick side of the frame 200 so that the view of the traffic surveillance camera faces the side of direction LAT2. Traffic monitoring cameras 460A, 460B provide for automatic merging of autonomous transport vehicles 110, for example, exiting pickup aisle 130A or lift transfer area 195 into transfer deck 130B (see Figure 1B). ). For example, autonomous transport vehicle 110 leaving lift transfer area 195 (FIG. 1B) detects autonomous transport vehicle 110T moving along transfer deck 130B. Here, the controller 122 based on information (e.g., distance, speed, etc.) of the vehicle 110V collected by the traffic surveillance cameras 460A, 460B and communicated and processed by the vision system controller 122VC. , autonomously enters the transfer deck in front or behind the transfer vehicle 110T, and plans acceleration to the transfer deck based on the speed of the approaching vehicle 110T.

One or more out-of-plane (e.g., upward or downward facing) localization cameras 477A, 477B are positioned on the ceiling 991 of the storage and retrieval system or on the rolling surface 284 of the storage and retrieval system. A reference is placed on the frame 200 of the autonomous transport vehicle 110 to sense/detect a reference (e.g., location mark 971, line 900). The location fiducial is located within the storage and retrieval system. May provide a unique identifying mark/pattern that has a known location and is recognized by the vision system controller 122VC (e.g., processing data obtained from positioning cameras 477A, 477B) based on the detected location reference. , the vision system controller 122VC compares the detected position reference to a known position reference (e.g., stored in or accessible to the memory of the vision system controller 122VC) to determine the autonomous transport vehicle within the storage structure 130 ( 110) determine the location.

The cameras of auxiliary navigation sensor system 288 may be configured in any suitable manner to effect detection of case units (CUs), storage structures (e.g., shelves, columns, etc.), and other structural features of the storage and retrieval system. For example, by intrinsic and external camera calibration). 4A, 4B, 5A, 5B, and 5C, a known object (e.g., case unit CU1, CU2, CU3 (or storage system structure)) may be detected by a camera of auxiliary navigation sensor system 288. The vehicle 110 may be positioned within the field of view (or the vehicle 110 may be positioned such that known objects are within the field of view of the cameras). These known objects may be imaged by the cameras from multiple angles/viewpoints so that each camera can be calibrated so that the vision system Controller 122VC is configured to detect known objects based on sensor signals from a calibrated camera.

For example, calibration of case unit monitoring cameras 410A, 410B will be described for case units CU1, CU2, CU3 with known physical properties/parameters. 5A-5C are example images captured from one of the case unit monitoring cameras 410A, 410B at three different viewpoints for illustrative purposes. Here, the physical characteristics/parameters (e.g., shape, length, width, height, etc.) of case units CU1, CU2, CU3 are known by the vision system controller 122VC (e.g., different case units (e.g., The physical characteristics of CU1, CU2, CU3) are stored in or accessible to the memory of the vision system controller (122VC). For example, based on three (or more) different viewpoints of case units CU1, CU2, and CU3 in the images of FIGS. 5A-5C, vision system controller 122VC monitors case unit monitoring camera 410A. , 410B) provides intrinsic and extrinsic camera and case unit parameters that affect the calibration. For example, from the images, vision system controller 122VC registers (e.g., stores in memory) the views of case units CU1, CU2, and CU3 with respect to case unit monitoring cameras 410A and 410B. The vision system controller 122VC estimates the poses of the case units CU1, CU2, and CU3 with respect to the case unit surveillance cameras 410A and 410B, and estimates the poses of the case units CU1, CU2, and CU3 with respect to each other. . The pose estimate (PE) of each case unit (CU1, CU2, CU3) is illustrated in FIGS. 5A-5C as being overlaid on each case unit (CU1, CU2, CU3).

Vehicle 110 is moved such that any suitable number of viewpoints of case units CU1, CU2, CU3 are acquired/imaged by case unit monitoring cameras 410A, 410B, so that known case units CU1, CU2, CU3 ) Perform convergence of case unit characteristics/parameters (e.g., estimated by the vision system controller 122VC) for each. Upon convergence of case unit parameters, case unit monitoring cameras 410A, 410B are calibrated. The calibration process is repeated for other case unit monitoring cameras (410A, 410B). Using both of the calibrated case unit monitoring cameras (410A, 410B), the vision system controller (122VC) creates a three-dimensional ray for each pixel of each of the case unit monitoring cameras (410A, 410B), as well as three rays that separate the cameras. The dimension consists of baseline segments and estimates of the relative poses of the case unit monitoring cameras 410A and 410B. The vision system controller 122VC configures the case unit monitoring cameras 410A, 410B to create a three-dimensional line for each pixel, an estimate of the three-dimensional baseline segments separating the cameras, and the case unit monitoring cameras 410A, 410B to each other. Configured to adopt an attitude relative to each other, case unit monitoring cameras 410A, 410B form passive stereo vision sensors where common features are visible within the field of view of case unit monitoring cameras 410AF, 410BF. As described above, calibration of case unit monitoring cameras 410A, 410B has been described for case units CU1, CU2, CU3, but can be used in any suitable structure of storage and retrieval system 100 in a substantially similar manner. For example, permanently or temporarily).

As may be realized, vehicle localization performed by physical characteristic sensor system 270 (e.g., a predetermined position along pickup aisle 130A or along transfer deck 130B relative to a pickup/placement location) Positioning the vehicle in the vehicle can be enhanced with pixel level positioning performed by the auxiliary navigation sensor system 288. Here, the controller 122 is configured to “approximately” locate the vehicle 110 with respect to a pick/place location by employing one or more sensors of the physical characteristic sensor system 270. The controller 122 uses secondary (e.g., pixel level) location information obtained from the vision system controller 122VC of the secondary navigation sensor system 288 to “fine-tune” the vehicle attitude and position for the pick/place location. tune), so that the positions of the vehicle 110 and the case unit (CU) placed at the storage location 130S by the vehicle 110 are adjusted to the physical characteristic sensor system 270 alone by the vehicle 110 or the case. It is configured so that smaller tolerances (i.e. increased positioning accuracy) can be maintained compared to the positioning of the unit (CU). Here, the pixel level positioning provided by secondary navigation sensor system 288 has a higher positioning sharpness/resolution than the electromagnetic sensor resolution provided by physical characteristic sensor system 270.

In aspects in which the case unit may be dimly illuminated, an illumination source may be provided in the vehicle 110 to illuminate the case unit (or other structure) to effect calibration of the camera in the manner mentioned above. The illumination may be diffuse illumination or may be applied to the surface of the case unit (or other structure) such that parameters of the case unit (or other structure) can be extracted by the vision system controller 122VC from the image of the case unit (or other structure) and convergence is obtained. It can have known patterns that are projected onto it. Suitable markers (e.g., known locations on the case unit or other structures) to facilitate feature extraction from images acquired by the case unit monitoring cameras 410A, 410B and correction of the effectiveness of the case unit monitoring cameras 410A, 410B. A calibration sticker located on) may also be placed on the case unit/structure. Other cameras of the auxiliary navigation sensor system 288 (e.g., front and rear navigation cameras 420A, 420B, 430A, 430B), traffic monitoring camera(s) 460A, 460B, and out-of-plane localization camera(s) Calibration of (477A, 477B, etc.) may be effected in a similar manner as described above.

1A, 2, 4A, 4B, and 11, the vision system controller 122VC of the autonomous transportation vehicle 110 controls various sensors (or groups of sensors) of the auxiliary navigation sensor system 288 depending on vehicle 110 operation. It is configured to dynamically select and access information from 11 is a diagram illustrating non-exhaustive sensor groupings 1111 - 1114 and associated non-exhaustive vehicle operations through which sensor groups can be accessed by vision system controller 122VC to affect vehicle operations. . Exemplary sensor group 1111 includes rear navigation cameras 230A and 230B. Exemplary sensor group 1112 includes front navigation cameras 420A and 420B. Exemplary sensor group 1113 includes out-of-plane cameras 477A and 477B. Exemplary sensor group 1114 includes case unit surveillance cameras 410A and 410B. For illustrative purposes only, for vehicle operations in which rear end 200E2 of vehicle 110 guides the direction of vehicle movement (e.g., backward movement on transfer deck 130B), sensor groups 1111, 1113 may be configured to: May be used by vision system controller 122VC (and controller 122). For vehicle operations (e.g., forward movement on transfer deck 130B) in which the front end 200E1 of vehicle 110 guides the direction of vehicle movement, sensor groups 1112, 1113 may be configured to communicate with vision system controller 122VC. (and may be used by the controller 122. For vehicle operation (e.g., forward movement along the picking aisle 130A), the front end 200E1 of the vehicle 110 directs the direction of vehicle movement, the sensor Groups 1112 and 1114 may be utilized by vision system controller 122VC (and controller 122) to control vehicle operation (e.g., , backward movement along the picking passage 130A), sensor groups 1111, 1114 may be used by the vision system controller 122VC (and controller 122). Sensor group 1114 may be used by the transfer arm For vehicle operations (e.g., picking placement operations) where 210A loads or unloads case units (CUs) from or to the payload bed (210B), sensor group 1114 may be configured to use a vision system. May be used by controller 122VC (and controller 122).

Also, referring to FIGS. 1A, 1B, 2, 4D, 17 and 18, autonomous transport vehicle 110 includes an auxiliary hazard sensor system 290, as described above. The auxiliary hazard sensor system 290 is coupled to the frame 200 of the autonomous transport vehicle 110 to provide bot operational control of the autonomous transport vehicle 110 in cooperation with the driver. Auxiliary hazard sensor system 290 provides data (images). Vision system data may either a) determine information characteristics (which are ultimately provided to controller 122), or b) the information is passed to controller 122 without characterizing it (of the object to predetermined criteria) and the characterization is performed by controller 122. Registered by the vision system controller 122VC, causing it to be performed by. It is the controller 122 that determines the choice to transition to the cooperative state in a) or b). After transition, coordinated operation is accomplished by a user accessing the auxiliary hazard sensor system 290 via vision system controller 122VC and/or controller 122. However, in its simplest form, secondary hazard sensor system 290 may be considered to provide a cooperative mode of operation for autonomous transportation vehicle 110. Auxiliary hazard sensor system 290 is configured to cooperatively identify and mitigate, for example, objects/hazards invading moving/rolling surfaces 284 of autonomous navigation/actuation sensor system 270 . and/or supplement the auxiliary sensor system 298. Supplementary hazard sensor system 290 forms at least in part vision system 400 and includes at least one camera 292. As used herein, the term "camera" refers to a two-dimensional camera, a two-dimensional camera with RGB (red, green, blue) pixels, a three-dimensional camera with the definition XYZ+A, where XYZ is the three-dimensional reference frame of the camera; A is a still or moving image device including one or more of a radar reflection intensity or time-of-flight stamp), and an RGB/XYZ camera containing both RGB and three-dimensional coordinate system information, non-limiting examples of which are provided herein. do. At least one camera 292 of the vision system 400 provides a view of the autonomous transport vehicle 110 at various locations in the facility 100 while executing autonomous navigation and transfer operations. It is arranged to capture image data indicative of objects and/or spatial features 299 (having unique physical characteristics) within at least a portion of the viewing facility 100 . As may be implemented, at least one camera 292 is shown in FIG. 4D as being separate and distinct from the camera shown in FIG. 4A for illustrative purposes only; however, at least one camera 292 is shown in FIG. 4A (e.g., camera 292 at end 200E1 of vehicle 110 may be camera 477A of FIG. 4A and camera 292 at end 200E2 of vehicle 110 may be part of the system shown in The camera 292 facing the side in the LAT1 direction of FIG. 4D may be the camera 477B of FIG. 4A, and the camera 292 facing the side in the LAT1 direction of FIG. 4D may be the cameras 460AF and 460BF of FIG. 4A.

As described above, vision system 400 includes at least one camera 292. Although aspects of the present disclosure have been described in the context of a forward-facing camera (i.e., a camera facing in the direction of travel ahead of end 200E1 of autonomous transport vehicle 110), the camera may be directed in any direction (rear, side, top, bottom). It will be understood that it can be arranged to face (etc., etc.). At least one camera 292 may be positioned on the longitudinal center line LAXCL and on either side of the longitudinal center line LAXCL, and one or more cameras 292 may be positioned along the length of the autonomous transport vehicle 110. It may be placed opposite the directional centerline LAXCL so that the field of view 292F provides stereo vision (e.g., cameras 420A, 420B) to the autonomous transport vehicle 110 or any suitable configuration. At least one camera 292 is any suitable camera configured to detect objects or spatial features 299. For example, at least one camera 292 may be any suitable high- or low-resolution video camera, 3D imaging system, time-of-flight camera, laser ranging camera, or autonomous navigation camera. and an object or spatial feature 299 within at least a portion of the facility 100 viewed by the at least one camera 292 while the autonomous transport vehicle 110 is at various locations in the facility 100 while executing the transfer operation. ) is any other suitable camera configured to detect. At least one camera 292 provides imaging and detection (when both ends 200E1 and 200E2 of the autonomous transport vehicle 110 are leading or lagging in the direction of travel). The object or spatial feature 299 detection may be compared to a reference floor map and structural information (e.g., stored in or accessible in memory) of the vision system controller 122VC. At least one camera 292 may also cause autonomous delivery vehicle 110 to initiate an autonomous stop (i.e., autonomous operation state) or to detect an object or spatial feature 299 as autonomous delivery vehicle 110 approaches an object or spatial feature 299 . or to identify spatial features 299 (e.g., other malfunctioning autonomous transport vehicles, dropped case units, debris, spills, or other transient objects within the storage and retrieval system 100), e.g., a transfer deck. A controller capable of entering a cooperative operating state to be started by an operator or stopped by an operator on an undetermined rolling surface of 130B or within picking aisle 130A (which may have a determined or undetermined rolling surface). (122)) (including or through the vision system controller 122VC).

Cameras 292 of auxiliary hazard sensor system 290 may be configured to affect detection/detection of objects or spatial features 299 in storage and retrieval system 100 (e.g., such as intrinsic and extrinsic camera calibration). ) may be corrected in any suitable manner. 4D and 5B-5C, a known object (e.g., a case unit (CU1, CU2, CU3) or storage system structure, with known physical properties such as shape, size, etc.) is connected to the secondary hazard sensor system ( The camera 292 may be positioned within the field of view 292F of the camera 290 (or the autonomous transport vehicle 110 may be positioned such that there is a known object within the field of view 292F of the camera 292). These known objects may be imaged by the cameras at various angles/views to calibrate each camera, such that the vision system controller 122VC can detect "unknown" (i.e., unidentifiable) objects based on the sensor signals of the calibrated cameras. ) can be configured to determine when an object is within the observation field of view (292F).

For example, calibration of camera 292 will be described in terms of case units CU1, CU2, CU3 with known physical properties/parameters. 5B and 5C are example images captured from camera 292 at two different viewpoints for illustrative purposes. Here, the physical properties/parameters (e.g., shape, length, width, height, etc.) of the case units (CU1, CU2, CU3) are stored so that they are “known” (i.e., identifiable) by the vision system controller 122VC. (For example, the physical characteristics of the different case units CU1, CU2, and CU3 are stored in a memory accessible to the vision system controller 122VC or in the memory of the vision system controller). For example, based on two (or more) different viewpoints of case units CU1, CU2, and CU3 in the images of FIGS. 5A-5B, vision system controller 122VC may be instructed to calibrate camera 292. Impacting intrinsic and extrinsic camera and case unit parameters are provided.

The autonomous transport vehicle 110 moves so that any suitable number of views of the case units CU1, CU2, CU3 can be acquired/imaged by the camera to generate convergence of case unit characteristics/parameters. do. Once the case unit parameters have converged, camera 292 is calibrated. With cameras 292 calibrated, vision system controller 122VC configures a three-dimensional ray for each pixel of each camera 292. As mentioned above, calibration of camera 292 has been described with respect to case units CU1, CU2, and CU3, but may be applied to any suitable structure (e.g., permanent or temporary) of storage and retrieval system 100. may be performed in a substantially similar manner.

As may be implemented, autonomous transport vehicle 110 (in one aspect a payload/case transport and/or transfer robot) moves autonomously along picking aisle 130A or along transport deck 130B. , the autonomous transport vehicle 110 may opportunistically detect other objects (e.g., other bots, dropped case units, spills, debris, etc.) within the facility 100 (e.g., retrieve the payload from storage). Incidentally or incidentally to certain autonomous tasks, such as autonomously picking/placing, moving payloads to transfer stations and/or charging stations, autonomously picking/placing/transferring payloads at transfer stations, and/or autonomous charging at charging stations. peripherally) can be done. The vision system controller 122VC determines whether the object is “unknown” (i.e., the object or spatial feature 299 is not expected to be within an area or space along the autonomous path of movement of the autonomous transport vehicle 110). It is configured to use the auxiliary navigation sensor system 288 and/or the auxiliary hazard sensor system 290 (i.e., image information obtained from cameras of one or more auxiliary sensor systems).

1A, 2, 4A, 4B, 4D, 10A, and 10B, the vision system 400 of the secondary navigation sensor system 288 and/or the secondary hazard sensor system 290 is configured to operate the autonomous transport vehicle 110. The autonomous transportation vehicle 110 is configured with a virtual model (400VM) of the operating environment 401. For example, the vision system controller 122VC may determine certain features (e.g., the fixed/permanent structure and and/or a temporary object), wherein the reference representation 400VMR of the predetermined feature defines the location or shape of at least a portion of the storage structure 130 or facility traversed by the autonomous transport vehicle 110. do. Here, the virtual model 400VM of the operating environment 401 (and the reference representation 400VMR of certain features thereof) may be stored in any suitable memory of the autonomous transport vehicle (e.g., the memory of the vision system controller 122VC) or the vision system controller 122VC. It is stored in memory accessible to the system controller (122VC). The virtual model 400VM provides the autonomous transport vehicle 110 with dimensions, locations, etc. of at least fixed (e.g., permanent) structural components in the operating environment 401 . The operating environment 401 and its virtual model 400VM include at least fixed/permanent structures (e.g., transfer deck 130B, picking aisle 130A, storage space 130S) of one or more storage structure levels 130L. , case unit transfer area, case unit buffer location, vehicle charging location, support pillar, etc.); In one or more aspects, the operating environment 401 and the virtual model 400VM include fixed structures of one or more storage structure levels 130L and temporary structures of at least a portion of the storage level 130L on which the autonomous transport vehicle 110 operates ( (e.g., a case unit (CU), etc.) stored or otherwise located in a case unit holding location of the storage and retrieval system 100; In one or more other aspects, the operating environment 401 and the virtual model 400VM may be configured to represent at least a portion of a temporary structure (e.g., a temporary structure (e.g., For example, a case unit) and at least a fixed structure; In another aspect, operating environment 401 and virtual model 400VM include the entire storage structure and at least a portion of a temporary structure (e.g., a temporary structure at a storage level at which an autonomous transportation vehicle operates).

Autonomous transport vehicle 110 may store thereon (or within a memory accessible thereto) a portion of virtual model 400VM that corresponds to a portion of the operating environment in which autonomous transport vehicle 110 operates. For example, autonomous transport vehicle 110 stores thereon (or in memory accessible thereto) only a portion of virtual model 400VM corresponding to the storage structure level 130L at which the autonomous transport vehicle is deployed. The virtual model 400VM of the operating environment 401 may be dynamically updated in any suitable manner to facilitate operation of the autonomous transport vehicle 110 in the storage structure 130. For example, when autonomous transport vehicle 110 is moved from one storage structure level 130L to another different storage structure level 130L, vision system controller 122VC (e.g., controller 122 and and/or wirelessly by the control server 120) to include portions of the virtual model 400VM corresponding to different different storage structure levels 130L. As another example, virtual model 400VM is dynamically updated as case units are added to and removed from storage structure 130 to create a predetermined (expected) number of case units (CUs) to be transported by autonomous transport vehicle 110. ) A dynamic virtual model case unit map indicating the location can be provided. In another aspect, the predetermined (expected) location of the case unit within the storage structure may not be included in the virtual model 400VM; However, the predetermined (expected) location, size, SKU, etc. of one or more case units to be transported by autonomous transport vehicle 110 are communicated, for example, from controller 120 to autonomous transport vehicle 110, and the vision The system 400 (and the vision system controller 122VC) may perform an operation as described herein (e.g., the vision system 400 may determine what it expects to "see" and what it actually does to ensure that the correct case unit is being transported). compare “what you see”) and/or case units at a predetermined location for detection/identification of other malfunctioning autonomous transport vehicles, fallen case units, debris, spills or other transient objects within the storage and retrieval system 100. Conduct verification.

Vision system controller 122VC registers image data captured by auxiliary navigation sensor system 288 and generates predetermined characteristics from the captured image data (e.g., storage structure 130 of the storage and retrieval system described herein). ) to generate at least one image (e.g., a still image and/or a video image) of one or more features of a fixed/permanent structure and/or a temporary object within the At least one image (e.g., see Figures 5A-5C, 9A, 10A, and 10B as example images) is formatted into a virtual representation (VR) of one or more (imaged) predetermined features, forming a reference representation ( Comparison of a given feature of the 400VMR) with one or more corresponding references (e.g., a corresponding feature of the virtual model 400VM that serves as a reference for identifying the shape and/or location of the imaged predetermined feature) (at least Provides one to up to six degrees of freedom (X, Y, Z, Rx, Ry, Rz).

13 is an example flow diagram of a comparison of at least one model 400VM of a storage and retrieval system stored in or accessible to a vision system controller 122VC. For illustrative purposes only, the storage facility information model, the storage structure/array information model, and the case input station model are provided, but in other embodiments, the vision system controller 122VC is provided with Any suitable model and number of models may be provided to provide virtual information. Different models may be combined to provide the vision system controller 122VC with a complete virtual operating environment in which the autonomous transport vehicle 110 operates. The sensors of vision system 400 (described herein) also provide sensor data to vision system controller 122VC. Sensor data representing a virtual representation (VR) image is processed by any suitable image processing method to detect edge features and/or regions of interest of objects in the image. The vision system controller 122VC predicts the field of view of the sensor providing image data within the model 400VM and determines the edge of the object and the region of interest within the predicted field of view. The edges and regions of interest of the virtual model 400VM are the edges and regions of interest of the virtual representation (VR) pose and position determination of one or more of the autonomous transport vehicle 110 and case units (payloads) as described herein. compared to

Vision system controller 122VC may control one or more imaged features (e.g., the features imaged in FIG. 9A ) (as described herein in conjunction with a portion of virtual model 400VM and appropriate image processing non-transitory computer program code). are the storage and retrieval system rack/column structure, the features imaged in FIG. 10A are the case units (CU1-CU3), and the features imaged in FIG. 10B are the case unit, storage rack structure, and payload bed 210B. A virtual representation (VR) of a virtual representation (which is part of ) presented in the reference representation (RPP) of ) is made in the autonomous transport vehicle 110 (see FIGS. 9A and 10A ). Here, autonomous transport vehicle 110 attitude determination and navigation are separate from each system controller (e.g., control server 120 or other appropriate controller of the storage and retrieval system) that sends commands to autonomous transport vehicle 110. independent and independent.

As described herein, controller 122 may use auxiliary (e.g., pixel level) location information obtained from vision system controller 122VC of auxiliary navigation sensor system 288 to determine vehicle posture and associated picking/placement location. configured to utilize "fine-tuning" positions, such that the positions of vehicle 110 and case units (CUs) placed in storage location 130S by vehicle 110 can be determined using only vehicle physical characteristic sensor system 270. 110 or a case unit (CU) can be maintained with smaller errors (i.e., increased positioning accuracy) compared to positioning. Fine-tuning of the attitude and position of the autonomous transport vehicle 110 is accomplished by the vision system controller 122VC, which determines the physical characteristics based on a comparison between the virtual representation (VR) and the reference representation (RPP). It is configured to check attitude and position information of the autonomous transport vehicle 110 registered by the vision system controller 122VC from the sensor system 270.

Comparison between the virtual representation (VR) and the reference representation (RPP) by the vision system controller 122VC by verifying the accuracy of the data with information obtained from the auxiliary navigation sensor system 288 and by the physical property sensor system 270. Build trust in the generated data. Here, the vision system controller 122VC identifies deviations in the attitude and position of the autonomous guided vehicle based on a comparison between the virtual representation (VR) and the reference representation (RPP) and moves the autonomous guided vehicle to a predetermined commanded position (e.g., Update attitude or position information of the autonomous transportation vehicle 110 from the physical property sensor system 270 based on that deviation (e.g., to effect final positioning of the transportation vehicle 110) configured to modify pose and/or position information from 270) or complete (if there is no pose and/or position information from physical characteristic system 270).

Vision system controller 122VC may determine a signal from physical characteristic sensor system 270 based on at least one of the identified deviations and image analysis of at least one image (from vision system 400 of auxiliary navigation sensor system 288). It is configured to determine the accuracy of the posture and position information of the autonomous guided vehicle 110 and the posture error of the information from the physical characteristic sensor system 270, and assign a reliability value according to at least one of the posture error and accuracy. If the confidence value is less than a predetermined threshold, the vision system controller 122VC guides autonomously based on the pose and position information generated from the virtual representation (VR) instead of the pose and position information from the physical property sensor system 270. It is configured to switch vehicle navigation. The conversion from physical attribute sensor system attitude and position information to virtual representation (VR) pose and position information is performed by vision system controller 122VC (or controller 122) using the pose and position generated from physical attribute sensor system 270. Opting out of location information, kinematic/dynamic algorithms (e.g., as described in U.S. Patent Application No. 16/144,668, entitled “Storage and Retrieval System,” filed September 27, 2018, the disclosures of which) This can be done by selecting/entering pose and position information from a virtual representation (VR), which is incorporated herein by reference in its entirety.

After vision system controller 122VC performs the foregoing transition, vision system controller 122VC may navigate autonomous transport vehicle 110 to any suitable destination (e.g., payload placement destination, charging destination, etc.). configured to continue performing; In another aspect, the vision system controller 122VC may direct the autonomous transport vehicle 110 from a diversion position to a safe location 157 (the safe location being a dedicated guidance/extraction area of the transfer deck, lift, etc.) for shutdown of the autonomous transport vehicle 110. The transfer area, or other area of the transfer deck 130B or picking aisle 130A, allows the operator to transport the autonomous transport vehicle 110 without interfering with the operation of other autonomous transport vehicles 110 operating in the storage and retrieval system 100. configured to select a safe route and trajectory for the autonomous transport vehicle 110 to be brought into the accessible area; In another aspect, vision system controller 122VC initiates communication with an operator of storage and retrieval system 100 identifying motion data of autonomous transport vehicle 110 and operator selection (e.g., user interface (UI)) provided above) is configured to identify the destination of the autonomous transport vehicle 110. Here, the operator may change control of the autonomous guided vehicle (e.g., via a user interface (UI)) from autonomous operation to semi-autonomous operation (e.g., autonomous transport vehicle 110 operates autonomously with limited manual input) or Manual operation may be selected or switched (e.g., an operator remotely controls the operation of autonomous transport vehicle 110 via a user interface (UI)). For example, the user interface (UI) may be a capacitive touch pad/screen, joystick, haptic screen, or user interface to effect operator control inputs in semi-autonomous and manual operation modes of autonomous transport vehicle 110. It may include other input devices that convey kinematic direction commands (e.g., turn, accelerate, decelerate, etc.) and/or picking placement commands from the interface (UI) to the autonomous guided vehicle 110 .

If the deviation described herein persists (within a predetermined tolerance), the vision system controller 122VC may be configured to apply the deviation as an offset that is automatically applied to the data in the physical characteristic system 270, whereby Accordingly, the autonomous transport vehicle 110 can be collectively positioned based on data from the physical property sensor system 270 modified by the offset, and comparison with the virtual representation (VR) and the reference representation (RPP) determines the validity of the offset. It is worth noting that the offset (and the attitude and position of the autonomous transport vehicle 110) are verified and adjusted according to any deviations. If the deviation reaches a predetermined threshold, vision system controller 122VC may alert the user of storage and retrieval system 100 that autonomous guided vehicle 110 may require servicing.

With continued reference to FIGS. 1A, 2, 4A, 4B, and 10A, although identification of attitude and position errors of autonomous transport vehicle 110 has been described above, vision system controller 122VC may be configured to, for example, store location 130S or It is configured to perform similar attitude and position error identification for case units (CUs) stored in other storage areas of the storage and retrieval system. For example, vision system controller 122VC may register by vision system controller 122VC from physical property sensor system 270 based on a comparison between a virtual representation (VR) and a reference representation (RPP) of virtual model 400VM. It is configured to check the attitude and location information of the payload. The vision system controller 122VC identifies deviations in the pose and position of the payload (case unit) based on a comparison between the virtual representation (VR) and the reference representation (RPP), and based on the deviations, retrieves the payload from the physical property sensor system. Update pose or position information of a rod (e.g., correct pose and/or position information from physics sensor system 270) or complete (if pose and/or position information is missing from physics sensor system 270). ) is configured to.

Vision system controller 122VC may provide information from physical characteristic sensor system 270 based on the at least one identified deviation and image analysis of at least one image from vision system 400 of auxiliary navigation sensor system 288. Attitude Error and Physical Properties sensor system 270 is configured to determine the accuracy of payload attitude and position information. The vision system controller 122VC assigns a reliability value based on at least one of payload pose error and accuracy. If the confidence value is less than a predetermined threshold, the vision system controller 122VC determines the autonomous transport based on the pose and position information generated from the virtual representation (VR) instead of the pose and position information from the physical property sensor system 270. Switches vehicle 110 payload processing.

After transition, vision system controller 122VC, in some aspects, directs the autonomous guided vehicle to a predetermined destination (e.g., a payload placement location or an area of a storage and retrieval system where the payload can be inspected by an operator). configured to continue processing; In another aspect, the vision system controller 122VC may, via a user interface device (UI), switch from an automatic payload processing operation to a semi-autonomous payload processing operation, wherein the operator provides limited input to the autonomous guided vehicle traverse movement and transfer arm 210A. ) or manual payload handling operations (the operator manually controls the movement of the autonomous guided vehicle traverse and the movement of the transfer arm 210A), which involves communication with the operator identifying payload data along with operator selection of the autonomous guided vehicle. It is configured to commence.

In a manner similar to that described above, vision system controller 122VC may communicate with an operator via a wireless communication system (e.g., network 180) that communicatively couples vision system controller 122VC and an operator interface (UI). configured to transmit a simulated image combining a virtual representation (VR) of one or more imaged predetermined features with one or more corresponding predetermined features of a reference, providing an augmented reality image in real time (predetermined features of the reference are 555, and the virtual representation includes representations of case units CU1 to CU3 (see FIG. 10A). Here, the vision system 400 of the auxiliary navigation sensor system 288 may include a "dashboard camera" ("dashboard camera") that transmits video and/or still images of the vehicle 110 to the operator to allow remote operation or monitoring of the vehicle 110. or dash-camera). The vision system 400 may also operate as a data recorder that periodically transmits still images acquired from the vision system camera to the memory of the user interface (UI), where the still images are stored/cached for operator review. (e.g., in addition to providing a real-time video stream, vision system 400 provides non-real-time review of still images). Still images may be captured and transmitted to the user interface for storage (e.g., every second, every 10 seconds, every 30 seconds, every minute, or any other suitable time interval) to the user interface for storage (e.g., every 10 seconds, every 30 seconds, every minute, or any other suitable time interval). excluding recording), the periodicity of still image capture/recording maintains adequate communication bandwidth between, for example, the control server 120 and the bot 110 (according to aspects of the disclosed embodiments, the storage and retrieval system 100 Note that the number of bots 110 operating/transporting case units may be in the hundreds or thousands. Here, a user interface (UI) with a stored still image history allows the user to review the still images to determine how and why an event occurred (e.g., case picking failure, bot breakage, product spill, presence of debris on transfer deck, etc.) Provides an interactive presentation/data interface to determine what occurred before and/or after an event.

Vision system controller 122VC is configured to receive real-time operator commands (e.g., commands from a user interface (UI)) for the traversing autonomous guided vehicle 110, such commands being transmitted to real-time augmented reality images (FIGS. 9A and 10A) and corresponds to changes in the real-time augmented reality image transmitted to the operator by the vision system controller 122VC. Video or still images may be stored (and time-stamped) in a memory onboard vehicle 110 and transmitted to control server 120 and/or operators upon request; In other aspects, the video and/or still images may be broadcast or otherwise transmitted in real time for viewing in an operator-accessible user interface (UI) (as described herein).

Vision system controller 122VC may also register image data captured by secondary hazard sensor system 290 and generate at least one image of one or more objects or spatial features 299 representing predetermined physical characteristics from the captured image data. configured to generate (e.g., still images and/or video images). At least one image (e.g., see FIGS. 5B, 5C and 15 as example images) is formatted as a virtual representation (VR) of one or more objects or spatial features 299 (see FIGS. 4D and 15), Comparison of a given feature of the representation 400VMR with one or more corresponding references (e.g., a corresponding feature of the virtual model 400VM that serves as a reference for identifying the shape and/or location of the given feature imaged) (Provides at least one and up to six degrees of freedom X, Y, Z, Rx, Ry, Rz (see Figure 2)). Controller 122VC is configured to determine (via comparison) the presence or absence of certain physical characteristics of an object or spatial feature 299 based on a comparison between the virtual representation and the reference representation (i.e., the object is “known”). comparison to determine whether it is known or “unknown”). If an object or spatial feature 299 is identified as “unknown” by controller 122VC, controller 122VC determines the dimensions of the desired physical feature (e.g., via controller 122 ) The autonomous transport vehicle 110 places the object 299 at a predetermined location (i.e., a predetermined location relative to the object or spatial feature 299) based on the location of the object or spatial feature 299 determined from the comparison. commands the bot to stop (as may be implemented, the command stop interrupts the autonomous routine of the vehicle's previous autonomous commands, effectively diverting the bot from autonomous operation). In response to detecting certain physical characteristics of at least one object or spatial feature 299, the controller 122 determines the object or spatial feature 299 as a hazard and identifies mitigation actions of the vehicle for the hazard. Selectively reconfigure the autonomous delivery vehicle 110 from an autonomous state to a cooperative vehicle state (i.e., selectively transition the autonomous delivery vehicle 110 from an autonomous operation state to a cooperative operation state and allow the vehicle to mitigate risks, e.g., failure). Identify whether the vehicle can be removed or act as a signal/beacon to warn other vehicles performing autonomous tasks). In the cooperative operation state, autonomous delivery vehicle 110 is positioned to receive operator commands for autonomous delivery vehicle 110 to continue performing vehicle operations to identify and mitigate objects or spatial features 299.

In one aspect, autonomous transport vehicle 110 may not include a reference map (e.g., virtual model 400VM). In this aspect, when the camera 292 detects an object or spatial feature 299, the controller 122VC determines the frame of reference of at least one camera 292 that is calibrated and has a predetermined relationship to the autonomous transport vehicle 110. Determine the position of an object within From the object pose in the camera frame of reference, controller 122VC determines the presence of certain physical characteristics of object 299 (i.e., whether object 299 extends across the bot's path, blocks the bot, or is an obstacle or hazard). Determine whether the bot path is considered to be within a predetermined distance. Once the presence of an object is determined and the transition from the autonomous state to the cooperative vehicle state is made, the controller 122VC switches to an operator (user) interface (UI) for cooperative operator operation of the autonomous transportation vehicle 110, as further described below. configured to initiate a transmission communicating the presence of certain physical characteristics as images/videos (wherein the vehicle 11 is configured as an observation platform and pointer for the user in a cooperative mode. In this mode the vehicle 11 also performs autonomous operation). A pointer for other running bots, which identifies the pointer bot (e.g. via control system 120 or a beacon) and automatically changes course to avoid the area until further commanded and if avoidance is not possible. (stop before encountering an object/hazard).

Vision system controller 122VC may be configured to operate a virtual representation (VR) of an imaged object or spatial feature 299 (as described herein with at least a portion of virtual model 400VM and appropriate image processing non-transitory computer program code). ) is performed in the autonomous transport vehicle 110 and is configured to create a virtual representation (VR) of one or more imaged objects or spatial features 299 and one or more corresponding reference predetermined features (RPF) (e.g., a virtual model ( 400VM), as presented in the reference representation (RPP)), is configured to make a comparison between the autonomous transport vehicle 110 (see Figure 15). A comparison between the virtual representation (VR) and the reference representation (RPP) by the vision system controller 122VC determines whether the object or spatial feature 299 is “unknown.” Vision system controller 122VC is configured to determine dimensions of an object or spatial feature 299 based on image analysis of at least one image (from vision system 400 of supplemental hazard sensor system 290). If the dimensions cannot be identified, vision system controller 122VC is configured to place autonomous transport vehicle 110 into a cooperative operation state to collaboratively identify object 299 with an operator. The transition from an autonomous state to a cooperative state is achieved by selectively reconfiguring the autonomous delivery vehicle 110 from an autonomous vehicle to a cooperative vehicle (i.e., selectively transitioning the autonomous delivery vehicle 110 from an autonomous operating state to a cooperative operating state). This may be done by system controller 122VC (or controller 122).

In one aspect, the transition performed by vision system controller 122VC (and controller 122) described above allows controller 122 to cause autonomous transport vehicle 110 to reach zero speed within a predetermined time period. The trajectory is configured to apply to the autonomous transport vehicle 110 to continue navigation of the autonomous transport vehicle 110 to any suitable destination associated with the detected object, wherein the motion of the autonomous transport vehicle 110 along the trajectory is configured to Coordinated with “known” and “unknown” objects positioned relative to vehicle 110. With autonomous transport vehicle 110 stationary, vision system controller 122VC allows the operator to identify objects 299 and perform mitigation actions such as maintenance (e.g., cleanup of spills, removal of misbehaving bots, etc.). and displaying an object or spatial feature 299 in a user interface (UI) to determine the location of the autonomous transport vehicle 110 (e.g., provided in the user interface (UI)) of the storage and retrieval system 100. Initiate communication to the operator. As discussed above, in one aspect, the controller 122 may at least initiate a signal/beacon to other bots to warn other bots of a traffic obstacle and to avoid the obstacle or indicate a detour area (and thus, in effect, a secondary hazard). Sensor system 290 provides one bot's danger pointer/indicator mode to other bots at the same level). In one aspect, a signal/beacon is transmitted via a local communication transmission to a system area bot task manager to manage the tasks of bots within a predetermined distance of a nearby bot or pointer bot. In another aspect, controller 122 further directs the operation of other autonomous transportation vehicles 110 operating in storage and retrieval system 100 based on vision system 400 and object information from vision system controller 122VC. It is configured to select a safe path and trajectory for the autonomous transport vehicle 110 to move the autonomous transport vehicle 110 from its position at the time of transition to a position 157 where the operator can observe the object 299 without further interference. . The vision system controller 122VC is configured to maintain the object or spatial feature 299 within the field of view 292F of at least one camera 292 and continue imaging the desired physical characteristic.

In one aspect, an operator may select or switch control of an autonomous guided vehicle (e.g., via a user interface (UI)) from autonomous operation to cooperative operation (e.g., via a user interface (UI)). remotely controls the operation of the autonomous transport vehicle 110). For example, the user interface (UI) may be a capacitive touch pad/screen, a joystick, a haptic screen, or a user interface (UI) to effect operator control input in a cooperative operating mode of autonomous transport vehicle 110. and other input devices that convey kinematic direction commands (e.g., turn, accelerate, decelerate, etc.) from the autonomous transport vehicle 110. For example, the vision system 400 of the secondary hazard sensor system 290 may be a "dashboard camera" that transmits video and/or still images of the autonomous transport vehicle 110 to the operator (via a user interface (UI)). (or a dash-camera) may be provided to allow remote operation or monitoring of areas relevant to the autonomous transportation vehicle 110 in a manner similar to that described herein with respect to the auxiliary navigation sensor system 288.

2, 4A, 4B, 9A, 10A, and 15, vision system controller 122VC (and/or controller 122) is configured to provide remote viewing to vision system 400 in one or more aspects; Such remote viewing may be provided to the operator in augmented reality or any other suitable manner (eg, non-augmented manner). For example, autonomous delivery vehicle 110 is communicatively coupled to warehouse management system 2500 (e.g., via control server 120) via network 180 (or any other suitable wireless network). do. Warehouse management system 2500 includes one or more warehouse control center user interfaces (UIs). The warehouse control center user interface (US) is configured to provide visual and/or auditory data obtained from the autonomous transport vehicle 110 or any suitable interface, such as a desktop computer, laptop computer, tablet, smartphone, virtual reality headset. It may be any other suitable user interface. In some aspects, vehicle 110 may include one or more microphones (MCP, FIG. 2), and the one or more microphones and/or remote viewing may be directed to vehicle 110, other vehicles, storage and retrieval systems, such as lifts, storage racks, etc. Can support preventive maintenance/troubleshooting diagnostics for components. The warehouse control center user interface (UI) allows the warehouse control center user to request or otherwise receive images from the autonomous transport vehicle 110 (e.g., detection of an unidentifiable object 299), and to display the requested/provided images. The image is configured for viewing in the warehouse control center user interface (UI).

Images provided and/or requested may be live video streams, pre-recorded (and stored in any suitable memory of autonomous transport vehicle 110 or warehouse management system 2500) images, or images (e.g., each image The request may be a specified (user selectable or preset) number of images or one or more static images and/or dynamic video images corresponding to a time interval, taken on request substantially in real time. Live video streams and/or image capture provided by vision system 400 and vision system controller 122VC can be captured remotely from autonomous transport vehicle 110 in real time by a warehouse control center user via a warehouse control center user interface (UI). Control operations (e.g., teleoperation) may be provided.

In some aspects, live video may be transmitted from the vision system 400 of the secondary navigation sensor system 288 and/or the secondary hazard sensor system 290 to a user interface (UI), as shown in FIGS. 9A and 15 , as shown in FIGS. Streamed as a stream (e.g., images are presented on a user interface without augmentation, and are presented as what the camera "sees"). In this aspect, Figure 9A shows live video streamed without enhancement from both front navigation cameras 420A, 420B (similar video streams may be provided in the opposite direction by rear navigation cameras 430A, 430B); 15 shows live video streamed without enhancement from the front camera 292/477A (a similar video stream can be provided in the opposite direction by the rear camera 292/477B). Similar video may be streamed from any of the cameras in the secondary navigation sensor system 288 and/or secondary hazard sensor system 290 described herein. Figure 9A shows a side-by-side presentation of front navigation cameras 420A and 420B, however, if the user requests, the video stream may be for only one of front navigation cameras 420A and 420B. When a user uses a virtual reality headset to view streamed video, an image from the right front navigation camera 420A may be presented to the viewfinder of the virtual reality headset corresponding to the user's right eye and the left front Images from navigation camera 420B may be presented to the viewfinder of the virtual reality headset corresponding to the user's left eye.

In some aspects, the live video is an augmented reality video stream (e.g., a combination of the live video and the virtual object) from the vision system 400 of the assistive navigation sensor system 288 to the user interface (UI), as shown in FIG. 10A. The combination is streamed as provided in the streamed video. In this aspect, FIG. 10A shows live video streaming augmentedly from one of the case unit monitoring cameras 410A, 410B (a similar video stream may be provided by the other of the case unit monitoring cameras 430A, 430B). but is offset by the separation distance between cameras 430A and 430B). Similar augmented video can be streamed from any camera of the assistive navigation sensor system 288 described herein. In Figure 10A, case units CU1, CU2, and CU3 are presented to the user through a user interface (UI) of a live video stream when the case unit is captured by one of the case unit monitoring cameras 410A, 410B. A virtual representation of the shelves 555 and slats 520L on which the case units CU1, CU2, and CU3 rest is live by the vision system controller 122VC or another suitable controller (e.g., control server 120). It can be inserted into the video stream to augment the live video stream. A virtual representation of shelves 555 and slats 520L (or other structures of storage and retrieval system 100) may be provided, for example, where portions of the structures are within the field of view 410AF, 410BF of case unit monitoring cameras 410A, 410B. ) (or if the field of view of the camera of the auxiliary navigation sensor system 288 is capturing video), it may be virtually inserted into the live video stream. Here, a virtual representation of the storage and retrieval structure complements/augments the live video stream with information that may be useful to the user (e.g. to provide a finished "picture" of what is being "observed" by an autonomous transport vehicle). may be virtually inserted into a live video stream, such information is not captured by the camera or is not clearly identified in the camera image data. A virtual representation of the storage and retrieval structure virtually inserted into the live video stream is obtained by the vision system controller 122VC (or control server 120) from the virtual model 400VM. Additionally, with reference to FIG. 9B , when autonomous transport vehicle 110 is in remote control operation, the video stream may be directed to a transport route (VTP) and/or destination that provides instructions to the operator as to the destination location of autonomous transport vehicle 110. It can be augmented to provide a location indicator (DL) to the operator. Transport route (VTP) and destination locator (DL) can also be provided in the video stream with autonomous transport vehicles operating in autonomous/autonomous and semi-autonomous operating modes to provide operators with an indication of the planned route and destination.

An exemplary method will be described according to aspects of the embodiment with reference to FIGS. 1A, 2, 4A, 4B, 9A, 10A and 12. The method includes providing an autonomous transportation vehicle 110 (Figure 12, block 1200) as described herein. The sensor data is generated by physical characteristic sensor system 270 (FIG. 12, block 1205), and as described herein, the sensor data includes at least one of vehicle navigation attitude or position information and payload attitude or position information. . Image data is captured with auxiliary navigation sensor system 288 (FIG. 12, block 1210), where the image data supplements information from physical characteristic sensor system 270 to determine vehicle navigation attitude or position and payload, as described herein. It tells you at least one of your posture or location.

The method also uses the vision system controller 122VC to determine vehicle attitude and position (FIG. 12, block 1220) from information in the physical property sensor system 270 to enable autonomous transportation across the storage and retrieval system 100 facility. It may include performing independent guidance of the vehicle 110. Vision system controller 122VC may also determine the pose and position (FIG. 12, block 1225) of the payload (e.g., case unit (CU)) from information in physical characteristic sensor system 270, as described herein. Together, independent underpicking and placement of payloads to and from storage locations, and independent underpicking and placement of payloads in payload bed 210B can be performed.

Vision system controller 122VC may register captured image data and generate at least one image from the image data in one or more of the predetermined features (Figure 12, block 1215), as described herein. The at least one image is formatted as a virtual representation (VR) of one or more predetermined features to match one or more corresponding criteria (e.g., shape and/or location of the predetermined features imaged) of the predetermined features of the reference representation 400VMR. It provides a comparison with the corresponding features of the virtual model (400VM), which is provided as a standard for identifying . As described herein, the vision system controller 122VC is configured to cause a virtual representation (VR) of one or more imaged features in the autonomous transportation vehicle 110, and the vision system controller 122VC is configured to create a virtual representation (VR) of one or more imaged features of the predetermined features in the autonomous transportation vehicle 110. A comparison between the virtual representation (VR) and certain features of one or more corresponding references (of the reference representation (400VMR)) is configured to be made in the autonomous transport vehicle (110). The vision system controller 122 receives the autonomous guided vehicle's attitude and position information registered by the vision system controller 122VC from the physical characteristic sensor system 270 based on a comparison between the virtual representation (VR) and the reference representation (400VMR), and Payload attitude and location information (FIG. 12, block 1230) can be checked.

The vision system controller 122VC identifies deviations in the attitude and position of the autonomous transport vehicle 110 or deviations in the attitude and position of the payload based on a comparison between the virtual representation (VR) and the reference representation (400VMR) (FIG. 12, Block 1235) may update and complete payload attitude and position information from physical characteristic sensor system 270 based on the deviation. In the method, vision system controller 122VC may detect attitude error (for the autonomous guided vehicle and/or payload) (FIG. 12, block 1240) and information from physical characteristic sensor system 270 (e.g., vision Attitude and position information (for the autonomous guided vehicle and/or payload) from the physical characteristic sensor system 270 based on image analysis of at least one image (from system 400) and at least one identified deviation. Accuracy can be determined, and a confidence value can be assigned based on at least one posture error and accuracy. If the confidence value is less than a predetermined threshold, the vision system controller 122VC may load the payload based on the pose and position information generated from the virtual representation (VR) instead of the pose and position information from the physical property sensor system 270. switch processing; and/or if the confidence value is less than a predetermined threshold, the vision system controller 122VC may base the pose and position information generated from the virtual representation (VR) instead of the pose and position information from the physical property sensor system 270. to switch the navigation of the autonomous guide vehicle 110. After the diversion, the controller continues navigation of the autonomous guided vehicle to the destination, selects a safe path and trajectory for the autonomous guided vehicle to move the autonomous guided vehicle from the diversion location to a safe location for shutdown, or communicates to the operator. configured to identify kinematic data and a destination of the autonomous guided vehicle for operator selection of autonomous guided vehicle control from automatic operation to semi-automatic operation or manual operation via the user interface device; and/or continue processing the autonomous guided vehicle to its destination, or initiate communication to the operator to change control of the autonomous guided vehicle from an automatic payload handling operation to a semi-autonomous payload handling operation or a manual payload handling operation via a user interface device. It is configured to identify payload data in conjunction with operator selection.

The controller provides real-time augmented reality images to the operator via a wireless communication system (e.g., network 180) that communicatively couples the vision system controller 122VC and an operator/user interface (UI) to one or more A simulated image (see FIGS. 9A, 10A, 10B) combining a virtual representation (VR) of an imaged predetermined feature and one or more corresponding reference predetermined features (RPF) of a reference representation (RPP) (FIG. 12, block 1245). ) is transmitted. Vision system controller 122VC receives real-time operator commands for the traversing autonomous guided vehicle 110, which commands are generated by real-time augmented reality images (see FIGS. 9A, 10A, 10B) and operator commands by vision system controller 122VC. Responds to changes in real-time augmented reality images transmitted to .

1A, 1B, 2, 4A, 4B, and 14, a mixed pallet load (MPL) (e.g., a pallet load is divided into units as shown in FIG. 1B) according to a predetermined order out sequence. Autonomous transport vehicle including case unit multi-picking batch operation with on the fly sortation of case units to produce cases with different stocks or mixed cases maintaining An example of a case unit transfer transaction of 110 will be described according to one aspect of the disclosed embodiment. Suitable examples of multi-picking/batch operations of autonomous transport vehicles 110 in which aspects of the disclosed embodiments may be utilized include, but are not limited to, U.S. Patent No. 10,562,705, entitled “Storage and Retrieval System,” issued February 18, 2020; U.S. Patent No. 10,839,347, entitled “Storage and Retrieval System,” issued November 17, 2020; U.S. Patent No. 10,850,921, entitled “Storage and Retrieval System,” issued December 1, 2020; U.S. Patent No. 10,954,066, entitled “Storage and Retrieval System,” issued March 23, 2021; and U.S. Patent No. 10,974,897, entitled “Storage and Retrieval System,” issued April 13, 2021, the disclosures of which are incorporated herein by reference in their entirety. Autonomous transport vehicle 110 picks at least a first case unit (CUA) from a first shelf at first storage location 130S1 in picking aisle 130A1 (FIG. 14, block 1400). As described above, localization of autonomous transport vehicle 110 relative to case unit (CUA) of storage location 130S1 may be performed using physical characteristic sensor system 270 and/or auxiliary navigation sensors in the manner described herein. This is done by system 288.

Autonomous transport vehicle 110 traverses picking aisle 130A1 and buffers at least a first case unit (CUA) within payload bed 210B (FIG. 14, block 1410). Autonomous transport vehicle 110 traverses picking aisle 130A1 to second storage location 130S2 and picks at least a second case unit (CUB) that is different from at least a first case unit (CUA) (FIG. 14, Block 1420). At least the second case unit (CUB) is described as being in the same picking aisle (130A1) as at least the first case unit (CUA), but in other embodiments at least the second case unit (CUB) is in a different aisle or storage and retrieval system. It may be in any other suitable storage location (e.g., transfer station, buffer, inbound lift, etc.). Localization of autonomous transport vehicle 110 relative to case unit (CUB) at storage location 130S2 may utilize physical characteristic sensor system 270 and/or auxiliary navigation sensor system 288 in the manner described herein. It is carried out. At least the first case unit (CUA) and at least the second case unit (CUB) may include one or more cases in an order order corresponding to a predetermined case order output order of the mixed case.

Autonomous guided vehicle 110 travels through picking aisle 130A1 and/or transfer deck 130B with both at least a first case unit (CUA) and at least a second case unit (CUB) maintained within payload bed 210B. Cross and move to a predetermined destination (for example, outbound lift (150B1)). The positions of at least the first case unit (CUA) and at least the second case unit (CUB) within the payload bed 210B are determined by at least one case unit monitoring camera (410A, 410B) and one or more three-dimensional imaging systems (440A, 440B). ) and one or more case edge detection sensors (450A, 450B), at least one case unit monitoring camera (410A, 410B), one or more three-dimensional imaging systems (440A, 440B), and one or more case edge detection sensors ( Arrange (e.g., alignment blade 471, pusher 470, and/or puller 472) relative to each other within payload bed 210B based on data obtained from 450A, 450B. (e.g., auxiliary navigation sensor system 288 may be installed in vehicle 110 in a manner substantially similar to that described in U.S. Patent No. 10, 850,921, the disclosure of which is incorporated herein by reference in its entirety). performs, at least in part, immediate justification and/or classification of mounted case units). Autonomous transport vehicle 110 is located (e.g., deployed) to a destination location using physical characteristic sensor system 270 and/or auxiliary navigation sensor system 288 in the manner described herein. At the destination location, autonomous transport vehicle 110 deploys at least a first case unit (CUA) and/or at least a second case unit (CUB) (FIG. 14, block 1430), wherein transfer arm 210A has one or more physical characteristics. Movement is based on data acquired by sensor system 270 and/or auxiliary navigation sensor system 288.

An exemplary method will be described in accordance with aspects of the embodiment with reference to FIGS. 1A, 2, 4D, 15 and 16. The method includes providing an autonomous transportation vehicle 110 (Figure 16, block 1700) as described herein. Autonomous transport vehicle 110 autonomously navigates to different locations with a navigation system and detects predetermined targets at different locations while incidentally capturing image data with auxiliary hazard sensor system 290 (FIG. 16, block 1710). It is configured to operate (FIG. 16, block 1705) by performing a transfer operation. As described herein, image data may be stored in the facility 100 as viewed by at least one camera 292 of the auxiliary hazard sensor system 290 along with the autonomous transport vehicle 110 at different locations in the facility 100. Announces objects and/or spatial features 299 (having unique physical properties) within at least some of them.

The method also uses the vision system controller 122VC to determine the presence of certain physical characteristics of at least one object or spatial feature from information in the auxiliary hazard sensor system 290 (FIG. 16, block 1715), and In response, the step may include selectively reconfiguring the vehicle from an autonomous state to a cooperative vehicle state to cooperate with the operator (FIG. 16, block 1720), wherein the vehicle in the cooperative state allows the vehicle to continue performing vehicle operations. arranged to receive operator commands to ultimately determine an object or spatial feature 299 as a hazard (FIG. 16, block 1725) and identify the vehicle's mitigation actions for the hazard (FIG. 16), as described herein. Block 1730).

The vision system controller 122VC may also register the captured image data and generate therefrom at least one image for the presence of certain physical characteristics of the at least one object or spatial feature 299 (Figure 16, block 1735), as described herein, the at least one image is formatted as a virtual representation (VR) of predetermined physical characteristics of at least one object or spatial feature 299, such that the predetermined characteristics of the reference representation 400VMR may provide a comparison with one or more corresponding criteria (e.g., corresponding features of the virtual model 400VM that serve as a reference for identifying the shape and/or location of the imaged object or spatial feature 299). there is. As described herein, the vision system controller 122VC is configured to cause a virtual representation (VR) of the imaged object or spatial feature 299 to be made on (e.g., onboard) the autonomous transport vehicle 110, or a comparison between a virtual representation (VR) of the spatial feature 299 and certain features of one or more corresponding references (of the reference representation 400VMR) is configured to be made in the autonomous transport vehicle 110 .

In the method, vision system controller 122VC may determine the presence of an unknown physical characteristic of at least one object or spatial feature and transition autonomous transportation vehicle 110 from an autonomous operating state to a cooperative operating state. Once the above-mentioned transition is made, the controller 122 stops the autonomous transportation vehicle 110 relative to the object or spatial feature 299, or moves the autonomous transportation vehicle 110 to the transition position by using a user interface device (UI). The autonomous guided vehicle is configured to select a path or trajectory to move the autonomous guided vehicle to a location to initiate communication with the operator to identify an object or spatial feature 299.

Controller 122VC provides real-time augmented (or non-augmented) reality via a wireless communication system (e.g., network 180) that communicatively couples vision system controller 122VC and an operator/user interface (UI). An image that provides an image to an operator to combine a virtual representation (VR) of one or more imaged objects or spatial features 299 with predetermined features (RPF) of one or more corresponding references in a reference representation (RPP) (see FIG. 15 ). ) (Figure 16, block 1740). Controller 122 receives real-time operator commands to autonomous transport vehicle 110, such commands may include real-time augmented reality or non-augmented images (see FIG. 15) and real-time augmented reality images transmitted to the operator by vision system controller 122VC. Or, it responds to changes in the non-augmented image.

2, 19, 20, 25A to 25C, the autonomous transport vehicle 110 includes a controller 122, a case handling assembly 210, and a peripheral electronics section 778, respectively coupled to the drive section 261D. and other components/features of autonomous transport vehicle 110 described herein (see FIGS. 25A-25C) to form control system 122CS. Control system 122CS performs each autonomous operation of autonomous transportation vehicle 110 described herein. Controller system 122CS may be configured to provide communications, supervisory control, vehicle location, vehicle navigation and motion control, payload sensing, payload transfer, and vehicle power management as described herein. In these and other aspects, the control system may also be configured to provide any suitable service to vehicle 110. Control system 122CS includes any suitable non-transitory program code and/or firmware to configure vehicle 110 to perform vehicle operations described herein. Control system 122CS is capable of remotely updating control system firmware/software, remote debugging of vehicle 110, remote operation of vehicle 110, location tracking of vehicle 110, tracking operating status of vehicle 110, and vehicle ( 110), but is not limited to one or more of the following:

For example, as shown in FIGS. 25A-25C, control system 122CS includes controller 122, vision system controller 122VC, and power management section 444 (switching device 449) as described herein. and a monitoring and control device (447). In some aspects, one or more of vision system controller 122VC and power management section 444 are at least partially integrated into controller 122; In another aspect, one or more of system controller 122VC and power management section 444 are separate but communicatively coupled to controller 122. Components of the control system (e.g., sensors, cameras, lights, drive sections, motors, etc.) may be distributed throughout autonomous transport vehicle 110 and in any suitable manner (as illustrated in FIGS. 25A-25C ). It can be communicatively coupled to the controller 122.

Controller 122 includes at least one of an autonomous navigation control section 122N and an autonomous payload processing control section 122H. The autonomous navigation control section 122N deterministically stores the current and expected state, attitude, and position of the autonomous transport vehicle 110 in volatile memory (e.g., memory 446 of the comprehensive power management section 444 of controller 122). It is configured to store and maintain status and attitude navigation information of the autonomous guided vehicle, which is described as deterministic (and provided in real time). Autonomous transport vehicle status and attitude navigation information includes both past and present autonomous guided vehicle status and attitude navigation information. State, attitude, and location information is deterministic (and available in real time), describing current and projected states, postures, and positions in up to six degrees of freedom (X, Y, Z, Rx, Ry, Rz), providing historical, current, and Ensure that expected conditions, postures, and positions are fully described. Autonomous payload processing control section 122H is configured to register and maintain current payload identification information, status and attitude information (e.g., both historical and current information) in volatile memory (e.g., memory 446). It is composed. Payload identification, state, and attitude information may include past and current payload identification, the autonomous transport vehicle's frame of reference (e.g., X, Y, Z coordinate axes, and appropriate data surfaces within payload bed 210B) Describes the payload attitude and state location for )) and the picking/placement location of current and past payloads.

As described herein, controller 122 has an overall power management section 444 (also referred to as a power distribution unit) that is separate and distinct from other sections of controller 122 (e.g., vision system controller 122VC). , see FIG. 26). As will be described herein, power distribution unit 444 is communicatively coupled with power supply section 481 to monitor the charge level (e.g., voltage level or current level) of power supply section 481. . As also described herein, power distribution unit 444 is a drive section that supplies power from power supply section 481 to drive section 261D, case handling assembly 210, and peripheral electronics section 778, respectively. 261D, case handling assembly 210 and peripheral electronics section 778 are each connected to a branch circuit 482 (also referred to as a branch power circuit, see FIG. 26 as a non-limiting example). The power distribution unit 444 is configured to comprehensively manage power consumption for each branch circuit 482 based on the demand level of each branch circuit 482 relative to the level of charge available from the power supply section 481. do.

Power distribution unit 444 includes a monitoring and control device 447 (herein referred to as monitoring device 447), a switching device 449 (including switch 449S), a memory 446, and a wireless communication module. 445 and an analog-to-digital converter 448 (referred to herein as AD converter 448). Monitoring device 447 is any suitable processing device configured to at least monitor the current usage and fuse status of branch power circuits 482 and control shutdown of one or more selected branch power circuits 482 as described herein. For example, the monitoring device 447 may be a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), or a system on chip integrated circuit. One or more of a SOC) and a central processing unit (CPU). Monitoring device 447 operates independently of controller 122 and vision system controller 122VC, and monitor device 447 manages one or more low-level systems of autonomous transport vehicle 110 (e.g., at least power distribution). ) is programmed with non-transitory code to do this.

1A, 1B, 2, 19, 20, 25A-25C and 26, power distribution unit 444 is configured to communicate with and control at least one branch device 483. For example, power distribution unit 444 may include analog sensors 483C (e.g., case edge detection sensors, line follow sensors 275 and other analog sensors described herein), digital sensors 483B (e.g. For example, cameras 410, 440, 450 of vision system 400 and other digital sensors described herein), lighting 483A, caster 250, drive/traction wheel 260, transfer arm ( 210A), extension motors (667A-667C), transfer arm lift motors (669), payload alignment motors (668A-668F) of payload bed (210B)/transfer arm (210A), suspension lock motors, and autonomous transport vehicles. communicatively coupled to one or more of any other suitable features of 110 (see FIGS. 20, 21 and 26), such as analog sensor 483C, digital sensor 483B and lighting 483A, caster 250, Drive/traction wheel (260), transfer arm (210A), extension motor (667), transfer arm lift motor (669), payload alignment motor (668) of payload bed (210B)/transfer arm (210A), suspension. Energize (e.g., turn on and maintain powered operation) the locking motor, and any other suitable features. Here, the power distribution unit 444 receives a command from the controller 122 to operate one or more of the analog sensor 483C and the digital sensor 483B, so that one or more of the analog sensor 483C and the digital sensor 483B One or more of the attitude and position information of autonomous transport vehicles within the storage structure 130 of the storage and retrieval system 100 may be obtained in a manner substantially similar to that described herein and in the references below, which references U.S. Patent No. 8,425,173, entitled “Autonomous Transportation of Storage and Retrieval Systems,” issued April 23, 2013; U.S. Patent No. 9,008,884, entitled “Bot Location Detection,” issued April 14, 2015; and U.S. Patent No. 9,946, 265, entitled “Bot with High-Speed Stability,” issued April 17, 2018, the disclosures of which are incorporated herein by reference in their entirety. Power distribution unit 444 is configured to process and filter (in any suitable manner) sensor data acquired by one or more of analog sensor 483C and digital sensor 483B. Power distribution unit 444 may also transmit control signals transmitted by controller 122 (or vision system controller 122VC) to one or more of analog sensors 483C and digital sensors 483B (in any suitable manner). Can be configured to process and filter. If the sensor is an analog sensor 483C, power distribution unit 444 includes an AD converter 448 that converts analog sensor data to digital sensor data for filtering and processing by power distribution unit 444.

The autonomous transport vehicle is coupled to frame 200 (or any other location on autonomous transport vehicle 110) and includes lights 483A that illuminate a portion of storage structure 130 adjacent to autonomous transport vehicle 110. (see lighting/LED in FIG. 20 and also FIGS. 25A-25C). Power distribution unit 444 is configured to control the operation of lighting 483A. For example, power distribution unit 444 is configured to provide a pulse width modulation control signal to light 483A to operate light 483A in a manner that minimizes power consumption. Here, the pulse width modulated control signal may be used to perform a given autonomous transportation vehicle task (e.g., reading a barcode using the vision system 400, detecting case unit features using the vision system, remote control performed by the vision system as described herein). It is configured to minimize the amount of power drawn from the power supply section 481 to illuminate the lighting 483A (to illuminate a portion of the storage and retrieval system 100 for operator observation). Light 483A may be any suitable light, including but not limited to light emitting diodes (LEDs).

With continued reference to FIGS. 2 , 19 , 20 , 25A-25C and 26 , power distribution unit 444 provides power distribution of autonomous transport vehicle 110 to preserve higher level functions/operations of autonomous transport vehicle 110 . Configured to manage power demands. Power distribution unit 444 is connected to each branch power circuit 482 (each branch device 483A-483F...483n, collectively referred to as branch device 483, where n represents the maximum number of branch devices. integers) are disposed thereon, see FIGS. 19, 20, 21, 25a to 25c and 26) and are configured to comprehensively manage the demand charge level of each branch circuit 482 and Each of the branch power circuits 482 can be turned off in a predetermined pattern based on the level of charge available from the power supply section 481. A predetermined pattern (e.g., a pattern for turning off branch power circuit 482) is arranged to turn off branch power circuit 482 as the level of charge available from power supply section 481 decreases, so that the controller ( It is possible to maximize the level of charge available from the power supply section 481 towards 122). The predetermined pattern is arranged to turn off the branch power circuit 482 as the level of charge available from the power supply section 481 decreases, such that the level of charge available to the controller 122 increases with the use of the power supply section 481. Based on the available charge level, the demand charge level of the controller 122 can be equal to or exceeded for a maximum period of time.

As an example of shutting down the branch power circuit 482 and maintaining the operation of the controller 122, the monitoring device 447 of the power distribution unit 444 monitors the voltage of the power supply section 481 as described herein (FIG. 23 , block 23800), and is configured to shut down components/systems (e.g., analog sensors, digital sensor drive systems, communication systems, etc.) of the autonomous transport vehicle 110 in a sequential shutdown sequence, wherein the sequential shutdown sequence: The respective shutdown operation of depends on the respective threshold voltage of the power supply section. For example, power supply section 481 has a fully charged voltage of V1. Once power distribution unit 444 detects voltage V1 , the components/systems of autonomous transport vehicle 110 may be substantially fully operational to perform transport of case units throughout storage structure 130 .

Depending on the operation of the autonomous transport vehicle 110, the voltage of the power supply section 481 may drop to a first predetermined threshold voltage V2, where V2 is less than V1 (and the power distribution unit 444 may detecting these voltage drops). The power distribution unit 444, which monitors the power supply section 481 voltage, detects that the power supply voltage drops to a voltage approximately equal to the first predetermined threshold voltage V2 (FIG. 23, block 23810); With the power supply section 481 voltage at approximately the first predetermined threshold voltage (V2), the power distribution unit 444 is configured to drive a case unit processing component/system (e.g., arm extension actuation) of the autonomous transport vehicle 110. Actuate a switch to remove power (e.g., shut down) from the branch power circuit corresponding to section 667, payload alignment drive section 668, arm/case unit alignment position sensor, suspension lock, etc. 23, block 23820), such that the remaining power in the power supply section 481 may be utilized to traverse the autonomous transport vehicle to a charging station/location or other desired location within the storage structure 130. In aspects where the power supply section 481 charge is insufficient to complete the traverse of the autonomous transport vehicle 110 to the charging station, the controller 122 may direct the autonomous transport vehicle 110 to a safe location, as described herein. For example, the autonomous vehicle may be traversed to a predetermined location in the storage and retrieval system to which an operator may access the autonomous vehicle for maintenance or removal from the storage structure 130. Suitable examples of charging stations that may be deployed in a storage and retrieval system include U.S. Patent No. 9,469,208, entitled “Rover Charging System,” issued October 18, 2016; U.S. Patent No. 11,001,444, entitled “Storage and Retrieval System Rover Interface,” issued May 11, 2021; and U.S. Patent Application No. 14/209,086, entitled “Rover Charging System,” filed March 13, 2014, the disclosures of which are incorporated herein by reference in their entirety.

Power distribution unit 444 continues to monitor the voltage of power supply section 481 for whether the power supply section voltage drops to a subsequent (e.g., next) lower threshold voltage (Figure 23, block 23830). For example, if a threshold voltage of the power supply section 481 of V3 (where V3 is less than V2) is detected by the power distribution unit 444, the power distribution unit 444 may Sections/systems (e.g., left and right drive/traction wheels 260A, 260B, Figures 2 and 21), caster wheel steering drive section 600M, Figure 2, traction control system 666, Figure 21, perform vehicle navigation Receive power from branch power circuits 482 (e.g., circuits 483D, 483F) corresponding to sensor controllers and sensors (e.g., vision systems, line tracking sensors, etc., provided with sensor system 270). Actuate switch 449S (FIG. 23, block 23840) to remove (e.g., shut down), thereby allowing residual power in power supply section 481 to operate controller 122 of autonomous transport vehicle 110. It can be used to perform. Here, basic communications between autonomous transport vehicle 110 and control server 120 and/or operator may also be shut down to conserve power for controller 122. As previously discussed, the communication module 445 of the power distribution unit 444 provides communication between the controller 122 and the control server 120 and/or an operator (e.g., via a laptop, smartphone/tablet, etc.). It operates to maintain primary communication channels.

As above, power distribution unit 444 continues to monitor the voltage of power supply section 481 for the next subsequent lower threshold voltage (Figure 23, block 23850). For example, if a threshold voltage (V4, where V4 is less than V3) of power supply section 481 is detected by power distribution unit 444, power distribution unit 444 may initiate shutdown of controller 122. configured to initiate (Figure 23, block 23860), so that controller 122 (and its software) may not be adversely affected by low voltage/low current faults or power losses. Here, the controller 122 is configured to, upon receiving an indication from the power distribution unit 444 that the level of available charge from the power supply section 481 to the controller 122 is decreasing and approaching below the demand level of the controller 122, Controller 122 is configured to enter suspension and hibernation. When controller 122 is in a stop and hibernation (e.g., shutdown) state, power distribution unit 444 may also shut itself down, effectively stopping all operations of autonomous transport vehicle 110.

It will be understood that threshold voltage V4 is described as the “lowest threshold voltage” such that detection of threshold voltage V4 initiates shutdown of controller 122. However, the above shutdown sequence performed by the power distribution unit 444 is exemplary only, and in other embodiments any suitable number of corresponding vehicle components/components may be used to conserve power in the power supply section 281 It will be appreciated that there may be any suitable number of threshold voltages that cause the system to shut down. For example, blocks 23830 and 23840 of FIG. 23 may be repeated in a loop until the next lowest threshold voltage is reached. Here, in descending order of values of the threshold voltage, each threshold voltage is known to the power distribution unit 444 (e.g. stored in memory 446 and accessible by the monitoring device 447), and thus the next lowest threshold voltage. The loop ends when .

With continued reference to FIGS. 2, 19, 20, 25A-25C, and 26, another exemplary shutdown operation will be described. Here the autonomous transport vehicle 110 has a power supply section 481 with a fully charged voltage of approximately 46V (in other embodiments the fully charged voltage may be greater or less than 46V). Power distribution unit 444 monitors the voltage output by power supply section 481 during operation of autonomous transport vehicle 110 in a manner similar to that described above with respect to FIG. 23 . Here, when the power supply section 481 outputs with a threshold voltage of about 22V (in other embodiments the output voltage may be greater or less than about 22V), the power distribution unit 444 is configured to drive the traction motor 261M of the autonomous transport vehicle. ) and other features (e.g., sensors related to navigation/traversing of the autonomous transportation vehicle) by operating switch 449S to disable driving of the autonomous transportation vehicle.

Power distribution unit 444 continuously monitors the output voltage of power supply section 481 for the next lowest threshold voltage of approximately 20V (in other embodiments the output voltage may be greater or less than 20V). When a threshold voltage of approximately 20V is detected, power distribution unit 444, via controller 122, places any case units (CUs) carried by autonomous transport vehicle 110 into a known safe state within payload bed 210B. (For example, it is positioned backward into the payload bed 210B at a predetermined alignment position). In another aspect, when the autonomous transport vehicle 110 is positioned in front of a predetermined destination/placement location for a case unit (CU) being carried by the autonomous transport vehicle 110, the controller 122 determines the case unit (CU) Instead of retracting into the payload bed 210B, the transfer arm 210A can be extended to place the case unit (CU) at the destination position (after placement of the case unit (CU), the transfer arm 210A is placed in a safe/home position. Note that it is retracted within the payload bed (21B) to the (home) position).

Power distribution unit 444, upon detecting the next lowest threshold voltage of about 18V of power supply section 481 (in other embodiments the output voltage may be greater or less than about 18V), powers the vision system 400 and the other 24V. It is configured to actuate switch 499S to shut down peripheral power supply sections (including, but not limited to, case locator sensors, vehicle location sensors, hot swap circuitry, etc.). Once the output threshold voltage of the next lowest power supply section 481 of approximately 14 V is detected (in other embodiments the output voltage may be greater or less than approximately 14 V), the power distribution unit 444 provides It is configured to operate switch 499S to disable onboard and off-board communications (e.g., wireless communications module 445 and onboard Ethernet communications). The power distribution unit 444 continuously monitors the power supply section 481 output voltage for the next lowest threshold voltage of approximately 12V (in other embodiments the output voltage may be greater or less than approximately 12V), when the output voltage of approximately 12V is reached. When detected, power distribution unit 444 turns off the lights (e.g., LEDs) of autonomous transport vehicle 110 and provides a command signal to controller 122 to cause controller 122 to hibernate/sleep ( It can be placed in sleep mode. Once the lowest power supply section 481 output threshold voltage of about 10 V is detected by power distribution unit 444 (in other embodiments the output voltage may be greater or less than about 10 V), power distribution unit 444 may Performing a complete shutdown of vehicle 444 such that controller 122, vision system controller 122VC, and other appropriate programmable devices (e.g., FPGA, CPLD, SOC, CPU, etc.) of the autonomous transportation vehicle are turned off/shutdown. It can be done as much as possible.

Monitoring device 447 is configured to monitor the operation and status of power supply section 481 substantially continuously (e.g., while autonomous transport vehicle 110 is in operation). For example, the monitoring device 447 monitors the voltage of the power supply section 481 substantially continuously (or at any suitable predetermined time interval) (e.g., using any suitable voltage sensor), is configured to communicate a low voltage condition (e.g., voltage drops below a predetermined voltage level) to controller 122 so that controller 122 can perform a safe state of autonomous transport vehicle 110. For example, if there is an indication from the power distribution unit 444 that a decrease in the available charge level of the power supply section 481 from the power supply section 481 to the branch power circuit of the drive section 261D is imminent (Figure 21), the controller 122 allows the autonomous transport vehicle 110 to follow a predetermined auxiliary path (AUXP) and auxiliary track (AUXT) (safely, without colliding with other vehicles 110, or interfering with other vehicle paths). To navigate to a predetermined bot secondary resting location 157 of a storage and retrieval facility 130 (e.g., a structure) (known as not to block and not to pass through the destination location, see FIG. 1B). It is configured to command the drive section 261D. The predetermined secondary stop location 157 is a human-accessible area or a safe, uncrowded area of the picking aisle 130A or transfer deck 130B (e.g., “Integrated Security,” published October 2, 2018). No. 10,088,840, entitled “Automatic Storage and Retrieval System with Personnel Access Areas and Remote Rover Shutdown,” the disclosure of which is incorporated herein by reference in its entirety.

Upon indication from the power distribution unit 444 that the available charge level of the power supply section 481 from the power supply section 481 to the branch circuit of the payload processing section 210 is decreasing (see Figure 21). , the controller 122 controls the payload handling actuator or transfer arm 210A (e.g., using one or more of the arm extension drive section 667 and the arm lift drive section 669) and any payload thereon. is configured to command the payload processing section 210 to move the load (e.g., using the payload alignment drive section 668) to a predetermined safe payload location within the payload bed 210B. A safe payload position may ensure that the payload is securely secured within the payload bed 210B without protruding outside the payload bed.

1A, 1B, 2, 19, 20, and 21, controller 122 may also actively monitor the health status of autonomous transport vehicle 110 and perform on-board diagnostics of vehicle systems, as described herein. It can be configured to perform. By way of example, vehicle system health may be monitored in any suitable manner, such as by monitoring the current used and fuse status of the vehicle system (and branch power circuit 482 of which branch device 483 is a part). do. Here, the controller 122 includes at least one of a vehicle status monitor 447V, a drive section status monitor 447D, a payload processing section status monitor 447H, and a peripheral electronics section status monitor 447P. Vehicle health monitor 447V, drive section health monitor 447D, payload processing section health monitor 447H, and peripheral electronics section health monitor 447P may be sections of monitoring device 447. The controller also includes a status register section 447M, which may be a section of memory 446 (or memory 122M or any other suitable memory accessible by controller 122).

Vehicle condition monitor 447V may monitor the dynamic response of frame 200 and wheel suspension using, for example, any suitable vehicle condition sensor (e.g., an accelerometer) coupled to the frame ( For example, as described in U.S. Provisional Patent Application No. 63/213,589, entitled “Autonomous Transportation Vehicle with Synergistic Vehicle Dynamic Response,” filed June 22, 2021, the disclosure of which is incorporated by reference in its entirety. (incorporated herein). If the dynamic response is outside a predetermined range, vehicle health monitor 447V may request maintenance (e.g., displayed in a user interface (UI)) to the operator of storage and retrieval system 100 (via controller 122). ) can be performed. In another aspect, any suitable characteristic of the vehicle may be monitored by vehicle health monitor 447V.

Drive section status monitor 447D monitors the power implemented by motors 261M of drive section 261D, drive section (e.g., wheel encoder, etc.) status, and the status of traction control system 666. . If the power usage of motor 261M, the responsiveness of drive section sensors, and/or the response of the traction control system deviate from predetermined operating characteristics, drive section status monitor 447D may notify the operator of storage and retrieval system 100 ( A maintenance request (e.g., displayed in a user interface UI) may be retrieved (via controller 122).

Payload handling section status monitor 447H may monitor the status of case handling assembly sensors and power sources drawn by motors (e.g., extension lift, alignment, etc.) of case handling assembly 210. If the power usage of the case handling assembly motors and/or case handling assembly sensor responses are outside of predetermined operating characteristics, the payload processing section health monitor 447H may notify the operator of the storage and retrieval system 100 (controller 122) You can retrieve a maintenance request (e.g. displayed in a user interface UI) via

Peripheral electronic section health monitor 447P may monitor sensor system 270 and at least one peripheral motor 777. If the power usage of at least one peripheral motor 777 and/or sensor (of sensor system 270) response deviates from predetermined operating characteristics, peripheral electronics section health monitor 447P may determine storage and retrieval system 100. A maintenance request (e.g., displayed in a user interface UI) may be brought to the operator (via controller 122).

As a non-limiting example of condition monitoring, power distribution device 444 may monitor the current of branch power circuits 482 (either directly with ammeters or indirectly by monitoring the voltage and/or resistance of each branch power circuit 482). etc.) and is configured to monitor the status of each fuse 484 of the branch power circuits 482. Temporal feedback (e.g., input data regarding current and fuse status is processed within milliseconds by the monitoring device 447 such that the processed data is substantially immediately available as feedback) can be provided to the autonomous transport vehicle 110 operator. and/or to one or more of the controller 122 and control server 120 to effect service/maintenance requests.

Real-time feedback performed by a monitoring device 447 that monitors at least branch power circuit 482 current and fuse 484 status provides on-board diagnostics and health monitoring of the autonomous transport vehicle system. Power distribution device 444 is configured to detect fuse 484 status (e.g., inoperable or operable) based, for example, on the current of each branch power circuit 482. If no current is detected in each branch power circuit 482, the monitoring device 447 determines that fuse 484 is inoperable and requires service; otherwise, if current is detected, fuse 484 is tripped. Determine that it is possible (i.e., an error condition (see Figure 5 for example) is detected). The monitoring device 447 feeds back the fuse status (e.g., fault status) to the control server 120 and/or the operator, etc. through the communication module 445 to plan the service schedule of the autonomous transportation vehicle 110. do. As may be realized, the power distribution device 444 is configured to monitor each branch power circuit 482 separately from each other with respect to the monitoring device 447 in case the fuse is determined to be inoperable. It also identifies the branch power circuit 482 of which the fuse is a part to reduce downtime and troubleshooting of the autonomous transport vehicle 110 for fuse 484 replacement.

Increased current in the branch power circuit as sensed by monitoring device 447 may indicate an impending drive motor failure, an impending bearing failure, or an impending electrical/mechanical failure. As described above, each branch power circuit is individually monitored to also identify the corresponding branch power circuit 482 where increased current is detected. The monitoring device 447 provides an increased current value (e.g. a fault condition) and, via the communication module 445, for example a branch power circuit with an overcurrent to the control server 120 and/or the operator ( 482) is identified so that service of the autonomous transportation vehicle 110 can be scheduled.

[0174] Power distribution device 444 may be configured to support peripheral devices (e.g., transfer arm 210A, payload alignment pusher/puller, wheel encoder, navigation sensor system (e.g., described herein), payload Alignment pusher/puller, voltage regulator 490, branch central processing unit (CPU) 491, and/or position sensor of a payload positioning sensor system (e.g., described herein). (492) (e.g., as described herein). A suitable example of a payload alignment pusher/puller is Attorney Docket No. 1127P015753-US(-#3), filed August 24, 2021. is described in U.S. Provisional Patent Application No. 63/236,591 with U.S. Patent No. 9187244, entitled “Storage and Retrieval System,” U.S. Patent No. 94991630, titled “Bot Payload Alignment and Sensing,” and filed on November 22, 2011, “Material Handling System Using Autonomous Transport and Transfer Vehicle.” ", U.S. Patent No. 7991505, entitled "Autonomous Transport Vehicle Charging System," U.S. Patent No. 9561905, issued February 7, 2017, U.S. Patent No. 9082112, entitled "Autonomous Transport Vehicle Charging System," issued December 2017 U.S. Patent No. 980079, entitled “Storage and Retrieval System Transport Vehicle,” dated 26 November 2015, and U.S. Patent 9187244, titled “Bot Payload Alignment and Sensing,” dated November 17, 2016. U.S. Patent No. 8965619, entitled “Autonomous Transport Arm Drive System,” followed by U.S. Patent No. 9,499,338, dated November 22, 2015; Described in 8965619; U.S. Patent No. 9008884, issued on April 14, 2015, for “Bot Position Sensing,” and U.S. Patent No. 8425173, issued on April 23, 2013, for “Autonomous Transports for Storage and Retrieve Systems,” on April 15, 2014. U.S. Patent No. 8696010, issued to “Suspension System for Autonomous Transports,” the disclosures of which were previously incorporated herein by reference. By way of example, the monitoring device 447 may be configured to monitor the communication between the position sensor 492 and the controller 122, the communication between the branch device controller(s) 491 and the controller 122, and the voltage from the voltage regulator 490. It is configured to monitor. When communication is expected from sensor 492 and/or branch device controller 491, monitoring device 447 registers a fault (e.g., time stamp) in memory 446 and reports such fault status to the control server ( 120) and/or may be communicated (e.g., via communication module 445) to the operator performing the maintenance request. If the branch device 483/branch power circuit 482 from which the fault is acquired is of lower operational significance, the monitoring device 447 continues to monitor and register faults from the branch device 483/branch power circuit 482 and A service request message may be sent to the control server 120 or operator based on the frequency of defects or any other suitable criterion.

As another example, monitoring device 447 is configured to monitor the voltage of voltage regulator 490 on one or more power branch circuits 482 in any suitable manner (such as feedback from a voltage regulator or voltmeter). If there is an overvoltage or undervoltage detected by the monitoring device 447, the monitoring device 447 registers a fault (e.g., a time stamp) in the memory 446 and reports such fault status to the control server 120 and/or Alternatively, the maintenance request may be communicated (e.g., via communication module 445) to an operator performing the maintenance request. If the branch device 483/branch power circuit 482 in which the fault is acquired is of lower operational importance, the monitoring device 447 continues to monitor and register faults from the voltage regulator 490 and determines the frequency or random number of faults. A service request message may be sent to the control server 120 or the operator according to other suitable criteria (eg, magnitude of overvoltage or undervoltage).

The description continues with reference to FIGS. 1 and 2. 1A, 1B, 2, 19, 20, and 21, the power distribution device 444 of the controller 122 monitors the device 447 during a cold start-up (initialization) of the autonomous transport vehicle 110. It is configured as a boot device to come online before other sections of the controller 122 and vision system controller 122VC, such that the initialization of the autonomous transport vehicle 110 prior to booting of the controller 122 and vision system controller 122VC. This is to set the (safety) state. To perform initialization of the autonomous transport vehicle 110, the controller 122 may, upon indication from the power distribution device 444, determine the imminent level of available power supply charge level from the power supply 481 to the controller 1222. Configured to reduce the reduction below the required amount, the controller 122 is configured to control each registry and memory (e.g., autonomous navigation control section 122N, autonomous payload processing control section 122H, and vision system control section (e.g., For example, status and attitude navigation information and payload identity, status and attitude of the autonomous guided vehicle held in memory 446 of the corresponding one of the vision system controller 122VC or other memory 122M. At least one of the information consists of an initialization file 122F (FIG. 2) that can be used when the controller 122 is rebooted. The controller 122 is configured to, upon indication from the power distribution device 444, indicate an impending decrease in the available power supply charge level from the power supply 481 to the controller 122 below the required amount. This uses the saved status information from at least one of the vehicle status monitor 447V, the driving section status monitor 447D, the payload handling section status monitor 447H, and the peripheral electronic section status monitor 447P when rebooting the controller 122. This is to configure it with a possible initialization file (122F).

Upon initialization of an autonomous transport vehicle. Monitoring device 447 of power distribution device 444 is configured to control controller 122 sections (e.g., autonomous navigation control section 122N, autonomous payload processing control section 122H, and vision system controller 122VC) and branches. Configured to control power-on sequencing of devices 483 (e.g., sensors, travel motors, caster motors, transfer arm motors, justification device motors, payload bed 210B motors, etc.). The sequence may be that the vision system controller (122VC) is powered on before the autonomous navigation control section (122N) and the branch device is powered on last; However, on the other hand, there can be any suitable power-up sequence so that the control device powers up first.

Referring also to FIG. 27 , the power-on or cold start-up process of an example autonomous transport vehicle 110 will be described using power distribution device 444 as a boot device. Here, when the autonomous transport vehicle 110 is powered on (FIG. 27, block 2200), the power distribution device 444 monitors the output voltage of the power supply 481 and determines that the output voltage is approximately equal to the start-up threshold voltage (V1a). Determine whether it is greater than 16V (Figure 27, block 2205) (in other respects, the startup threshold voltage V1a may be greater or less than about 16V). When the power supply 481 output voltage is greater than the start-up threshold voltage (V1a), the power distribution device 444 connects, for example, the controller 122, the vision system controller 122VS, the wireless communication module 445, and the autonomous transport. Operate switch 499S to power other suitable programmable devices (e.g., FPGA, CPLD, SOC, CPU, etc.) of vehicle 110 (FIG. 27, block 2210). Here, initialization file 122F (described above) can be used to configure controller 122, vision system controller 122VS, wireless communication module 445, and other suitable programmable devices (e.g., FPGA, CPLD, SOC, CPU, etc.). ), it can be used to perform startup and operation of controlled devices based on the information in the initialization file 122F (FIG. 27, block 2215).

Power distribution device 444 is configured to detect the voltage output by power supply 481 and the output voltage is higher than the next higher startup threshold voltage V2a (FIG. 27, block 2220), approximately 18V (in another aspect, startup threshold Voltage V2a may be greater or less than about 18V) and power distribution unit 444 operates switch 449S to turn on the lights (e.g., LEDs - FIGS. 10A to 10A) of autonomous transportation vehicle 110. (see FIG. 10C) is turned on (FIG. 27, block 2225). If the next higher startup threshold voltage V2a has not been reached, the power distribution device 444 continues the shutdown sequence until the next higher startup threshold voltage V2a is reached (such as when the autonomous transport vehicle 110 is charging). Continue to monitor the power supply 481 output voltage until it starts (see Figure 8 illustrated here).

Power distribution unit 444 continues to monitor the voltage output of power supply 481, and with the next higher startup threshold voltage (V3a) detected by the power distribution unit (FIG. 27, block 2230), the power distribution unit 444 continues to monitor the voltage output of power supply 481. Distribution 444 may be used to support 24V peripheral devices and instruments of autonomous transportation vehicle 110 (see FIGS. 25A-25C), as well as, for example, front and rear alignment modules 210ARJ, 210AFJ, payload bed 210B. , and operating the switch 449S to turn on or power up the case handling drive of the transfer arm 210A (FIG. 27, block 2235). Here, the threshold voltage V3a may be about 24V, but in other respects the threshold voltage V3a may be greater or less than about 24V. If the voltage output of power supply 481 is less than about 24V, power distribution device 444 will stop until the next higher startup threshold voltage V3a is reached (such as when autonomous transport vehicle 110 is charging) or stop. Continue to monitor the power supply 481 output voltage until the sequence begins (see Figure 23 described herein).

Power distribution unit 444 monitors the voltage output of power supply 481 and, upon detection of the next higher starting threshold voltage V4a (FIG. 27, block 2240), powers up/turns on traction drive motor 261M. To do so (Figure 27, block 2245), power distribution device 444 activates switch 449S. Here, the threshold voltage V4a may be about 28V, but in other respects the threshold voltage V4a may be greater or less than about 28V. If the voltage output of power supply 481 is less than about 28V, power distribution device 444 will stop until the next higher startup threshold voltage V4a is reached (such as when autonomous transport vehicle 110 is charging) or stop. Continue to monitor the power supply 481 output voltage until the sequence begins (see Figure 23 described here).

As may be realized, when the threshold voltage V4a is detected by the power distribution device 444 upon a cold start of the autonomous transport vehicle 110, the power distribution device 444 performs the manner/sequence described above with respect to FIG. 27 supplies power to components of autonomous transport vehicle 110 (e.g., consisting of any suitable non-transitory computer program code). Here, the power distribution unit 444 is configured to be turned on before the device controlled by the control device.

1 and 2, as illustrated in FIGS. 2, 19, 20, 21, and 28, the controller 122 controls the onboard power supply charging mode, active control of the inrush current to shunt 483 ( For example, autonomous transportation vehicles) and regenerative power supply (481) bring charging.

When autonomous transport vehicle 110 is at a charging station (FIG. 28, block 1300), power distribution device 444 detects the presence of transverse surface charging pad(s) (see FIGS. 21 and 28, block 1310). As described herein, power distribution device 444 is configured to monitor the output voltage of power supply 481 and perform control operations based on the output voltage level. Here, charging control of the power supply section 481 is performed based on the output voltage of the power supply section 481 detected by the power distribution unit 444. Here, the monitoring device 447 of the power distribution device 444 is configured to control the low-level charging logic of the autonomous transport vehicle 110. An example charging logic block diagram for power distribution device 444 is shown in FIG. 21. As can be seen in FIG. 21 , autonomous transport vehicle 110 receives charging current from a charging pad, the transverse surface of transfer deck 130B, picking aisle 130A, and/or any of the storage and retrieval systems. It consists of a vehicle-mounted charging contact located on another suitable transverse surface of the device. Transverse surface-mounted charging pads and vehicle-mounted charging contacts are covered by U.S. Patent Nos. 9,469,208, entitled “Rover Charging System,” issued October 18, 2016, and U.S. Patent Nos. 11,001,444, entitled “Storage and Retrieve System Rover Interface,” issued 2021 issued May 11, 2014, and is substantially similar to U.S. Patent Application No. 14/209,086, entitled “Rover Charging System,” filed March 13, 2014. Autonomous transport vehicle 110 may also have a remote mounted on the front end 200E1 or rear end 200E2 of frame 200 that engages (e.g., plugs into) a corresponding charging port mounted on storage structure 130. It may consist of charging ports or a plug that the operator plugs into a remote charging port of the autonomous transport vehicle 110.

The monitoring device 447 controls the charging mode/speed of the power supply device 481 to maximize the number of charging cycles of the power supply device 481. For example, the monitoring device 447 is configured with a trickle charging mode (e.g., with a charging rate below a set threshold voltage), a slow charging mode, and an ultra-fast (e.g., high current) charging mode. Here the charging current is limited to the maximum charging voltage threshold set by the monitoring device 447 to substantially prevent adverse effects resulting from charging on the power supply 481. Here, the charging current and voltage may vary depending on the capacity and type of the power supply device 481. Power supply 481 may have any suitable voltage and charging capacity and may be an ultracapacitor or any other suitable power source (eg, lithium ion battery pack, lead acid battery pack, etc.). As can be seen in FIG. 6 , autonomous transport vehicle 110 includes appropriate active reverse voltage protection for power source 481 .

As an example of charge rate control, with the vehicle charging contacts coupled to a transverse surface charging pad (see FIG. 21), power distribution device 444 may cause the output voltage from power supply 481 to reach a threshold charging voltage of approximately 23V. 28, block 1320) and detecting that the threshold charge voltage V1c is below 23V (in other embodiments the threshold charge voltage V1c may be greater or less than 23V), the monitoring device 477 of the power distribution unit 444 monitors the voltage of the power supply 1330. Affects limited current charging. For example, the limited charging current may be the slow charging mode described above. The above-described slow charging mode may have a higher charging current than the trickle charging mode, but may have a lower charging current than the full charging current. Power distribution device 444 continuously monitors the output voltage of power supply 481 during charging and detects that the output voltage of power supply 481 is above the threshold charging voltage V1c (Figure 28, block 1320). Monitoring device 477 of power distribution device 444 implements different charging modes, such as full charge current mode (Figure 28, block 1350). Power distribution unit 444 monitors the output voltage of power supply 481 while charging at full charge current and, if the output voltage is above the next highest threshold charge voltage V2c (FIG. 13, block 1340) of approximately 44 V (other On the side (the output voltage may be greater or less than about 44V), the monitoring device 477 of the power distribution device 444 terminates charging. In another aspect, if the output voltage is detected to be above about 44V, the monitoring device 477 connects the power supply 481 to a vehicle charging contact of the autonomous transport vehicle 110 engaged/coupled with transverse surface charging pads. The trickle charge mode can be influenced to maintain peak/maximum charge (see Figure 21).

Continue referring to Figures 1 and 2. 2, 19, 20, and 21, the autonomous transport vehicle 110 includes one or more of inrush current protection, overvoltage/current protection, and undervoltage/current protection. For example, autonomous transportation vehicle 110 may include Hot Swap Circuitry (substantially similar to that described in U.S. Patent No. 9,469,208, entitled “Rover Charging System,” issued October 18, 2016). U.S. Patent No. 11,001,444, entitled “Rover Charging System,” issued May 11, 2021, and U.S. Patent Application No. 14/209,086, entitled “Rover Charging System,” issued March 13, 2014. (applied on) may be included. Here, the power distribution device 444 is configured to actively control the inrush current to the branch devices 483A-483F.... 483n of each branch power circuit 482 (collectively branch devices 483, where n represents an integer indicating the maximum number of branch devices) is received by the power distribution device 444 from the controller 122, and the controller 122 is configured to generate a pulse width modulated signal that effects active control of switches 449S to limit inrush current (e.g., charging or power surges) into branch devices 483. Example For example, upon initial contact between a vehicle charging contact and a transverse surface mounted charging pad, power distribution device 444 may cause switch 449S to open one or more switches to prevent inrush current from flowing to shunt devices 483. One or more of these may work.

2, 19 and 22, one or more of the branch power circuits include an electrical protection circuit 700 configured to protect the branch device 483 (the sensor is shown in FIG. 22 for illustrative purposes, but may not be used in other respects). Any suitable branching device may be provided as described herein.). The electrical protection circuit 700 is configured to substantially protect branch device 483 (and any control/measurement instrumentation associated therewith) against, for example, short circuits, overvoltages and overcurrents. For example, branch device 483 (sensor in this example) operates with a signal of about 4 mA to about 20 mA. Electrical protection circuit 700 includes an adjustable three-terminal positive voltage regulator 710 and a single resistor 720 for illustrative purposes only. Voltage regulator 710 is configured to supply at least about 1.5A over an output voltage range of about 1.25V to about 37V. The voltage regulator 710 coupled with the resistor 720 uses the internal reference voltage of the voltage regulator 710 to limit the current to about 27 mA. Inserting electrical protection circuit 700 into branch power circuit 482 does not substantially affect the signal from about 4 mA to about 20 mA, while (with respect to current flow) upstream and downstream of electrical protection circuit 700. Provides control/measurement protection for devices deployed on both devices. The configuration of electrical protection circuit 700 is by way of example only, and electrical protection circuit 700 may include any suitable voltage regulator and resistor ( It should be noted once again that it can be configured with appropriate specifications.

As shown in FIGS. 19, 20, and 21, the power distribution device 444 is configured to perform regenerative charging of the power supply device 481. For example, when the right and left running wheels 260A, 260B are rolling but not energized (e.g., during braking), the back electromotive force (EMF) voltage generated by each motor 261M will be applied to each branch power circuit. It is fed back to (483E, 483F). Monitoring device 447 may actuate switch 449S (e.g., Vcap_IN switch - see FIG. 21) such that the back electromotive voltage (and current) regenerates power supply 481. Using the motor 261M supplied with power to drive the driving wheels 260A and 260B, the monitoring device 447 can prevent power from being consumed from the power supply device 481 by closing the Vcap_IN switch.

As shown in FIGS. 2, 19, and 20, the power distribution unit 444 includes a wireless communication module 445. Wireless communication module 445 may be configured for any suitable wireless communication, including but not limited to Wi-Fi, Bluetooth, cellular, etc. Wireless communication module 445 configures power distribution device 444 to at least partially control it. For example, communication is possible between autonomous transport vehicle 110 and other functions of the storage and retrieval system, including but not limited to control server 120, over any suitable network, such as network 180. Here, the wireless communication module 445 and monitoring device 447 monitor processing function errors of the controller 122 (e.g., safety-related functions including remote shutdown, communications, or other common component errors). Configure power distribution unit 444 as an auxiliary processor/controller as detected by device 447. If the controller 122 experiences an error in communications or control effected by the controller 122, the power distribution device 444 may be inactive with the control server 120 (and the operators of the storage and retrieval system 100). and maintaining (secondary) communication between different components of the autonomous transport vehicle 110 (e.g., via communication module 445), so that the autonomous transport vehicle 110 can operate in the manner described in this application. It can be remotely shut down (autonomously, semi-autonomously, or under manual remote control by an operator).

The wireless communications module 445 may also be configured to enable "over the air" programming of the controller 122, the vision system controller 122VC, and the monitoring device 447 of the autonomous transport vehicle 110, or other suitable programmable device. Provides for updating firmware/programming of (e.g. FPGA, CPLD, SOC, CPU, etc.). Here, the operator of the storage and retrieval system 100 may operate via a control server 120 or via a network 180 (which is at least partially a wireless network) or using another suitable device such as a laptop, smartphone/tablet, etc. Software updates may be pushed or uploaded to the autonomous vehicle 110 . Power distribution unit 444 is for installation in monitoring device 447, controller 122, vision system controller 122VC, and/or other suitable programmable devices (e.g., FPGA, CPLD, SOC, CPU, etc.). Includes any suitable memory 446 capable of buffering software updates.

The wireless communication module 445 of the power distribution unit 444 may be configured as an Ethernet switch or bridge. Here, the wireless communication module 455 of the autonomous transportation vehicle 110 moving throughout the storage structure 130 may form a mesh network. In this way, wireless communication from, for example, control server 122 or other suitable devices such as laptops, smartphones/tablets, etc. can be extended to cover substantially the entirety of storage structure 130 without a dedicated Ethernet switch. and the bridges are positioned (e.g. mounted) at fixed/predetermined locations throughout the storage structure 130.

1A, 2, 19, 20, 21, and 24, exemplary methods for autonomous guided vehicle power management will now be described in accordance with aspects of the disclosed embodiments. The method includes providing autonomous transportation 110 as described herein (Figure 24, block 24900). Autonomous operation of autonomous transport vehicle 110 is performed by controller 122 (FIG. 24, block 24910), and the charge level of power supply 481 of autonomous transport vehicle 110 is controlled by power distribution unit 444. (Figure 24, block 24910). Block 24920). As described herein, a predetermined pattern (herein 24, block 24930).

Impending availability of power supply charge levels directed from power distribution section 444, from power supply 481 to drive section 261D (see FIG. 21) and/or branch circuit 482 of case handling assembly 210. Upon direction of the reduction, controller 122 commands drive section 261D to move autonomous delivery vehicle 110 to a safe location and/or case handles to move the payload to a safe location as described herein. Command assembly 210 (Figure 24, block 24960).

Upon receiving an indication from power distribution device 444 that the level of available power charge from power supply 481 to controller 122 is imminent to decrease below the required level of controller 122, controller 122 may perform the following: It stops operation as done and enters Hibernation (Figure 24, block 24950).

Additionally, as described herein, an indication from power distribution device 444 of an impending decrease in the available power supply charge level from power supply 481 to controller 122 may result in a level below the required level of controller 122. As a result, controller 122 creates at least one initialization file (Figure 24, block 24940). As described herein, controller 122 is configured to access the respective registries and memories of corresponding controller sections (e.g., autonomous driving control section 122N, autonomous payload handling control section 122H, and vision system control section). Autonomous guided vehicle status and attitude navigation information and payload identity held in (e.g., the corresponding memory 446 of the vision system control section 122VC or another memory 122M); At least one of the status and posture information may be configured as an initialization file 122F that can be used when the controller 122 is rebooted. The controller 122 sends status information of at least one of the vehicle status monitor 447V, the driving section status monitor 447D, the payload handling section status monitor 447H, and the peripheral electronic section status monitor 447P to the controller 122. It can be saved to an initialization file (122F) (or a different initialization file) that will be available upon reboot.

According to one or more aspects of the disclosed embodiments, an autonomous guided vehicle:

Frame with payload hold;

a drive section coupled to a frame having running wheels that support a vehicle on the crossing surface, the driving wheels effecting vehicle crossing on the crossing surface to move an automatically guided vehicle over the crossing surface within the facility;

In a storage array, a payload handler coupled to a frame configured to transmit a payload, the payload handler having a flat non-deterministic seating surface seated on the payload hold and a storage location for the payload;

A physical property sensor system connected to a frame having electromagnetic sensors, each responsive to the interaction or interface of the sensor to emit or generate an electromagnetic beam or field having a physical property, and the electromagnetic beam or field by interaction or interface with the physical property. The field is disturbed, and the disturbance is sensed and affected by electromagnetic sensors of a physical nature, and the physical sensor system detects and affects the vehicle's navigation pose or location information and a physical characteristic sensor system configured to generate sensor data including at least one of load posture or position information; and

Coupled to the frame, complementary to the physical characteristic sensor system, and arranged to capture image data that is indicative, at least in part, of the vehicle's navigation attitude, position, and payload attitude or position complementary to information from the physical characteristic sensor system. It includes an auxiliary sensor system, which is a vision system equipped with a camera.

According to one or more aspects of the disclosed embodiments, the autonomous guided vehicle further includes a controller coupled to the frame, operably coupled to the drive section or payload handler, and communicatively coupled to the physical characteristic sensor system, wherein the controller It is configured to determine, from information from the characteristic sensor system, a vehicle attitude and position that affects independent guidance of an autonomous guided vehicle traversing the facility.

In accordance with one or more aspects of the disclosed embodiments, a controller may be configured to determine independent underpick and place of a payload relative to a storage location and an independent underpick and place of a payload within a payload hold from information of a physical characteristic sensor system payload pose. It is structured to determine the location of influence.

According to one or more aspects of the disclosed embodiments, the controller is programmed with a reference representation of predetermined features that define at least a portion of the facility to be traversed by the autonomous guided vehicle.

In accordance with one or more aspects of the disclosed embodiment, a controller is configured to register captured image data and generate therefrom at least one image of one or more predetermined features, wherein the at least one image is a predetermined image of the reference representation. Formatted as a virtual representation of the one or more predetermined features to provide comparison with one or more corresponding references to the features.

In accordance with one or more aspects of the disclosed embodiment, a controller is configured to register captured image data and generate therefrom at least one image of one or more predetermined features, wherein the at least one image is a predetermined image of the reference representation. Formatted as a virtual representation of the one or more predetermined features to provide comparison with one or more corresponding references to the features.

According to one or more aspects of the disclosed embodiments, the controller is configured to verify autonomous guided vehicle attitude and position information registered by the controller from a physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

In accordance with one or more aspects of the disclosed embodiments, the controller identifies a variance of the autonomous guided vehicle attitude and position based on a comparison between the virtual representation and the reference representation and determines the autonomous guided vehicle attitude or position from the physical characteristic sensor system based on the variance. It is configured to update or complete information.

In accordance with one or more aspects of the disclosed embodiments, the controller may be configured to provide information from the physical property sensor system and fidelity of the autonomously guided vehicle attitude and position information from the physical property sensor system based on at least one of the distribution and analysis of the identified at least one image. and determine a posture error of and assign a reliability value based on at least one of posture error and fidelity.

In accordance with one or more aspects of the disclosed embodiments, the controller determines that, with a confidence value below a predetermined threshold, the controller determines the position of an autonomous guided vehicle based on posture and position information generated from a virtual representation on behalf of the posture and position information from a physical property sensor system. It is configured to switch navigation.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue self-guided vehicle navigation to your destination, or

Select a safe path for the autonomous guided vehicle and a trajectory to bring the autonomous guided vehicle from a switching position to a safe position for stopping, or

Initiates communication with the operator, through a user interface device, identifying the destination and autonomous guided vehicle kinematic data for operator selection of autonomous guided vehicle control from automatic operation to semi-autonomous operation or manual operation.

According to one or more aspects of the disclosed embodiments, the controller is configured to verify payload attitude and position information registered by the controller from a physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

In accordance with one or more aspects of the disclosed embodiments, a controller identifies a variance in payload pose and position based on a comparison between a virtual representation and a reference representation and updates payload pose or position information from a physical characteristic sensor system based on the variance. or configured to complete.

In accordance with one or more aspects of the disclosed embodiments, the controller may determine the fidelity of the payload pose and position information from the physical property sensor system based on at least one of the identified distribution and analysis of the at least one image. It is configured to determine a posture error and assign a reliability value based on at least one of posture error and fidelity.

In accordance with one or more aspects of the disclosed embodiments, with a confidence value below a predetermined threshold, the controller determines the autonomous guided vehicle payload based on location information generated from a virtual representation on behalf of the posture and location information from a physical property sensor system. It is configured to switch handling.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue handling the automated guided vehicle to your destination, or

The user interface device initiates communication with the operator identifying payload data along with operator selection of autonomous guided vehicle control from automatic payload processing operations to semi-autonomous payload processing operations or manual payload processing operations.

In accordance with one or more aspects of the disclosed embodiments, a controller may include a virtual representation of one or more corresponding reference presentations of the reference presentation to present augmented reality images in real time to the operator via a wireless communication system communicatively coupling the controller and an operator interface; and transmit a simulated image combining one or more corresponding reference predetermined features.

In accordance with one or more aspects of the disclosed embodiments, a controller is configured to receive real-time operator commands to the autonomous guided vehicle responsive to the real-time augmented reality image, the commands being changes to the real-time augmented reality image transmitted to the operator.

According to one or more aspects of the disclosed embodiments, the auxiliary sensor system at least partially affects immediate justification and/or sorting of case devices mounted on an autonomous guided vehicle.

In accordance with one or more aspects of the disclosed embodiments, one or more of the assistive vehicle navigation attitude or position, and the vehicle navigation attitude or position may be provided via assistive information from an assistive sensor system, one or more of an assistive vehicle navigation attitude or position, and an assistive payload attitude or position. One or more auxiliary payload attitude or position information and one or more of the payload attitude or position information are coated on the reference model to improve the resolution of one or more of the auxiliary payload attitude or position, peripheral facility features and interface facility features.

According to one or more aspects of the disclosed embodiments, an autonomous guided vehicle:

Frame with payload hold;

a drive section coupled to a frame having running wheels that support a vehicle on the crossing surface, the driving wheels effecting vehicle crossing on the crossing surface to move an automatically guided vehicle over the crossing surface within the facility;

In a storage array, a payload handler coupled to a frame configured to transmit a payload, the payload handler having a flat, non-deterministic seating surface seated on the payload hold and a storage location for the payload;

A physical property sensor system connected to a frame having electromagnetic sensors, each responsive to the interaction or interface of the sensor to emit or generate an electromagnetic beam or field having a physical property, and the electromagnetic beam or field by interaction or interface with the physical property. The field is disturbed, and the disturbance is sensed and affected by electromagnetic sensors of a physical nature, and the physical sensor system detects and affects the vehicle's navigation pose or location information and a physical characteristic sensor system configured to generate sensor data including at least one of load posture or position information; and

Coupled to the frame, complementary to the physical characteristic sensor system, and arranged to capture image data that is indicative, at least in part, of the vehicle's navigation attitude, position, and payload attitude or position complementary to information from the physical characteristic sensor system. It includes an auxiliary sensor system, which is a vision system equipped with a camera.

According to one or more aspects of the disclosed embodiments, the autonomous guided vehicle further includes a controller coupled to the frame, operably coupled to the drive section or payload handler, and communicatively coupled to the physical characteristic sensor system, wherein the controller It is configured to determine, from information from the characteristic sensor system, a vehicle attitude and position that affects independent guidance of an autonomous guided vehicle traversing the facility.

In accordance with one or more aspects of the disclosed embodiments, a controller may be configured to determine independent underpick and place of a payload relative to a storage location and an independent underpick and place of a payload within a payload hold from information of a physical characteristic sensor system payload pose. It is structured to determine the location of influence.

According to one or more aspects of the disclosed embodiments, the controller is programmed with a reference representation of predetermined features that define at least a portion of the facility to be traversed by the autonomous guided vehicle.

In accordance with one or more aspects of the disclosed embodiment, a controller is configured to register captured image data and generate therefrom at least one image of one or more predetermined features, wherein the at least one image is a predetermined image of the reference representation. Formatted as a virtual representation of the one or more predetermined features to provide comparison with one or more corresponding references to the features.

In accordance with one or more aspects of the disclosed embodiments, the controller is configured to cause a virtual representation of one or more imaged features to reside on the autonomous guided vehicle, and between the virtual representations of the one or more imaged certain features and one or more corresponding reference certain features. The comparison of is configured to reside on the autonomous guided vehicle.

According to one or more aspects of the disclosed embodiments, the controller is configured to verify automated guided vehicle attitude and position information registered by the controller from a physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

In accordance with one or more aspects of the disclosed embodiments, the controller identifies a variance of the autonomous guided vehicle attitude and position based on a comparison between the virtual representation and the reference representation, and determines the autonomous guided vehicle attitude or position from the physical characteristic sensor system based on the variance. It is configured to update or complete information.

In accordance with one or more aspects of the disclosed embodiments, the controller may determine the fidelity of the autonomous guided vehicle attitude and the position information from the physical property sensor system and the information from the physical property sensor system based on at least one of the distribution and analysis of the identified at least one image. and determine a posture error of and assign a reliability value based on at least one of posture error and fidelity.

In accordance with one or more aspects of the disclosed embodiments, the controller determines that, with a confidence value below a predetermined threshold, the controller determines the position of an autonomous guided vehicle based on posture and position information generated from a virtual representation on behalf of the posture and position information from a physical property sensor system. It is configured to switch navigation.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue navigation of the autonomously guided vehicle to the destination, or select a safe path for the autonomously guided vehicle and a trajectory that will bring the autonomously guided vehicle from the transition position to a safe position for stopping, or

Initiates communication with the operator through a user interface device identifying a destination and autonomous guided vehicle kinematic data for operator selection of autonomous guided vehicle control from automatic operation to semi-autonomous operation or manual operation.

According to one or more aspects of the disclosed embodiments, the controller is configured to verify payload attitude and position information registered by the controller from a physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

In accordance with one or more aspects of the disclosed embodiments, a controller identifies a variance in payload pose and position based on a comparison between a virtual representation and a reference representation and updates payload pose or position information from a physical characteristic sensor system based on the variance. or configured to complete.

In accordance with one or more aspects of the disclosed embodiments, the controller may be configured to determine the fidelity of the payload pose and location information from the physical property sensor system based on at least one of the identified distribution and analysis of the at least one image. It is configured to determine a posture error and assign a reliability value based on at least one of posture error and fidelity.

In accordance with one or more aspects of the disclosed embodiments, the controller may, with a confidence value below a predetermined threshold, determine the autonomous guided vehicle payload based on location information generated from a virtual representation on behalf of the posture and location information from a physical property sensor system. It is configured to switch handling.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue handling the self-guided vehicle to your destination, or

The user interface device initiates communication with the operator identifying payload data along with operator selection of autonomous guided vehicle control from automatic payload processing operations to semi-autonomous payload processing operations or manual payload processing operations.

In accordance with one or more aspects of the disclosed embodiments, a controller may include a virtual representation of one or more corresponding reference presentations of the reference presentation to present augmented reality images in real time to the operator via a wireless communication system communicatively coupling the controller and the operator interface; and transmit a simulated image combining one or more corresponding reference predetermined features.

In accordance with one or more aspects of the disclosed embodiments, a controller is configured to receive real-time operator commands to the autonomous guided vehicle responsive to the real-time augmented reality image, the commands resulting in changes to the real-time augmented reality image transmitted to the operator. .

According to one or more aspects of the disclosed embodiments, the auxiliary sensor system at least partially effects immediate justification and/or classification of case devices mounted on an autonomous guided vehicle.

According to one or more aspects of the disclosed embodiments, an auxiliary vehicle navigation attitude or position, and assist, from an auxiliary sensor system, via one or more of an auxiliary vehicle navigation attitude or position, an auxiliary vehicle navigation attitude or position, and an auxiliary vehicle navigation attitude or position. One or more of the vehicle navigation attitude or position, and one or more of the auxiliary vehicle navigation attitude or position information, to improve the resolution of the payload attitude or position of the auxiliary vehicle, one or more criteria of peripheral facility features and interface facility features It is characterized by being applied to the model.

According to one or more aspects of the disclosed embodiments, the method includes:

We provide self-guided vehicles equipped with:

A frame with a payload hold,

a drive section coupled to a frame having running wheels that support a vehicle on the crossing surface, the driving wheels effecting vehicle crossing on the crossing surface to move an automatically guided vehicle over the crossing surface within the facility;

In a storage array, a payload handler coupled to a frame configured to transmit a payload, the payload handler having a flat, non-deterministic seating surface seated on the payload hold and a storage location for the payload;

A physical property sensor system connected to a frame having electromagnetic sensors, each responsive to the interaction or interface of the sensor to emit or generate an electromagnetic beam or field having a physical property, and to cause the electromagnetic beam or field to be emitted or generated by the interaction or interface with the physical property. The field is disturbed, and the disturbance is sensed and affected by electromagnetic sensors of a physical nature, and the physical sensor system detects and affects the vehicle's navigation pose or location information and a physical characteristic sensor system configured to generate sensor data including at least one of load posture or position information; and

Coupled to the frame, complementary to the physical characteristic sensor system, and arranged to capture image data that is indicative, at least in part, of the vehicle's navigation attitude, position, and payload attitude or position complementary to information from the physical characteristic sensor system. It includes an auxiliary sensor system, which is a vision system equipped with a camera.

A method according to one or more aspects of the disclosed embodiments further comprises the step of determining a controller from information of the physical characteristic sensor system's vehicle attitude and position affecting independent guidance of the autonomous guided vehicle traversing the facility, wherein the controller determines a frame and operably connected to a drive section or payload handler and communicatively connected to a physical property sensor system.

A method according to one or more aspects of the disclosed embodiments includes determining from information of a physical characteristic sensor system payload pose, independent underpick and location of the payload relative to a storage location and independent underpick and location of the payload within the storage location and payload hold. It further includes positions that influence picks and positions.

According to one or more aspects of the disclosed embodiments, the controller is programmed with a reference representation of predetermined features that define at least a portion of the facility to be traversed by the autonomous guided vehicle.

According to one or more aspects of the disclosed embodiment, the method further comprises registering the captured image data with the controller and generating from at least one image of one or more of the predetermined features, the at least one The image is formatted as a virtual representation of the one or more predetermined features to provide comparison with one or more corresponding references of the predetermined features of the reference representation.

In accordance with one or more aspects of the disclosed embodiments, the controller is configured to cause a virtual representation of one or more imaged features to reside on the autonomous guided vehicle, wherein the virtual representation of the one or more imaged features and the one or more corresponding reference features are configured to reside on the autonomous guided vehicle. The comparison of is configured to reside on the autonomous guided vehicle.

A method according to one or more aspects of the disclosed embodiments further includes confirming with the controller autonomous guided vehicle attitude and position information registered by the controller from the physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

A method according to one or more aspects of the disclosed embodiments includes, with a controller, identifying a variance of an autonomous guided vehicle attitude and position based on a comparison between a virtual representation and a reference representation, and determining the autonomous guided vehicle posture from a physical property sensor system based on the variance. Or, it further includes updating or completing location information.

In accordance with one or more aspects of the disclosed embodiments, the controller determines the fidelity of the autonomous guided vehicle attitude and position information and information from the physical property sensor system based on at least one of the distribution and analysis of the identified at least one image, and determines the attitude error. and assigning a reliability value according to at least one of fidelity.

According to one or more aspects of the disclosed embodiments, based on a confidence value below a predetermined threshold, the controller determines the autonomous guided vehicle It is characterized by switching navigation.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue navigation of the autonomous guided vehicle to the destination, or select a safe path for the autonomous guided vehicle and a trajectory that will bring the autonomous guided vehicle from the transition location to a safe location for termination, or

Initiates communication with the operator, through a user interface device, identifying the destination and autonomous guided vehicle kinematic data for operator selection of autonomous guided vehicle control from automatic operation to semi-autonomous operation or manual operation.

According to one or more aspects of the disclosed embodiments, the controller verifies payload attitude and position information registered by the controller from a physical characteristic sensor system based on a comparison between the virtual representation and the reference representation.

In accordance with one or more aspects of the disclosed embodiments, a controller identifies a variance in payload pose and position based on a comparison between a virtual representation and a reference representation and updates payload pose or position information from a physical characteristic sensor system based on the variance. or is characterized by completion.

In accordance with one or more aspects of the disclosed embodiments, the controller may determine the fidelity of the information and payload attitude from the physical property sensor system and the position information from the physical property sensor system based on at least one of the distribution and analysis of the at least one image identified. Determining the posture error and assigning a reliability value based on at least one of the posture error and fidelity.

According to one or more aspects of the disclosed embodiments, based on a confidence value below a predetermined threshold, the controller determines the autonomous guided vehicle Switch payload handling.

In accordance with one or more aspects of the disclosed embodiment after conversion, the controller is configured as follows:

Continue handling the self-guided vehicle to your destination, or

The user interface device initiates communication with the operator identifying payload data along with operator selection of autonomous guided vehicle control from automatic payload processing operations to semi-autonomous payload processing operations or manual payload processing operations.

In accordance with one or more aspects of the disclosed embodiments, a controller provides a virtual representation of one or more imaged features and a reference presentation together with the augmented reality image to the operator in real time via a wireless communication system communicatively coupling the controller and the operator interface. Transmit a simulated image combining certain features with one or more corresponding references.

In accordance with one or more aspects of the disclosed embodiments, a controller receives real-time operator commands to the autonomous guided vehicle responsive to the real-time augmented reality image, and the commands cause changes in the real-time augmented reality image transmitted to the operator to the controller. Received by.

According to one or more aspects of the disclosed embodiments, a controller effect with at least an auxiliary sensor system provides alignment and/or sorting of case devices mounted on an autonomous guided vehicle.

In accordance with one or more aspects of the disclosed embodiments, one or more of the assistive vehicle navigation attitude or position, and the vehicle navigation attitude or position may be provided via assistive information from an assistive sensor system, one or more of an assistive vehicle navigation attitude or position, and an assistive payload attitude or position. One or more auxiliary payload attitude or position information and one or more of the payload attitude or position information are coated on the reference model to improve the resolution of one or more of the auxiliary payload attitude or position, peripheral facility features and interface facility features.

According to one or more aspects of the disclosed embodiments, an autonomous guided vehicle:

Frame with payload hold;

a drive section coupled to a frame having running wheels that support a vehicle on the crossing surface, the driving wheels effecting vehicle crossing on the crossing surface to move an automatically guided vehicle over the crossing surface within the facility;

a payload handler coupled to the frame configured to hold the payload of the vehicle within the storage array and transfer the payload to a storage location of the payload;

An auxiliary sensor system connected to the frame for collaboration between vehicle and operator assists a vehicle autonomous navigation/manipulation sensor system configured to collect at least sensory data embodying vehicle attitude and location information for automatic navigation by vehicles of the installation,

The auxiliary sensor system may, at least in part, be configured such that the at least one camera is arranged to capture image data indicative of objects and/or spatial features within at least a portion of the facility along with vehicles at other locations of the facility. It is a vision system equipped with one camera,

coupled to the frame and communicatively coupled to the auxiliary sensor system to register the information from image data of the at least one camera, wherein the controller determines a predetermined physical characteristic of the at least one object or spatial feature from the information. Determining the presence of and, in response, selectively reconfiguring the vehicle into a cooperative vehicle state arranged to receive operator commands for the vehicle to continue vehicle operation.

In accordance with one or more aspects of the disclosed embodiments, the predetermined physical characteristics are such that at least one object or spatial feature is characterized by a lateral surface, a vehicle crossing path across the lateral surface, or through the space of the vehicle, or that of another vehicle crossing the traversing surface. extends over at least part of it.

According to one or more aspects of the disclosed embodiments, a controller is programmed with a reference representation of predetermined characteristics that at least partially define a facility through which a vehicle passes.

According to one or more aspects of the disclosed embodiments, the controller is configured to register captured image data and generate therefrom at least one image of at least one object or spatial feature representing predetermined physical characteristics.

In accordance with one or more aspects of the disclosed embodiments, the at least one image is formatted as a virtual representation of at least one object or spatial feature to provide comparison with one or more reference features of predetermined features of the reference representation.

According to one or more aspects of the disclosed embodiments, the controller may identify the presence of a predetermined physical characteristic of an object or spatial feature based on a comparison between the virtual representation and the reference representation, and based on the preset physical characteristic, determine the presence of a predetermined physical characteristic of the object or spatial feature based on the comparison between the virtual representation and the reference representation. The vehicle is commanded to stop on a preset trajectory based on the location or spatial characteristics of the object.

According to one or more aspects of the disclosed embodiments, the stopping position on the predetermined trajectory maintains an object or spatial reference within the field of view of the at least one camera and performs continuous imaging of predetermined physical characteristics, such as traffic obstacles, areas to avoid, etc. or initiate a signal to one or more other vehicles in the detour area.

According to one or more aspects of the disclosed embodiments, the predetermined physical characteristics are calibrated and by determining a position of the object within a frame of reference of at least one camera having a predetermined relationship to the vehicle and from an object pose within the frame of reference of the at least one camera. , which determines the presence of predetermined physical properties, which are determined by the controller.

In accordance with one or more aspects of the disclosed embodiments, the controller may be configured to: identify the presence and transition from an autonomous state to a collaborative vehicle state, where the controller transmits communication images to an operator interface for operator collaborative operation of the vehicle, the presence of predetermined physical characteristics; It is configured to initiate identification.

In accordance with one or more aspects of the disclosed embodiments, the controller can autonomously create a trajectory that brings the autonomous guided vehicle to zero speed within a predetermined time period where the movement of the autonomous guided vehicle along the trajectory is coordinated with the positions of objects and/or spatial features. It is configured to be applied to guided vehicles.

According to one or more aspects of the disclosed embodiments, the capture of image data indicative of object and/or spatial features is an opportunity during payload hold of a vehicle or transfer of the payload to/from a storage location in a storage array.

According to one or more aspects of the disclosed embodiments, a controller is programmed to command a vehicle to another location in a facility associated with the vehicle to perform an autonomous transfer of one or more predetermined payloads, wherein the controller performs an autonomous transfer of one or more predetermined payloads. A task is a separate task that captures image data viewed from different locations by at least one camera.

In accordance with one or more aspects of the disclosed embodiments, the controller determines the presence of a predetermined physical characteristic of at least one object or spatial feature that is at least partially consistent with the vehicle's ability to affect each of one or more predetermined payload automatic transfer tasks. It is structured to be complementary and peripheral to the actions.

In accordance with one or more aspects of the disclosed embodiments, the controller may determine that the presence of a predetermined physical characteristic of at least one object or spatial feature opportunistically determines vehicle actions that affect each of the one or more predetermined payload automatic transfer tasks. It is configured to do this.

According to one or more aspects of the disclosed embodiments, at least one of the one or more predetermined automatic payload transfer operations is performed at at least one of the different locations.

According to one or more aspects of the disclosed embodiments, the cooperative vehicle state assists the autonomous state of the vehicle in influencing each of one or more predetermined autonomous payload transfer tasks.

According to one or more aspects of the disclosed embodiments, the method includes:

We provide self-guided vehicles equipped with:

Frame with payload hold;

a drive section coupled to a frame having running wheels that support a vehicle on the crossing surface, the driving wheels effecting vehicle crossing on the crossing surface to move the vehicle over the crossing surface within the facility;

a payload handler coupled to the frame configured to transfer the payload between the vehicle's payload hold and a storage location for the payload in the storage array;

Using an auxiliary sensor system coupled to the frame for vehicle and operator collaboration, at least one camera generates image data informing objects and/or spatial features within at least a portion of the facility as viewed with vehicles at different locations within the facility. steps; and the auxiliary sensor system is, at least in part, a vision system comprising at least one camera arranged to capture image data, and the auxiliary sensor system is at least in part a vehicle attitude and A vehicle autonomous navigation/actuation sensor system configured to collect detection data embodying location information;

registering with a controller coupled to the frame and communicatively coupled to the auxiliary sensor system, and registering information from image data of the at least one camera; and

Using the controller, determine from the information the presence of a predetermined physical characteristic of at least one object or spatial feature, and in response thereto, move the vehicle from an autonomous state to a collaborative vehicle state arranged to receive operator commands for the vehicle. and selectively reconfiguring the vehicle to continue to operate effectively.

In accordance with one or more aspects of the disclosed embodiments, the predetermined physical characteristic is at least one object or spatial feature that determines whether at least one object or spatial feature is present in a vehicle crossing path or at least a portion of another vehicle crossing the crossing surface, across the crossing surface, or through the space of the vehicle. extends across.

According to one or more aspects of the disclosed embodiments, a controller is programmed with a reference representation of predetermined characteristics that at least partially define a facility through which a vehicle passes.

According to one or more aspects of the disclosed embodiments, the method further includes generating at least one image of at least one object or spatial feature showing predetermined physical characteristics from registered captured image data.

In accordance with one or more aspects of the disclosed embodiments, the at least one image is formatted as a virtual representation of the at least one object or spatial feature, comparing the virtual representation to one or more reference features of the predetermined feature of the reference representation. Includes more steps.

A method according to one or more aspects of the disclosed embodiments includes, based on a comparison between a virtual representation and a reference representation, identifying with a controller the presence of a predetermined physical feature of an object or spatial feature, determining dimensions of the predetermined physical feature, and comparing and commanding the vehicle to stop on a predetermined trajectory based on the location of the object or spatial feature determined from.

A method according to one or more aspects of the disclosed embodiments includes maintaining an object or spatial reference within the field of view of at least one camera and performing continued imaging of predetermined physical characteristics, with the vehicle at a resting position within a predetermined trajectory, It further includes initiating a signal to another vehicle in at least one of an obstacle, an area to be avoided, or a detour area.

According to one or more aspects of the disclosed embodiments, the predetermined physical characteristics are calibrated and determine a position of the object within a frame of reference of at least one camera having a predetermined relationship to the vehicle, and determine a position of the object within the frame of reference of the at least one camera. The decision is made by the controller by determining the presence of predetermined physical characteristics of the object.

In accordance with one or more aspects of the disclosed embodiments, a controller may be configured to: identify the presence of a predetermined physical characteristic of at least one object or spatial feature and transfer information from an autonomous state to a collaborative vehicle state to an operator interface for operator collaborative operation of the vehicle; The transmitted communication image is configured to initiate identification of the presence of predetermined physical characteristics.

A method according to one or more aspects of the disclosed embodiments further includes applying, with a controller, a trajectory to the autonomous guided vehicle to bring the autonomous guided vehicle to zero speed within a predetermined time period, wherein the autonomous guided vehicle follows the trajectory. The movement of is coordinated with the positions of objects and/or spatial features.

According to one or more aspects of the disclosed embodiments, the capture of image data indicative of object and/or spatial features is an opportunity during payload hold of the vehicle or transfer of the payload to/from a storage location in a storage array.

According to one or more aspects of the disclosed embodiments, an autonomous guided vehicle:

The vehicle chassis on which the power supply is mounted and the power section connected to the chassis and each powered by the power supply include:

a drive section supporting the vehicle chassis and having motorized running wheels arranged to traverse the autonomously guided vehicle on the transverse surface of the autonomously guided facility;

a payload handling section having at least one payload handling actuator configured such that operation of the at least one payload handling actuator affects payload delivery to the payload bed, vehicle chassis, and storage within the facility;

At least one of an autonomous attitude and navigation sensor, a payload handling sensor, and at least one peripheral motor, wherein the at least one peripheral motor is separate and distinct from each of the motors of the drive section and each actuator of the payload handling section. , peripheral electronic components; and

A controller communicatively coupled to the driving section, the payload handling section, and the peripheral section, respectively, to effectively perform each autonomous operation of the autonomous guided vehicle; and a comprehensive power management section, wherein the controller is communicatively coupled to the power source to monitor the charge level of the power source,

The comprehensive power management section is connected from the power source to each branch circuit of the drive section, the payload handling section and the peripheral electronic device section, and is connected to the power source to the drive section, the payload handling section and the peripheral electronic device section. It is characterized in that power is supplied to each branch circuit of the section.

A comprehensive power management section according to one or more aspects of the disclosed embodiments includes each branch switching each branch circuit on or off based on the demand charge level of each branch circuit with respect to the other branch circuits and the charge level available from the power supply. It is configured to manage the demand charge level of the circuit in a predetermined pattern.

In accordance with one or more aspects of the disclosed embodiments, a pattern is arranged to switch off branch circuits with a reduction in the level of charge available from the power supply, thereby maximizing the level of charge available from the power supply to the controller.

In accordance with one or more aspects of the disclosed embodiments, the predetermined pattern is to change the available charge from the power supply such that the available charge level indicated to the controller equals or exceeds the controller's desired charge level for a maximum period of time based on the available charge level of the power supply. It is arranged to switch off the branch circuits with a decrease in level.

According to one or more aspects of the disclosed embodiments, the controller has at least one of the following:

an autonomous navigation control section configured to register and maintain past and present autonomous guided vehicle state and attitude navigation information in volatile memory to determine and describe the current and predicted state, attitude and position of the autonomous guided vehicle; and

an automatic payload processing control section configured to register and maintain current payload identity, state, and attitude information (historical and current) in volatile memory;

The controller, in accordance with an indication from a comprehensive power management section of an impending decrease in the available charge level indicated by the power supply to the controller, below the required level of the controller, the controller determines the state and attitude of the autonomous guided vehicle. At least one of search information and payload identity, status, and attitude information held in each registry and memory of the corresponding controller section is configured as an initialization file that can be used when rebooting the controller.

In accordance with one or more aspects of the disclosed embodiments, the controller is configured to cause the controller to cease operation and enter hibernation upon instructions from the comprehensive power management section of an impending decrease in available charge level, as indicated by a power supply to the controller. .

According to one or more aspects of the disclosed embodiment, when the controller indicates from the comprehensive power management section an impending decrease in the available charge level from the power supply to the branch circuit of the drive section, the controller determines the predetermined secondary path and secondary trajectory. It is configured to command the drive section to drive the autonomous guided vehicle along (to a predetermined autonomous guided vehicle assistance stopping location within the facility).

In accordance with one or more aspects of the disclosed embodiments, the controller is configured to: upon an indication from the comprehensive power management section of an impending decrease in the available charge level directed from the power supply to a branch circuit of the payload handling section, the controller configures the payload handling actuator to: , and instruct the payload handling section to move any payload thereon to a predetermined safe payload location within the payload bed.

According to one or more aspects of the disclosed embodiments, the controller includes at least one of the following:

vehicle health monitor,

drive section health monitor,

payload processing section status monitor, and

Peripheral electronics section health monitor; and

State registration section; and

The controller, upon reboot of the controller, monitors the vehicle status, the driving section, upon indication from the comprehensive power management section of an impending decrease in the available charge level indicated by the power supply to the controller below the demand of the controller. The stored status information from at least one of a status monitor, the payload handling section status monitor, and the peripheral electronic section status monitor is configured as an initialization file of the controller.

According to one or more aspects of the disclosed embodiments, the power supply is an ultra-capacitor or the charge level is a voltage level.

According to one or more aspects of the disclosed embodiments, a method for autonomous guided vehicle power management is provided. This method includes:

It provides an autonomous guided vehicle having a vehicle chassis equipped with a power supply and a power section connected to the chassis and each powered by a power supply, the power section comprising:

a drive section supporting the vehicle chassis and having motorized running wheels arranged to traverse the autonomously guided vehicle on the transverse surface of the autonomously guided facility;

a payload handling section having at least one payload handling actuator configured such that operation of the at least one payload handling actuator affects payload delivery to the payload bed, vehicle chassis, and storage within the facility;

At least one of an autonomous attitude and navigation sensor, a payload handling sensor, and at least one peripheral motor, wherein the at least one peripheral motor is separate and distinct from each of the motors of the drive section and each actuator of the payload handling section. , peripheral electronic components; and

performing respective autonomous operations of the autonomous guided vehicle using a controller communicatively coupled to each of the drive section, payload handling section, and peripheral section; and

The charge level of the power supply is monitored through a comprehensive power management section of the controller, said comprehensive power management section being connected to each branch circuit of the drive section, payload handling section and peripheral electronics section, each of which is connected to a power supply section. Provides power to the drive section, payload handling section, and peripheral electronics section from the supply unit, and the comprehensive power management section provides power to the branch circuits based on the level of charge available from the power supply depending on the demand level of each branch circuit. It includes steps to manage consumption.

A comprehensive power management section according to one or more aspects of the disclosed embodiments includes each branch switching each branch circuit on or off based on the demand charge level of each branch circuit with respect to the other branch circuits and the charge level available from the power supply. The demand charge level of the circuit is managed in a predetermined pattern.

In accordance with one or more aspects of the disclosed embodiments, a pattern is arranged to switch off branch circuits with a reduction in the level of charge available from the power supply, thereby maximizing the level of charge available from the power supply to the controller.

According to one or more aspects of the disclosed embodiments, the predetermined pattern is arranged to turn off the branch circuit as the available charge level from the power supply decreases such that the available charge level to the controller is equal to or greater than the available charge level from the power supply. Based on the charge level, the demand charge level of the controller is exceeded for a maximum period of time.

According to one or more aspects of the disclosed embodiments, the method further includes at least one of the following:

Using the autonomous navigation control portion of the controller, registering and maintaining volatile memory autonomous guided vehicle state and attitude navigation information (historical and current) and conclusively describing the current and predicted state, attitude, and position of the autonomous guided vehicle; and

registering and maintaining current payload identity, state and attitude information (historical and current) in volatile memory using the automatic payload processing control section of the controller;

When the controller receives an indication from the comprehensive power management section of an impending decrease in the available charge level indicated by the power supply to the controller below the demand level of the controller, the controller receives the autonomous guided vehicle status and attitude navigation information and At least one of the payload identity, status, and posture information held in each registry and memory of the corresponding controller section is configured as an initialization file that can be used when the controller is rebooted.

According to one or more aspects of the disclosed embodiments, when the comprehensive power management section indicates an impending decrease in the available charge level from the power supply to the controller, to a level lower than the controller's required level, the controller suspends operation and hibernates. Go in.

In accordance with one or more aspects of the disclosed embodiments, upon indication from the comprehensive power management section of the impending decrease in the available charge level from the power supply to the branch circuit of the drive section, the controller directs the autonomous guided vehicle to predetermined secondary paths and secondary trajectories. Commands the drive section to navigate to a predetermined autonomous guided vehicle secondary stop location within the facility.

In accordance with one or more aspects of the disclosed embodiments, the controller, in accordance with instructions from the comprehensive power management section of the impending decrease in available charge level, from the power supply to the branch circuit of the payload handling section, controls the payload handling actuator and the same. Commands the payload handling section to move any of the above payloads to a predetermined safe payload location within the payload bed.

According to one or more aspects of the disclosed embodiment method of claim 11,

Provide at least one of the following to the controller:

vehicle health monitor,

drive section health monitor,

payload processing section status monitor, and

Peripheral electronics section health monitor; and

State registration section; and

When the controller receives an indication from the comprehensive power management section of an imminent decrease in the available charge level indicated by the power supply to the controller below the controller's required level, the controller monitors the vehicle status. , the stored status information from at least one of the driving section status monitor, the payload handling section status monitor, and the peripheral electronic product section status monitor is configured into an initialization file that can be used when the controller is rebooted.

According to one or more aspects of the disclosed embodiments, the power supply is an ultra-capacitor or the charge level is a voltage level.

It should be understood that the foregoing description is merely illustrative of aspects of the disclosed embodiments. Various alternatives and modifications may occur to those skilled in the art without departing from the aspects of the disclosed embodiments. Accordingly, aspects of the disclosed embodiments are intended to cover all alternatives, modifications and variations that fall within the scope of any claims appended hereto. Moreover, the mere fact that different features are recited in different dependent or independent claims does not mean that combinations of these features cannot be advantageously used, and such combinations remain within the scope of aspects of the disclosed embodiments.

100: Storage and retrieval system
110: Autonomous transport vehicle
122: controller
276: Physical property sensor system
288: Auxiliary navigation sensor system
290: Secondary hazard sensor system
400: Vision system

Claims (107)

In an autonomous guided vehicle, the autonomous guided vehicle:
Frame with payload hold;
a drive section coupled to a frame having drive wheels supporting the autonomous guided vehicle on a crossing surface, the drive wheels traversing the vehicle on the crossing surface moving the autonomous guided vehicle over the crossing surface within the facility;
a payload handler coupled to the frame adapted to transfer a payload having a flat, indeterministic seat surface seated on the payload hold in the storage array between the payload hold of the autonomous guided vehicle and a storage location for the payload;
A physical property sensor system coupled to a frame having electromagnetic sensors, each responsive to the interaction or interface of an emitted or generated electromagnetic beam or electromagnetic field having a physical property, and wherein the electromagnetic beam or electromagnetic field is responsive to an interaction or interface to the physical property. Disturbed by, and the disturbance is sensed and sensed by a physical property electromagnetic sensor, the physical property sensor system to generate sensor data embodying at least one of vehicle navigation attitude or position information and payload attitude or position information. being a physical property sensor system;
An auxiliary sensor system coupled to the frame and complementary to a physical characteristics sensor system, the auxiliary sensor system arranged to capture image data at least in part indicative of one or more of vehicle navigation attitude or position information and payload attitude or position. An auxiliary sensor system, which is a vision system equipped with a camera and assists with information from the physical characteristic sensor system. According to claim 1,
further comprising a controller coupled to the frame, operably connected to a drive section or payload handler, and communicatively coupled to a physical characteristic sensor system, wherein, from information from the physical characteristic sensor system, the controller determines that the controller is configured to provide an autonomous vehicle to traverse the facility. An autonomous guide vehicle, characterized in that it determines the vehicle attitude and position to perform independent guidance of the guide vehicle. According to claim 2,
From information from the physical characteristic sensor system, the controller is configured to determine payload attitude and position to perform independent underpick and storage locations and placement of the payload relative to the underpick, and placement of the payload within the payload hold. Features an autonomous guided vehicle. According to claim 2,
and wherein the controller is programmed with a reference representation of certain features forming at least part of the installation through which the autonomous guided vehicle passes. According to claim 4,
The controller is configured to register the captured image data and generate therefrom at least one image of one or more of the predetermined features, wherein the at least one image is adapted to be compared against a corresponding reference of one or more of the predetermined features of the reference representation. An autonomous guided vehicle, characterized in that it is formatted as a virtual representation of one or more predetermined characteristics to provide. According to claim 5,
The controller is configured to cause a virtual representation of one or more imaged features of the predetermined feature to reside in the autonomous guided vehicle, and a comparison between the virtual representation of the one or more imaged predetermined features and one or more corresponding reference predetermined features is performed on the autonomous guided vehicle. An autonomous guided vehicle, characterized in that it runs permanently. According to claim 5,
The controller is configured to confirm attitude and position information of the autonomous guided vehicle registered to the controller from the physical characteristic sensor system based on a comparison between the virtual representation and the reference representation. According to claim 7,
wherein the controller identifies changes in autonomous guided vehicle attitude and position based on a comparison between the virtual representation and the reference representation, and updates or completes autonomous guided vehicle attitude or position information from a physical characteristic sensor system based on the changes. autonomous guided vehicle. According to claim 8,
The controller is configured to: determine the accuracy of the posture and position information of the autonomous guided vehicle from the physical property sensor system based on analysis of at least one image and at least one identified change in attitude error of the information from the physical property sensor system; An autonomous guided vehicle that determines fidelity and assigns a reliability value based on at least one of posture error and accuracy. According to clause 9,
wherein the controller is configured to switch autonomous guided vehicle navigation based on attitude and position information generated from the virtual representation instead of attitude and position information from the physical characteristic sensor system when the confidence value is below a predetermined threshold. Features an autonomous guided vehicle. According to claim 10,
After switching, the controller:
The automatic guided vehicle continues driving to the destination, or
Select a safe path and trajectory for the autonomous guided vehicle to bring the autonomous guided vehicle from the transition point to a safe location for termination, or
An autonomous guided vehicle, characterized in that it is arranged to initiate communication with an operator identifying a destination and autonomous guided vehicle motion data for operator selection of autonomous guided vehicle control from automatic operation to semi-autonomous operation or manual operation via a user interface device. According to claim 5,
wherein the controller verifies payload attitude and position information registered to the controller from the physical characteristic sensor system based on a comparison of the virtual representation and the reference representation. According to claim 12,
wherein the controller is configured to identify changes in payload attitude and position based on a comparison between the virtual representation and the reference representation and update or complete payload attitude or position information from a physical property sensor system based on the changes. Autonomous guided vehicle. According to claim 13,
The controller determines the accuracy of the payload pose and position information of the information from the physical attribute sensor system based on at least one of the analysis of the at least one image and pose errors and identified changes in the information from the physical attribute sensor system. And, an autonomous guided vehicle characterized in that a reliability value is assigned according to at least one of posture error and accuracy. According to claim 14,
wherein the controller is configured to switch autonomous guided vehicle payload handling based on attitude and position information generated from the virtual representation instead of attitude and position information from a physical characteristic sense system when the confidence value is below a predetermined threshold. Features an autonomous guided vehicle. According to claim 15,
After switching, the controller:
Continue handling the automated guided vehicle to your destination, or
characterized by initiating communication with the operator identifying payload data together with the operator's selection of control of the autonomous guided vehicle from an automated payload processing operation to a semi-automated payload processing operation or a manual payload processing operation via a user interface device. Autonomous guided vehicle. According to claim 5,
The controller creates a simulation image that combines one or more imaged predetermined features and one or more corresponding reference predetermined features of the reference representation presented to the operator with a real-time augmented reality image through a wireless communication system communicatively combining the controller and the operator interface. An autonomous guided vehicle, characterized in that it is adapted to transmit. According to claim 17,
wherein the controller is configured to receive real-time operator commands for a traversing autonomous guided vehicle, the commands reacting to real-time augmented reality images and causing changes to the real-time augmented reality images transmitted by the controller to the operator. Guide vehicle. According to claim 1,
An autonomous guided vehicle, wherein the auxiliary sensor system at least partially affects the immediate justification and/or classification of case units mounted on the autonomous guided vehicle. According to claim 1,
imaged as described by one or more of ancillary information from the auxiliary sensor system, auxiliary vehicle navigation attitude or position, auxiliary payload attitude or position, or
The objects shown are, through auxiliary information, auxiliary vehicle navigation attitude or position, and payload attitude or vehicle navigation attitude or position information and auxiliary payload attitude or position, vehicle navigation attitude or position information and payload attitude or position information. An autonomous guided vehicle, characterized in that it is coupled to an existing model with one or more of the surrounding equipment features and interfacing equipment features to improve the resolution of one or more of the following. In an autonomous guided vehicle, the autonomous guided vehicle:
Frame with payload hold;
a drive section coupled to the frame with drive wheels supporting the vehicle at the crossing surface, the drive wheels moving the autonomous guided vehicle over the crossing surface of the facility to cause the vehicle to traverse the crossing surface;
A payload handler coupled to a frame configured to transfer a payload having a flat, indeterministic seat surface seated on the payload hold between the payload hold of the autonomous guided vehicle and the storage location of the payload in the storage array. ;
A physical property sensor system coupled to a frame having electromagnetic sensors, each responsive to the interaction or interface of an emitted or generated electromagnetic beam or electromagnetic field having a physical property, and wherein the electromagnetic beam or electromagnetic field is responsive to an interaction or interface to the physical property. is perturbed by, and the disturbance is detected and sensed by an electromagnetic sensor of a physical nature, wherein the physical characteristic sensor system is configured to generate sensor data embodying at least one of vehicle navigation attitude or position information and payload attitude or position information. characteristic sensor system; and
An auxiliary sensor system coupled to the frame and separate and distinct from the physical characteristic sensor system, the auxiliary sensor system arranged to capture image data conveying, at least in part, at least one of a vehicle navigation attitude or position and a payload attitude or position. An autonomous guided vehicle comprising a vision system with a camera, wherein the image data is auxiliary information to the information of the physical characteristic sensor system. According to claim 21,
further comprising a controller coupled to the frame, operably coupled to the drive section or payload handler, and communicatively coupled to the physical characteristic sensor system, wherein the controller is configured to traverse the facility. An autonomous guide vehicle, characterized in that it determines the vehicle posture and position to perform independent guidance of the autonomous guide vehicle. According to claim 22,
From information from the physical property sensor system, the controller determines the payload attitude and position to perform an independent underpick, position the payload between a storage location and the independent underpick, and place the payload in a payload hold. An autonomous guided vehicle characterized by being designed to do so. According to claim 22,
and wherein the controller is programmed with reference representations of certain features forming at least part of the installation through which the autonomous guided vehicle passes. According to claim 24,
The controller is configured to register captured image data and generate therefrom at least one image for one or more of the predetermined features, wherein the at least one image is configured to compare against one or more corresponding criteria of the predetermined features of the reference representation. An autonomous guided vehicle characterized in that one or more predetermined characteristics are formatted as a virtual representation. According to claim 25,
The controller is configured to cause a virtual representation of the one or more imaged features of the predetermined feature to reside on the autonomous guided vehicle, and a comparison between the virtual surface of the one or more imaged predetermined features and the one or more corresponding predetermined features to reside on the autonomous guided vehicle. An autonomous guided vehicle characterized in that: According to claim 25,
The controller is configured to confirm attitude and position information of the autonomous guided vehicle registered by the controller from the physical characteristic sensor system based on a comparison of the virtual representation and the reference representation. According to clause 27,
wherein the controller is configured to identify changes in the autonomous guided vehicle attitude and position based on a comparison between the virtual representation and the reference representation, and update or finalize the autonomous guided vehicle attitude or position from the physical characteristic sensor system based on the changes. Features an autonomous guided vehicle. According to clause 28,
the controller determines the accuracy of the posture and position information of the autonomous guided vehicle from the physical property sensor system based on analysis of at least one image and posture errors and identified changes in the information from the physical property sensor system; An autonomous guided vehicle, characterized in that a reliability value is assigned according to at least one of posture error and accuracy. According to clause 29,
wherein the controller is configured to switch autonomous guided vehicle navigation based on attitude and position information generated from the virtual representation instead of attitude and position information from the physical characteristic sensor system when the confidence value is below a predetermined threshold. Features an autonomous guided vehicle. According to claim 30,
After switching, the controller:
continue navigation of the autonomous guided vehicle to the destination or select a safe path and trajectory for the autonomous guided vehicle to bring the autonomous guided vehicle from a transition location to a safe location for stopping; or
An autonomous guided vehicle, characterized in that it is arranged to initiate communication with an operator identifying a destination and autonomous guided vehicle motion data for operator selection of autonomous guided vehicle control from automatic operation to semi-autonomous operation or manual operation via a user interface device. According to claim 25,
and the controller is configured to confirm payload attitude and position information registered by the controller from the physical characteristic sensor system based on a comparison of the virtual representation and the reference representation. According to claim 32,
wherein the controller is configured to identify changes in payload pose and position based on a comparison between the virtual representation and the reference representation and update or complete payload pose or position information from physical characteristic sensors based on the changes. Guide vehicle. According to claim 33,
the controller determines an attitude change in information from the physical property sensor system and determines accuracy of payload attitude and position from the physical property sensor system based on at least one of the identified change and analysis of the at least one image; An autonomous guided vehicle, characterized in that a reliability value is assigned according to at least one of posture error and accuracy. According to claim 34,
The controller is configured to switch autonomous guided vehicle payload handling based on attitude and position information generated from the virtual representation instead of attitude and position information from a physical characteristic sensor system when the confidence value is below a predetermined threshold. An autonomous guided vehicle characterized by According to claim 35,
After switching, the controller:
Continue handling the automated guided vehicle to your destination, or
characterized by initiating communication with the operator identifying payload data together with the operator's selection of control of the autonomous guided vehicle from an automated payload processing operation to a semi-automated payload processing operation or a manual payload processing operation via a user interface device. Autonomous guided vehicle. According to claim 25,
The controller provides a virtual representation of one or more imaged predetermined features and a real-time augmented reality image to the operator, via a wireless communication system that communicatively connects the controller and the operator interface. An autonomous guided vehicle, characterized in that it is adapted to transmit simulation images that combine. According to clause 37,
The controller is adapted to receive real-time operator commands for a traversing autonomous guided vehicle, the commands being responsive to real-time augmented reality images and changes in the real-time augmented reality images delivered by the controller to the operator. Guide vehicle. According to claim 21,
An autonomous guided vehicle, wherein the auxiliary sensor system at least partially affects the immediate alignment and/or sorting of case units mounted on the autonomous guided vehicle. According to claim 21,
The imaged or viewed object described by one or more of supplementary information from the supplementary sensor system, supplementary vehicle navigation attitude or position, and supplementary payload attitude or position is coupled to a reference model of one or more of the surrounding installation features and interfacing facility features. (coapted), characterized in that the resolution of one or more of the vehicle navigation attitude or location information and payload attitude or location information is improved through one or more of auxiliary information, auxiliary vehicle navigation attitude or position, and auxiliary payload attitude or position. autonomous guided vehicle. In the method, said method:
providing an autonomous guided vehicle, wherein the autonomous guided vehicle:
Frame with payload holds,
a drive section coupled to a frame having drive wheels that support an autonomous guided vehicle on a crossing surface, the drive wheels effecting vehicle crossing on a crossing surface that moves the autonomous guided vehicle over the crossing surface within the facility; and
A payload handler coupled to a frame configured to transfer a payload having a flat, indeterministic seat surface seated on the payload hold between the payload hold of the autonomous guided vehicle and the storage location of the payload in the storage array. Providing an autonomous guided vehicle, including;
Generating sensor data using a physical characteristic sensor system, wherein the sensor data implements at least one of vehicle navigation attitude or position information and payload attitude or position information, wherein the physical characteristic sensor system is connected to a frame and uses a physical characteristic as a physical characteristic. Equipped with electromagnetic sensors that each respond to an interface or interaction of a sensor that emits or generates an electromagnetic beam or electromagnetic field, wherein the electromagnetic beam or electromagnetic field is disturbed by interacting with or interfacing with physical properties, and the disturbance is an electromagnetic field of physical properties. generating sensor data, which is detected by a sensor and results in detection;
Capturing image data with an auxiliary sensor system, wherein the image data informs at least one of a vehicle navigation attitude or position and a payload attitude or position and supplements information from a physical characteristic sensor system, the auxiliary sensor system coupled to the frame. Capturing image data, supplementing a physical property sensor system, wherein the secondary sensor system is at least partially a vision system having a camera positioned to capture the image data. According to claim 41,
determining, from information in the physical property sensor system, a vehicle attitude and position that affects independent guidance of the autonomous guided vehicle traversing the facility, using a controller, the controller being coupled to the frame; A method operably connected to a drive section or payload handler and communicatively connected to a physical property sensor system. According to claim 42,
a payload that uses the controller from information from the physical property sensor system to execute an independent underpick, placing a payload between the storage location and an independent underpick, and placing the payload on a payload hold; A method further comprising the step of determining posture and position. According to claim 42,
and wherein the controller is programmed with a reference representation of certain characteristics that define at least a portion of the facility through which the autonomous guided vehicle passes. According to claim 44,
Using the controller, registering captured image data and generating therefrom at least one image for one or more of the predetermined features, wherein the at least one image is a virtual representation of the one or more features. A method characterized in that it is formalized to provide a comparison of one or more of the predetermined features of the reference expression to a corresponding reference. According to claim 45,
The controller is configured to cause a virtual representation of the imaged one or more features of the predetermined feature to reside in the autonomous guided vehicle, wherein a comparison between the imaged virtual representation of the one or more predetermined features and one or more corresponding references of the predetermined feature is performed on the autonomous guided vehicle. A method characterized in that it is executed to reside in the vehicle. According to claim 45,
The method further comprising confirming, via the controller, attitude and position information of the autonomous guided vehicle registered by the physical characteristic sensor system based on a comparison of the virtual representation and the reference representation. According to clause 47,
Identifying changes in the autonomous guided vehicle attitude and position based on a comparison between the virtual representation and the reference representation via the controller, and updating or completing the autonomous guided vehicle attitude or position information from the physical property sensor system based on the changes. A method characterized in that it further comprises steps. According to clause 48,
The controller determines position errors in information from the physical property sensor system and adjusts the accuracy of the autonomous guided vehicle attitude and position from the physical property sensor system based on at least one of the identified changes and analysis of the at least one image. and determining a reliability value based on at least one of the location error and accuracy. According to clause 49,
When the reliability value is less than a predetermined threshold, the controller switches the autonomous guided vehicle navigation based on the posture and position information generated from the virtual representation instead of the posture and position information of the physical characteristic sensor system. method. According to claim 50,
After switching, the controller:
continue navigation of the autonomous guided vehicle to the destination or select a safe path and trajectory for the autonomous guided vehicle to bring the autonomous guided vehicle from a transition location to a safe location for stopping; or
A method characterized in that the method is adapted to initiate, via a user interface device, communication with an operator identifying a destination and autonomous guided vehicle movement data for operator selection of autonomous guided vehicle control from automatic operation to semi-autonomous operation or manual operation. According to claim 45,
wherein the controller verifies payload attitude and position information registered to the controller from the physical characteristic sensor system based on a comparison of the virtual representation and the reference representation. According to claim 52,
wherein the controller identifies changes in payload pose and position based on a comparison between the virtual representation and the reference representation, and updates or completes payload pose or position information from a physical property sensor system based on the changes. . According to claim 53,
wherein the controller determines an attitude error from information from the physical property sensor system and determines an accuracy of payload attitude and position from the physical property sensor system based on at least one of the identified changes and analysis of the at least one image; A method characterized by assigning a reliability value according to at least one of the posture error and accuracy. According to claim 54,
If the confidence value is below a predetermined threshold, the controller switches autonomous guided vehicle payload handling based on attitude and position information generated from the virtual representation instead of attitude and position information from the physical characteristic sensor system. How to feature. According to claim 55,
After switching, the controller:
Continue handling the automated guided vehicle to your destination, or
characterized by initiating communication with the operator identifying payload data together with the operator's selection of control of the autonomous guided vehicle from an automated payload processing operation to a semi-automated payload processing operation or a manual payload processing operation via a user interface device. How to do it. According to claim 42,
The controller may display a virtual representation of one or more imaged predetermined features and one or more corresponding reference predetermined features of the reference representation to provide the operator with a real-time augmented image, via a wireless communication system communicatively combining the controller and the operator interface. A method characterized by transmitting a simulation image combining. According to clause 57,
wherein the controller receives real-time operator commands for a traversing autonomous guided vehicle, the commands being responsive to real-time augmented reality images and changes in the real-time augmented reality images transmitted by the controller to the operator. According to claim 41,
The method of claim 1 , wherein the auxiliary sensor system at least partially influences on-the-fly alignment and/or sorting of case units mounted on the autonomous guided vehicle. According to claim 41,
The imaged or viewed object described by one or more of ancillary information from the auxiliary sensor system, auxiliary vehicle navigation attitude or position, auxiliary payload attitude or position is coupled to a reference model of one or more of peripheral facility features and interfacing facility features. , a method characterized in that the resolution of at least one of vehicle navigation attitude or location information and payload attitude or location information is improved through at least one of auxiliary information, auxiliary vehicle navigation attitude or position, and auxiliary payload attitude or position. . In an autonomous guided vehicle, the autonomous guided vehicle:
Frame with payload hold;
a drive section coupled to the frame with drive wheels supporting the vehicle on the transverse surface, the drive wheels effecting vehicle traversal of the transverse surface to move the vehicle over the transverse surface of the facility;
a payload handler coupled to the frame configured to transfer the payload between the vehicle's payload hold and a storage location for the payload in the storage array;
An auxiliary sensor system connected to the frame for cooperation between vehicle and driver, said auxiliary sensor system configured to collect at least sensing data implementing vehicle attitude and position information for autonomous navigation by the vehicle of the installation Includes an auxiliary sensor system that assists the sensor system,
The auxiliary sensor system may comprise, at least in part, at least one camera positioned to capture image data indicative of objects and/or spatial features within at least a portion of the facility as viewed by the camera to the vehicle at different locations in the facility. It is a vision system with
a controller coupled to the frame and communicatively coupled with the auxiliary sensor system to register information from image data of the at least one camera, wherein the controller registers, from the information, a predetermined number of objects or spatial features; , and in response thereto selectively reconfigure the vehicle from an autonomous state to a cooperative vehicle state where the vehicle is positioned to receive operator commands for the vehicle to continue to influence vehicle operation. Autonomous guided vehicle. According to claim 61,
The predetermined physical characteristic is such that at least one object or spatial feature extends across a crossing surface, at least a portion of a vehicle crossing path across the crossing surface, or through the space of a vehicle or through another vehicle crossing the crossing surface. An autonomous guided vehicle characterized by being According to claim 61,
and wherein the controller is programmed with a reference representation of certain characteristics that at least partially define the facility through which the vehicle passes. According to claim 61,
and the controller is arranged to register the captured image data and generate therefrom at least one image of the at least one object or spatial feature. According to claim 1,
An autonomous guided vehicle, wherein the at least one image is formatted into a virtual representation of at least one object or spatial feature to provide comparison to one or more predetermined features of the reference representation. According to item 65,
The controller identifies the presence of a predetermined physical feature or spatial feature of the object based on a comparison between the virtual representation and the reference representation, determines the size of the predetermined physical feature, and determines the location or location of the object determined through the comparison. An autonomous guided vehicle characterized by commanding the vehicle to stop at a predetermined trajectory based on spatial characteristics. According to clause 66,
The stopping position on the predetermined trajectory maintains an object or spatial reference within the field of view of at least one camera, maintains continuous imaging of the predetermined physical characteristic, and signals to one or more other vehicles an area to avoid or a detour area. An autonomous guided vehicle, characterized in that starting. According to claim 61,
Predetermined physical characteristics are determined by the controller by determining the position of the object within a reference frame of at least one camera that is calibrated and has a predetermined relationship to the vehicle, wherein the object pose within the reference frame of the at least one camera is determined by determining a predetermined physical characteristic of the object. An autonomous guided vehicle characterized by determining the presence of physical characteristics of. According to clause 68,
wherein the controller identifies the presence and transition from an autonomous state to a collaborative vehicle state and initiates transmission of a communication image, identification of the presence or absence of certain physical characteristics, to the driver interface for driver collaboration tasks in the vehicle. Guide vehicle. According to claim 61,
The controller is characterized in that it applies a trajectory to the autonomous guide vehicle that causes the autonomous guide vehicle to reach zero speed within a predetermined time in which the movement of the autonomous guide vehicle along the track is adjusted according to the position and / or spatial characteristics of the object. Autonomous guided vehicle. According to claim 61,
An autonomous guided vehicle, wherein the capture of image data indicative of object and/or spatial features occurs opportunistically while transferring the payload between payload holds on the vehicle or storage locations in a storage array. According to claim 61,
The controller is programmed to command the vehicle to a different location in the facility associated with the vehicle performing one or more predetermined automated payload autonomous transport operations, each of which is configured to have at least one camera on a different An autonomous guided vehicle characterized by tasks that are independent of the captured image data viewed from the location. According to clause 68,
wherein the controller determines the presence of predetermined physical characteristics of at least one object or spatial feature, each of which is at least partially consistent with, but secondary and peripheral to, the movement of the vehicle performing an autonomous motion task with one or more predetermined payloads. autonomous guided vehicle. According to clause 68,
wherein the controller is such that the determination of the presence of certain physical characteristics of at least one object or spatial feature is opportunistic to the vehicle's operation to perform each of one or more certain automatic delivery tasks. Guide vehicle. According to clause 74,
An autonomous guided vehicle, wherein at least one of the one or more predetermined autonomous transport operations affects at least one of the different locations. According to claim 61,
An autonomous guided vehicle wherein the cooperative vehicle state supplements the vehicle's autonomous state to influence each of one or more predetermined automatic payload delivery tasks. In the method, said method:
providing an autonomous guided vehicle, wherein the autonomous guided vehicle:
Frame with payload holds;
a drive section coupled to a frame having drive wheels that support a vehicle on a transverse surface, the drive wheels effecting vehicle traversal of the transverse surface to move the vehicle over the transverse surface of the installation;
a payload handler coupled to the frame configured to transfer the payload between the vehicle's payload hold and a storage location for the payload in the storage array;
Image data indicating objects and/or spatial features within at least a portion of the facility as viewed by at least one camera when the vehicle is at different locations in the facility, using an auxiliary sensor system connected to the frame for vehicle and operator coordination. generating a auxiliary sensor system, wherein the auxiliary sensor system is a vision system having at least one camera positioned to capture, at least in part, image data, wherein the auxiliary sensor system determines at least a vehicle attitude and position for autonomous navigation by a vehicle in the facility; generating image data to supplement a vehicle autonomous navigation/actuation sensor system adapted to collect sensor data embodying information;
registering information from image data of at least one camera with a controller coupled to the frame and communicatively coupled to an auxiliary sensor system; and
Using the controller, arranged to determine from said information the presence of predetermined physical characteristics of at least one object or spatial feature and, in response thereto, to receive operator commands for the vehicle to continue performing vehicle operations from an autonomous state A method comprising: selectively reconfiguring a vehicle into a cooperative vehicle state. According to clause 77,
Predetermined physical properties are characterized in that at least one object or spatial feature extends at least a portion of the crossing surface, across a vehicle crossing path crossing the crossing surface, or through the space of a vehicle or another vehicle crossing the crossing surface. method. According to clause 77,
and wherein the controller is programmed with a reference representation of certain characteristics that at least partially define the facility through which the vehicle passes. According to clause 77,
The method further comprising generating, from the registered and captured image data, at least one image of at least one object or spatial feature representing predetermined physical characteristics. According to clause 77,
At least one image is formatted as a virtual representation of at least one object or spatial feature, the method further comprising comparing the virtual representation to one or more reference features of a predetermined feature of the reference representation. . According to claim 81,
Using the controller, identifying the presence of a predetermined physical characteristic of the object or spatial feature based on a comparison between the virtual representation and the reference representation, determining a magnitude of the predetermined physical characteristic, and making the comparison. A method further comprising commanding the vehicle to stop on a predetermined trajectory based on the location or spatial characteristics of the object determined through the method. According to item 82,
With the vehicle in a stationary position on a predetermined trajectory, maintaining an object or spatial reference within the field of view of at least one camera and continuously imaging predetermined physical characteristics, and traffic obstacles, avoidance zones, and detour zones. A method further comprising the step of sending a signal to at least another vehicle. According to clause 77,
The predetermined physical characteristic is determined by determining the position of the object within a frame of reference of at least one camera with which the controller is calibrated and has a predetermined relationship to the vehicle, wherein the controller determines the position of the object from the object pose in the frame of reference of the at least one camera. A method characterized by determining the presence of a predetermined physical property. According to item 84,
The controller identifies the presence of a predetermined physical characteristic of the at least one object or spatial feature, switches from an autonomous state to a cooperative vehicle state, initiates transmission of communication images, and identifies the predetermined physical characteristic to a cooperative operator of the vehicle. A method characterized in that it is adapted to provide an operator interface for operation. According to clause 77,
Using the above controller,
Further comprising the step of applying a trajectory to the autonomous guided vehicle that causes the autonomous guided vehicle to reach zero speed within a predetermined time in which the movement of the autonomous guided vehicle along the trajectory is adjusted according to the location and/or spatial characteristics of the object. Features an autonomous guided vehicle. According to clause 77,
An autonomous guided vehicle, wherein the capture of image data indicative of object and/or spatial features occurs opportunistically while transferring the payload between payload holds on the vehicle or storage locations in a storage array. In an autonomous guided vehicle, the autonomous guided vehicle:
A vehicle chassis equipped with a power supply and a power section connected to the chassis and each powered by a power supply, the power section comprising:
a drive section having motor driven wheels, supporting the vehicle chassis, and arranged to traverse the autonomously guided vehicle on the traversing surface of the autonomously guided facility;
a payload handling section having at least one payload handling actuator wherein actuation of the at least one payload handling actuator is such that operation of the at least one payload handling actuator performs payload transfer between a payload bed in a vehicle chassis and storage within the facility;
A peripheral electronics section having at least one of an autonomous attitude and navigation sensor, at least one of a payload handling sensor, and at least one peripheral motor, wherein the at least one peripheral motor is configured for each motor in the drive section and each of the payload handling sections. a vehicle chassis and power section comprising: a peripheral electronics section that is separate and distinct from the actuator;
A controller communicatively connected to a drive section, a payload handling section, and a peripheral section respectively to perform each autonomous operation of the autonomous vehicle, said controller communicatively connected to a power supply to monitor the charge level of the power supply. A controller, including a power management section;
The integrated power management section is configured to each branch circuit of the drive section, the payload handling section, and the peripheral electronics section respectively supplying power to the drive section, the payload handling section, and the peripheral electronics section from the power supply. connected, and the comprehensive power management section manages the power consumption of each branch circuit based on the required level of each branch circuit with respect to the available charge level from the power supply. According to clause 88,
The comprehensive power management section controls the requirements of each branch circuit, switching each branch circuit on or off in a predetermined pattern based on the required charge level of each branch circuit with respect to other branch circuits and charge levels available from the power supply. An autonomous guided vehicle characterized by managing its charge level. According to clause 88,
wherein the predetermined pattern switches the branch circuit off as the available charge level from the power supply decreases, in order to maximize the available charge level from the power supply to the controller. According to clause 88,
The predetermined pattern determines that the available charge level toward the controller is
An autonomous guided vehicle, characterized in that the branch circuit is switched off when the available charge level from the power supply decreases to be above the required charge level of the controller for a maximum period of time based on the available charge level of the power supply. According to clause 88,
The controller:
an autonomous navigation control section, configured to register and maintain autonomous guided vehicle status and navigation information, and historical and current, determining and describing current and predicted status, attitude and position of the autonomous guided vehicle, in a volatile memory; and
an automatic payload processing control section configured to register and maintain current payload identity, state and attitude information, history and presence in volatile memory; It has at least
When the controller receives an indication from the comprehensive power management section that the available charge level from the power supply to the controller is decreasing below the controller's demand level, the controller determines the autonomous guided vehicle status stored in the respective registries and memories of that controller section. , An autonomous guided vehicle characterized in that at least one of the attitude navigation information payload identity, status, and attitude information is reconfigured into an available initialization file upon reboot of the controller. According to clause 88,
wherein if the controller receives an indication from the comprehensive power management section that the available charge level from the power supply to the controller may imminently decrease to a level below the demand level of the controller, the controller will cease operation and enter a hibernation mode. Autonomous guided vehicle. According to clause 88,
When the controller indicates from the comprehensive power management section an impending decrease in the available charge level from the power supply to the branch circuit of the drive section, the controller determines the auxiliary trajectory and the predetermined auxiliary path to the predetermined autonomous guided vehicle auxiliary stopping position of the installation. An autonomous guided vehicle, characterized in that the driving section is commanded to operate the autonomous guided vehicle along . According to clause 88,
When the controller indicates from the comprehensive power management section an impending decrease in the available charge level from the power supply to the branch circuits of the payload handling section, the controller instructs the payload handling section to activate the payload handling actuator and any payloads thereon. An autonomous guided vehicle characterized by commanding the load to move to a predetermined safe payload location in the payload bed. According to clause 88,
The controller:
vehicle health monitor,
drive section health monitor,
payload handling section status monitor, and
Peripheral electronics section health monitor; and
State registration section; Contains at least one of
The controller may, upon receiving an indication from the comprehensive power management section that an impending decrease in the available charge level from the power supply to the controller be lower than the controller's demand level, monitor the vehicle health section, the drive section health monitor, the payload handling section health monitor, and An autonomous guided vehicle, characterized in that the stored status information from at least one of the status monitors of the peripheral electronics section of the registration section is configured into an initialization file that is possible upon reboot of the controller. 89. The autonomous guided vehicle of claim 88, wherein the power supply is an ultracapacitor or the charging level is a voltage level. In the power management method of an autonomous guided vehicle, the method includes:
Providing an autonomous guided vehicle having a vehicle chassis equipped with a power supply and a power section connected to the chassis and each powered by a power supply, wherein the power section:
a drive section having motor driven wheels, supporting the vehicle chassis, and arranged to traverse the autonomously guided vehicle on the traversing surface of the autonomously guided facility;
a payload handling section having at least one payload handling actuator wherein operation of the at least one payload handling actuator is such that operation of the at least one payload handling actuator transfers the payload between a payload bed of the vehicle chassis and a storage unit of the facility;
A peripheral electronics section having one or more autonomous attitude and navigation sensors, one or more payload handling sensors, and one or more peripheral motors, wherein at least one peripheral motor is separate from each motor in the drive section and each actuator in the payload handling section. is distinguished by a peripheral electronics section; containing. providing autonomous guided vehicles;
performing respective autonomous operations of the autonomous guided vehicle using a controller communicatively coupled to each of the drive section, payload handling section, and peripheral section; and
Monitoring the charge level of the power supply with a comprehensive power management section of the controller, wherein the comprehensive power management section includes a drive section, a payload handling section, and a peripheral electronics section that supplies power to each drive section and payload handling section, respectively. connected to each branch circuit of the section and peripheral electronics from the power supply, the comprehensive power management section is responsible for managing the power consumption of the branch circuit based on the main level of each branch circuit with respect to the available charge level from the power supply. How to feature. According to clause 98,
The comprehensive power management section manages the demand charge level of each branch circuit by switching each branch circuit on or off at the available charge level from the power supply and the demand charge level of each branch circuit with respect to the other branch circuits. A method characterized by: According to clause 98,
wherein the predetermined pattern is such that branch circuits are switched off as the available charge level from the power supply decreases to maximize the available charge level from the power supply to the controller. According to clause 98,
The predetermined pattern is to turn off the branch circuit when the available charge level from the power supply decreases such that the available charge level to the controller is above the controller's demand charge level for a maximum period of time based on the available charge level of the power supply. A method characterized in that it is arranged to switch. According to clause 98,
registering and maintaining, using an autonomous navigation control section of the controller, past and present autonomous guided vehicle status and attitude navigation information in volatile memory that is conclusive and descriptive of current and predicted status, attitude and position of the autonomous guided vehicle; and
registering and maintaining current payload identity, state and attitude information, history and current in volatile memory using an automatic payload processing control section of the controller; Additionally include at least one of the following:
When there is an indication from the comprehensive power management section that a reduction in the available charge level below the controller's demand level is imminent from the power supply to the controller, the controller will determine the autonomous guided vehicle status maintained in the respective registries and memories of the corresponding controller section, and A method characterized by configuring at least one of attitude navigation information, payload identity, status, and attitude information into an initialization file available when the controller is rebooted. According to clause 98,
and wherein, if an indication from the comprehensive power management section is indicated from the power supply that a decrease in the available charge level below the demand level of the controller is imminent, the controller postpones operation and enters hibernation. According to clause 98,
When the controller receives an indication from the comprehensive power management section that a decrease in the available charge level from the power supply section to the branch circuit of the drive section is imminent, the controller directs the autonomous guided vehicle along predetermined auxiliary paths and auxiliary trajectories. and commanding the drive section to drive to a predetermined autonomous guided vehicle assisted stopping position of the installation. According to clause 98,
Upon indication from the comprehensive power management section that a decrease in the available charge level from the power supply to the branch circuits of the payload handling section is imminent, the controller instructs the payload handling section to direct the payload to a predetermined safe payload location in the payload bed. A method characterized by commanding a load handling actuator and any payload thereon to move. According to clause 98,
Providing the controller with one or more of the following:
vehicle health monitor,
drive section health monitor,
payload handling section status monitor, and
Peripheral electronics section health monitor; and
Further comprising providing a status registration section;
If an impending decrease in the available charge level from the power supply to the controller is indicated from the comprehensive power management section below the controller's demand level, the controller
Characterized by reconstructing the stored status information from at least one of the vehicle status monitor, the drive section status monitor, the payload handling status monitor, and the peripheral electronics section status monitor of the status registration section into an available initialization file upon reboot of the controller. method. According to clause 98,
The power supply is an ultra capacitor, or the charging level is a voltage level.