JP2022171625A - Oversight system to autonomous vehicle communications - Google Patents

Abstract

To provide an oversight system for autonomous vehicle communications.SOLUTION: A system comprises an autonomous vehicle (AV) and an operation server operably coupled with the AV. The operation server accesses environmental data associated with a road traveled by the AV. The environmental data is associated with a time window during which the AV is traveling along the road. The operation server compares the environmental data with map data that comprises expected road conditions ahead of the AV. The operation server determines whether the environmental data comprises an unexpected road condition that is not included in the map data. In response to determining that the environmental data comprises the unexpected road condition that is not included in the map data, the operation server determines a location coordinate of the unexpected road condition, and communicates a command to the AV to maneuver to avoid the unexpected road condition.SELECTED DRAWING: None

Description

The present disclosure relates generally to autonomous vehicles. More specifically, the present disclosure relates to supervisory systems for autonomous vehicle communications.

One of the goals of autonomous vehicle technology is to provide vehicles that can safely navigate to their destination. In some cases, unexpected road conditions can occur on roads on which autonomous vehicles travel. For example, unexpected weather conditions or unexpected traffic may be observed on the road ahead of the autonomous vehicle. Current autonomous vehicle technology is not configured for such unforeseen road conditions.

Current autonomous vehicle technology is not configured for unpredictable road conditions. The present disclosure recognizes various problems and unmet needs associated with communicating commands to autonomous vehicles (AVs). Certain embodiments of the present disclosure are unique technical solutions to technical problems of current autonomous vehicle technology, including the above problem for communicating commands to one or more AVs heading into unexpected road conditions. I will provide a. The disclosed system is configured to determine different levels of granularity of commands for one or more AVs. In some examples the command may be a general command, while in other examples the command may be a specific command.

A global command may relate to a particular road condition and may be directed to one or more AVs directed to the particular road condition. Unexpected road conditions may include, but are not limited to, unexpected weather conditions, unexpected traffic jams, road closures, road obstructions, construction areas, etc., among others. For example, an unexpected road condition on a road on which one or more AVs travel may be detected by an operations server operably coupled to the one or more AVs as the one or more AVs travel along the road. It can be detected by analyzing the environmental data associated with the time window during which one is on and comparing the environmental data with map data including expected road conditions. Environmental data may include weather data and traffic data. The operations server may access environmental data from external sources such as live weather forecasts, live traffic information, and the like. Upon detecting an unexpected road condition, a global command may be communicated to one or more AVs indicating that an unexpected road condition has been detected on the road and to divert.

Special commands may relate to special road conditions and may be directed to specific AVs in special road conditions. For example, suppose a particular AV is traveling along a road that is blocked by a road closure. Upon detection of a road closure, a special command may be communicated to the particular AV by an operations server operatively coupled to the particular AV to inform it to take a particular action (e.g., take the next exit, Make the first right turn, go 50 feet and pull over to the side of the road).

The present disclosure contemplates detecting various unexpected road conditions and issuing commands (eg, broad or special commands) to one or more AVs heading into the unexpected road conditions. In some cases, similar to the example above, commands are communicated from the operations server to one or more AVs, ie, this is an operations server to AV command.

In some cases, upon receiving a command, the AV (or an in-vehicle controller associated with the AV) may update the command based on analysis of sensor data received from sensors on the AV. For example, if the in-vehicle controller determines that there are obstacles in the path for navigating the AV according to the commands issued, the AV may update the commands. For example, assume the command sent to the AV indicates to pull over to the first location on the side of the road. Also assume that the AV's sensors detect that the first location is occupied by an obstacle. Thus, in this example, the onboard controller determines that an obstacle prevents the fulfillment of the issued command. Therefore, the onboard controller can search for an obstacle-free location. Upon finding a second, clear location, the in-vehicle controller may update the command to bring the AV to a second location, eg, 50 feet beyond the first location.

In some cases, the AV may communicate commands to the operations server, ie, AV to operations server commands. For example, when an AV detects an unexpected object in its travel path, it may communicate a command containing one or more navigation instructions to the operations server. The AV (or an on-board controller associated with the AV) analyzes the sensor data received from the sensors and uses the sensor data to identify areas in front of the AV, such as road signs, buildings, terrain, lane markings, road boundaries, traffic lights, etc. Unexpected objects may be detected by comparison with map data containing expected objects. By comparing sensor data and map data, the AV may detect that unexpected objects included in sensor data are not included in map data. The AV may determine suggested commands or navigation instructions to avoid unexpected objects. The AV may communicate the proposed command to the operations server for confirmation, affirmation, or approval. The operations server may confirm, modify, or overwrite the proposed command and communicate the confirmation or modification to the AV. Upon receiving confirmation or modification, the AV may perform the original command or the modified command.

In some cases, a first AV may communicate commands to a second AV, ie, this is an AV to AV command. For example, assume that a first and second AV are on the same road and the first AV is ahead of the second AV. Also assume that the first AV received a command from the operations server to avoid unexpected road conditions ahead of the first AV. The first AV may or may not update commands based on analysis of sensor data, similar to above.

In one embodiment, the first AV may indirectly communicate the command (or updated command) to the second AV. For example, a first AV may communicate commands (or updated commands) to an operations server, and the operations server may communicate commands (or updated commands) to a second AV.

In another embodiment, if the second AV is within vehicle-to-vehicle (V2V) communication range of the first AV, the first AV uses, for example, a vehicle-to-vehicle (V2V) network interface to command (or updated commands) directly to the second AV. In this embodiment, the first AV accesses environmental data associated with the portion of the road between the first AV and the second AV. The environmental data includes position coordinates of a plurality of objects detected by sensors of the first AV while the first AV was traveling along the portion of the road between the first AV and the second AV. can include Since the first AV has already traveled the portion of the road between the first AV and the second AV, the first AV has experienced and recorded environmental data. The first AV determines if the object detected in the sensor data and/or environmental data (related to the portion of the road between the first AV and the second AV) is the result of the command issued by the second AV. can decide whether to impede performance. In this process, for example, the first AV receives the trajectory of the second AV (eg, from the operations server and/or the second AV), and the objects detected in the sensor data and/or environmental data 2 AVs on the travel path. If an object is on the path of travel of the second AV, the first AV may determine that the object prevents the second AV from fulfilling the command. If the first AV determines that an object detected in the sensor data and/or environmental data will prevent the second AV from fulfilling the issued command, the first AV determines that the updated command will Update the commands of the second AV to include one or more navigation instructions therein for avoiding objects on the path of travel of the second AV. Similarly, if a third AV is within V2V communication range of the first AV, the first AV may transmit a set of environmental data associated with roads between the first AV and the third AV. Based on the analysis, a second updated command may be communicated to a third AV moving behind the second AV.

In the case where the third AV is not within V2V communication range of the first AV, but the third AV is within V2V communication range of the second AV, the second AV and the second AV Access and analyze environmental data associated with roads to and from the third AV to avoid objects on the third AV's travel path while the third AV performs the command. 3 AV can communicate the command. As such, the present disclosure contemplates communication between network meshes of AVs, where different subsets of AVs may be in varying V2V communication range from other subsets of AVs. Thus, the present disclosure contemplates one-to-one, one-to-many, and many-to-many communication between the AV's network mesh and the operations server.

Regarding communication of commands from the operations server to the AV, in one embodiment, the system includes an AV and an operations server. The AV includes at least one vehicle sensor, where the AV is configured to travel along the road. The operations server is communicatively coupled with the AV. The operations server includes a processor. The processor is configured to access environmental data associated with a road ahead of the AV. Environmental data is associated with the timeframe during which the AV is traveling along the road. The processor compares the environmental data with map data including expected road conditions in front of the AV. Based on comparing the environmental data and the map data, the processor determines whether the environmental data includes unexpected road conditions not included in the map data. In response to determining that the environmental data includes an operational design domain corresponding to unanticipated road conditions not included in the map data and/or conditions under which the AV can safely operate, the processor outputs the unanticipated road conditions. and communicate commands to the AV to maneuver around unexpected road conditions.

Regarding receipt of commands from the operations server by the AV, in one embodiment, the system includes an AV and a controller. The AV includes at least one vehicle sensor, where the AV is configured to travel along the road. A controller is associated with the AV. The controller includes a processor. The processor is configured to receive commands from the operations server to navigate the AV to avoid unexpected road conditions. A processor receives sensor data including position coordinates of a plurality of objects in front of the AV from at least one vehicle sensor. The processor determines whether at least one object from the plurality of objects interferes with fulfillment of the command. In response to determining that at least one object from the plurality of objects interferes with performance of the command, the processor updates the command so that the updated command includes at least one object during performance of the command. contains one or more navigation instructions for avoiding The processor navigates the AV according to the updated commands.

Regarding the lead AV that communicates commands to follow-on AVs, in one embodiment, the system includes a lead AV and a follow-on AV. The lead AV includes at least one vehicle sensor. The lead AV is configured to move along the road. The trailing AV differs from the leading AV and is communicatively coupled with the leading AV. The trailing AV is moving along the road following behind the lead AV. The top AV is associated with the controller. The controller includes a first processor. A first processor is configured to receive commands to navigate the lead AV to avoid unexpected road conditions ahead of the lead AV. A first processor receives sensor data from at least one vehicle sensor including position coordinates of a first plurality of objects in front of the lead AV. A first processor accesses a first set of environmental data associated with a portion of the road between the leading AV and the trailing AV. The first set of environmental data includes position coordinates of a second plurality of objects between the leading AV and the trailing AV. A first processor determines whether at least one object from the first and second plurality of objects interferes with performance of the command by the subsequent AV. In response to determining that at least one object prevents the subsequent AV from performing the command, the processor updates the subsequent AV's command based at least in part on the first set of sensor data and environmental data; As a result, the updated command includes one or more navigation instructions for avoiding at least one object during execution of the command. The first processor communicates the updated command to the subsequent AV.

The disclosed system offers several practical applications and technical advantages, including: 1) each level of granularity of commands can be associated with a particular road condition; Techniques for determining various levels of granularity of commands, including broad and special commands, that can be directed at one or more AVs to which they are headed; 2) associated with windows of time during which the first AV travels along the road; communication of a command (e.g., a wide area or special command) to the first AV to avoid the unexpected road condition on the road on which the first AV travels, in response to detection of an unexpected road condition by the 3) detecting an obstacle in the path of navigating the first AV according to the received command and determining that the obstacle prevents the first AV from navigating according to the received command; 4) techniques for enabling indirect data communication between the first and second AVs via the operations server; 5) techniques for enabling indirect data communication between the first and second AVs within V2V communication range; and 6) communicating commands (or updated commands) to the second AV, according to the commands issued, for unexpected road conditions and obstacles that impede navigation. A technique for notifying a second AV.

Therefore, the system described in this disclosure has practical application in determining a more efficient, safe, and reliable solution for determining and communicating commands to AVs to avoid unexpected road conditions. can be integrated into Additionally, the system described in this disclosure is more efficient for AVs heading into unexpected road conditions, even when the unexpected road conditions are not within the field of view or detection zone of the AV's sensors. and can be integrated into additional practical applications to determine solutions for safe, reliable navigation. For example, from live weather and/or traffic information, the operations server detects unexpected road conditions outside the detection zone of the AV's sensors and issues commands to the AV to avoid the unexpected road conditions. obtain. As such, the disclosed system may improve current autonomous vehicle technology.

Additionally, the disclosed system may improve data communication between the AV and the operations server responsible for monitoring the operation of the AV. For example, the disclosed system may establish a communication channel between a first AV and one or both of an operations server and subsequent AVs within V2V communication range of the first AV. As such, the first AV can send and receive data (eg, commands or any other data) to and from the operations server and/or subsequent AVs.

Therefore, by improving data communication between AVs and operations servers, the disclosed system provides flexibility in frequency bandwidth allocation to different AVs when fleets of AVs are traveling on the same road. do. Thus, the disclosed system leverages V2V communication for load balancing and processing resource distribution at different network/access nodes (e.g., operations servers or any AV in a fleet), and data in one access/network. Transmission overhead may be reduced or minimized.

Furthermore, by improving data communication between the AV and the operations server, the disclosed system can further improve tracking of the AV and monitoring of the autonomous operation of the AV. For example, current data communication technology used in current autonomous vehicle technology may suffer from the limitations of existing network infrastructure already in place. The disclosed system may implement one or more computing agent units in the AV that are configured to operate in various communication states.

For example, if an AV is connected to an operations server and a successor AV, one or more computing agent units transition to the state "connected to an operations server and successor AV". In this state, the AV can send and receive data (eg, commands or any other data) between the operations server and/or subsequent AVs.

In another example, when an AV is connected to a successor AV, one or more computing agent units transition to a "connected to successor AV" state. Thus, if communication between the AV and the operations server is lost, the AV can send and receive data (eg, commands or any other data) to and from the successor AV.

In another example, when an AV is connected to an operations server, one or more computing agent units transition to a "connected to operations server" state. Thus, if communication between an AV and a successor AV is lost, the AV can send and receive data (eg commands or any other data) to and from the operations server.

In another example, if connectivity between the AV and both the operations server and successor AVs is lost, one or more of the computing agent units "executes the command without connectivity to the operations server and succession AVs." transition to state. In this state, the controller associated with the AV pulls the AV off the road until communication is re-established with either or both the operations server and the trailing AV.

In this manner, the disclosed system can improve data communication with AVs, track AVs, and monitor autonomous operation of AVs.

Particular embodiments of the present disclosure may include some, all, or none of these advantages. These advantages and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.
The present application provides, for example, the following.
(Item 1)
A system, the system comprising:
an autonomous vehicle (AV) comprising at least one vehicle sensor, the AV configured to travel along a road;
an operations server communicatively coupled to the AV;
The operations server comprises a processor, the processor comprising:
accessing environmental data associated with the road ahead of the AV, wherein the environmental data is associated with a window of time during which the AV travels along the road; ,
comparing the environmental data with map data including expected road conditions in front of the AV;
determining whether the environmental data includes unexpected road conditions not included in the map data based at least in part on comparing the environmental data to the map data;
in response to determining that the environmental data includes the unexpected road conditions not included in the map data;
determining location coordinates of the unexpected road condition;
and communicating commands to the AV to steer around the unexpected road condition.
(Item 2)
further comprising a controller associated with the AV, the controller comprising a second processor, the second processor comprising:
receiving sensor data from the at least one vehicle sensor including position coordinates of a plurality of objects on the road ahead of the AV;
Based at least in part on the sensor data, updating the command such that the updated command includes one or more instructions for the AV to avoid the plurality of objects during performance of the command. and
and navigating the AV according to the updated command.
(Item 3)
The system of any of the preceding items, wherein the second processor is further configured to communicate the updated command to one or more subsequent AVs moving in line behind the AV. .
(Item 4)
The second processor,
comparing the sensor data with the map data, the map data including position coordinates of expected objects on the road in front of the AV;
determining, based at least in part on comparing the sensor data to the map data, whether the sensor data indicates that the unexpected object is absent from the map data;
Responsive to determining that the sensor data indicates that the unexpected object is not in the map data, the AV determines suggested navigation instructions for avoiding the unexpected object. A system according to any of the preceding items, further configured to:
(Item 5)
The second processor is further configured to perform the suggested navigation instructions in response to determining that the sensor data indicates that the unexpected object is absent from the map data. A system according to any of the preceding items.
(Item 6)
The second processor,
in response to determining that the sensor data indicates that the unexpected object is not in the map data;
communicating the suggested navigation instructions to the operations server;
determining whether confirmation has been received from the operations server to perform the suggested navigation instructions;
and performing the suggested navigation instructions in response to receiving the confirmation from the operations server.
(Item 7)
The second processor,
in response to determining that the sensor data indicates that the unexpected object is not in the map data;
communicating the sensor data to the operations server;
any of the preceding items, further configured to: request the operations server to provide instructions to avoid the unexpected object based at least in part on the sensor data. system.
(Item 8)
further comprising one or more subsequent AVs different from the AV;
the one or more trailing AVs are on the road behind the AVs;
The one or more subsequent AVs are communicatively coupled to the operations server;
The above processor
in response to determining that said environmental data includes said unexpected road conditions not included in said map data;
receiving the sensor data from the controller;
generating a second command including one or more instructions for avoiding the unexpected road condition while the one or more subsequent AVs are avoiding the plurality of objects; the second command includes the position coordinates of the unexpected road condition indicated in the command and the position coordinates of the plurality of objects indicated in the sensor data;
and communicating the second command to the one or more subsequent AVs.
(Item 9)
A method, the method comprising:
accessing environmental data associated with a road ahead of an autonomous vehicle (AV), the environmental data associated with a window of time during which the AV travels along the road; and
comparing the environmental data with map data including expected road conditions in front of the AV;
determining whether the environmental data includes unexpected road conditions not included in the map data based at least in part on comparing the environmental data to the map data;
in response to determining that the environmental data includes the unexpected road conditions not included in the map data;
determining location coordinates of the unexpected road condition;
and communicating commands to the AV to steer around the unexpected road condition.
(Item 10)
The unexpected road conditions include at least one of unanticipated weather conditions, unanticipated traffic congestion, unanticipated road closures, and unanticipated construction areas not included in the map data. , item 9.
(Item 11)
A method according to any of the preceding items, wherein the environmental data includes at least one of weather data and traffic data.
(Item 12)
The above command is
Transition from autonomous operation to manual operation by the above AV,
avoidance of obstacles on the road in front of the AV by the AV;
avoidance by the AV of one or more certain lanes in which one or more obstacles are detected;
avoidance by the AV of one or more fixed routes where the unexpected road conditions are detected;
certain route changes by the AV, and driving by the lead AV at speeds slower or faster than indicated in driving instructions associated with the lead AV;
The method of any of the preceding items, associated with at least one of
(Item 13)
the command includes a broad command directed to the AV and one or more subsequent AVs behind the AV on the road;
A method according to any of the preceding items, wherein the global command includes one or more navigation instructions for avoiding certain unanticipated road conditions ahead of the AV.
(Item 14)
said command includes a special command directed to said AV;
A method according to any of the preceding items, wherein the special commands include one or more navigation instructions for avoiding certain unexpected road conditions ahead of the AV.
(Item 15)
A computer program comprising executable instructions stored on a non-transitory computer readable medium, the instructions being executed by one or more processors to cause the one or more processors to:
accessing environmental data associated with a road ahead of an autonomous vehicle (AV), the environmental data associated with a window of time during which the AV travels along the road; and
comparing the environmental data with map data including expected road conditions in front of the AV;
determining whether the environmental data includes unexpected road conditions not included in the map data based at least in part on comparing the environmental data to the map data;
in response to determining that the environmental data includes the unexpected road conditions not included in the map data;
determining location coordinates of the unexpected road condition;
and communicating commands to the AV to steer around the unexpected road condition.
(Item 16)
The instructions, when executed by the one or more processors, are responsive to determining to the one or more processors that the environmental data includes the unexpected road conditions not included in the map data. , further causing the command to be propagated to one or more subsequent AVs traveling along the road behind the AV.
(Item 17)
A computer program according to any of the preceding items, wherein the command comprises a setting command.
(Item 18)
A computer program according to any of the preceding items, wherein the setting command includes at least one of changing the orientation of the at least one vehicle sensor and changing the data sampling frequency of the at least one vehicle sensor.
(Item 19)
of the preceding items, wherein the AV comprises at least one vehicle sensor, the at least one vehicle sensor including at least one of a camera, a light detection and ranging (LiDAR) sensor, a motion sensor, and an infrared sensor. A computer program according to any of the preceding claims.
(Item 20)
A computer program according to any of the preceding items, wherein the AV is a tractor unit attached to a trailer.
(summary)
The system includes an autonomous vehicle (AV) and an operations server operably coupled to the AV. The operations server accesses environmental data associated with roads traveled by the AV. Environmental data are associated with the time frame during which the AV is traveling along the road. The operations server compares the environmental data with map data including expected road conditions ahead of the AV. The operations server determines whether the environmental data includes unexpected road conditions not included in the map data. In response to determining that the environmental data includes unexpected road conditions not included in the map data, the operations server determines location coordinates of the unexpected road conditions and avoids the unexpected road conditions. Communicate commands to the AV to maneuver.

For a more complete understanding of the present disclosure, reference is made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

FIG. 1 illustrates an embodiment of a system configured to communicate commands between one or more autonomous vehicles (AVs) and an operations server.

FIG. 2 illustrates an embodiment of a system configured to communicate commands from an AV to an operations server and/or subsequent AVs.

FIG. 3 shows an exemplary flowchart of a method for communicating commands between one or more AVs and an operations server.

FIG. 4 shows an exemplary flowchart of a method for communicating commands from an AV to an operations server and/or subsequent AVs.

FIG. 5 shows a first exemplary embodiment of a system configured to communicate commands between AVs.

FIG. 6 illustrates a second exemplary embodiment of a system configured to communicate commands between AVs.

FIG. 7 shows an exemplary flowchart of a method for communicating commands between AVs.

FIG. 8 illustrates a block diagram of an exemplary autonomous vehicle configured to perform autonomous driving operations.

FIG. 9 shows an exemplary system for providing autonomous driving operation used by the AV of FIG.

FIG. 10 shows a block diagram of an onboard control computer included in the AV of FIG.

As noted above, previous techniques fail to provide an efficient, reliable, and safe solution for determining and communicating commands to AVs to avoid unexpected road conditions. The present disclosure is directed to 1) communicating commands to a moving AV to avoid unpredictable road conditions, 2) improving data communication with the AV, and 3) improving the navigation of the AV. 4) to improve tracking of AVs; 5) to improve monitoring of autonomous operation of AVs; and 6) to provide a safe driving experience for AVs, other vehicles, and pedestrians. , provides various systems, methods and devices.

In one embodiment, a system 100 and method 300 for communicating commands between an operations server and one or more AVs are described herein with reference to FIGS. 1 and 3, respectively. In one embodiment, a system 200 and method 400 for receiving commands from an operations server and/or subsequent AVs by an AV are described herein with respect to FIGS. 2 and 4, respectively. Systems 500 and 600 and method 700 for communicating commands between AVs in one embodiment are described herein with reference to FIGS. 5-7, respectively. In one embodiment, an exemplary AV and its various systems and devices for implementing autonomous driving operations with the AV are described herein with reference to FIGS. 8-10.
An Exemplary System for Communicating Commands Between an Operations Server and One or More AVs

FIG. 1 illustrates an embodiment of system 100 for communicating commands 130 between operations server 120 and one or more AVs 802 . FIG. 1 further shows a simplified schematic diagram of a roadway 102 traveled by one or more AVs 802, e.g., AV 802a and trailing AV 802b, where one or more AVs 802 represent various exemplary anticipatory traffic. We are heading towards road condition 156. In one embodiment, system 100 includes an operations server 120 and one or more AVs 802 (see FIG. 8 for details of AVs 802). System 100 includes application server 160 , remote operator 162 , and network 110 . Network 110 provides communication paths between the components of system 100 . System 100 may be configured as shown or in any other suitable configuration.

In general, system 100 (at operations server 120 ) accesses environmental data 150 , including weather data 152 and traffic data 154 that provide information about road 102 conditions. Environmental data 150 may further include traffic safety regulation data, data related to obstacles 118 (eg, location coordinates, shape, size, and type of road closures 104, objects 106, construction areas 108, etc.), among others. For example, operations server 120 receives environmental data 150 from one or more external sources such as live weather forecasts, live traffic information, and the like. The operations server 120 (via the processor 122) compares the environmental data 150 with the map data 142 including the expected road conditions in front of the AV 802a. Anticipated road conditions may include anticipated weather conditions and traffic congestion ahead of AV 802a on roadway 102 . Expected road conditions relate to, among other things, expected traffic safety data associated with road 102, expected road elevation data (e.g., elevation information for various road segments), expected objects in front of AV 802a. (eg, location coordinates, shape, size and type of lane markings, road signs, traffic lights, crosswalks, etc.), and historical driving information for one or more AVs 802 on roadway 102 . Based on the comparison of environmental data 150 and map data 142 , operations server 120 determines whether environmental data 150 includes unexpected road conditions 156 not included in map data 142 . For example, unexpected road conditions 156 may include unexpected weather conditions, unexpected traffic jams, unexpected road closures 104, unexpected objects 106, unexpected construction areas 108, or map data 142. Other unexpected road conditions 156 may be included. In response to determining that environmental data 150 includes unexpected road conditions 156, operations server 120 determines location coordinates 158 of unexpected road conditions 156 and communicates commands 130 to AV 802a to determine unexpected road conditions 156. Maneuver around road conditions 156. The corresponding descriptions below include brief descriptions of specific components of system 100 .
system components

Network 110 includes the Internet, intranets, private networks, public networks, peer-to-peer networks, public switched telephone networks, cellular networks, local area networks (LAN), metropolitan area networks (MAN), wide area networks (WAN), and satellite networks. It may be any suitable type of wireless and/or wired network, including but not limited to all or part. Network 110 may be configured to support any suitable type of communication protocol, as would be understood by those skilled in the art.

Operations server 120 is generally configured to oversee the operation of AV 802 . Operations server 120 includes processor 122 , network interface 124 , user interface 126 , and memory 128 . The components of operations server 120 are operatively coupled to each other.

Processor 122 may include one or more processing units that perform various functions as described herein. Memory 128 stores data and/or instructions used by processor 122 to perform its functions. For example, memory 128 stores software instructions 138 that, when executed by processor 122, cause operations server 120 to perform one or more functions described herein.

Operations server 120 is in signal communication with AV 802 and its components. Operations server 120 is further configured to detect objects on and around roadway 102 by analyzing sensor data 148 . For example, operations server 120 may detect objects on and around roadway 102 by implementing object detection machine learning module 140 . Object detection machine learning module 140 may be implemented with neural networks and/or machine learning algorithms for detecting objects from images, videos, infrared images, point clouds, radar data, and the like. Object detection machine learning module 140 is described in further detail below.

In one embodiment, the AV 802 may include a semi-truck tractor unit attached to a trailer for transporting cargo or cargo from one location to another (see FIG. 8). AV 802 is navigated by a number of components detailed in Figures 8-10. The operation of AV 802 is described in further detail in FIG. The corresponding descriptions below include brief descriptions of specific components of AV 802 . Briefly, AV 802 includes an onboard control computer 850 that operates to facilitate autonomous operation of AV 802 . In this disclosure, onboard control computer 850 may be interchangeably referred to as controller 850 .

Controller 850 is generally configured to control the operation of AV 802 and its components. Controller 850 is further configured to determine a path ahead of AV 802 that is safe to travel and free of objects/obstacles, and to navigate AV 802 to travel that path. This process is described in more detail in Figures 8-10. Controller 850 generally includes one or more computing devices in signal communication with other components of AV 802 (see FIG. 8). Controller 850 determines a safe route to travel by receiving sensor data 148 from one or more sensors 846 located on AV 802 . Sensor data 148 includes data captured by sensors 846 . Sensor 846 is configured to capture any object within its detection zone or field of view, such as landmarks, lane markings, lane boundaries, road boundaries, vehicles, pedestrians, road/traffic signs, among others. ing. Sensors 846 may include cameras, LiDAR sensors, motion sensors, infrared sensors, and the like. In one embodiment, sensors 846 may be positioned around AV 802 to capture the environment surrounding AV 802 .

Controller 850 is in signal communication with operations server 120 . Controller 850 is configured to communicate sensor data 148 to operations server 120 , for example, via network 110 . Controller 850 periodically (eg, every minute, every few minutes, or any other suitable interval), continuously, and/or upon receiving a request to send sensor data 148 from operations server 120 . At the same time, sensor data 148 may be communicated to operations server 120 . Sensor data 148 may include data describing the environment surrounding AV 802 , such as image feeds, video feeds, LiDAR data feeds, and other data captured from the field of view of sensor 846 . Sensor data 148 may further include location coordinates of AV 802 . For further description of controller 850, please refer to the corresponding description of FIG.

Communications subsystem 210 is a component of controller 850 and is generally configured to establish communications between AV 802a and operations server 120 and/or subsequent AVs 802b. Communications subsystem 210 is further configured to receive commands 130 or other suitable communications from operations server 120 and/or subsequent AVs 802b. Upon receiving command 130, communication subsystem 210 (or controller 850) may implement command 130 or may update command 130 prior to implementation. For further description of communication subsystem 210, please refer to the corresponding description of FIG. The corresponding descriptions below describe various examples of commands 130 .
Example command

Commands 130 generally include one or more instructions that may be used to navigate and/or configure one or more AVs 802 . For example, the command 130 may be: 1) transition from autonomous to manual driving by the AV 802; , avoidance of one or more certain lanes (e.g., lanes occupied by obstacles 118); 4) one or more certain routing plans where unexpected road conditions 156 or obstacles 118 are detected by the AV 802; 5) constant rerouting by the AV 802 (e.g., exit 112); and 6) behavior by the AV 802 in a manner different from the driving instructions 146 (e.g., driving at speeds slower or faster than speed).

In some cases, command 130 may be a configuration command 130 related to aspects other than navigating AV 802 . For example, configuration commands 130 may relate to changing sensor parameters, including changing the orientation of sensor 846 (eg, tilting the orientation of sensor 846), changing the data sampling frequency of sensor 846, and the like.

Commands 130 may include operations server to AV commands 132 , AV to operations server commands 134 , and AV to AV commands 136 . An operations server-to-AV command 132 is a command 130 that the operations server 120 communicates with one or more AVs 802 . AV to operations server commands 134 are commands 130 that AV 802 communicates with operations server 120 . AV-to-AV commands 136 are commands 130 in which a first AV 802 (eg, AV 802a) communicates with a second AV 802 (eg, AV 802b) through operations server 120, either directly or indirectly.

In one embodiment, remote operator 162 may determine and issue commands 130 from operations server 120 . Remote operator 162 may be an individual who is associated with and has access to operations server 120 . For example, teleoperator 162 may be an administrator who can access and view information about AV 802 , such as data available on memory 128 . Remote operator 162 may access operations server 120 directly or indirectly. For example, remote operator 162 may directly access operations server 120 via communications path 164 . By reviewing operations server 120 and reviewing environmental data 150 , map data 142 , routing plans 144 , and sensor data 148 on user interface 126 of operations server 120 , teleoperator 162 can communicate with one or more AVs 802 . A command 130 may be issued to the In another example, remote operator 162 may access operations server 120 indirectly through network 110 from application server 160 acting as a presentation layer. Operations server 120 may forward or communicate commands 130 to one or more AVs 802 .

In another embodiment, the process of determining and issuing commands 130 may be computerized and performed by operations server 120 and/or controller 850 .

Commands 130 may have different levels of granularity. For example, different granularity levels of commands 130 may indicate whether commands 130 are general commands 130a or special commands 130b, as described below.

The commands 130, including the broad commands 130a and the specific commands 130b described herein, are exemplary and extend the scope of the invention as otherwise presented or described in the claims appended hereto. It is not meant to be limiting.
Extensive command examples

A broad command 130a may relate to a particular unexpected road condition 156 that generally applies to one or more AVs 802 . The global command 130a may be directed to one or more AVs 802 on a particular road 102 heading toward a particular unexpected road condition 156, for example. As shown in FIG. 1, AVs 802 are traveling along road 102, where AV 802a is ahead of AV 802b, and these AVs 802 may encounter different and unexpected road conditions 156. As described below, the operations server 120 may issue a special command 130 (ie, operations server to AV command 132) for each unexpected road condition 156. FIG.

For example, the operations server 120 may communicate a broad command 130a to one or more AVs 802 indicating that there is a road closure 104 on the road 102 and an alternate routing plan 144 (e.g., safest route) to reach the destination. It informs that it will find a routing plan 144) that provides a better driving experience. In another example, operations server 120 may communicate global command 130a to one or more AVs 802 indicating that severe weather has been detected (eg, based on weather data 152) on road 102 ahead of lead AV 802a; and to find the next exit (eg, exit 112) and pull over to the side of the road. In another example, the operations server 120 may communicate a pervasive command 130a to one or more AVs 802 that an unexpected object 106 (not detected in the map data 142) was detected on the roadway 102; and to navigate around the unexpected object 106. In another example, operations server 120 may communicate global command 130a to one or more AVs 802 indicating that an unexpected construction area 108 (not detected in map data 142) was detected on roadway 102. , and to find a side on the road 102 from which the vehicle can be pulled.
Special command example

Special commands 130b may relate to special, unexpected road conditions 156 that apply to a particular AV 802 . A special command 130 b may be directed to a particular AV 802 encountering a special unexpected road condition 156 . For example, the operations server 120 may communicate a special command 130b to the AV 802a indicating that there is a road closure 104 on the road 102, and take the next exit 112, make the first right turn after the traffic lights, drive 50 feet and pull over to the side of the road to inform them that further instructions are awaited. In another example, assume AV 802a pulls off road 102, but AV 802a drives two feet into the side lane and stops. The operations server 120 may communicate a special command 130b to the AV 802a telling it to turn the steering wheel to the right for 2 seconds, move at a speed of 2 miles per hour (mph), and stop. In another example, the special command 130b may provide turn-by-turn instructions to a particular AV 802a without taking over autonomous driving of the AV 802a.

Controller 850 and/or operations server 120 use global commands 130a and special commands 130b as a training data set to identify associations and relationships between each unexpected road condition 156 and its corresponding issued command 130. can learn. Thus, controller 850 and/or operations server 120 can be trained to issue commands 130 with minimal (or no) human intervention (ie, remote operator 162 intervention).
operation server

Aspects of embodiments of the operations server 120 have been described above, and additional aspects are provided below. Operations server 120 includes at least one processor 122 , at least one network interface 124 , at least one user interface 126 , and at least one memory 128 . Operations server 120 may be configured as shown or in any other suitable configuration.

In one embodiment, operations server 120 may be implemented by a cluster of computing devices that may help oversee the operation of AV 802 . For example, operations server 120 may be implemented by multiple computing devices using a distributed computing system and/or a cloud computing system. In another example, operations server 120 may be implemented by multiple computing devices within one or more data centers. Thus, in one embodiment, operations server 120 may contain more processing power than controller 850 . Operations server 120 is in signal communication with one or more AVs 802 and their components (eg, controller 850). In one embodiment, operations server 120 is configured to determine a particular routing plan 144 for AV 802 . For example, operations server 120 may determine a particular routing plan for AV 802 that results in shorter driving times and a safer driving experience to reach the destination of AV 802 .

In one embodiment, when unexpected road conditions 156 are detected by comparing environmental data 150 and map data 142, operations server 120 evaluates environmental data 150 and map data 142 to determine AV 802. A command 130 to navigate may be determined. For example, in response to detecting certain unexpected road conditions 156 , operations server 120 may determine commands 130 and communicate to AV 802 to avoid unexpected road conditions 156 . In one embodiment, remote operator 162 may review, modify, and/or override suggested commands 130 (or navigation plans) from operations server 120, as described below.

In one embodiment, a navigation solution or routing plan 144 for AVs 802 may be determined from V2V communications such as one AV 802 and another AV 802 . In one embodiment, a navigation solution or routing plan 144 for AVs 802 may be determined from vehicle-to-cloud (V2C) communications, such as between operations server 120 and AVs 802 .

In one embodiment, commands 130, navigation solutions, and/or routing plans 144 for AVs 802 are vehicle-to-cloud-to-human (V2C2H), vehicle-to-human (V2H), vehicle-to-cloud-to-vehicle (V2C2V), and/or vehicle-to-vehicle It can be determined by human-to-vehicle (V2H2V) communication, where human intervention is incorporated in determining the navigation solution for the AV802. For example, remote operator 162 may review environmental data 150 and map data 142 from user interface 126 and confirm, modify, and/or override commands 130 to AV 802 determined by operations server 120 and/or controller 850. can. Teleoperator 162 may add a human perspective to AV 802's navigation plan determination that control device 850 and/or operations server 120 do not provide. In some cases, the human point of view is preferable to the machine point of view for safety, fuel saving, etc. reasons.

In one embodiment, commands 130 to AV 802 may be determined by any combination of V2V, V2C, V2C2H, V2H, V2C2V, V2H2V communication, among other types of communication.

As shown in FIG. 1, remote operator 162 can access application server 160 via communications path 166 , and likewise can access operations server 120 via communications path 164 . In one embodiment, operations server 120 transmits sensor data 148, commands 130, environmental data 150, and/or any other data/instructions wirelessly via network 110 and/or via wired communication, for example. , can be sent to application server 160 and reviewed by remote operator 162 . Thus, in one embodiment, remote operator 162 can remotely access operations server 120 through application server 160 .

Processor 122 comprises one or more processors operatively coupled to memory 128 . Processor 122 may be a state machine, one or more central processing unit (CPU) chips, logic units, cores (e.g., multicore processors), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or digital signal processors. Any electronic circuit, including but not limited to (DSP). Processor 122 may be a programmable logic device, microcontroller, microprocessor, or any suitable combination of the foregoing. Processor 122 is communicatively coupled to and in signal communication with network interface 124 , user interface 126 , and memory 128 . One or more processors are configured to process data and may be implemented in hardware or software. For example, processor 122 may be of 8-bit, 16-bit, 32-bit, 64-bit, or any other suitable architecture. Processor 122 includes an arithmetic logic unit (ALU) for performing arithmetic and logical operations, processor registers for supplying operands to the ALU and storing results of ALU operations, and fetching instructions from memory, the ALU, registers, and a control unit that executes them by directing the coordinated operation of other components. One or more processors are configured to implement various instructions. For example, one or more processors are configured to execute software instructions 138 to perform the functions disclosed herein, such as some or all of those described with respect to FIGS. 1-7. In some embodiments the functions described herein are implemented using logic units, FPGAs, ASICs, DSPs, or any other suitable hardware or electronic circuitry.

Network interface 124 is configured to allow wired and/or wireless communication. Network interface 124 is configured to communicate data between operations server 120 and other network devices, systems, or domains. For example, network interface 124 may comprise a WIFI interface, a local area network (LAN) interface, a wide area network (WAN) interface, modem, switch, or router. Processor 122 is configured to send and receive data using network interface 124 . Network interface 124 may be configured to use any suitable type of communication protocol.

User interface 126 may include one or more user interfaces configured to interact with a user, such as remote operator 162 . Remote operator 162 may access operations server 120 via communications path 164 . User interface 126 may include peripherals of operations server 120 such as a monitor, keyboard, mouse, trackpad, touchpad, and the like. Remote operator 162 may access memory 128 using user interface 126 to review sensor data 148 , environmental data 150 , unexpected road conditions 156 , and other data stored in memory 128 .

Memory 128 may be any of the data, instructions, logic, rules, or code described in FIGS. 1-7, along with other data, instructions, logic, rules, or code operable to perform the functions described herein when executed by processor 122. store any of the information For example, memory 128 stores commands 130, software instructions 138, object detection machine learning module 140, map data 142, routing plans 144, driving instructions 146, sensor data 148, environmental data 150, unexpected road conditions 156, unexpected Location coordinates 158 for road conditions, obstacles 118, and/or other data/indications may be stored. Software instructions 138 include code that, when executed by processor 122, causes operations server 120 to perform the functions described herein, such as some or all of those described in FIGS. Let Memory 128, which comprises one or more disks, tape drives, or solid state drives, serves as an overflow data storage device for storing programs when such programs are selected for execution and for executing programs. It can be used to store instructions and data read in. Memory 128 can be volatile or non-volatile and includes read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), dynamic random access memory (DRAM), and static random access memory ( SRAM). Memory 128 may include one or more of local databases, cloud databases, network attached storage (NAS), and the like.

Object detection machine learning module 140 may be implemented by processor 122 executing software instructions 138 and generally detects objects or obstacles 118, including road closures 104, unexpected objects 106, and construction areas 108, from sensor data 148. configured to detect Object detection machine learning module 140 may be implemented with neural networks and/or machine learning algorithms for detecting objects from any data type, such as images, videos, infrared images, point clouds, radar data, and the like.

In one embodiment, object detection machine learning module 140 may be implemented using machine learning algorithms such as Support Vector Machines (SVM), Naive Bayes, Logistic Regression, k-Nearest Neighbors, Decision Trees, and the like. In one embodiment, the object detection machine learning module 140 may utilize multiple neural network layers, convolutional neural network layers, and/or other, where the weights and biases of these layers are used by the object detection machine learning module. Optimized with 140 training processes. Object detection machine learning module 140 may be trained with a training data set containing samples of the data type labeled with one or more objects in each sample. For example, the training data set may include sample images of objects (eg, vehicles, lane markings, pedestrians, road signs, obstacles 118, etc.) labeled with the objects in each sample image. Similarly, the training data set may contain samples of other data types such as video, infrared imagery, point clouds, radar data, etc. labeled with the object of each sample data. Object detection machine learning module 140 may be trained, tested, and improved with training datasets and sensor data 148 . The object detection machine learning module 140 uses sensor data 148 (not labeled with objects) to improve their predictive accuracy in detecting objects. For example, supervised and/or unsupervised machine learning algorithms may be used to validate predictions of object detection machine learning module 140 in detecting objects in sensor data 148 .

Map data 142 may include a virtual map of a city or region containing roads 102 . In some examples, map data 142 may include map 958 and map database 936 (see FIG. 9 for description of map 958 and map database 936). Map data 142 includes drivable areas such as roads 102, trails, highways, and non-drivable areas such as terrain (as determined by occupancy grid module 960, see FIG. 9 for description of occupancy grid module 960). can include Map data 142 may specify location coordinates for road signs, lanes, lane markings, lane boundaries, road boundaries, traffic lights, obstacles 118, and the like.

A routing plan 144 is a plan for traveling from a starting location (eg, a first AV launch pad/landing pad) to a destination (eg, a second AV launch pad/landing pad). For example, routing plan 144 may specify a combination of one or more streets, roads, and highways in a particular order from origin to destination. The routing plan 144 includes a first stage (e.g., starting from a starting location), multiple intermediate stages (e.g., traveling along a particular lane of one or more particular streets/roads/highways), and a final stage. Steps may be specified, including (eg, entering a destination). The routing plan 144 may include other information regarding the route from the starting location to the destination, such as road/traffic signs within the routing plan 144 .

Driving directions 146 may be implemented by planning module 962 (see description of planning module 962 in FIG. 9). Driving instructions 146 may include instructions and rules for adapting autonomous driving of AV 802 according to the driving rules of each stage of routing plan 144 . For example, driving instructions 146 may indicate changes in AV 802 observed by sensors 846, such as instructions to stay within the speed range of road 102 traveled by AV 802, the speed of surrounding vehicles, objects within the detection zone of sensors 846, and so on. instructions for adapting speed, and so on.

Environmental data 150 may include weather data 152 and traffic data 154 that provide information about roadway 102 conditions. Weather data 152 may include weather information for zones or regions ahead of AV 802 . For example, weather data 152 may be accessed from weather servers, live weather reports, and the like. Environmental data 150 may further include, among other things, road safety regulation data and data related to obstacles 118 (eg, location coordinates, shape, size, and type of road closures 104, objects 106, construction areas 108, etc.).

Traffic data 154 may include road/street/highway traffic data in map data 142 . Operations server 120 may use traffic data 154 collected by one or more mapping vehicles. The operations server 120 may use traffic data 154 captured from any source, such as from external sources such as Waze and Google Maps, live traffic reports, and the like. and crowdsourced traffic data 154 .
application server

Application server 160 is generally any computing device configured to communicate over network 110 with other servers (eg, operations server 120), AVs 802, databases, and other devices. Application server 160 is configured to perform the functions described herein and interact with teleoperator 162, for example, via communication path 166 using its user interface. Examples of application servers 160 include, but are not limited to, desktop computers, laptop computers, servers, and the like. In one example, application server 160 may serve as a presentation layer for remote operator 162 to access operations server 120 . Accordingly, operations server 120 may transmit sensor data 148 , commands 130 , environmental data 150 , and/or any other data/instructions to application server 160 via network 110 , for example. After establishing a communication path 166 with the application server 160, the remote operator 162 reviews the received data and confirms, modifies, and executes the suggested commands 130 determined by the operations server 120, as described below. /or can be overwritten.
operational flow
Command communication from operation server to AV

The operational flow for communicating commands 130 from operations server 120 to AV 802 may begin when unexpected road conditions 156 are detected in environmental data 150 . For example, operations server 120 (and/or teleoperator 162) reviews environmental data 150 and compares environmental data 150 to map data 142, including expected road conditions ahead of AV 802 (eg, AV 802a). By doing so, it may detect that environmental data 150 includes unexpected road conditions 156 . Environmental data 150 may include weather data 152 and traffic data 154 within the time window during which AV 802 (eg, AV 802 a ) is traveling along roadway 102 .

Based on comparing environmental data 150 to map data 142 , operations server 120 (and/or remote operator 162 ) determines that environmental data 150 includes unexpected road conditions 156 not included in map data 142 . to decide whether Examples of unexpected road conditions 156 are described above.

In response to determining that environmental data 150 includes unexpected road conditions 156, operations server 120 (and/or teleoperator 162) determines location coordinates 158 of unexpected road conditions 156 and command 130 (ie, operations server to AV commands 132) to the AV 802a may maneuver to avoid unexpected road conditions 156.
Communication of AV to Operation Server Commands

In one embodiment, in response to receiving commands 130 from operations server 120 , controller 850 (eg, via communications subsystem 210 ) may navigate AV 802 a according to commands 130 .

In alternate embodiments, controller 850 may update command 130 (eg, via communications subsystem 210). For example, in some cases, operations server 120 (and/or remote operator 162) may not be able to access (or have not yet received) sensor data 148, and the comparison of environmental data 150 and map data 142 may result in command 130 may be determined based on the Therefore, in some cases, command 130 may not be the optimal navigation solution for AV 802a. In such cases, controller 850 may update commands 130 based on analysis of sensor data 148 .

To this end, controller 850 receives sensor data 148 from sensors 846 . By processing sensor data 148 , controller 850 may determine position coordinates of multiple objects on and around roadway 102 . Controller 850 may update command 130 so that updated command 130-1 is 1 for avoiding objects on and around road 102 while performing command 130. Contains one or more instructions.

For example, assume road 102 is closed by road closure 104 . Also assume that road closure 104 is outside the field of view or detection zone of sensor 846 of AV 802a. Operations server 120 may access environment data 150 indicating that road 102 is closed due to road closure 104 . From environmental data 150 , operations server 120 (and/or teleoperator 162 ) determines the location coordinates of road closure 104 .

Operations server 120 (and/or remote operator 162) may issue command 130 to AV 802a to indicate that road 102 is closed ahead and to pull over to the side of road 102 at location 114. Controller 850 receives commands 130 from operations server 120 (eg, via communications subsystem 210). Controller 850 also receives sensor data 148 from sensors 846 (eg, via communications subsystem 210). By processing sensor data 148, controller 850 may determine (eg, via communication subsystem 210) that a particular location 114 is occupied by an obstacle. In this case, controller 850 may avoid obstacles on particular locations 114 by updating commands 130 (eg, via communications subsystem 210). For example, by processing sensor data 148 , controller 850 determines that position coordinates 116 are free of obstructions, and by updating command 130 , position 116 instead of position 114 as indicated by command 130 . can bring the AV 802a to.

In one embodiment, controller 850 may perform updated command 130-1. For example, controller 850 may navigate AV 802a according to updated command 130-1.

In an alternative embodiment, controller 850 communicates sensor data 148 and updated command 130-1 (i.e., AV to operations server command 134) to operations server 120, and provides a command for performing updated command 130. I can wait for confirmation. Upon receiving sensor data 148 and updated commands 130-1, operations server 120 (and/or remote operator 162) analyzes the received data and confirms, modifies, or overwrites updated commands 130-1. can. Operations server 130 may then communicate the determination of operations server 120 (and/or remote operator 162) to AV 802a.
Inciting Communication from AV to Operations Server Commands

In some cases, controller 850 may communicate to operations server 120 (eg, via communication subsystem 210) a suggested command 130 (i.e., AV to operations server command 134) for navigating AV 802a. can incite For example, assume sensor 846 detects obstacle 118 on roadway 102 . For example, obstacles 118 can be any of road closures 104 , objects 106 , and construction areas 108 . Accordingly, sensor data 148 may include an indication of the presence of obstacles 118 on roadway 102 .

Controller 850 analyzes sensor data 148 to determine the presence of obstacles 118 on roadway 102 . Controller 850 compares sensor data 148 with map data 142 , which includes position coordinates of expected objects on and around roadway 102 . Based on comparing sensor data 148 and map data 142 , controller 850 determines that obstacle 118 is not included in map data 142 . In response, controller 850 may determine suggested commands 130 or suggested navigation instructions for AV 802 a to avoid obstacle 118 . For example, the suggested commands 130 may include pulling the vehicle off the road, diverting to another lane not occupied by the obstacle 118, reselecting the route, and the like.

In one embodiment, controller 850 may carry out suggested command 130 .

In an alternative embodiment, controller 850 sends suggested command 130 (i.e., AV to operations server command 134) for confirmation or modification by operations server 120 (and/or remote operator 162), as described above. (see FIG. 1)) may be communicated to the operations server 120 .

Controller 850 may determine whether confirmation or correction is received from operations server 120 . In response to receiving confirmation, controller 850 performs proposed command 130 . In response to receiving the modification, controller 850 performs modified command 130 .

In one embodiment, controller 850 communicates sensor data 148 indicating the presence of observed obstacles 118 and provides remote operator 162 (and/or operations server 120) with commands to avoid obstacles 118. 130 may be requested to be provided.

Thus, in the various embodiments described above, controller 850 negotiates with operations server 120 (and/or teleoperator 162) in response to detecting different obstacles 118 and/or unexpected road conditions 156. can determine a more optimal command 130 for navigating the AV 802a.
Communicating Commands from an AV to Another AV

In one embodiment, communicating commands 130 from an AV 802 (eg, AV 802 a ) to another AV 802 (eg, AV 802 b ) may be initiated from operations server 120 . For example, as described above, in response to detecting unexpected road conditions 156 ahead of AV 802a, operations server 120 (and/or teleoperator 162) communicates command 130 to AV 802a to may steer to avoid rough road conditions 156. Similar to the above, controller 850 may perform or update commands 130 . This command 130 (or updated command 130-1) may be propagated to one or more subsequent AVs 802, such as AV 802b. For example, operations server 120 communicates command 130 (or updated command 130-1) to AV 802b to indicate unexpected road condition 156, location coordinates 158 of unexpected road closure 104, and unexpected road condition 156 may communicate one or more navigation instructions to the AV 802b to avoid

In one embodiment, communicating the command 130 from the first AV 802a to the second AV 802b may be initiated from the first AV 802a. For example, in response to detecting unexpected road conditions 156 and/or obstacles 118, controller 850 may navigate first AV 802a (eg, via communication subsystem 210) in a manner similar to that described above. A suggested command 130 for gating may be determined. Controller 850 may communicate command 130 directly to second AV 802b (eg, via communication subsystem 210), for example, if second AV 802b is within V2V communication range of first AV 802a. This process is described in more detail in Figure 2, and various scenarios for communicating commands 130 between AVs 802 are described in Figures 5-7.
An exemplary communication subsystem and its operation

FIG. 2 illustrates an embodiment of system 200 configured to establish communication between AV 802 and operations server 120 . In one embodiment, system 200 includes a communication subsystem 210 (see FIG. 8), which is a component of controller 850, AV 802a, and operations server 120. FIG. In some embodiments, system 200 may further include AV 802b and network 110 . Network 110 establishes communications between components of system 200 . System 200 may be configured as shown or in any other suitable configuration. Aspects of embodiments of specific components of system 200 are described above in FIG. 1, and additional aspects are provided below.

In one embodiment, network 110 may be implemented using any suitable cloud computing service and network infrastructure, including Amazon Web Services (AWS). Network 110 may transfer data between devices using any suitable data communication protocol, including Message Queuing Telemetry Transport (MQTT). Network 110 1) provides a secure connection between operations server 120 and AVs 802, 2) stores data received from AVs 802 (eg, AV 802a), such as commands 130, sensor data 148, and 3). 4) providing access to data communicated between AV 802a and operations server 120; and 4) transferring data between AV 802 and operations server 120.

Controller 850 includes a number of components including communication subsystem 210 operably coupled to computing device 260 . Other components of controller 850 are described in FIG. The corresponding description below describes the components of communication subsystem 210 .

A communication subsystem 210 is provided to facilitate data communication between AV 802 (eg, AV 802 a ) and other components of system 200 . In some embodiments, communications subsystem 210 may include a controller area network (CAN) controller for establishing communications between components within communications subsystem 210 . Communications subsystem 210 may include various components that support the operation of communications subsystem 210 . Communication subsystem 210 includes at least one processor 212 in signal communication with at least one memory 242 . Memory 242 stores software instructions 244 that, when executed by processor 212, cause processor 212 to perform one or more functions described herein. For example, when software instructions 244 are executed, processor 212 receives commands 130 from operations server 120 and/or subsequent AV 802b and executes commands 130 . In another example, when software instructions 244 are executed, processor 212 updates commands 130 and sends updated commands 130 to operations servers 120 and/or in response to receipt of commands 130 from subsequent AVs 802b. 120 and/or subsequent AVs 802b, and upon receipt of the confirmation (or correction), implement the command 130. Communication subsystem 210 may further include at least one sensor unit 224 , radio frequency (RF) gateway 228 , long term evolution (LTE) modem 232 , and V2V modem 234 .

Processor 212 includes one or more processors such as computing unit 214 and vehicle control unit 222 . Processor 212 is operatively coupled to sensor unit 224 , RF gateway 228 , LTE modem 230 , V2V modem 234 and memory 242 . Processor 212 may be a state machine, one or more central processing unit (CPU) chips, logic units, cores (such as multi-core processors), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or digital signal processors ( any electronic circuit including but not limited to DSP). Processor 212 may be a programmable logic device, microcontroller, microprocessor, or any suitable combination of the foregoing. Processor 212 is configured to process data and may be implemented in hardware and/or software. For example, processor 212 may be of 8-bit, 16-bit, 32-bit, 64-bit, or any other suitable architecture. Processor 212 includes an arithmetic logic unit (ALU) for performing arithmetic and logical operations, processor registers for supplying operands to the ALU and storing results of ALU operations, and fetching instructions from memory, the ALU, registers, and other components by directing the coordinated operation thereof. One or more processors are configured to implement various instructions. For example, one or more processors are configured to execute software instructions 244 to perform the functions disclosed herein, such as some or all of those described with respect to FIGS. 1-10. In some embodiments the functions described herein are implemented using logic units, FPGAs, ASICs, DSPs, or any other suitable hardware or electronic circuitry.

Each computing unit 214 may be implemented by a processor 212 executing software instructions 244, generally to support autonomous driving functions and to facilitate data communication between AV 802a and other components of system 200. is configured to

In one embodiment, computing units 214 are redundant backups of each other. Processor 212 may include any suitable number of computing units 214, such as one, two, three, and so on. Each computing unit 214 1) receives commands 130 from operations server 120 and/or subsequent AVs 802b, including global commands 130a, special commands 130b, and configuration commands 130; 3) communicating the location coordinates of the AV 802a to the operations server 120 and/or the successor AV 802b; 4) communicating the health information of the controller 850 components (e.g., hardware and software components); collecting, where the health information of the components of the control device 850, such as memory utilization, processor utilization, average response time per second, average error rate per minute, etc. and 5) communicating component health information of the controller 850 to the operations server 120 and/or subsequent AVs 802b. may perform one or more functions of controller 850).

Each computing unit 214 may include an LTE agent 216 , a V2V agent 218 and a Human Machine Interface (HMI) agent 220 .

LTE agent 216 may be implemented in software and/or hardware modules and is generally configured to establish communication between AV 802a and operations server 120 over network 110 .

V2V agent 218 may be implemented in software and/or hardware modules and is generally configured to receive commands 130 from AV 802b and use V2V modem 234 to transmit commands 130 to subsequent AVs 802b.

HMI agent 220 may be a software and/or hardware module and is generally configured to establish communication with subsequent AVs 802b to provide visitor mode HMI services so that operators can , the controller 850 of the AV 802a can be accessed from the computing device 260 of the AV 802a.

Each sensor unit 224 may be implemented in software and/or hardware modules. For example, each sensor unit 224 may be implemented on one or more microprocessors. Each sensor unit 224 generally: 1) collects sensor data 148 from one or more sensors 846; 2) generates a timestamp when each sensor data 148 is received; Detecting instructions 246 are used to perform one or more functions, including processing sensor data 148 to detect objects, and 4) transmitting processed sensor data 148 to processor 212 . is configured to

In one embodiment, sensor unit 224 may process sensor data 148 to determine and identify objects indicated in sensor data 148 . For example, the sensor unit may use obstacle detection instructions 246 to determine or identify objects indicated in sensor data 148 . Sensor unit 224 uses obstacle detection instructions 246 to determine if an object is detected in sensor data 148 (e.g., image/video feeds, LiDAR data feeds, motion sensor data feeds, infrared data feeds, etc.). obtain. For example, obstacle detection instructions 246 may include code for implementing an object detection method from a feed of images corresponding to frames of video (eg, object detection from images or videos). Similarly, the obstacle detection instructions 246 include LiDAR data, motion sensor data (e.g., detecting motion of an object), sound (e.g., detecting sound near the AV802a), and infrared data (e.g., can include code for detecting objects from the Obstacle detection instructions 246 may include code for detecting objects using other data types as well. In one embodiment, sensor units 224 are redundant backups of each other. Communication subsystem 210 may include any suitable number of sensor units 224, such as one, two, three, and so on.

Vehicle control unit 222, which may be implemented in software and/or hardware modules, generally implements commands 130 received from either operations server 120 or another AV 802 (e.g., AV 802b) and transmits commands 130 to processor 212. Configured to send the status of the process it implements.

LTE modem 232 may be implemented in software and/or hardware modules and is generally configured to provide Internet connectivity to communication subsystem 210 via network 110, for example. LTE modem 232 is configured to enable wired and/or wireless communication. LTE modem 232 may be referred to as a first network interface. LTE modem 232 is further configured to communicate data between communication subsystem 210 and other devices, systems (eg, AV 802b), servers (eg, operations server 120), or domains. For example, LTE modem 232 may comprise a WIFI interface, local area network (LAN) interface, wide area network (WAN) interface, modem, switch, or router.

Processor 212 is configured to send data to and receive data from other devices such as operations server 120 using LTE modem 232 . LTE modem 232 may be configured to use any suitable type of communication protocol. LTE modem 232 may provide data security to communication subsystem 210 by providing firewall functionality. For example, LTE modem 232 may block or filter data communications from unauthorized or unknown sources and allow data communications from operations server 120 and other AVs 802 . Therefore, outsiders may not be able to access the AV 802a and its components.

V2V modem 234 may be implemented in software and/or hardware modules and is generally configured to establish communication channels between AVs 802 . For example, AV 802a may use V2V modem 234 to transmit command 130 (or any other data/instructions) to subsequent AV 802b. Processor 212 is configured to send and receive data using V2V modem 234 to other devices, such as other AVs 802, such as AV 802b. V2V modem 234 may be referred to as a second network interface.

RF gateway 228 may be implemented in software and/or hardware modules and is generally configured to bridge communications between vehicle control unit 222 and V2V modem 234 . For example, vehicle control unit 222 may support a CAN connection (eg, using a CAN connector) and V2V modem 234 may support an Ethernet connection (eg, using an Ethernet port). can. Thus, RF gateway 228 may be used to transfer data between the CAN connector of vehicle control unit 222 and the Ethernet port of V2V modem 234 .

RF gateway 228 may include V2V agent 230 . V2V agent 230 may be implemented in software and/or hardware modules and is generally configured to receive commands 130 (or any other data/instructions) from subsequent AV 802b via V2V modem 234 .

Memory 242 stores any of the information described with respect to FIGS. 1-10 and other data, instructions, logic, and other data operable to implement the functions described herein when executed by processor 212. Store with rules or code. For example, memory 242 may store software instructions 244, obstacle detection instructions 246, sensor data 148, commands 130, and/or any other data/instructions. Software instructions 244 include code that, when executed by processor 212, causes communications subsystem 210 to perform functions described herein, such as some or all of those described in FIGS. carry out Memory 242, which includes one or more disks, tape drives, or solid-state drives, serves as overflow data storage for storing such programs when such programs are selected for execution and during program execution. can be used to store instructions and data that are read into the Memory 242 can be volatile or non-volatile and includes read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), dynamic random access memory (DRAM), and static random access memory ( SRAM). Memory 242 may include one or more of local databases, cloud databases, network attached storage (NAS), and the like.

Computing device 260 is generally any device configured to process data and interact with users. Examples of computing device 260 include, but are not limited to, personal computers, desktop computers, workstations, servers, laptops, tablet computers, and the like. Computing device 260 is configured to run a software application configured to provide a user with a monitoring interface for monitoring sensor data 148, commands 130, and/or any other data/instructions. . In one embodiment, computing device 260 may be configured to not allow control of AV 802 to mitigate unauthorized access and control of AV 802 via computing device 260 .
Communication between components of the communication subsystem

In some embodiments, processor 212 and its components may be configured to connect to other components of communication subsystem 210 using wired and/or wireless communications. For example, computing unit 214 may be configured to establish communications with other components of communications subsystem 210 using wired and/or wireless communications.

Each computing unit 210 establishes a separate communication channel over the network 110 with the operations server 120 and/or subsequent AVs 802a. Communications subsystem 210 periodically (eg, every 0.02 seconds, 0.5 seconds, or any other suitable duration) sends messages or acknowledgments between computing units 214 and operations server 120 . can detect loss of communication.

Each computing unit 214 connects to network 110 via an LTE modem 232 . Each computing unit 214 may connect to an LTE modem 232 using an Ethernet switch. Each computing unit 214 may be connected to vehicle control unit 222 using a CAN connector. Each computing unit 214 may be connected to V2V modem 234 using an Ethernet connection.

At AV 802a, each computing unit 214 and RF gateway 228 may establish a separate communication channel with computing device 260 on AV 802b. Communication subsystem 210 periodically (e.g., every 0.02 seconds, 0.5 seconds, or any other suitable duration) between RF gateway 228 and computing device 260 on AV 802b. By communicating messages, loss of communication can be detected.
Communication between AV and operation server
TRANSITION TO CONNECTED TO OPERATIONS SERVER AND SUBSEQUENT AV

When the engine 842a of the AV 802a is ignited or powered up (see FIG. 8), the components of the communication subsystem 210 are initialized. Components of processor 212, including computing unit 214 and vehicle control unit 222, establish communication channels to operations server 120 and/or subsequent AVs 802b. In this way, the communication subsystem 210 becomes "connected to the operations server and subsequent AVs". In this state, operations server 120 (or remote operator 162 (see FIG. 1)) can monitor the operation of AV 802a and issue commands 130 in a manner similar to that described above in FIG.
TRANSITION TO 'CONNECTED TO SUCCESSOR AV' STATE

The communication subsystem 210 transitions to the "connected to subsequent AV" state when the communication channel 270 with the operations server 120 is lost. Loss of communication channel 270 with operations server 120 is based on periodic acknowledgment messages (or lack of periodic acknowledgment messages) between RF gateway 228 and computing device 260 at subsequent AV 802b, as described above. detected by

In this way, AV 802a can be tracked, can send commands 130 (and receive commands 130 from it), and communication between AV 802a and operations server 120 can be lost. Even if it is, its autonomous operation can be monitored. For example, if a periodic acknowledgment message is not received from operations server 120 at a corresponding periodic interval, communication subsystem 210 determines that communication channel 270 with operations server 120 has been lost. In response, communication subsystem 210 may transition to a “connected to successor AV” state, where AV 802 a is connected to successor AV 802 b via second communication channel 280 .
Transition to "connected to operations server" state

The communication subsystem 210 transitions to the "connected to operations server" state when the communication channel 280 with the successor AV 802b is lost. Loss of communication channel 280 with subsequent AV 802b is detected based on periodic acknowledgment messages (or lack of periodic acknowledgment messages) between computing unit 214 and operations server 120 . In this state, AV 802a can maintain a connection with operations server 120 as long as AV 802a can connect to network 110 using LTE modem 232, for example.

In this way, AV 802a can be tracked, commands 130 can be sent (and commands 130 received therefrom), and communication between AV 802a and subsequent AVs 802b can be lost. Even if it is, its autonomous operation can be monitored. For example, if periodic acknowledgment messages are not received from the successor AV 802b at corresponding periodic intervals, the communication subsystem 210 determines that the communication channel 280 with the successor AV 802b has been lost. In response, communications subsystem 210 may transition to a “connected to operations server” state, where AV 802 a is connected to operations server 120 via communications channel 270 .
Transition to "execute command without connection to operations server and subsequent AV" state

The communication subsystem 210 transitions to the "execute command without connection to the operations server and successor AV" state when communication with the operations server 120 and successor AV 802b is lost, i.e. communication channels 270 and 280 are lost. do. In this state, processor 212 may execute command 130 to pull AV 802a to the side of the road until communication subsystem 210 establishes communication with operations server 120 and either or both of trailing AV 802b. Therefore, even if communication with both the operations server 120 and the successor AV 802b is lost, the communication subsystem 210 will remain on the side of the road until communication is re-established with either or both of the operations server 120 and the successor AV 802b. A submit command 130 can be performed.
An Exemplary Method of Communicating Commands to an AV

FIG. 3 shows an exemplary flowchart of a method 300 for communicating commands 130 to AV 802. As shown in FIG. Modifications, additions, or omissions may be made to method 300 . Method 300 may include more, fewer, or other steps. For example, steps may be performed in parallel or in any suitable order. Although AV 802, operations server 120, controller 850, communication subsystem 210, or components of any of them may be discussed as performing the steps, any suitable system or component thereof may perform one or more steps of method 300 . For example, one or more steps of method 300 may be performed, at least in part, on a non-transitory, tangible, machine-readable medium (eg, memories 128, 242, 890 and 890 from FIGS. 1, 2, 8 and 10, respectively). 1002), which may be implemented in the form of software instructions 138, 244 and 880, respectively, from FIGS. , 2, 8 and 10) may cause the one or more processors to perform steps 302-310.

Method 300 begins at step 302 where operations server 120 accesses environmental data 150 associated with road 102 in front of AV 802a, where environmental data 150 is accessed while AV 802a is traveling along road 102. includes weather data 152 and traffic data 154 associated with the time window of . For example, operations server 120 may receive environmental data 150 from one or more external sources such as live weather bulletin news, live traffic bulletin news, and the like.

At step 304 , operations server 120 compares environmental data 150 with map data 142 including expected road conditions ahead of AV 802 . For example, expected road conditions may include expected weather and traffic congestion ahead of AV 802a on roadway 102 .

At step 306 , operations server 120 determines whether environmental data 150 includes unexpected road conditions 156 not included in map data 142 . Unexpected road conditions 156 may include unexpected weather conditions, unexpected traffic jams, unexpected road closures 104, unexpected objects 106, and unexpected construction areas 108, among others. For example, based on comparing environmental data 159 to map data 142 , operations server 120 may determine whether environmental data 150 includes unexpected road conditions 156 . Remote operator 162 may confirm, modify, or override operations server 120 decisions, similar to those described above in FIG. If it is determined that environmental data 150 includes unexpected road conditions 156 not included in map data 142, method 300 proceeds to step 308; Otherwise, method 300 ends.

At step 308 , operations server 120 determines location coordinates 158 for unexpected road conditions 156 . For example, operations server 120 may determine location coordinates 158 of unexpected road conditions 156 from environmental data 150, similar to those described above in FIG.

In step 310, operations server 120 maneuvers to avoid unexpected road conditions 156 by communicating commands 130 (or operations server to AV commands 132) to AV 802a. Commands 130 may be general commands 130a or special commands 130b. Examples of global commands 130a and specific commands 130b are described in FIG.
An Exemplary Method for Navigating an AV Based on Received Commands

FIG. 4 shows an exemplary flow chart of a method 400 for navigating AV 802 based on received commands 130 . Modifications, additions, or omissions may be made to method 400 . Method 400 may include more, fewer, or other steps. For example, steps may be performed in parallel or in any suitable order. Although AV 802, operations server 120, controller 850, communication subsystem 210, or components of any of them may be discussed as performing the steps, any suitable system or component thereof may perform one or more steps of method 400 . For example, one or more steps of method 400 may be performed, at least in part, on a non-transitory, tangible, machine-readable medium (eg, memories 128, 242, 890 and 890 from FIGS. 1, 2, 8 and 10, respectively). 1002), which may be implemented in the form of software instructions 138, 244 and 880, respectively, from FIGS. 2, 8 and 10) may cause the one or more processors to perform steps 402-412.

Method 400 begins at step 402, where controller 850 associated with AV 802 (e.g., AV 802a in FIG. 1) receives from operations server 120 (e.g., via communications subsystem 210) the information described in FIG. Commands 130 are received for navigating the AV 802 to avoid unexpected road conditions 156, including unexpected road conditions 156 that have been detected.

At step 404 , controller 850 receives sensor data 148 including position coordinates of a plurality of objects in front of AV 802 from vehicle sensors 846 (eg, via communication subsystem 210 ). Sensor data 148 may further include indications that describe the identity of multiple objects. For example, sensor data 148 may indicate the presence of vehicles, pedestrians, obstacles 118, lane markings, road boundaries, road signs, etc. on and around road 102 in FIG.

At step 406 , controller 850 determines whether any object from plurality of objects indicated in sensor data 148 (eg, via communication subsystem 210 ) prevents fulfillment of command 130 . For example, controller 850 may determine (eg, via communication subsystem 210) that any object from the plurality of objects occupies the travel path of AV 802a, thereby navigating AV 802a in accordance with command 130. It can decide whether to block the gate. If any object from the plurality of objects is determined to interfere with the performance of command 130 , method 400 proceeds to step 410 . Otherwise, method 400 proceeds to step 408 .

At step 408 , controller 850 navigates AV 802 a according to commands 130 . For example, assuming command 130 is to pull over AV 802a at position coordinate 114, controller 850 pulls AV 802a over the road at position coordinate 114, similar to that described above in FIG.

At step 410 , controller 850 updates command 130 (eg, via communications subsystem 210 ) so that updated command 130 - 1 includes multiple objects while performing command 130 . contains one or more navigation instructions for avoiding For example, assuming command 130 is to pull over AV 802a at position coordinate 114, if controller 850 determines that the spot with position coordinate 114 is occupied by an obstacle or object: Controller 850 pulls AV 802a off the road at position coordinates 116, similar to that described in FIG. 1, by updating command 130 (eg, via communication subsystem 210).

At step 412, controller 850 navigates AV 802a according to updated command 130-1.
A First Exemplary System for Communicating Commands Between Two or More AVs

FIG. 5 shows an embodiment of a system 500 for communicating commands 130 between two or more AVs 802. As shown in FIG. FIG. 5 also shows a simplified schematic diagram of roadway 102 traveled by AV 802, including leading AV 802a, first trailing AV 802b, and second trailing AV 802n. In FIG. 5, AV 802 is heading towards an unexpected road condition 156 similar to that described in FIG. In one embodiment, the system 500 includes an operations server 120, a lead AV 802a, and a first trailing AV 802b. In some embodiments, system 500 further includes a second subsequent AV 802 n and network 110 . Network 110 enables communication between components of system 500 . System 500 may be configured as shown or in any other suitable configuration.

As shown in FIG. 5, the leading AV 802a, trailing AV 802b, and trailing AV 802n are within V2V communication range 510. FIG. Although three AVs 802 are shown in FIG. 5, those skilled in the art will recognize that any number of AVs 802 may be within V2V communication range 510 in light of this disclosure. V2V communication range 510 may correspond to a threshold distance range of operation for V2V modem 234 implemented in communication subsystem 210 described in FIG. Thus, using V2V modem 234 (see FIG. 2), AVs 802 can communicate data, including commands 130, to each other. The corresponding description below describes the communication of commands 130 between AVs 802 in V2V communication range 510 .
Communication of command to first successor AV

Generally, the system 500 (at the lead AV 802a) receives commands 130 (eg, operations server to AV commands 132) to navigate the lead AV 802a to avoid unexpected road conditions 156 ahead of the lead AV 802a. . For example, similar to that described in FIG. may receive command 130 .

Controller 850a associated with lead AV 802a receives sensor data 148a from sensor 846, where sensor data 148a includes position coordinates of a first plurality of objects in front of lead AV 802a. For example, sensor data 148a may include location coordinates of other vehicles, road signs, lane markings, road boundaries and traffic lights, among any other objects that sensor 846 may detect. Controller 850a may be an example of controller 850 described in FIG. Controller 850a may send and receive commands 130 via communication subsystem 210 described in FIG. Controller 850a navigates lead AV 802a (eg, via communication subsystem 210 (see FIG. 2)) based on analysis of sensor data 148a, similar to that described in FIGS. It may or may not update the command 130 for Controller 850a may determine and communicate commands 130-1 (eg, AV to AV command 136) to subsequent AVs 802b, as described below.

Controller 850a accesses environmental data 150b associated with the portion of road 102 between leading AV 802a and trailing AV 802b. Because lead AV 802a has traveled the portion of road 102 between lead AV 802a and trailing AV 802b, lead AV 802a has experienced and recorded environment data 150b. Environment data 150b may include position coordinates of a second plurality of objects between leading AV 802a and trailing AV 802b. Environmental data 150b relates to road safety regulation data, and obstacles 118, among other arbitrary data experienced and recorded while lead AV 802a travels the portion of road 102 between lead AV 802a and trailing AV 802b. It may further include data (see Figure 1). Environmental data 150b is associated with a window of time during which trailing AV 802b travels along roadway 102 .

Controller 850a determines whether at least one object detected in sensor data 148a and any of environmental data 150a and 150b prevents performance of command 130 by subsequent AV 802b. For example, controller 850a receives the trajectory of trailing AV 802b (e.g., from operations server 120 and/or trailing AV 802b) and detects objects detected in sensor data 148a and any of environmental data 150a and 150b to determine the movement of trailing AV 802b. can determine whether it is on the path. In another example, controller 850a may use sensors 846 similar to those described in FIG. 8 to determine the trajectory of trailing AV 802b. Controller 850a may determine that an object prevents subsequent AV 802b from performing command 130 if the object is in the path of travel of subsequent AV 802b.

In response to determining that at least one object (detected in sensor data 148a and any of environmental data 150a and 150b) prevents subsequent AV 802b from performing command 130, controller 850a directs subsequent AV 802b to Update command 130 . In this process, controller 850a generates updated command 130-1 (eg, AV to AV command 136). Updated command 130-1 includes one or more navigation instructions for subsequent AV 802b to avoid at least one object detected in its travel path while subsequent AV 802b was performing command 130. FIG. Controller 850a communicates updated command 130-1 to subsequent AV 802b.

Controller 850b associated with successor AV 802b receives updated command 130-1 and navigates AV 802b according to updated command 130-1. Controller 850b may be an example of controller 850 described in FIG.

In one embodiment, in response to determining that an object detected in sensor data 148b prevents performance of command 130-1, similar to that described in FIGS. 1 and 2 above, controller 850b , may further update command 130-1.
COMMUNICATION OF COMMANDS TO SECOND SUCCESSOR AV

Similar to that described above, controller 850a generates updated command 130-2 and communicates updated command 130-2 to second successor AV 802n within V2V communication range 510 of lead AV 802a. obtain. In this process, controller 850a accesses environmental data 150 associated with the portion of road 102 between leading AV 802a and second trailing AV 802n. Environmental data 150 associated with the portion of road 102 between leading AV 802a and second trailing AV 802n may include a set of environmental data 150b and 150n. The set of environment data 150b and 150n may include position coordinates of multiple objects between the leading AV 802a and the second trailing AV 802n. The set of environmental data 150b and 150n includes road safety regulation data, among other data experienced and recorded while the lead AV 802a traveled the portion of the road 102 between the lead AV 802a and the second trailing AV 802n; and data related to obstacles 118 (see FIG. 1). A set of environmental data 150b and 150n are associated with a window of time during which the second subsequent AV 802n travels along road 102. FIG.

Controller 850a determines whether at least one object detected in sensor data 148a and any of environmental data 150a, 150b, and 150n prevents performance of command 130 by second subsequent AV 802n. For example, the controller 850a receives the trajectory of the second successor AV 802n (eg, from the operations server 120 and/or the successor AV 802n) detected in the sensor data 148a and any of the environmental data 150a, 150b, and 150n It may be determined whether the object is on the travel path of the second subsequent AV 802n.

In response to determining that at least one object (detected in sensor data 148a and any of environmental data 150a, 150b, and 150n) prevents performance of command 130 by second subsequent AV 802n, control Device 850a updates command 130 for the second successor AV 802n. In this process, controller 850a generates a second updated command 130-2 by updating command 130-1. Second updated command 130-2 causes second successor AV 802n to avoid at least one object detected on its travel path while second successor AV 802n was performing command 130-1. contains one or more navigation instructions for Controller 850a communicates second updated command 130-2 (eg, AV to AV command 136) to second successor AV 802n so that AV 802n is navigated according to command 130-2. allows avoiding unexpected road conditions 156 and objects on its travel path.

Thus, according to the set of sensor data 148a, environmental data 150a, and corresponding environmental data 150 between the leading AV 802a and the nth trailing AV 802, the leading AV 802a updates the command 130 to It may communicate to subsequent AVs 802 .

In one embodiment, any of the AVs 802 (eg, AVs 802a-n) may, based on analysis of their corresponding sensor data 148, receive commands 130 ( Or the updated commands 130-1, 130-2) can be further updated or modified.

In one embodiment, any of AVs 802 (e.g., AVs 802a-n) receive commands 130 (or updated commands 130-1, 130-2) similar to those described in FIGS. to the operations server 120 to request further instructions.

In one embodiment, any of AVs 802 (e.g., AVs 802a-n) receive commands 130 (or updated commands 130-1, 130-2) similar to those described in FIGS. to the operations server 120 and request confirmation.

In one embodiment, AV 802b may be the leading AV 802 within V2V communication range 510 that includes one or more AVs 802 traveling behind AV 802b.

5, while AVs 802 (eg, AVs 802a-n) are traveling along roadway 102, AVs 802 communicate their corresponding sensor data 148 to operations server 120. As shown in FIG. For example, similar to that described in FIGS. 1 and 2, the lead AV 802a communicates sensor data 148a, the trailing AV 802b communicates sensor data 148b, and the second trailing AV 802n communicates sensor data 148n to operations server 120. FIG. Operations server 120 uses sensor data 148a-n, similar to that described in FIGS. may confirm, modify, or override that decision.
A Second Exemplary System for Communicating Commands Between Two or More AVs

FIG. 6 shows an embodiment of a system 600 for communicating commands 130 between two or more AVs 802. As shown in FIG. FIG. 6 also shows a simplified schematic diagram of roadway 102 traveled by AV 802, including lead AV 802a, first trailing AV 802b, and second trailing AV 802c. In FIG. 6, AV 802 is heading towards an unexpected road condition 156 similar to that described in FIG. In one embodiment, the system 600 includes an operations server 120, a lead AV 802a, and a first trailing AV 802b. In some embodiments, system 600 further includes a second subsequent AV 802c and network 110. FIG. Network 110 enables communication between components of system 600 . System 600 may be configured as shown or in any other suitable configuration.

As shown in FIG. 6, the lead AV 802a and the first trailing AV 802b are within a first V2V communication range 510 of each other, and the first trailing AV 802b and the second trailing AV 802c are in a second V2V communication range of each other. within range 510. AVs 802 within V2V communication range 510 can communicate data, including commands 130, to each other. FIG. 6 illustrates a scenario where different sets of AVs 802 are within different V2V communication ranges 510. FIG. The corresponding description below describes the communication of commands 130 between AVs 802 in different V2V communication ranges 510 .
Communication of command to first successor AV

The process of lead AV 802a communicating updated command 130-1 (eg, AV to AV command 136) to follow-on AV 802b may be similar to that described in FIG. In this process, lead AV 802a receives commands 130 to avoid unexpected road conditions 156 ahead of lead AV 802a, where command 130 is triggered upon detection of unexpected road conditions 156 in environment data 150a. Issued in response. Controller 850a associated with lead AV 802a receives sensor data 148a from sensor 846, where sensor data 148a includes position coordinates of a first plurality of objects in front of lead AV 802a. Controller 850a accesses environmental data 150b associated with the portion of road 102 between leading AV 802a and trailing AV 802b. Controller 850a determines whether at least one object detected in sensor data 148a and any of environmental data 150a and 150b prevents performance of command 130 by subsequent AV 802b. In response to determining that at least one object (detected in sensor data 148a and any of environmental data 150a and 150b) prevents subsequent AV 802b from performing command 130, controller 850a causes command 130 to By updating, an updated command 130-1 is generated, so that the updated command 130-1 contains at least one detected on its travel path while the subsequent AV 802b is performing the command 130. contains one or more navigation instructions for the subsequent AV 802b to avoid one object. Controller 850a communicates updated command 130-1 to subsequent AV 802b. Upon receiving updated commands 130-1, controller 850b may navigate AV 802b according to updated commands 130-1.

In scenarios where a particular AV 802 is not within V2V communication range 510 of the leading AV 802a, the particular AV 802 may, as described below, from operations server 120 (similar to those described in FIGS. 1 and 2) and/or Or it may receive commands 130 to avoid unexpected road conditions 156 from another AV 802 within V2V communication range 510 from the particular AV 802 . In FIG. 6, the second successor AV 802c is not within the V2V communication range 510 of the lead AV 802a, but the second successor AV 802c is within the V2V communication range 510 of the first successor AV 802b. Accordingly, the second successor AV 802c may receive commands 130-2 to avoid the unexpected road condition 156 from the first successor AV 802b. This process is described below.
COMMUNICATION OF COMMANDS TO SECOND SUCCESSOR AV

To generate command 130-2 (e.g., AV to AV command 136) for second subsequent AV 802c, controller 850b accesses environmental data 150c associated with the portion of road 102 between AV 802b and AV 802c. can. Environment data 150c may include position coordinates of multiple objects between AV 802b and AV 802c. Environmental data 150c includes, among other data experienced and recorded by AV 802b while traveling on the portion of road 102 between AV 802b and AV 802c, traffic safety regulation data, and data relating to obstacles 118 (see FIG. 1). Environmental data 150c is associated with a window of time during which second trailing AV 802c is traveling along roadway 102 . Controller 850b may further access and analyze sensor data 148b in a manner similar to that described in FIG. 5 with respect to controller 850a.

Controller 850b determines whether at least one object detected in sensor data 148b and any of environmental data 150a, 150b and 150c prevents AV 802c from performing command 130-1. For example, controller 850b receives the trajectory of AV 802c (e.g., from operations server 120 and/or AV 802c), and detects objects detected in sensor data 148b and any of environmental data 150a, 150b, and 150c to determine the movement path of AV 802c. can determine whether it is on. In response to determining that at least one object (detected in sensor data 148b and any of environmental data 150a, 150b and 150c) prevents AV 802c from performing command 130-1, controller 850b: Command 130-2 is generated by updating command 130-1, so that command 130-2 is the value of at least one detected on its travel path while AV 802c was performing command 130-1. Contains one or more navigation instructions for the AV 802c to avoid an object. Controller 850b communicates command 130-2 to AV 802c so that AV 802c is navigated according to command 130-2 to avoid unexpected road conditions 156 and objects in its travel path. be able to.

In one embodiment, any of the AVs 802 (eg, AVs 802a-c) may, based on analysis of their corresponding sensor data 148, receive commands 130 (e.g., AVs 802a-c) ( Or the updated commands 130-1, 130-2) can be further updated or modified.

In one embodiment, any of AVs 802 (e.g., AVs 802a-c) receive commands 130 (or updated commands 130-1, 130-2) similar to those described in FIGS. to the operations server 120 to request further instructions.

In one embodiment, any of AVs 802 (e.g., AVs 802a-c) receive commands 130 (or updated commands 130-1, 130-2) similar to those described in FIGS. to the operations server 120 and request confirmation.

As further shown in FIG. 6, as AVs 802 (eg, AVs 802a-c) travel along roadway 102, AVs 802 communicate their corresponding sensor data 148 to operations server 120. As shown in FIG. For example, similar to that described in FIGS. 1 and 2, the lead AV 802a communicates sensor data 148a, the trailing AV 802b communicates sensor data 148b, and the second trailing AV 802c communicates sensor data 148c to operations server 120. FIG. 1 and 2, the operations server 120 uses the sensor data 148a-c to direct commands 130, 130-1 and 130-2 or any of the controllers 850a-c for any navigation solution. may confirm, modify, or override that decision.

Although three AVs 802 are shown in FIG. 6, the present disclosure describes how to communicate command 130 (or updated commands 130-1, 130-2, etc.) to one or more other AVs 802 on road 102. Any number of AVs 802 above is assumed. As such, the present disclosure contemplates one-to-one, one-to-many, and many-to-many communications between the network mesh of AVs 802 and the operations server 120 .
Exemplary Method for Communicating Commands Between AVs

FIG. 7 shows an exemplary flowchart of a method 700 for communicating commands 130 between AVs 802. As shown in FIG. Modifications, additions, or omissions may be made to method 700 . Method 700 may include more, fewer, or other steps. For example, steps may be performed in parallel or in any suitable order. Although AV 802, operations server 120, controller 850, communication subsystem 210, or components of any of them may be discussed as performing the steps, any suitable system or component thereof may perform one or more steps of method 700 . For example, one or more steps of method 700 may be performed, at least in part, on a non-transitory, tangible, machine-readable medium (eg, memories 128, 242, 890 and 890 from FIGS. 1, 2, 8 and 10, respectively). 1002), which may be implemented in the form of software instructions 138, 244 and 880, respectively, from FIGS. 2, 8 and 10) may cause the one or more processors to perform steps 702-714.

Method 700 begins at step 702 , where controller 850 a associated with lead AV 802 a receives commands 130 to navigate lead AV 802 a to avoid unexpected road conditions 156 . Controller 850a may receive commands 130 via communications subsystem 210 described in FIG. For example, similar to that described in FIGS. 1, 5 and 6, that operations server 120 detects unexpected road conditions 156 in environmental data 150a associated with the portion of road 102 ahead of leading AV 802a. In response, controller 850 a may receive command 130 from operations server 120 .

At step 704, controller 850a receives sensor data 148a from sensor 846 associated with lead AV 802a, where sensor data 148a includes position coordinates of a first plurality of objects in front of lead AV 802a. For example, controller 850 a may be triggered to receive sensor data 148 a from sensor 846 in response to receiving command 130 . In another example, controller 850a periodically (eg, every 0.02 seconds, every 0.5 seconds, or any other suitable duration) similar to that described in FIG. Sensor data 148a may be received.

At step 706, controller 850a accesses environmental data 150b associated with the portion of road 102 between leading AV 802a and trailing AV 802b. Because the lead AV 802a traveled the portion of the road 102 between the lead AV 802a and the trailing AV 802b, the lead AV 802a experienced and recorded environmental data 150b. Environment data 150b may include position coordinates of a second plurality of objects between leading AV 802a and trailing AV 802b. Environmental data 150b relates to road safety regulation data, and obstacles 118, among other arbitrary data experienced and recorded while lead AV 802a travels the portion of road 102 between lead AV 802a and trailing AV 802b. It may further include data (see Figure 1). Environmental data 150 a is associated with a window of time during which trailing AV 802 b travels along roadway 102 .

At step 708, controller 850a determines whether at least one object detected in sensor data 148a and any of environmental data 150a and 150b prevents performance of command 130 by subsequent AV 802b. For example, controller 850a receives the trajectory of trailing AV 802b (e.g., from operations server 120 and/or trailing AV 802b) and detects objects detected in sensor data 148a and any of environmental data 150a and 150b to determine the movement of trailing AV 802b. can determine whether it is on the path. If controller 850a determines that at least one object detected in sensor data 148a and any of environmental data 150a and 150b prevents subsequent AV 802b from performing command 130, method 700 proceeds to step 712; Otherwise, method 700 proceeds to step 710 .

At step 710, controller 850a communicates command 130 to subsequent AV 802b. For example, controller 850a communicates original command 130 to subsequent AV 802b via communication subsystem 210. FIG.

At step 712, controller 850a updates command 130 for subsequent AV 802b so that updated command 130-1 is one for avoiding at least one object while performing command 130. Including the above commands.

At step 714, controller 850a communicates updated command 130-1 to successor AV 802b.

In one embodiment, the lead AV 802a further includes sensor data 148a, environmental data 150a, and environmental data 150 associated with the portion of the road 102 between the lead AV 802 and the particular AV 802, similar to that described in FIG. command 130 (or updated commands 130-1, 130-2) to communicate to any particular AV 802 within V2V communication range 510 of the lead AV 802a.

In the event that a particular AV 802 is not within V2V communication range 510 of the leading AV 802, the particular AV 802 can be sent from another AV 802 within V2V communication range 510 of the particular AV 802 (eg, It may receive command 130 (or updated commands 130-1, 130-2) indirectly from head AV 802a. For example, multiple AVs 802 on roadway 102 may form a network mesh of AVs 802, in which any AV 802 communicates command 130 (or updated commands 130-1, 130) with any other AV 802. -2) or any other data/instructions may be sent or received.
Exemplary AV and its operation

FIG. 8 shows a block diagram of an example vehicle ecosystem 800 in which autonomous driving behavior may be determined. As shown in FIG. 8, AV 802 may be a semi-trailer truck. Vehicle ecosystem 800 includes a number of systems and components that can generate one or more sources of information/data and related services and/or deliver to onboard control computer 850, which may be located on AV 802. . Onboard control computer 850 may be in data communication with multiple vehicle subsystems 840 , all of which may reside on AV 802 . Vehicle subsystem interface 860 is provided to facilitate data communication between onboard control computer 850 and multiple vehicle subsystems 840 . In some embodiments, vehicle subsystem interface 860 may include a controller area network (CAN) controller to communicate with devices within vehicle subsystem 840 .

AV 802 may include various vehicle subsystems that support operation of AV 802 . Vehicle subsystems 840 may include communications subsystem 210 , vehicle operation subsystem 842 , vehicle sensor subsystem 844 , and/or vehicle control subsystem 848 . Examples are the components or devices of the vehicle operating subsystem 842, vehicle sensor subsystem 844, and vehicle control subsystem 848 shown in FIG. AV 802 may be configured as shown or in any other configuration.

Vehicle operating subsystem 842 may include components operable to provide powered motion for AV 802 . In the exemplary embodiment, vehicle operating subsystem 842 may include engine/motor 842a, wheels/tires 842b, transmission 842c, electrical subsystem 842d, and power source 842e.

Vehicle sensor subsystem 844 may include a number of sensors 846 configured to sense information regarding the environment or condition of AV 802 . The vehicle sensor subsystem 844 includes one or more cameras 846a or image capture devices, a radar unit 846b, one or more temperature sensors 846c, a wireless communication unit 846d (eg, cellular communication transceiver), an inertial measurement unit (IMU) 846e, It may include a laser rangefinder/LiDAR unit 846f, a global positioning system (GPS) transceiver 846g, and/or a wiper control system 846h. Vehicle sensor subsystem 844 may also include sensors configured to monitor internal systems of AV 802 (eg, _O2 monitor, fuel gauge, engine oil temperature gauge, etc.).

IMU 846e may include any combination of sensors (eg, accelerometers and gyroscopes) configured to sense changes in AV 802 position and orientation based on inertial acceleration. GPS transceiver 846g may be any sensor configured to estimate the geographic location of AV 802 . To this end, GPS transceiver 846g may include a receiver/transmitter operable to provide information regarding the position of AV 802 relative to Earth. Radar unit 846b may represent a system that utilizes radio signals to sense objects in the AV 802's local environment. In some embodiments, in addition to sensing objects, radar unit 846b may be further configured to sense the speed and heading of objects proximate to AV 802 . Laser rangefinder or LiDAR unit 846f may be any sensor configured to sense objects in the environment in which AV 802 is located using a laser. Camera 846a may include one or more devices configured to capture multiple images of the AV 802 environment. Camera 846a may be a still image camera or a motion video camera.

Vehicle control subsystem 848 may be configured to control the operation of AV 802 and its components. Accordingly, vehicle control subsystem 848 may include various elements such as throttle and gear selector 848a, brake unit 848b, navigation unit 848c, steering system 848d, and/or autonomous control unit 848e. Throttle 848 a may be configured, for example, to control the operating speed of the engine, which in turn controls the speed of AV 802 . Gear selector 848a may be configured to control gear selection of the transmission. Brake unit 848b may include any combination of mechanisms configured to slow AV 802 . The brake unit 848b can slow the AV in standard ways, including using friction to slow the wheels or using engine braking. Brake unit 848b may include an anti-lock braking system (ABS) that can prevent the brakes from locking when the brakes are applied. Navigation unit 848 c may be any system configured to determine driving paths or routes for AV 802 . The navigation 848c unit may be further configured to dynamically update the driving route while the AV 802 is operating. In some embodiments, navigation unit 848c incorporates data from GPS transceiver 846g and one or more predetermined maps to determine a driving route for AV 802 (eg, along road 102 in FIG. 1). can be configured as Steering system 848d may represent any combination of mechanisms that may be operable to adjust the orientation of AV 802 in autonomous mode or driver-controlled mode.

Autonomous control unit 848e may represent a control system configured to identify, evaluate, and avoid or otherwise maneuver through potential obstacles or obstacles in AV 802's environment. Generally, autonomous control unit 848e may be configured to control AV 802 to operate without a driver or to provide driver assistance in controlling AV 802 . In some embodiments, autonomous control unit 848e incorporates data from GPS transceiver 846g, radar 846b, LiDAR unit 846f, camera 846a, and/or other vehicle subsystems to determine the driving path or trajectory of AV 802. can be configured to

Many or all of the functions of AV 802 can be controlled by onboard control computer 850 . Onboard control computer 850 has at least one (which may include at least one microprocessor) data processor 870 that executes processing instructions 880 stored in a data storage device 890 or non-transitory computer readable media such as memory. can include Onboard control computer 850 may also represent multiple computing devices that may serve to control individual components or subsystems of AV 802 in a distributed fashion. In some embodiments, data storage device 890 stores processing instructions 880 ( program logic).

Data store 890 may also include additional instructions that transmit data to one or more of vehicle operating subsystem 842, vehicle sensor subsystem 844, and vehicle control subsystem 848. contains instructions for receiving, interacting with, or controlling the Onboard control computer 850 may be configured to include data processor 870 and data storage device 890 . Onboard control computer 850 may control the functions of AV 802 based on inputs received from various vehicle subsystems (eg, vehicle driving subsystem 842, vehicle sensor subsystem 844, and vehicle control subsystem 848).

FIG. 9 shows an exemplary system 900 for providing accurate autonomous driving behavior. System 900 includes several modules that can operate in onboard control computer 850, as described in FIG. Onboard control computer 850 includes a sensor fusion module 902 shown in the upper left corner of FIG. 9, where sensor fusion module 902 may perform at least four image or signal processing operations. A sensor fusion module 902 acquires images from cameras located on the autonomous vehicle and performs image segmentation 904 to identify moving objects (e.g., other vehicles, pedestrians, etc.) and/or stationary objects around the autonomous vehicle. The presence of physical obstacles (eg, stop signs, speed bumps, terrain, etc.) can be detected. A sensor fusion module 902 obtains LiDAR point cloud data items from LiDAR sensors located on the autonomous vehicle and performs LiDAR segmentation 906 to detect the presence of objects and/or obstacles located around the autonomous vehicle. can do.

A sensor fusion module 902 performs instance segmentation 908 on image and/or point cloud data items to identify contours (e.g., boxes) around objects and/or obstacles placed around the autonomous vehicle. can do. The sensor fusion module 902 can perform temporal fusion 910, where objects and/or obstacles from one image and/or one frame of a point cloud data item are subsequently received in time 1 Correlated or associated with objects and/or obstacles from one or more images or frames.

The sensor fusion module 902 can fuse objects and/or obstacles from images obtained from cameras and/or point cloud data items obtained from LiDAR sensors. For example, based on the positions of the two cameras, the sensor fusion module 902 may determine that an image from one of the cameras containing the half of the vehicle positioned in front of the autonomous vehicle is captured by the other camera and positioned. can be determined to be the same as the vehicle that Sensor fusion module 902 transmits fused object information to interference module 946 and fused obstacle information to occupancy grid module 960 . The onboard control computer includes an occupancy grid module 960 that can retrieve landmarks from a map database 958 stored on the onboard control computer. Occupancy grid module 960 may determine drivable areas and/or obstacles from the fused obstacles obtained from sensor fusion module 902 and landmarks stored in map database 958 . For example, the occupancy grid module 960 may determine that the drivable area may include speed bump obstacles.

Underneath the sensor fusion module 902, the onboard control computer 850 includes a LiDAR-based object detection module 912 that can perform object detection 916 based on point cloud data items obtained from LiDAR sensors 914 located on the autonomous vehicle. include. An object detection 916 technique can provide the position (eg, in 3D world coordinates) of an object from the point cloud data item. Below the LiDAR-based object detection module 912, the onboard control computer includes an image-based object detection module 918 that can perform object detection 924 based on images obtained from cameras 920 located on the autonomous vehicle. Object detection 918 techniques can provide the position (eg, in 3D world coordinates) of objects from images provided by camera 920 by utilizing deep machine learning techniques 924 .

Radar 956 on the autonomous vehicle can scan the area in front of the autonomous vehicle or the area to which the autonomous vehicle is being driven. The radar data is sent to the sensor fusion module 902, which uses the radar data to convert objects and/or obstacles detected by the radar 956 into LiDAR point cloud data items and camera images. Both can be correlated with detected objects and/or obstacles. The radar data is also transmitted to an interference module 946, which performs data processing on the radar data to enable object tracking by an object tracking module 948, as further described below. be able to.

The onboard control computer includes an interference module 946 that receives objects from the point cloud and object positions from the image, as well as the fused objects from the sensor fusion module 902 . The interferometric module 946 also receives radar data, which the interferometric module 946 converts from one point cloud data item and one image acquired at a time to another (or subsequent) point cloud data. data that allows tracking of the object by the object tracking module 948 to the item and another image acquired at another subsequent time.

Interference module 946 may perform object attribute estimation 950 to estimate one or more attributes of objects detected in an image or point cloud data item. One or more attributes of an object may include the type of object (eg, pedestrian, car, truck, etc.). Interference module 946 may perform behavior prediction 952 to estimate or predict motion patterns of objects detected in images and/or point clouds. Behavior prediction 952 is for detecting the position of an object in a set of images received at different times (e.g., sequential images) or a set of point cloud data items received at different times (e.g., sequential point cloud data items). can be performed in In some embodiments, behavior prediction 952 may be performed for each image received from a camera and/or each point cloud data item received from a LiDAR sensor. In some embodiments, interferometry module 946 is performed (e.g., run or executed) and every other or predetermined number of images received from cameras or point cloud data items received from LiDAR sensors ( For example, computational load can be reduced by performing behavior prediction 952 every two images, or every three point cloud data items).

The behavior prediction 952 feature can determine the speed and direction of objects surrounding the autonomous vehicle from radar data, where the speed and direction information is used to predict or determine the object's motion pattern. can be done. A motion pattern may include predicted trajectory information of an object over a predetermined period of time in the future after the image is received from the camera. Based on the predicted motion pattern, interference module 946 may assign a motion pattern context tag to the object (e.g., "located at coordinates (x,y)", "stopped", "driving at 50 mph"). ”, “accelerating” or “decelerating”). A context tag can describe the motion pattern of an object. Interference module 946 transmits one or more object attributes (eg, object type) and motion pattern context tags to planning module 962 . Interference module 946 may perform environmental analysis 954 using any information obtained by combination of system 900 and any number of its components.

The onboard control computer includes a planning module 962 that incorporates object attribute and motion pattern context tags, drivable areas and/or obstacles from interference module 946, and vehicle position and attitude information from fusion localization module 926. Receive (described further below).

Planning module 962 may determine a set of trajectories on which the autonomous vehicle can drive by executing navigation plan 964 . A set of trajectories may be determined based on drivable area information, one or more object attributes of the object, motion pattern context tags of the object, locations of obstacles, and drivable area information. In some embodiments, navigation plan 964 may include determining areas next to roads where autonomous vehicles can be safely parked in an emergency. The planning module 962 responds to changing conditions on the road (e.g., a traffic light has turned yellow, or the autonomous vehicle has another vehicle driving in front of it, pre-determined from the autonomous vehicle's position). (e.g., steering, braking, throttling) in response to determining that the vehicle is in an unsafe driving condition (because it is within the safe distance region). . Planning module 962 performs trajectory generation 968 to select a trajectory from the set of trajectories determined by navigation plan operation 964 . Selected trajectory information is sent by planning module 962 to control module 970 .

The onboard control computer includes a control module 970 that receives the proposed trajectory from planning module 962 and the position and attitude of the autonomous vehicle from fusion localization module 926 . Control module 970 includes system identifier 972 . Control module 970 can refine the proposed trajectory by performing model-based trajectory refinement 974 . For example, control module 970 can apply filtering (eg, a Kalman filter) to smooth the proposed trajectory data and/or minimize noise. Based on the refined and proposed trajectory information and the current position and/or attitude of the autonomous vehicle, the control module 970 determines the amount of brake pressure to apply, the steering angle, the amount of throttle to control the speed of the vehicle, and/or Alternatively, robust control 976 may be performed by determining the transmission gear. The control module 970 controls and facilitates precise driving behavior of the autonomous vehicle by transmitting the determined brake pressure, steering angle, throttle amount, and/or transmission gear to one or more devices within the autonomous vehicle. be able to.

Deep image-based object detection 924 performed by image-based object detection module 918 is also used to detect landmarks (eg, stop signs, speed bumps, etc.) on roads (eg, road 102 in FIG. 1). be able to. The on-board control computer includes a fusion localization module 926 that integrates landmarks detected from images, landmarks obtained from a map database 936 stored in the on-board control computer, by the LiDAR-based object detection module 912 . Landmarks detected from point cloud data items, velocity and displacement from odometer sensors 944, and GPS/IMU sensors 938 (i.e., GPS sensor 940 and IMU sensor 942) located on or within the autonomous vehicle. ) to obtain the estimated position of the autonomous vehicle from Based on this information, fusion localization module 926 can determine the position of the autonomous vehicle by performing localization operations 928 , which can be transmitted to planning module 962 and control module 970 .

Fused localization module 926 may estimate pose 930 of the autonomous vehicle based on GPS and/or IMU sensors 938 . The attitude of the autonomous vehicle can be sent to planning module 962 and control module 970 . The fusion localization module 926 may also, for example, determine the state of the trailer unit (e.g., position, possible movement angle) can be estimated. Fused localization module 926 may also check map content 932 .

FIG. 10 shows an exemplary block diagram of an onboard control computer 850 included in the autonomous AV 802. As shown in FIG. Onboard control computer 850 includes at least one processor 1004 and a memory 1002 in which instructions are stored (eg, software instructions 138, 244 and processing instructions 880 of FIGS. 1, 2 and 8, respectively). The instructions, when executed by processor 1004, configure onboard control computer 850 and/or various modules of onboard control computer 850 to perform the operations described in FIGS. 1-7. Transmitter 1006 transmits or transmits information or data to one or more devices within the autonomous vehicle. For example, the transmitter 1006 can send instructions to one or more motors of the steering wheel to navigate the autonomous vehicle. Receiver 1008 receives information or data transmitted or transmitted by one or more devices. For example, receiver 1008 receives the current speed status from an odometer sensor or the current transmission gear from a transmission. Transmitter 1006 and receiver 1008 are also configured to communicate with a plurality of vehicle subsystems 840 and onboard control computer 850 described in FIGS. 8 and 9 above.

Although several embodiments have been provided in this disclosure, it is understood that the disclosed system and method may be embodied in many other specific forms without departing from the spirit or scope of this disclosure. want to be The examples are to be considered illustrative and non-limiting, and the intent is not to be limited to the details given herein. For example, various elements or components may be combined, integrated into another system, or certain features omitted or not implemented.

Moreover, the techniques, systems, subsystems and methods described and illustrated in various embodiments as discrete or separate may be combined with other systems, modules, techniques or methods without departing from the scope of the present disclosure. or can be integrated. Other items shown or described as being coupled or directly coupled or in communication with each other may not be electrically, mechanically, or otherwise indirectly through any interface, device, or intermediate component. may be combined or communicated. Other examples of changes, substitutions, and modifications are identifiable to those skilled in the art and can be made without departing from the spirit and scope disclosed herein.

To assist the Patent Office, and the reader of the patent issued in this application, to interpret the claims appended hereto, Applicants may use the words "means for" or "steps for" Unless expressly used in a particular claim, none of the appended claims are intended to invoke Section 112(f), title 35, United States Code, as it exists on the filing date of this specification. Note that the

Implementations of the disclosure can be described in view of the following paragraphs, the features of which can be combined in any reasonable way.

Item 1. A system, the system comprising:
an autonomous vehicle (AV) comprising at least one vehicle sensor, the AV configured to travel along a road;
an operations server communicatively coupled to the AV;
The operations server comprises a processor, the processor comprising:
accessing environmental data associated with the road ahead of the AV, wherein the environmental data is associated with a window of time during which the AV travels along the road; ,
comparing the environmental data with map data including expected road conditions in front of the AV;
determining whether the environmental data includes unexpected road conditions not included in the map data based at least in part on comparing the environmental data to the map data;
in response to determining that the environmental data includes the unexpected road conditions not included in the map data;
determining location coordinates of the unexpected road condition;
and communicating commands to the AV to steer around the unexpected road condition.

Item 2. further comprising a controller associated with the AV, the controller comprising a second processor, the second processor comprising:
receiving sensor data from the at least one vehicle sensor including position coordinates of a plurality of objects on the road ahead of the AV;
Based at least in part on the sensor data, updating the command such that the updated command includes one or more instructions for the AV to avoid the plurality of objects during performance of the command. and
and navigating the AV according to the updated command.

Item 3. 3. The system of clause 2, wherein the second processor is further configured to communicate the updated command to one or more subsequent AVs moving in line behind the AV.

Item 4. The second processor,
comparing the sensor data with the map data, the map data including position coordinates of expected objects on the road in front of the AV;
determining, based at least in part on comparing the sensor data to the map data, whether the sensor data indicates that the unexpected object is absent from the map data;
Responsive to determining that the sensor data indicates that the unexpected object is not in the map data, the AV determines suggested navigation instructions for avoiding the unexpected object. 4. The system of clause 2 or 3, further configured to:

Item 5. The second processor is further configured to perform the suggested navigation instructions in response to determining that the sensor data indicates that the unexpected object is absent from the map data. 5. The system of any one of clauses 2-4, wherein the system comprises:

Item 6. The second processor,
in response to determining that the sensor data indicates that the unexpected object is not in the map data;
communicating the suggested navigation instructions to the operations server;
determining whether confirmation has been received from the operations server to perform the suggested navigation instructions;
6. The system of any of clauses 2-5, further configured to: perform the suggested navigation instruction in response to receiving the confirmation from the operations server.

Item 7. The second processor,
in response to determining that the sensor data indicates that the unexpected object is not in the map data;
communicating the sensor data to the operations server;
7. Any of clauses 2-6, further configured to: request the operations server to provide instructions to avoid the unexpected object based at least in part on the sensor data. The system described in .

Item 8. further comprising one or more subsequent AVs different from the AV;
the one or more trailing AVs are on the road behind the AVs;
The one or more subsequent AVs are communicatively coupled to the operations server;
The above processor
in response to determining that said environmental data includes said unexpected road conditions not included in said map data;
receiving the sensor data from the controller;
generating a second command including one or more instructions for avoiding the unexpected road condition while the one or more subsequent AVs are avoiding the plurality of objects; the second command includes the location coordinates of the unexpected road condition indicated in the command and the location coordinates of the plurality of objects indicated in the sensor data;
and communicating the second command to the one or more subsequent AVs.

Item 9. A method, the method comprising:
accessing environmental data associated with a road ahead of an autonomous vehicle (AV), the environmental data associated with a window of time during which the AV travels along the road; and
comparing the environmental data with map data including expected road conditions in front of the AV;
determining whether the environmental data includes unexpected road conditions not included in the map data based at least in part on comparing the environmental data to the map data;
in response to determining that the environmental data includes the unexpected road conditions not included in the map data;
determining location coordinates of the unexpected road condition;
and communicating commands to the AV to steer around the unexpected road condition.

Item 10. The unexpected road conditions include at least one of unanticipated weather conditions, unanticipated traffic congestion, unanticipated road closures, and unanticipated construction areas not included in the map data. 9. The method of claim 9.

Item 11. 11. The method of any of clauses 9-10, wherein the environmental data includes at least one of weather data and traffic data.

Item 12. The above command is
Transition from autonomous operation to manual operation by the above AV,
avoidance by the AV of obstacles on the road in front of the AV;
avoidance by the AV of one or more certain lanes in which one or more obstacles are detected;
avoidance by the AV of one or more fixed routes where the unexpected road conditions are detected;
certain route changes by the AV, and driving by the lead AV at speeds slower or faster than indicated in driving instructions associated with the lead AV;
12. The method of any of paragraphs 9-11, associated with at least one of

Item 13. the command includes a broad command directed to the AV and one or more subsequent AVs behind the AV on the road;
13. The method of any of clauses 9-12, wherein the global command includes one or more navigation instructions for avoiding certain unanticipated road conditions ahead of the AV.

Item 14. said command includes a special command directed to said AV;
14. The method of any of clauses 9-13, wherein the special commands include one or more navigation instructions for avoiding certain unexpected road conditions ahead of the AV.

Item 15. A computer program comprising executable instructions which, when executed by one or more processors, cause the one or more processors to:
accessing environmental data associated with a road ahead of an autonomous vehicle (AV), the environmental data associated with a window of time during which the AV travels along the road; and
comparing the environmental data with map data including expected road conditions in front of the AV;
determining whether the environmental data includes unexpected road conditions not included in the map data based at least in part on comparing the environmental data to the map data;
in response to determining that the environmental data includes the unexpected road conditions not included in the map data;
determining location coordinates of the unexpected road condition;
and communicating commands to the AV to steer around the unexpected road condition.

Item 16. The instructions, when executed by the one or more processors, cause the one or more processors to respond to determining that the environmental data includes the unexpected road conditions not included in the map data. 16. The computer program of clause 15, further causing the command to be propagated to one or more subsequent AVs traveling along the road behind the AV.

Item 17. 17. Computer program according to any of clauses 15-16, wherein said command comprises a setting command.

Item 18. 18. The computer program product of any of clauses 15-17, wherein the configuration command includes at least one of changing the orientation of the at least one vehicle sensor and changing the data sampling frequency of the at least one vehicle sensor. .

Item 19. From paragraph 15, wherein the AV comprises at least one vehicle sensor, the at least one vehicle sensor including at least one of a camera, a light detection and ranging (LiDAR) sensor, a motion sensor, and an infrared sensor. 19. The computer program according to any of 18.

Item 20. 20. The computer program product of any of clauses 15-19, wherein the AV is a tractor unit attached to a trailer.

Item 21. A system, the system comprising:
an autonomous vehicle (AV) comprising at least one vehicle sensor, the AV configured to travel along a road;
a controller associated with the AV and comprising a processor;
The processor is
receiving commands from an operations server to navigate the AV to avoid unexpected road conditions;
receiving sensor data including position coordinates of a plurality of objects in front of the AV from the at least one vehicle sensor;
determining whether at least one object of the plurality of objects interferes with fulfillment of the command such that the at least one object is on a path for navigating the AV according to the command;
updating the command in response to determining that the at least one object interferes with performance of the command, so that the updated command updates the at least one object during performance of the command; including one or more navigation instructions to avoid
and navigating the AV according to the updated command.

Item 22. The above processor
comparing the sensor data with map data, the map data including position coordinates of expected objects on the road in front of the AV;
determining whether the sensor data indicates an unexpected object not in the map data based at least in part on comparing the sensor data to the map data;
in response to determining that the sensor data indicates the unexpected object not in the map data;
determining position coordinates of the unexpected object;
and determining suggested navigation instructions for avoiding the unexpected object.

Item 23. The processor is further configured to perform the suggested navigation instructions in response to determining that the sensor data indicates that the unexpected object is not shown in the map data. 23. The system of clause 22, wherein the system comprises:

Item 24. The processor, in response to determining that the sensor data indicates that the unexpected object is not in the map data,
communicating the suggested navigation instructions to the operations server;
determining whether confirmation has been received from the operations server to perform the suggested navigation instructions;
24. The system of any of clauses 22-23, further configured to: perform the suggested navigation instructions in response to receiving the confirmation from the operations server.

Item 25. The processor, in response to determining that the sensor data indicates that the unexpected object is not in the map data,
communicating the sensor data to the operation server;
25. Any of clauses 22-24, further configured to: request the operations server to provide instructions to avoid the unexpected object based at least in part on the sensor data. System as described.

Item 26. The operations server comprises a second processor, the second processor comprising:
accessing environmental data associated with the road ahead of the AV, comprising:
the environmental data includes at least one of weather data and traffic data;
the environmental data is associated with a window of time during which the AV is traveling along the road;
comparing the environmental data with the map data;
determining whether the environmental data includes the unexpected road condition not included in the map data based at least in part on comparing the environmental data to the map data;
In response to determining that the environmental data includes the unexpected road condition not included in the map data, the command is issued to the AV to maneuver to avoid the unexpected road condition. 26. The system of any of clauses 21-25, wherein the system is configured to:

Item 27. the processor is further configured to communicate the sensor data to the operations server in response to receiving the command from the operations server;
The second processor,
receiving the sensor data from the controller;
generating a second command including one or more instructions for avoiding the unexpected road condition while the one or more subsequent AVs are avoiding the plurality of objects; the second command includes the position coordinates of the unexpected road condition indicated in the command and the position coordinates of the plurality of objects indicated in the sensor data;
the one or more trailing AVs are on the road behind the AVs;
the one or more successor AVs are communicatively coupled to the operations server;
27. The system of clause 26, further configured to: and communicate the second command to the one or more subsequent AVs.

Item 28. A method, the method comprising:
Receiving commands from an operations server to navigate an autonomous vehicle (AV) to avoid unexpected road conditions,
the AV comprises at least one vehicle sensor;
the AV is configured to travel along a road;
receiving sensor data including position coordinates of a plurality of objects in front of the AV from the at least one vehicle sensor;
determining whether at least one object of the plurality of objects interferes with performance of the command;
updating the command in response to determining that at least one object of the plurality of objects prevents performance of the command, so that the updated command is executed during performance of the command; including one or more navigation instructions for avoiding the at least one object;
navigating the AV according to the updated command.

Clause 29. Collecting, by one or more computing agent units, one or more health information reports from one or more components of said AV, each of said one or more health information reports comprising: indicating the performance status of the corresponding component of the AV;
30. The method of clause 28, further comprising communicating, by the one or more computing agent units, the one or more health information reports to the operations server.

Item 30. establishing a first communication channel between the AV and the operations server by one or more computing agent units;
receiving, by the one or more computing agent units, a first periodic message from the operations server at periodic intervals and determining whether the first communication channel is lost; determining that the first communication channel is lost if the first periodic message has not been received in one periodic interval;
In response to determining that the first communication channel has been lost, a second communication between the AV and a subsequent AV on the road behind the AV is performed by the one or more computing agent units. 30. The method of clause 28 or 29, further comprising: establishing a communication channel.

Item 31. establishing, by one or more computing agent units, a third communication channel between the AV and a subsequent AV on the road behind the AV;
receiving by the one or more computing agent units from the subsequent AV a second periodic message at regular time intervals and determining whether the third communication channel has been lost; , it is determined that the third communication channel is lost if the second periodic message has not been received in at least one periodic interval;
establishing, by the one or more computing agent units, a fourth communication channel between the AV and the operations server in response to determining that the third communication channel has been lost. 31. The method of any of paragraphs 28-30, further comprising.

Item 32. The unexpected road conditions include at least one of unanticipated weather conditions, unanticipated traffic congestion, unanticipated road closures, and unanticipated construction areas not included in the map data. 32. The method of any one of paragraphs 28-31.

Item 33. The above command is
Transition from autonomous operation to manual operation by the above AV,
avoidance by the AV of obstacles on the road in front of the AV;
avoidance by the AV of one or more certain lanes in which one or more obstacles are detected;
avoidance by the AV of one or more fixed routes where the unexpected road conditions are detected;
certain route changes by the AV, and driving by the lead AV at speeds slower or faster than indicated in driving instructions associated with the lead AV;
33. The method of any of paragraphs 28-32, associated with at least one of

Item 34. the command includes a broad command directed to the AV and one or more subsequent AVs behind the AV on the road;
34. The method of any of clauses 28-33, wherein the global command includes one or more navigation instructions for avoiding certain unanticipated road conditions ahead of the AV.

Item 35. said command includes a special command directed to said AV;
35. A method according to any of clauses 28-34, wherein the special commands comprise one or more navigation instructions for avoiding certain unexpected road conditions ahead of the AV.

Item 36. A computer program comprising executable instructions which, when executed by one or more processors, cause the one or more processors to:
Receiving commands from an operations server to navigate an autonomous vehicle (AV) to avoid unexpected road conditions,
the AV comprises at least one vehicle sensor;
the AV is configured to travel along a road;
receiving sensor data including position coordinates of a plurality of objects in front of the AV from the at least one vehicle sensor;
determining whether at least one object of the plurality of objects interferes with fulfillment of the command such that the at least one object is on a path for navigating the AV according to the command;
updating the command in response to determining that the at least one object interferes with performance of the command, so that the updated command updates the at least one object during performance of the command; including one or more navigation instructions to avoid
navigating the AV according to the updated commands.

Item 37. The instructions, when executed by the one or more processors, cause the one or more processors to respond to determining that the environmental data includes the unexpected road conditions not included in the map data. 37. The computer program of clause 36, further causing the command to be propagated to one or more subsequent AVs traveling along the road behind the AV.

Item 38. The above commands include setting commands,
38. Computer program according to clause 36 or 37, wherein the setting command comprises at least one of changing the orientation of the at least one vehicle sensor and changing the data sampling frequency of the at least one vehicle sensor.

Item 39. 39. The computer program of any of clauses 36-38, wherein the at least one vehicle sensor comprises at least one of a camera, a light detection and ranging (LiDAR) sensor, a motion sensor, and an infrared sensor.

Item 40. 40. The computer program of any of clauses 36-39, wherein the AV is a tractor unit mounted on a trailer.

Item 41. A system, the system comprising:
a lead autonomous vehicle (AV) comprising at least one vehicle sensor, the lead autonomous vehicle (AV) configured to travel along a road;
a trailing AV that is different from the lead AV and communicatively coupled to the lead AV, the trailing AV moving along the road behind the lead AV;
a first controller associated with the top AV and comprising a first processor, the first processor comprising:
receiving commands to navigate the lead AV to avoid unexpected road conditions ahead of the lead AV;
receiving sensor data from the at least one vehicle sensor including position coordinates of a first plurality of objects in front of the lead AV;
accessing a first set of environmental data associated with a portion of a road between the leading AV and the trailing AV, wherein the first set of environmental data comprises the leading AV and the trailing AV; including position coordinates of a second plurality of objects between
determining whether at least one object of the first and second plurality of objects prevents performance of the command by the subsequent AV;
issuing a command to the subsequent AV based at least in part on the first set of sensor data and the environmental data in response to determining that the at least one object prevents the subsequent AV from performing the command; updating so that the updated command includes one or more navigation instructions for avoiding the at least one object during performance of the command;
and communicating the updated command to the successor AV.

Item 42. 42. The system of Clause 41, wherein the unexpected road conditions include at least one of unexpected weather conditions and unexpected traffic jams.

Item 43. 43. The system of clause 41 or 42, wherein the first set of environmental data is associated with a window of time during which the subsequent AV travels along the road.

Item 44. further comprising a second controller associated with the successor AV and comprising a second processor, the second processor:
receiving the updated command from the top AV;
44. The system of any of clauses 41-43, wherein the system is configured to: and navigate the successor AV according to the updated command.

Item 45. The successor AV includes a first successor AV, the system further includes a second successor AV communicatively coupled to the first successor AV, the second successor AV comprising the first successor AV. 1 is moving along the road behind the following AV,
The second processor,
accessing a second set of environmental data associated with the portion of the road between the first successor AV and the second successor AV, the second set of environmental data being , position coordinates of a third plurality of objects between the first successor AV and the second successor AV;
determining whether at least one object of the third plurality of objects interferes with performance of the updated command;
responsive to a determination that the at least one object prevents performance of the updated command, based at least in part on a set of the first and the second sets of environmental data; so that the second updated command generates the third plurality while performing the updated command one or more navigation instructions for avoiding the at least one of the objects of
45. The system of clause 44, further configured to: and communicate the second updated command to the second successor AV.

Item 46. the leading AV and the trailing AV are within vehicle-to-vehicle (V2V) communication range;
of paragraphs 41 to 45, wherein the V2V communication range corresponds to a threshold distance of V2V modules implemented in the lead AV and the trailing AV to establish a communication path between the lead AV and the trailing AV A system according to any of the preceding claims.

Item 47. The first processor,
responsive to the leading AV and the second trailing AV being within vehicle-to-vehicle (V2V) communication range, based at least in part on a collection of the first and second sets of environmental data; generating the second updated command for the second successor AV;
46. The system of clause 45, further configured to: and communicate the second updated command to the second successor AV.

Item 48. A method, the method comprising:
Receiving a command to navigate the lead autonomous vehicle (AV) to avoid unexpected road conditions ahead of the lead AV, comprising:
the lead AV comprises at least one vehicle sensor;
the lead AV is configured to travel along roads;
receiving sensor data including position coordinates of a plurality of objects in front of the lead AV from the at least one vehicle sensor;
accessing a first set of environmental data associated with a portion of the road between the leading AV and the trailing AV;
the trailing AV is communicatively coupled to the lead AV and is traveling along the road behind the lead AV;
the first set of environmental data includes position coordinates of a second plurality of objects between the leading AV and the trailing AV;
determining whether at least one object of the first and second plurality of objects prevents performance of the command by the subsequent AV;
issuing a command to the subsequent AV based at least in part on the first set of sensor data and the environmental data in response to determining that the at least one object prevents the subsequent AV from performing the command; updating so that the updated command includes one or more navigation instructions for avoiding the at least one object during performance of the command;
and communicating the updated command to the successor AV.

Item 49. generating a second updated command by updating a command for navigating the top AV based at least in part on the sensor data, resulting in the second updated command the command includes one or more navigation instructions for avoiding the first plurality of objects while performing the command;
49. The method of Clause 48, further comprising navigating the top AV according to the second updated command.

Item 50. accessing environmental data associated with a portion of the road ahead of the lead AV, the environmental data associated with a window of time during which the lead AV travels along the road; that there is
comparing the environmental data with map data including expected road conditions ahead of the lead AV;
determining whether the environmental data includes unexpected road conditions not included in the map data based at least in part on comparing the environmental data to the map data;
in response to determining that the environmental data includes the unexpected road conditions not included in the map data;
determining location coordinates of the unexpected road condition;
50. The method of clause 48 or 49, further comprising communicating said command to said lead AV to maneuver to avoid said unexpected road condition.

Item 51. comparing the sensor data with the map data, the map data including location coordinates of expected objects on the road ahead of the lead AV;
determining whether the sensor data indicates an unexpected object not in the map data based at least in part on comparing the sensor data to the map data;
in response to determining that the sensor data indicates the unexpected object not in the map data;
determining position coordinates of the unexpected object;
51. The method of any of clauses 48-50, further comprising determining suggested navigation instructions for the lead AV to avoid the unexpected object.

Item 52. 52. The method of Clause 51, further comprising performing the suggested navigation instructions in response to determining that the sensor data indicates the unexpected object not in the map data.

Item 53. in response to determining that the sensor data indicates the unexpected object not in the map data;
communicating the suggested navigation instructions to an operations server;
determining whether confirmation has been received from the operations server to perform the suggested navigation instructions;
53. The method of clause 51 or 52, further comprising: responsive to receiving the confirmation from the operations server, performing the suggested navigation instructions.

Item 54. in response to determining that the sensor data indicates the unexpected object not in the map data;
communicating the sensor data to an operation server;
54. The method of any of clauses 51-53, further comprising requesting the operations server to provide instructions to avoid the unexpected object based at least in part on the sensor data.

Item 55. The above command is
Transition from autonomous operation to manual operation by the above leading AV,
Avoidance of obstacles on the road in front of the leading AV by the leading AV;
Avoidance of one or more fixed lanes with one or more obstacles detected by the leading AV;
Avoidance of one or more fixed routes with said unanticipated road conditions detected by said leading AV;
certain route changes by the lead AV and driving by the lead AV at speeds slower or faster than indicated in driving instructions associated with the lead AV;
55. The method of any of paragraphs 48-54, associated with at least one of

Item 56. A computer program comprising executable instructions which, when executed by one or more processors, cause the one or more processors to:
Receiving a command to navigate the lead autonomous vehicle (AV) to avoid unexpected road conditions ahead of the lead AV, comprising:
the lead AV comprises at least one vehicle sensor;
the lead AV is configured to travel along roads;
receiving sensor data from the at least one vehicle sensor including position coordinates of a first plurality of objects in front of the lead AV;
accessing a first set of environmental data associated with a portion of the road between the leading AV and the trailing AV;
the trailing AV is communicatively coupled to the lead AV and is traveling along the road behind the lead AV;
the first set of environmental data includes position coordinates of a second plurality of objects between the leading AV and the trailing AV;
determining whether at least one object of the first and second plurality of objects prevents performance of the command by the subsequent AV;
issuing a command to the subsequent AV based at least in part on the first set of sensor data and the environmental data in response to determining that the at least one object prevents the subsequent AV from performing the command; updating so that the updated command includes one or more navigation instructions for avoiding the at least one object during performance of the command;
and communicating the updated command to the successor AV.

Item 57. the command includes a broad command directed to the leading AV and one or more trailing AVs behind the leading AV on the road;
57. The computer program of clause 56, wherein the global command includes one or more navigation instructions for avoiding certain unanticipated road conditions ahead of the lead AV.

Item 58. The command includes a special command directed to the top AV,
58. Computer program according to clause 56 or 57, wherein the special command comprises one or more navigation instructions for avoiding certain unexpected road conditions ahead of the lead AV.

Item 59. The above commands include setting commands,
59. The computer program product of any of clauses 56-58, wherein the setting command includes at least one of changing the orientation of the at least one vehicle sensor and changing the data sampling frequency of the at least one vehicle sensor. .

Item 60. 60. The computer program of any of clauses 56-59, wherein the at least one vehicle sensor comprises at least one of a camera, a light detection and ranging (LiDAR) sensor, a motion sensor, and an infrared sensor.

Claims (20)

A system, the system comprising:
an autonomous vehicle (AV) comprising at least one vehicle sensor, the AV configured to travel along a road;
an operations server communicatively coupled to the AV;
The operations server comprises a processor, the processor comprising:
accessing environmental data associated with the road ahead of the AV, wherein the environmental data is associated with a window of time during which the AV travels along the road; ,
comparing the environmental data with map data including expected road conditions in front of the AV;
determining whether the environmental data includes unexpected road conditions not included in the map data based at least in part on comparing the environmental data to the map data;
in response to determining that the environmental data includes the unexpected road conditions not included in the map data;
determining location coordinates of the unexpected road condition;
and communicating commands to the AV to steer around the unexpected road condition. further comprising a controller associated with the AV, the controller comprising a second processor, the second processor comprising:
receiving sensor data from the at least one vehicle sensor including position coordinates of a plurality of objects on the road in front of the AV;
Based at least in part on the sensor data, updating the command such that the updated command includes one or more instructions for the AV to avoid the plurality of objects during performance of the command. and
2. The system of claim 1, wherein the system is configured to: navigate the AV according to the updated command. 3. The system of claim 2, wherein the second processor is further configured to communicate the updated command to one or more subsequent AVs moving in line behind the AV. the second processor,
comparing the sensor data with the map data, the map data including position coordinates of expected objects on the road in front of the AV;
determining, based at least in part on comparing the sensor data to the map data, whether the sensor data indicates that the unexpected object is absent from the map data;
Responsive to determining that the sensor data indicates that the unexpected object is not in the map data, the AV determines suggested navigation instructions for avoiding the unexpected object. 3. The system of claim 2, further configured to: The second processor is further configured to perform the suggested navigation instructions in response to determining that the sensor data indicates that the unexpected object is absent from the map data. 3. The system of claim 2, wherein: the second processor,
in response to determining that the sensor data indicates that the unexpected object is not in the map data;
communicating the suggested navigation instructions to the operations server;
determining whether confirmation has been received from the operations server to perform the suggested navigation instructions;
3. The system of claim 2, further configured to perform the suggested navigation instructions in response to receiving the confirmation from the operations server. the second processor,
in response to determining that the sensor data indicates that the unexpected object is not in the map data;
communicating the sensor data to the operations server;
3. The system of claim 2, further configured to: request the operations server to provide instructions to avoid the unexpected object based at least in part on the sensor data. . further comprising one or more subsequent AVs different from said AV;
the one or more trailing AVs are on the road behind the AVs;
the one or more subsequent AVs are communicatively coupled to the operations server;
The processor
in response to determining that the environmental data includes the unexpected road conditions not included in the map data;
receiving the sensor data from the controller;
generating a second command including one or more instructions for avoiding the unexpected road condition while the one or more subsequent AVs are avoiding the plurality of objects; the second command includes the position coordinates of the unexpected road condition indicated in the command and the position coordinates of the plurality of objects indicated in the sensor data;
and communicating the second command to the one or more subsequent AVs. A method, the method comprising:
accessing environmental data associated with a road ahead of an autonomous vehicle (AV), the environmental data associated with a window of time during which the AV travels along the road; and
comparing the environmental data with map data including expected road conditions in front of the AV;
determining whether the environmental data includes unexpected road conditions not included in the map data based at least in part on comparing the environmental data to the map data;
in response to determining that the environmental data includes the unexpected road conditions not included in the map data;
determining location coordinates of the unexpected road condition;
and communicating commands to the AV to steer around the unexpected road condition. The unexpected road conditions include at least one of unanticipated weather conditions, unanticipated traffic congestion, unanticipated road closures, and unanticipated construction areas not included in the map data. 10. The method of claim 9. 10. The method of claim 9, wherein the environmental data includes at least one of weather data and traffic data. Said command is
Transition from autonomous operation to manual operation by the AV;
avoidance of obstacles on the road in front of the AV by the AV;
avoidance by the AV of one or more certain lanes in which one or more obstacles are detected;
avoidance by the AV of one or more fixed routes where the unexpected road conditions are detected;
certain route changes by the AV, and driving by the lead AV at speeds slower or faster than indicated in driving instructions associated with the lead AV;
10. The method of claim 9, associated with at least one of said command includes a broad command directed to said AV and one or more subsequent AVs behind said AV on said road;
10. The method of claim 9, wherein the global command includes one or more navigation instructions for avoiding certain unanticipated road conditions ahead of the AV. said command includes a special command directed to said AV;
10. The method of claim 9, wherein the special commands include one or more navigation instructions for avoiding certain unexpected road conditions ahead of the AV. A computer program comprising executable instructions stored on a non-transitory computer readable medium, the instructions being executed by one or more processors to cause the one or more processors to:
accessing environmental data associated with a road ahead of an autonomous vehicle (AV), the environmental data associated with a window of time during which the AV travels along the road; and
comparing the environmental data with map data including expected road conditions in front of the AV;
determining whether the environmental data includes unexpected road conditions not included in the map data based at least in part on comparing the environmental data to the map data;
in response to determining that the environmental data includes the unexpected road conditions not included in the map data;
determining location coordinates of the unexpected road condition;
and communicating commands to the AV to steer around the unexpected road condition. The instructions, when executed by the one or more processors, cause the one or more processors to respond to determining that the environmental data includes the unexpected road conditions not included in the map data. 16. The computer program of claim 15, further causing the command to propagate to one or more subsequent AVs traveling along the road behind the AV. 16. A computer program as recited in claim 15, wherein said commands comprise configuration commands. 16. The computer program product of claim 15, wherein the setting command includes at least one of changing the orientation of the at least one vehicle sensor and changing the data sampling frequency of the at least one vehicle sensor. 16. The AV comprises at least one vehicle sensor, the at least one vehicle sensor including at least one of a camera, a light detection and ranging (LiDAR) sensor, a motion sensor, and an infrared sensor. the computer program described in . 16. The computer program product of claim 15, wherein the AV is a tractor unit attached to a trailer.

Images