JPWO2020185665A5 - - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of US Application No. 16/351,124, filed March 12, 2019, the disclosure of which is incorporated herein by reference.

Autonomous vehicles, such as vehicles that do not require a human driver, may be used to help transport passengers or goods from one location to another. Such vehicles may operate in a fully autonomous mode in which the passenger may provide some initial input such as a pick-up location or destination, and the vehicle steers itself to that location.

A person (or user) may use any number of taxi services when they wish to be physically transported between two locations via a vehicle. To date, these services typically involve a human driver who is given dispatch instructions to the location to pick up and drop off the user. Generally, these locations require physical cues (i.e., raising your hand to stop the driver), a phone call explaining where the user is actually located, or a direct discussion between the driver and the user. understand through. While these services are convenient, they typically cannot provide users with precise information as to where pick-up or drop-off will occur.

Aspects of the present disclosure provide a method for facilitating communication from an autonomous vehicle to a user. The method comprises inputting the vehicle's current location and map information into a model by one or more processors of the vehicle before the user enters the autonomous vehicle while attempting to pick the user up; , identifying a type of transmission action for communicating the location of the vehicle to the user; enabling, by one or more processors, a first transmission based on the type of transmission action; determining, by one or more processors, whether the user is moving toward the vehicle from the received sensor data after enabling .

In one example, the type of transmission action is automatically generating an audible transmission by the vehicle, and enabling the first transmission includes instructing the vehicle to make the audible transmission . In this example, the first transmission is to honk the vehicle. In another example, the type of communication action is to automatically surface an option on the user's client computing device to allow the user to cause the vehicle to generate an audible communication . In another example, the type of communication action is automatically generating a visual communication with the vehicle, and enabling the first communication includes the vehicle performing the visual communication . In this example, the first transmission is to flash the vehicle's headlights. In another example, a type of communication action is to automatically surface an option on the user's client computing device to allow the user to cause the vehicle to generate a visual communication . In another example, the received sensor data includes location information generated by the user's client computing device. In another example, the received sensor data includes data generated by the vehicle's sensory system, the sensory system including at least one sensor. In another example, the method also includes using the model of the escalated transfer to determine a type of transfer action for the second transfer ; and enabling a second transmission based on the determination of whether the second transmission is moving, wherein the type of transmission action for the second transmission is further used to enable the second transmission . be done. In this example, the type of communication action for the second communication is to automatically surface an option on the user's client computing device to allow the user to cause the vehicle to generate an audible communication . is. Alternatively, the type of communication action for the second communication may automatically surface an option on the user's client computing device to allow the user to have the vehicle generate the visual communication . is. In another example, the first transmission includes the vehicle automatically flashing its lights and the second transmission includes the vehicle automatically honking its horn. In another example, the first transmission includes the vehicle automatically honking the vehicle's horn, and the second transmission automatically causes the vehicle to contact a customer service representative to the user's client computing device. including requesting In another example, the model is a machine learning model.

Another aspect of the disclosure provides a method of training a model to facilitate communication from an autonomous vehicle to a user. The method comprises, by one or more computing devices, a first training input indicative of a vehicle location, a second training input indicative of map information, a third training input indicative of a user's location; receiving training data including a fourth training input characterizing sensor data identifying one or more objects and a target output indicating a type of transfer ; one or more computing devices to train a model to produce a set of output values that indicate a level of adequacy for the type of transmission , and using the target output and the set of output values to determine a difference value and using the difference value to adjust the current value of the parameter of the model.

In one example, the training data corresponds to a request by a user to have the vehicle perform a type of transmission to communicate with the user. In another example, the type of transmission is auditory transmission . In another example, the type of communication is visual communication . In another example, the training data further includes ambient lighting conditions.

1 is a functional diagram of an exemplary vehicle, in accordance with an exemplary embodiment; FIG. 1 is an example of map information according to aspects of the present disclosure; 1 is an exemplary exterior view of a vehicle in accordance with aspects of the present disclosure; FIG. 1 is an illustration of an exemplary system in accordance with aspects of the present disclosure; FIG. 5 is a functional diagram of the system of FIG. 4, in accordance with aspects of the present disclosure; FIG. 1 is an example of a client computing device and displayed options in accordance with aspects of the present disclosure; 1 is an example of a client computing device and displayed options in accordance with aspects of the present disclosure; 1 is an example of a client computing device and displayed options in accordance with aspects of the present disclosure; 4 is an example of map information in accordance with aspects of the present disclosure; 4 is an example of map information in accordance with aspects of the present disclosure; FIG. 4 is an exemplary flow diagram in accordance with aspects of the present disclosure;

Overview This technology facilitates the pick-up and drop-off of passengers (or users) or cargo in autonomous vehicles using auditory and/or visual communication , or indeed any situation where pedestrians need to reach the vehicle. related to making In many situations, autonomous vehicles do not have human drivers who can communicate with humans to help them find the vehicle or the correct drop-off location (ie, for pickup). As such, autonomous vehicles can use various auditory and/or visual transmissions to proactively attempt to communicate with humans in a useful and effective manner. For example, a model may be generated to allow the vehicle to determine when to provide an audible and/or visual communication to a person and/or whether the person surfaces options for doing so. .

To generate the model, the person may be provided with the option to have the vehicle provide auditory communication , for example, via an application on the person's computing device (e.g., cell phone or other client computing device). . This data may be recorded when the person uses the option. Each time an option is used, a message may be provided to the vehicle that causes the vehicle to communicate . This message may include information such as the date and time the request was generated, the type of transmission being made, and the location of the person. This message and other information may also be sent to the server computing system, for example, by the vehicle and/or the client computing device.

The messages and other information may then be processed by the server computing device to generate models for enabling the vehicle's computing devices to better communicate with people. For example, a model may be trained to indicate whether a particular type of transmission is appropriate. If so, the type of transmission may be made available as an option of an application on the person's client computing device and/or automatically generated by the vehicle's computing device.

To train the model, the person's location, other information, and map information can be used as training input, and the type of transmission (from the message) can be used as training output. The more training data that is used to train the model, the more accurate the model will be in determining when to provide transmissions or options to provide transmissions . The model can be trained to distinguish between situations in which visual communication is appropriate and situations in which auditory communication is appropriate.

In some cases, a model can be trained for a specific purpose, depending on the amount of training data available. For example, a model may be trained for a particular person or group of people based on characteristics of the person or group's history of using the service.

The trained model can then be provided to vehicles to enable one or more of the vehicle's computing devices to better communicate with humans. As the vehicle approaches or waits at a pickup or drop-off location, the vehicle's computing device 110 uses the model to determine if the transfer is appropriate, and if so, what type. can decide. This can be done, for example, depending on whether a person (or possible passenger) has a clear line of sight to the vehicle, or vice versa, based on the environment of the vehicle.

In one aspect, the model may be used to determine if the options described above should be surfaced in the application. Additionally, if the output of the model indicates that visual communication is more appropriate than auditory communication , the options surfaced may only allow visual communication . Similarly, if the model's output indicates that auditory communication is more appropriate than visual communication , the options surfaced may only allow visual communication . In another aspect, rather than providing the user with options for auditory or visual communication , the model is used to determine whether the vehicle should automatically perform auditory or visual communication . can be done. Additionally or alternatively, the output of the model may be used to determine initial actions, and subsequent actions may be automatically performed in response to the initial actions.

User responses to subsequent actions may be used to build a model of escalated communication . For example, results may be tracked for each situation in which a subsequent action was used. This information can then be analyzed to identify patterns that increase the likelihood that the user will enter the vehicle more quickly in response to the vehicle transmission . A model of escalated communication can be trained to determine what the next action should be to make it easiest for the user to reach the vehicle, based on previous or initial actions. Again, the more training data used to train the model, the more accurately the model determines how to escalate from previous actions. Similar to the first model, a trained model of escalated transmission is used in one or more vehicles to enable the computing devices in those vehicles to better communicate with people, including potential passengers. can be provided to

The features described herein may enable autonomous vehicles to improve passenger pick-up and drop-off. For example, the user may have the vehicle visually and/or audibly communicate with the user, either by themselves or by prompting them to use surfaced options. This allows the user to more easily identify the position of the vehicle. Additionally or alternatively, the vehicle may use the model to pre-determine if and how to communicate with the user and how to escalate those communications over time.

Exemplary System As shown in FIG. 1, a vehicle 100 according to one aspect of the present disclosure includes various components. Some aspects of the present disclosure are particularly useful in connection with certain types of vehicles, although vehicles of any type include, but are not limited to, automobiles, trucks, motorcycles, buses, recreational vehicles, and the like. may be A vehicle may have one or more computing devices, such as computing device 110, which includes one or more processors 120, memory 130, and other components typically found in general purpose computing devices.

Memory 130 stores information accessible by one or more processors 120 , including instructions 134 and data 132 that may be executed or otherwise used by processors 120 . Memory 130 may be any type of memory capable of storing information that is accessible by the processor, including computing device readable media, or hard drives, memory cards, ROM, RAM, DVD, or other media that store data that can be read with electronic devices such as other optical discs and other writable and read-only memories. Systems and methods may include different combinations of the above, whereby different portions of instructions and data are stored on different types of media.

Instructions 134 may be any sequence of instructions that are executed directly (such as machine code) or indirectly (such as scripts) by a processor. For example, the instructions may be stored as computing device code on a computing device readable medium. In that regard, the terms "instructions" and "programs" may be used interchangeably herein. The instructions may be in object code form for direct processing by the processor, or any other computing device language, including scripts or collections of independent source code modules that are interpreted on demand or precompiled. may be stored in The functions, methods and routines of the instructions are described in further detail below.

Data 132 may be retrieved, stored, or modified by processor 120 according to instructions 134 . For example, although claimed subject matter is not limited to any particular data structure, data may reside in computing device registers, tables with multiple different fields and records, XML documents, or as flat files in relational databases. may be stored. The data may also be formatted in any computing device readable format.

The one or more processors 120 may be any conventional processor such as a commercially available CPU or GPU. Alternatively, one or more processors may be dedicated devices such as ASICs or other hardware-based processors. Although FIG. 1 functionally illustrates the processor, memory, and other elements of computing device 110 as being in the same block, the processor, computing device, or memory may actually be in the same physical block. It will be appreciated by those skilled in the art that it may include multiple processors, computing devices, or memories that may or may not be housed within a single enclosure. For example, the memory may be a hard drive or other storage medium located within a different housing than the housing of computing device 110 . Thus, reference to a processor or computing device will be understood to include reference to a collection of processors or computing devices or memories that may or may not operate in parallel.

Computing device 110 includes a processor and memory as described above, as well as user input devices 150 (e.g., mouse, keyboard, touch screen, and/or microphone) and various electronic displays (e.g., monitor having a screen or display of information). may include all components normally used in connection with a computing device, such as any other electrical device operable to In this example, the vehicle includes an electronic display 152 as well as one or more speakers 154 to provide information or an audiovisual experience. In this regard, electronic display 152 may be located within vehicle 100 and may be used by computing device 110 to provide information to passengers within vehicle 100 . In some cases, electronic display 152 may be an internal display that is visible to a person outside the vehicle through a vehicle window or other transparent vehicle enclosure and/or projects an image through a window or other transparent vehicle enclosure. and may provide information to persons outside the vehicle. Alternatively, the electronic display 152 can be an externally mounted display that can project information to passengers inside the vehicle (i.e., under the roof pod that can be viewed through the glass roof) and/or provide information to persons outside the vehicle. It may be an externally mounted display that

Computing device 110 also includes one or more wireless network connections 156 to facilitate communication with other computing devices, such as the client and server computing devices described in detail below. . Wireless network connections include short-range communication protocols such as Bluetooth, Bluetooth low energy (LE), cellular connections, as well as the Internet, World Wide Web, intranets, virtual private networks, wide area networks, local networks, one or more businesses. Various configurations and protocols including private networks using proprietary communication protocols, Ethernet, WiFi, and HTTP, and various combinations of the above may be included.

In one example, computing device 110 may be part of a communication system of an autonomous driving computing system embedded in vehicle 100 . In this regard, a transmission system may be included or configured to transmit a signal that causes an audible transmission to be played through speaker 154 . The transmission system also provides visual indication, for example, by flashing or otherwise controlling the vehicle's headlights 350, 352 (shown in FIG. 3), or by displaying information on the internal electronic display 152. It may be configured to send a signal that causes the communication to occur.

Autonomous control system 176 may include various computing devices, configured similarly to computing device 110, that can communicate with various components of the vehicle to control the vehicle in autonomous modes of operation. For example, returning to FIG. 1, the autonomous control system 176 controls the deceleration system 160, the acceleration system 162, the steering system, etc. to control the movement, speed, etc., of the vehicle 100 according to the instructions 134 in the memory 130 in the autonomous driving mode. 164 , routing system 166 , planner system 168 , positioning system 170 , and perception system 172 .

As an example, a computing device of autonomous control system 176 may interact with deceleration system 160 and acceleration system 162 to control the speed of the vehicle. Similarly, steering system 164 may be used by autonomous control system 176 to control the direction of vehicle 100 . For example, if the vehicle 100 is configured for road use, such as a car or truck, the steering system may include components that control the angle of the wheels to turn the vehicle. Autonomous control system 176 may also use the signaling system to signal the vehicle's intent to other drivers or vehicles, for example, by illuminating turn signals or brake lights as appropriate. .

Routing system 166 may be used by autonomous control system 176 to generate routes to destinations. Planner system 168 may be used by computing device 110 to follow a route. In this regard, planner system 168 and/or routing system 166 may provide detailed map information such as road geometry and elevation, lane markings, intersections, crosswalks, speed limits, traffic lights, buildings, signs, real-time traffic information, and more. , curbside spots, vegetation, or other such objects and information may be stored.

FIG. 2 is an example of map information 200 for a section of roadway including an intersection 202 adjacent to a parking lot 210 for a building 220 . Map information 200 may be a local version of the map information stored in memory 130 of computing device 110 . Other versions of map information may also be stored in storage system 450, discussed further below. In this example, the map information 200 describes the shape, location, and other characteristics of lane markings 230, 232, 234, 236, lanes 240, 242, 244, 246, stop signs 250, 252, 254, 256, etc. Contains identifying information. In this example, map information 200 also includes information identifying features of parking lot 210 and building 220 . For example, as parking spaces 260, 262, 264, 266, 268, drivable areas 270, 272, 274, 276, etc. Additionally, in this example, the map information identifies entrances and exits 282 , 284 , 286 of building 220 . Although only a few features are shown in the map information 200 of FIG. 2, the map information 200 may contain significantly more features and details in order to allow the vehicle 100 to be controlled in an autonomous mode.

Although the map information is depicted herein as an image-based map, the map information need not be entirely image-based (eg, raster). For example, map information may include one or more road graphs or graph networks of information such as roads, lanes, intersections, and connections between these features, which may be represented by road segments. Each feature may be stored as graphical data, may be associated with information such as geographic location, whether or not linked to other related features, e.g., stop signs link to roads and intersections, etc. can be In some examples, the associated data may include a grid-based index of road graphs to enable efficient retrieval of particular road graph features.

Positioning system 170 may be used by autonomous control system 176 to determine the vehicle's relative or absolute position on a map or earth. For example, positioning system 170 may include a GPS receiver for determining the latitude, longitude, and/or altitude position of the device. Other location systems such as laser-based location systems, inertial-assisted GPS, or camera-based location can also be used to identify the location of the vehicle. A vehicle's position may include absolute geographic location information, such as latitude, longitude, and altitude, as well as relative location information, such as its position relative to other vehicles in its immediate surroundings, which is often , it can be determined with less noise than the absolute geographic position.

The positioning system 170 may also use other computing devices in communication with the autonomous control system 176, such as an accelerometer, gyroscope, or another direction/speed sensing device for determining vehicle direction and speed, or changes thereof. may include devices. By way of example only, the acceleration device may determine the pitch, yaw, or roll (or changes thereof) of the vehicle with respect to the direction of gravity or a plane perpendicular to gravity. The device can also track speed increases and decreases, and the direction of such changes. Providing device position and orientation data as described herein may be automatically provided to computing device 110, other computing devices, and combinations of the above.

Perception system 172 also includes one or more components to detect objects external to the vehicle such as other vehicles, obstacles in the roadway, traffic lights, signs, trees, and the like. For example, perception system 172 may include lasers, sonar, radar, cameras, and/or any other detection device that records data that can be processed by the computing devices of autonomous control system 176 . If the vehicle is a passenger vehicle such as a minivan, the minivan may include lasers or other sensors mounted on the roof or other convenient location. For example, FIG. 3 is an exemplary exterior view of vehicle 100 . In this example, roof top housing 310 and dome housing 312 may include LIDAR sensors as well as various camera and radar units. Additionally, housing 320 located at the front end of vehicle 100 and housings 330, 332 on the driver and passenger sides of the vehicle may each store a LIDAR sensor. For example, housing 330 is located in front of driver door 360 . Vehicle 100 also includes housings 340 , 342 for radar units and/or cameras that are also located on the roof of vehicle 100 . Additional radar units and cameras (not shown) may be located at the front and rear ends of vehicle 100 and/or on other locations along roof or roof top housing 310 .

Autonomous control system 176 may be able to communicate with various components of the vehicle to control movement of vehicle 100 according to primary vehicle control code in memory of autonomous control system 176 . For example, returning to FIG. 1, the autonomous control system 176 may operate a deceleration system 160, an acceleration system 162, a steering system 164, a routing system 166 to control the movement, speed, etc. of the vehicle 100 according to the instructions 134 in the memory 130. , planner system 168, positioning system 170, perception system 172, and power system 174 (ie, the vehicle's engine or motor).

Various systems of the vehicle may function using autonomous vehicle control software to determine and control how to control the vehicle. As an example, the perception system software module of perception system 172 uses sensor data generated by one or more sensors of the autonomous vehicle, such as cameras, LIDAR sensors, radar units, sonar units, etc., to detect objects and their features. and can be identified. These features may include position, type, direction of travel, orientation, velocity, acceleration, change in acceleration, size, shape, and the like. In some cases, the features are input to a motion prediction system software module that uses different motion models based on object type to output the predicted future motion of the detected object. In other examples, the features include a traffic light detection system software module configured to detect known traffic light conditions, a construction zone from sensor data generated by one or more sensors on the vehicle. and one or more detection system software modules, such as a construction zone detection system software module configured to detect emergency vehicles, and an emergency vehicle detection system configured to detect emergency vehicles from sensor data generated by the vehicle's sensors. . Each of these detection system software modules may use different models to output the likelihood that a construction zone or object is an emergency vehicle. Detected objects, predicted future behavior, various possibilities from the detection system software modules, map information identifying the vehicle's environment, location information from the positioning system 170 identifying the vehicle's position and orientation, the vehicle's The destination, as well as feedback from various other systems of the vehicle, may be input into the planner system software module of planner system 168 . The planner system can use this input to generate a trajectory that the vehicle will follow over some short period of time in the future based on the route generated by the routing module of routing system 166 . A control system software module of the autonomous control system 176 may be configured to control the motion of the vehicle, for example, by controlling the braking, acceleration, and steering of the vehicle to follow a trajectory.

Autonomous control system 176 may control the vehicle in an autonomous mode by controlling various components. For example, by way of example, autonomous control system 176 may use detailed map information and data from planner system 168 to navigate the vehicle to its destination fully autonomously. The autonomous control system 176 uses the positioning system 170 to determine the position of the vehicle and uses the perception system 172 to detect and respond to objects when needed to reach that position safely. can respond with Again, to do so, the computing device 110 generates trajectories, causes the vehicle to follow these trajectories, and (eg, supplies fuel or other energy to the engine or power system 174 via the acceleration system 162). Accelerate the vehicle (e.g., by reducing fuel supplied to the engine or power system 174, change gears, and/or brake with the reduction system 160), slow the vehicle (e.g., by steering The system 164 allows the vehicle 100 to change direction (by turning the front or rear wheels of the vehicle 100) and to signal such changes (eg, by illuminating turn signals). As such, acceleration system 162 and deceleration system 160 may be part of a drivetrain that includes various components between the vehicle's engine and the vehicle's wheels. Again, by controlling these systems, the autonomous control system 176 can also control the vehicle's drivetrain to steer the vehicle autonomously.

Computing device 110 of vehicle 100 may also receive or transfer information to and from other computing devices, such as computing devices that are part of transportation services, as well as other computing devices. 4 and 5 are illustrative and functional diagrams, respectively, of an exemplary system 400 that includes multiple computing devices 410, 420, 430, 440 and storage systems connected via a network 460. 450 included. System 400 also includes vehicle 100 and vehicles 100A, 100B that may be configured the same as or similar to vehicle 100 . Only a few vehicles and computing devices are shown for simplicity, but a typical system may include many more.

As shown in FIG. 5, each of computing devices 410, 420, 430, 440 may include one or more processors, memory, data, and instructions. Such processors, memory, data, and instructions may be configured similarly to one or more of processor 120 , memory 130 , data 132 , and instructions 134 of computing device 110 .

Network 460 and intermediary nodes can be Bluetooth, Bluetooth LE, Internet, World Wide Web, intranets, virtual private networks, wide area networks, local networks, private networks using communication protocols proprietary to one or more enterprises, Ethernet, WiFi. , and HTTP, as well as various combinations of the above, including short-range communication protocols. Such communication can be facilitated by any device capable of transmitting data to and from other computing devices, such as modems and wireless interfaces.

In one example, one or more computing devices 410 exchange information with different nodes of a network for the purpose of receiving, processing, and transmitting data to other computing devices, e.g., a load balancing server farm. , etc., may include one or more server computing devices having multiple computing devices. For example, one or more computing devices 410 may communicate with computing device 110 of vehicle 100, or similar computing devices of vehicle 100A, as well as computing devices 420, 430, 440 via network 460. It can include one or more server computing devices. For example, vehicles 100, 100A may be part of a fleet of vehicles that may be dispatched to various locations by a server computing device. In this regard, server computing device 410 functions as a dispatch server computing system that can be used to dispatch vehicles, such as vehicle 100 and vehicle 100A, to different locations to pick up and drop off passengers. can. In addition, server computing device 410 may use network 460 to provide information to users, such as users 422, 432, 442, on displays such as displays 424, 434, 444 of computing devices 420, 430, 440. to send and present. In this regard, computing devices 420, 430, 440 may be considered client computing devices.

As shown in FIG. 5, each client computing device 420, 430, 440 may be a personal computing device intended for use by users 422, 432, 442 and may include one or more processors ( memory (e.g., RAM and internal hard drives) for storing data and instructions; displays such as displays 424, 434, 444 (e.g., monitors with screens, touch screens, projectors, televisions , or other device operable to display information), and user input devices 426, 436, 446 (e.g., a mouse, keyboard, touch screen, or microphone). It can have all the components normally used. A client computing device may also include cameras for recording video streams, speakers, network interface devices, and all of the components used to connect these elements together.

Client computing devices 420, 430, and 440 may each comprise a full-sized personal computing device, but may alternatively exchange data wirelessly with a server over a network such as the Internet. It may comprise a mobile computing device. By way of example only, client computing device 420 may be a device such as a mobile phone or wireless-enabled PDA, tablet PC, wearable computing device or system, or netbook capable of obtaining information over the Internet or other network. may be In another example, client computing device 430 may be a wearable computing system shown as a wristwatch, as shown in FIG. As an example, a user may enter information using a miniature keyboard, keypad, microphone, visual signals using a camera, or a touch screen.

Similar to memory 130, storage system 450 stores information accessible by server computing device 410, such as hard drives, memory cards, ROM, RAM, DVDs, CD-ROMs, writable memory, and read-only memory. It can be any type of computerized storage device capable of. Additionally, storage system 450 may include a distributed storage system in which data is stored on multiple different storage devices that may be physically located at the same or different geographic locations. Storage system 450 may be connected to computing devices via network 460 and/or directly to any of computing devices 110, 410, 420, 430, 440, etc., as shown in FIGS. can be implemented or incorporated.

Storage system 450 may store various types of information. For example, storage system 450 may also store the above-described autonomous vehicle control software used by a vehicle, such as vehicle 100, to operate the vehicle in an autonomous mode of operation. This autonomous vehicle control software stored in storage system 450 includes various disabled and verified versions of autonomous vehicle control software. Once verified, the autonomous vehicle control software may be transmitted to the memory 130 of the vehicle 100 for use by the vehicle's computing devices, for example, to control the vehicle in an autonomous mode.

Storage system 450 may store various types of information, as described in more detail below. This information may be retrieved or otherwise accessed by one or more server computing devices, such as server computing device 410, to perform some or all of the features described herein. For example, the storage system can store various models and model parameter values that can be updated through training as described further below. Storage system 450 may also store log data. This log data may include, for example, sensor data generated by a sensory system, such as sensory system 172 of vehicle 100 . A sensory system may include multiple sensors that generate sensor data. As an example, sensor data includes not only raw sensor data, but also the perception of the shape, position, orientation, speed, etc. of objects such as vehicles, pedestrians, bicycles, vegetation, curbs, lane lines, sidewalks, pedestrian crossings, and buildings. It may contain data specifying defining characteristics of objects (including other road users) that have been detected. The log data may also include "event" data that identifies various types of auditory transmissions that the vehicle produces in response to the vehicle's environment and/or demands, as described further below.

Exemplary Methods Various operations are described herein in addition to those described above and illustrated in the Figures. It should be appreciated that the following operations do not have to be performed in the exact order described below. Rather, various steps may be processed in different orders or simultaneously, and steps may also be added or omitted.

To generate and train the model, users of the service are provided the option to request that the vehicle provide transmission , for example, via an application on the user's computing device (i.e., mobile phone). obtain. In this regard, an option may be used to have the vehicle provide transmission . This data may be recorded when the user uses the option. FIG. 6 is an exemplary illustration of client computing device 420 including options 610, 620 displayed on display 424. FIG. In this example, option 610 is for the client computing device to send a request to the vehicle, e.g., over network 460 or other wireless connection, and play a corresponding sound by honking its horn or through speaker 154. This may allow the vehicle to generate an audible transmission . Option 620 allows the client computing device to send a request to the vehicle, e.g., via network 460 or other wireless connection, to e.g. Displaying information may allow the vehicle to generate a visual communication . In some examples, option 630 may be provided to allow the user not to request transmission , such as if the user is confident that they have identified their vehicle.

For example, the user can use option 620 in a dark parking lot to have the autonomous vehicle flash its headlights. As another example, in a well-lit parking lot with few other pedestrians, the user may use option 610 to have the vehicle honk its horn or provide some other audible communication . . If there are more pedestrians, the user may select option 620 over option 610 . As another example, the user may use option 610 to have the vehicle honk when in a large parking lot or near a large building. As yet another option, the user may use option 620 to cause the vehicle to flash its headlights when there are multiple autonomous vehicles nearby. Alternatively, other types of visual communication options may be provided, such as displaying information on the electronic display 152 rather than flashing the headlights.

Each time one of the options, such as options 610, 620, is used to request a transmission , a message may be provided to the vehicle to cause the vehicle's computing device 110 to make or generate the transmission . This message may include information such as the date and time the request was generated, the type of transmission being made, as well as the location of the user. This message, as well as other message information, is sent, for example, by the vehicle and/or the user's client computing device to a server computing system, such as server computing system 410, where the message can be stored in storage system 450. may be By way of example, other message information may include the location of the vehicle, the type of transmission (flashing lights, displaying information on the electronic display 152, honking, etc.), other road usage detected by the vehicle's perception system 172, etc. It may include data generated by the vehicle's computing system, such as the location and/or characteristics of persons (vehicles, pedestrians, cyclists, etc.), ambient lighting conditions, and the like.

As an example, ambient lighting conditions may be determined in any number of different ways. For example, the computing device 110 may, in some cases, be used to control the state of the vehicle's headlights and adjust the brightness of internal electronic displays, such as the electronic display 152 . Feedback from the optical sensor may be received. If feedback from light sensors is not directly available to computing device 110, this information may also be gleaned from the status of the vehicle's headlights and/or internal electronic displays. In other words, the computing device 110 may be able to determine from this information whether the vehicle is "dark enough" to turn on the headlights or internal electronic displays of a particular brightness. Additionally or alternatively, ambient lighting conditions may be determined from data generated by the vehicle's perception system. As previously mentioned, perception system 172 may include multiple different sensors, some of which may be used to determine ambient lighting conditions, such as still cameras or video cameras. For example, "live" camera images of the vehicle's environment may be analyzed to determine ambient lighting conditions. This may involve processing the pixels to determine if the area the camera is pointing at is a bright area. If the pixel is bright and the image has a short exposure time, this may indicate that the area is also bright. As another example, ambient lighting conditions can be determined in real-time by using camera exposure values. As an example, when capturing an image, the cameras of perception system 172 can automatically recalibrate exposure values given ambient lighting conditions. In this regard, the exposure value can be considered a proxy for the brightness of the area currently visible by the vehicle's camera. For example, real-time exposure values may be used to determine ambient lighting conditions. The longer the exposure value, the darker the scene, or rather the lower the ambient lighting conditions. Similarly, the shorter the exposure value, the brighter the scene, or rather the higher the ambient lighting conditions. In addition, it is possible to look at exposure values during periods when the sun is not out (i.e., from dusk to dawn on any given day of the year) and identify those with shorter exposure times that indicate brighter artificial illumination. can.

Messages containing sensor data and other message information may then be processed by server computing device 410 to generate and train models. The models can be machine learning models such as decision trees (such as random forest decision trees), deep neural networks, logistic regression, neural networks, and the like. To train the model, the user's location, other message information (including sensor data generated by the various vehicle perception systems 172 that generated the messages), and map information can be used as training inputs, The type of transmission (from messages) can be used as training output.

Thus, training may include receiving training data, including training outputs or target outputs, as well as various training inputs. A model can be trained on training data using the current values of the model's parameters to produce a set of output values. These output values may indicate the level of adequacy of the type of transfer or other output data determined using the model. A target output and a set of output values can be compared to each other to determine one or more difference values that indicate how far these values are from each other. Based on the one or more difference values, the current values of the parameters of the model can be adjusted. Repeated training and tuning can improve the accuracy of the model. Therefore, the more training data that is used to train the model, the more the model can determine whether and which transmission types to automatically provide, or which types of transmission options to provide or enable as described further below. be more accurate in determining In addition, by using map information as training data, the model is trained to determine what type of environment, e.g., road or area (e.g., residential or commercial) the vehicle and/or pedestrian is located. The type of communication or how the user's desire for communication is influenced in determining whether to provide or enable communication options can be incorporated.

Additionally, by using time of day and/or ambient lighting conditions as training data, the model can be trained to distinguish between different times of day and lighting conditions for different types of transmissions output by the model. For example, the time of day as well as ambient lighting conditions can be used as training inputs. Again, the more training data used to train the model, the more accurate the model can be in determining when and what types of transmissions to enable and/or provide.

In addition, through feedback and hand training, various transmission options can be used to reduce the likelihood of false positives or to indicate that the vehicle should generate transmissions at inappropriate or inappropriate times. Weights can be assigned (or generated). Since such weights are likely to be highly dependent on environmental factors (such as map information and sensor data), such inputs can affect the corresponding weighted factors of the model. For example, when training the model, the model should avoid honking when the pedestrian is within a short distance, such as 1-2 meters, from the vehicle, as this may cause pedestrian discomfort. can be weighted more.

Similarly, the model can be trained to distinguish between situations in which visual communication is appropriate and cases in which auditory communication is appropriate. For example, it may not be appropriate (or more appropriate) to honk the horn (or play a corresponding auditory transmission via speaker 154) in a crowded area, and using lights during the day may be It may not be appropriate (or more appropriate). In this regard, user feedback regarding the effectiveness or usefulness of various transmissions can also be used to train the model. As an example, the user may ask whether a particular transmission was inappropriate or inconvenient, and why (e.g., whether a person was standing in front of the vehicle when the headlights flashed, causing pain to the person's eyes). information displayed on the horn or electronic display has disturbed others in the vehicle's environment, etc.). In addition, based on laws and regulations, audible communication may or may not be more appropriate (e.g., it is illegal to flash your headlights or honk your horn in certain areas). sometimes). Such instances of inappropriate, ineffective, inconvenient, or less helpful communication are generated and/or labeled (e.g., manually by a human operator) as inappropriate, and/or can be used as training data. Thus, as above, the model is trained to output whether the transmission is appropriate and, if so, the type of transmission , or rather whether the transmission is auditory or visual. can be As an example, the model may identify a list of possible transfer types and a corresponding level of appropriateness for each. In some cases, users may provide positive and/or negative feedback about their experience. This information can also be used to help train the model to select the transmissions that the user finds most useful.

In some cases, a model can be trained for a specific purpose, depending on the amount of training data available. For example, a model may be trained for a particular user or type of user based on the type of user or user history picked up at a particular location. In this way, the model may allow the vehicle to provide advance notice to the user in situations where the vehicle should deviate from its user's typical pick-up location. For example, a user is normally picked up in one corner of a building, but there is an obstacle (construction, parked vehicle, fallen tree debris, etc.) causing the vehicle to move to another location, such as another corner of the building. If the vehicle is forced to do so, the model will inform the user in advance through visual and/or auditory communication (honking, flashing lights, etc., or by displaying information on the electronic display 152). It can be trained to be notified to get the user's attention when they leave the building. In this way the vehicle can respond as needed. In some cases, the model may be trained to further notify the user via an application on the user's client computing device in conjunction with visual and/or auditory communication .

The trained model, or rather the model and parameter values, are then provided to one or more vehicles, such as vehicles 100, 100A, to enable the computing devices 110 of these vehicles to better communicate with humans. be able to. When the vehicle is approaching or waiting for a pick-up location (or dropping off goods). The vehicle's computing device 110 can use the model to determine if transmission is appropriate, and if so, what type. This may be done, for example, based on the environment of the vehicle and/or whether the user (or possible passenger) has a clear line of sight to the vehicle, or vice versa.

In one aspect, the model and parameter values can be used to determine whether options such as those described above should be surfaced to the application. For example, sensor data generated by the vehicle's perception system, local map information for the area around the vehicle, and the vehicle's current position can be input to the model. Map information may, for example, indicate the distance to the nearest curbs, stairs, entrances or exits, and/or whether the vehicle is close to another object such as a wall or tree that may block the vehicle's user's view. may include various related information such as This determination may be made, for example, if the vehicle's computing device finds a place for the vehicle to stop and wait for the user, pulls to that location, and/or if the vehicle is already parked (i.e. already parked). can be The model can then output values indicating whether the transfer is appropriate and the level of appropriateness for each type of transfer .

In one example, if the output of the model indicates that visual communication is more appropriate than auditory communication , the options surfaced may only allow visual communication . In other words, if the value indicative of adequacy of visual communication is greater than the value indicative of auditory communication , the surfaced option may only allow visual communication . For example, looking to FIG. 7, option 620 for providing visual communication is not available, but option 610 is available for providing audible communication .

Similarly, if the model's output indicates that auditory communication is more appropriate than visual communication , the options surfaced may only allow visual communication . Again, in other words, if the value indicating adequacy of auditory communication is greater than the value indicating visual communication , the surfaced option may only allow visual communication . For example, looking at FIG. 8, option 610 for providing audible communication is not available, but option 620 for providing visual communication is available.

As an example, turning to FIG. 9, which corresponds to the map information 200 of FIG. , options 610 or 620 to indicate a tendency to request audio or visual communication . Thus, when the trained model is used, users exit building 220, for example, via entrances and exits 286, and are tracked by their client computing device's GPS (and possibly the vehicle's perception system). When on a trajectory toward or near area 910 (as confirmed by pedestrian detection by 172), the application may communicate (e.g., , visual, audible, or both) can be automatically surfaced.

In another aspect, the model can be used to determine whether the vehicle should automatically make audible transmissions , rather than just surfacing options as described above. Again, this determination can be made, for example, by the vehicle's computing device finding a place where the vehicle is parked and waiting for the user, pulling over to that location, and/or if the vehicle is already parked (i.e. already parked). ) can be executed. For example, turning to FIG. 10, the training data indicates that when the user exits building 220, e.g., via entrance and exit 282, stands near area 1010 and user option 610 and displays the indicates a tendency to honk the horn (or produce a corresponding audible transmission via speaker 154). Thus, when the trained model is used, users leave the building 220 and are tracked by their client computing device's GPS (and possibly confirmed by pedestrian detection by the vehicle's perception system). ), the vehicle's computing device 110 may automatically honk its horn (or generate a corresponding audible communication via speaker 154) when on track toward or near area 1010. can. In another example, the training data indicates that when the user exits the building 220, e.g., via the entrance and exit 286, the user stands near the area 1020 and uses option 620 to turn on the headlights to the vehicle's computing device. It may indicate a tendency to blink 350,352. Thus, when the trained model is used, the user leaves building 220 via entrance and exit 286 and is on track towards or near area 1020 with many pedestrians around. If there are many pedestrians around, the vehicle's computing devices as tracked by their client computing device's GPS (and possibly confirmed by pedestrian detection by the vehicle's perception system) 110 can automatically flash the headlights 350,352.

In some cases, the vehicle's computing device may use information from the user's account information and/or other input from the user to determine the appropriate type of transmission . For example, if the user's account information or other input indicates that the user has a disability (visual or auditory related), this can be used by the computing device to "override" the model's output and/or Or it may be an input to a mode. For example, blind people may benefit more from auditory communication . However, if there are many other people around, the system may prefer to give instructions through the user's device rather than honking. Similarly , deaf people may benefit more from visual than auditory communication . Also, there may be higher thresholds for various parameters related to the distance that the user must cover to reach the vehicle. For example, a vehicle's computing device should avoid directing or encouraging a visually impaired person to cross a road or other pedestrian-unfriendly area (heavy traffic) to reach the vehicle.

Additionally or alternatively, the output of the model can be used to determine and perform initial actions, and subsequent actions can be performed automatically in response to the initial actions. Again, this determination can be made, for example, by the vehicle's computing device finding a place where the vehicle is parked and waiting for the user, pulling over to that location, and/or if the vehicle is already parked (i.e. already parked). ) can be executed. For example, when the user exits the building 220 and approaches the area 1020, the vehicle's computing device 110 may automatically cause the headlights 350, 352 to flash. With no immediate change in the user's trajectory (e.g., toward the vehicle), options such as option 610 are surfaced via the user's client computing device, and the user is directed to the vehicle's computing device, as in the example of FIG. may allow the driving device 110 to honk its horn (or generate a corresponding audible transmission via the speaker 154). As another example, when the user exits the building 220 and approaches the area 1020, the vehicle's computing device 110 may automatically flash the headlights 350, 352. FIG. If there is no immediate change in the user's trajectory (eg, towards the vehicle), the vehicle's computing device 110 automatically honks the vehicle's horn (or generates a corresponding auditory communication via speaker 154). )be able to. In some cases, in addition to automatically honking, options such as option 610 also surface to allow the user to cause the vehicle to honk the vehicle (or via speaker 154), as shown in the example of FIG. to generate a corresponding auditory transmission ). Alternatively, rather than surface the option, a notification may be displayed informing the user that the vehicle is honking its horn. At least initially, these subsequent actions can be chosen randomly or by using human-tuned heuristics. In some cases, these heuristics may involve responding to specific audio cues or other information about the vehicle's environment (such as other message information). For example, when there is a lot of ambient noise, a loud voice transmission may serve as a first action.

User responses to subsequent actions can be used to build models of escalated communication . Models of escalated transfer can be machine learning models such as decision trees (such as random forest decision trees), deep neural networks, logistic regression, neural networks, etc. For example, results may be tracked for each situation where a subsequent action was used. This information is then analyzed, for example, by server computing device 410 to train a model of the escalated transition , thereby increasing the likelihood that the user will enter the vehicle more quickly in response to the vehicle transition . Patterns can be identified. For example, this analysis may include both "time to board" + "time/consistency of trajectory change" depending on the transmission . As an example, when a typical (or average) user exits building 220, it takes N seconds to board. However, if the vehicle's computing device 110 provides an audible transmission , it is reduced, for example, to N/2, resulting in significantly improved discoverability of the vehicle. The same is true for trajectory changes. If the user is generally leaving the vehicle at the building 220, but eventually finds the vehicle in the average case, then ideally the trajectory that the user heads toward the vehicle after the vehicle has provided audible communication . time to fix can be greatly improved.

It then trains a model of escalated communication to determine what the next action should be to make it easiest for the user to reach the vehicle, based on previous or initial actions. be able to. As an example, a model can be trained using the user's orientation. For example, training inputs to the model include the actual time for the user to reach and/or board the vehicle, what actions the user employed over time, and the user's original orientation. obtain. These combinations indicate whether escalated transmissions , e.g., user-initiated second or third transmissions , reduced boarding time by correcting the user's orientation when the user triggered the action. can show So if the user exits the building heading north (where the vehicle is actually in the opposite direction, south here), use the previously mentioned options to have the vehicle honk its horn, then head toward the vehicle. By changing , the model can be trained to cause the vehicle to honk its horn early when following a user heading north out of an exit. Similarly, if the first action does not force the user to change orientation, the model of escalated communication can be used to, if necessary, base the second communication on the user's reaction (e.g., change orientation, etc.). , a third transmission , etc. can be determined. Again, the more training data that is used to train the model, the more accurate the model will be in determining how to escalate from previous actions.

As an example, a model of escalated transmission is that for a user exiting building 220 and standing in area 1010, the vehicle first flashes the lights without any response from the user, and then the vehicle automatically turns on the vehicle. horn (or produce a corresponding auditory transmission via speaker 154). If there is no immediate change in the user's trajectory (e.g., towards the vehicle), the vehicle's computing device 110 may call a customer service representative, e.g., a customer service representative such as user 442 using computing device 440. can be called automatically. Personnel are sensor data generated by and received from the vehicle's perception system 172, the vehicle's position generated by and received from the vehicle's positioning system 170, as well as the user's client computing device, and The user's location received therefrom may be used to communicate with the user and direct the user to the vehicle.

As another example, a model of escalated transmission is that for a user exiting building 220 and standing in area 1020, the vehicle then waits each time to see if there is a change in the user's trajectory. can be trained to automatically honk the vehicle's horn three times while As another example, the model of escalated communication is that instead of surfacing options for users leaving building E at night, the vehicle's computing device 110 always automatically calls a customer service representative. can be trained to

Similar to the first model, the trained model of escalated communication is then applied to one of the vehicles 100, 100A, etc. to enable the computing devices 110 of those vehicles to better communicate with people. can be provided for more than one vehicle.

In addition to using messages and other information to train models, data can be analyzed to make pick-ups and drop-offs easier. For example, if the user is generally located in area 1010 for pickup and would normally use the option to activate the vehicle horn when the vehicle is at 1020, this could be used to move the vehicle near the area 1010 location. can be stopped by

FIG. 11 illustrates an example according to aspects of the disclosure that may be executed by one or more processors of one or more computing devices, such as processor 120 of computing device 110, to facilitate communication from an autonomous vehicle to a user. 11 is a typical flow diagram 1100. FIG.

While attempting to pick up the user by the vehicle, as shown in block 1110, before the user enters the vehicle, the vehicle's current location and Map information is input to the model. This may include models and/or models of escalated communication as described above. Thus, as above, the model can output whether the transmission is appropriate, and if so, the type of transmission , or rather whether the transmission is auditory or visual. .

At block 1120, a first propagation is enabled based on the type of propagation action. This activation may include, for example, surfacing options as described above and/or automatically generating an audible or visual communication as described above.

At block 1130, it is determined from the received sensor data whether the user is moving toward the vehicle after enabling the first transmission . In other words, it can be determined whether the user has responded to the first transmission . This sensor data may include sensor data generated by the vehicle's perception system 172 and/or sensor data from the user's client computing device. From this sensor data, the vehicle's computing device determines, for example, whether the user is moving toward the vehicle, is facing toward the vehicle, and/or has changed orientation to move toward the vehicle. can decide whether

At block 1140, a second transmission is enabled based on a determination of whether the user is moving toward the vehicle. As an example, the second transmission may indicate that the user is not moving toward the vehicle or changes the orientation or orientation of the user to move toward the vehicle in response to activation of the first transmission . can be enabled when not Activation may include, for example, surfacing options as described above and/or automatically generating an audible or visual communication as described above.

The features described herein may enable autonomous vehicles to improve pick-up and drop-off of passengers or users. For example, the user can have the vehicle visually and/or audibly communicate with the user, either by themselves or by prompting them to use surfaced options. This allows the user to more easily identify the position of the vehicle. Additionally or alternatively, the vehicle may use the model to pre-determine if and how to communicate with the user and how to escalate those communications over time.

Unless stated otherwise, the aforementioned alternative examples, although not mutually exclusive, can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above may be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments is subject to the claims. It should be viewed as an illustration and not as a limitation of the subject matter provided. In addition, the examples described herein and the provision of phrases expressed as "such as,""including," etc. should not be construed as limiting the claimed subject matter to the particular examples. rather, the example is intended to illustrate only one of many possible embodiments. Further, the same reference numbers in different drawings may identify the same or similar elements.