CN115705723A - Computer-implemented method, system for a vehicle, and computer-readable medium - Google Patents

Abstract

The invention provides a computer-implemented method, a system for a vehicle, and a computer-readable medium. A method for scalable and realistic camera occlusion data set generation is provided that may include generating a composite image depicting occlusions on or near a camera sensor. A composite image may be created by combining one or more chroma-key extracted partial blocking images with one or more background images, which combination may provide an extensible blocking data set. By labeling the part of the composite image represented by the partial block image extracted by the chroma key as constituting a block, metadata of the respective composite images can be generated together with the composite image. The composite image may be used to increase the accuracy of a machine learning model trained to identify obstructions by increasing the amount of data available for such training.

Description

Computer-implemented method, system for a vehicle, and computer-readable medium

Technical Field

The invention relates to a computer-implemented method, a system for a vehicle, and a computer-readable medium.

Background

Autonomous vehicles may be used to transport people and/or goods (e.g., packages, objects, or other items) from one location to another. For example, the autonomous vehicle may navigate to a location of a person, wait for the person to board the autonomous vehicle, and navigate to a specified destination (e.g., a location selected by the person). For navigation in an environment, these autonomous vehicles are equipped with various types of sensors to detect objects in the surrounding environment. A full or partial obstruction of one or more of these sensors may result in degraded performance of the autonomous vehicle.

Disclosure of Invention

According to one aspect of the invention, there is provided a computer-implemented method, implemented by one or more hardware processors, the method comprising: obtaining, using the one or more hardware processors, data representing partial block images, each partial block image depicting a block in front of a chroma-key background; generating, using the one or more hardware processors, a training data set comprising a plurality of synthetic partial occlusion images, wherein generating the training data set comprises: for each of the synthetic partial barrier images, performing a chroma keying operation to extract a shadow of an obstacle from a partial barrier image of the partial barrier images, superimposing the extracted shadow of the obstacle on a background image to generate the synthetic partial barrier image, and generating annotation data for the synthetic partial barrier image, the annotation data being for distinguishing a portion of the synthetic partial barrier image representing the shadow of the extracted obstacle from a portion of the synthetic partial barrier image representing the background image; and training, using the one or more hardware processors, a neural network with the plurality of synthetic partial obstruction images to identify portions of the input image data corresponding to data from a sensor of an autonomous vehicle that represent obstructions on the sensor of the autonomous vehicle.

According to another aspect of the present invention, there is provided a system for a vehicle, comprising: a data storage device comprising computer executable instructions; and a processor configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to: obtaining data representing partial block images, each partial block image depicting a block in front of a chroma-key background; generating a training data set comprising a plurality of synthetic partial occlusion images by at least: for each of the synthetic partial barrier images, performing a chroma keying operation to extract a shadow of an obstacle from a partial barrier image of the partial barrier images, superimposing the extracted shadow of the obstacle on a background image to generate the synthetic partial barrier image, and generating annotation data for the synthetic partial barrier image, the annotation data being for distinguishing a portion of the synthetic partial barrier image representing the shadow of the extracted obstacle from a portion of the synthetic partial barrier image representing the background image; and training a neural network with the plurality of synthetic partial obstruction images to identify portions of the input image data corresponding to data from a sensor of an autonomous vehicle that represent obstructions on the sensor of the autonomous vehicle.

According to yet another aspect of the invention, one or more non-transitory computer-readable media are provided that include computer-executable instructions that, when executed by a computing system that includes a processor, cause the computing system to: obtaining data representing partial block images, each partial block image depicting a block in front of a chroma-key background; generating a training data set comprising a plurality of synthetic partial occlusion images by at least: for each of the synthetic partial barrier images, performing a chroma keying operation to extract a shadow of an obstacle from a partial barrier image of the partial barrier images, superimposing the extracted shadow of the obstacle on a background image to generate the synthetic partial barrier image, and generating annotation data for the synthetic partial barrier image, the annotation data being for distinguishing a portion of the synthetic partial barrier image representing the shadow of the extracted obstacle from a portion of the synthetic partial barrier image representing the background image; and training a neural network with the plurality of synthetic partial obstruction images to identify portions of the input image data corresponding to data from a sensor of an autonomous vehicle that represent obstructions on the sensor of the autonomous vehicle.

Drawings

Fig. 1 illustrates an example of an autonomous vehicle having autonomous capabilities.

FIG. 2 illustrates an example cloud computing environment.

Fig. 3 illustrates a computer system.

Fig. 4 illustrates an example architecture of an autonomous vehicle.

FIG. 5 shows an example of inputs and outputs that the perception module can use.

Fig. 6A-6C illustrate examples of how debris can at least partially block the field of view of the optical sensor.

Fig. 7 shows a system for capturing images (images) of various types of debris at different distances from an optical sensor.

Fig. 8 illustrates how the imagery generated by the system depicted in fig. 7 can be used to synthesize imagery for training a neural network.

FIG. 9 illustrates a neural network training architecture for training a neural network to detect and characterize optical sensor barriers.

FIG. 10 depicts an example routine for generation of a synthetic partial occlusion training data set.

FIG. 11 depicts an example routine for training a neural network with training data having multiple levels of granularity.

Detailed Description

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, that the embodiments described in this disclosure may be practiced without these specific details. In some instances, well-known structures and devices are illustrated in block diagram form in order to avoid unnecessarily obscuring aspects of the present disclosure.

In the drawings, the specific arrangement or order of schematic elements (such as those representing systems, devices, modules, instruction blocks, and/or data elements, etc.) is illustrated for ease of description. However, it will be appreciated by those of ordinary skill in the art that the particular order or arrangement of elements illustrated in the figures is not intended to imply that a particular order or sequence of processing, or separation of processing, is required unless explicitly described. Moreover, unless explicitly described, the inclusion of a schematic element in a drawing is not intended to imply that such element is required in all embodiments, nor that the feature represented by such element cannot be included or combined with other elements in some embodiments.

Further, in the figures, connecting elements (such as solid or dashed lines, arrows, and the like) are used to illustrate a connection, relationship, or association between or among two or more other illustrated elements, and the absence of any such connecting element is not intended to imply that a connection, relationship, or association cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the disclosure. Further, for ease of illustration, a single connected element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connection element represents a communication of signals, data, or instructions (e.g., "software instructions"), those skilled in the art will appreciate that such element may represent one or more signal paths (e.g., a bus) that may be required to affect the communication.

Although the terms first, second, third, etc. may be used to describe various elements, these elements should not be limited by these terms. The terms "first," second, "and/or third" are used merely to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact, without departing from the scope of the described embodiments. Both the first contact and the second contact are contacts, but they are not identical contacts.

The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, and may be used interchangeably with "one or more than one" or "at least one" unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises," "comprising," "includes" and/or "including," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, the terms "communicate" and "communicating" refer to at least one of the receipt, transmission, and/or provision of information (or information represented by, for example, data, signals, messages, instructions, and/or commands, etc.). For one unit (e.g., a device, a system, a component of a device or system, and/or combinations thereof, etc.) to communicate with another unit, this means that the one unit can directly or indirectly receive information from and/or send (e.g., transmit) information to the other unit. This may refer to a direct or indirect connection that may be wired and/or wireless in nature. In addition, two units may communicate with each other even though the transmitted information may be modified, processed, relayed and/or routed between the first and second units. For example, a first unit may communicate with a second unit even if the first unit passively receives information and does not actively transmit information to the second unit. As another example, if at least one intermediary element (e.g., a third element located between the first element and the second element) processes information received from the first element and transmits the processed information to the second element, the first element may communicate with the second element. In some embodiments, a message may refer to a network packet (e.g., a data packet, etc.) that includes data.

As used herein, the term "if" is optionally interpreted to mean "when 8230;," at 8230;, "responsive to a determination," and/or "responsive to a detection," etc., depending on the context. Similarly, the phrase "if determined" or "if [ stated condition or event ] is detected" is optionally to be construed to mean "upon determination of 8230a", "in response to a determination of" or "upon detection of [ stated condition or event ], and/or" in response to detection of [ stated condition or event ] ", etc., depending on the context. Further, as used herein, the terms "having," "having," or "possessing," etc., are intended to be open-ended terms. Further, the phrase "based on" is intended to mean "based, at least in part, on" unless explicitly stated otherwise.

Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. It will be apparent, however, to one skilled in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail as not to unnecessarily obscure aspects of the embodiments.

General overview

In general, aspects of the present disclosure relate to training a neural network to identify a camera obstruction (such as an object attached to a lens of a camera of an autonomous vehicle). More particularly, aspects of the present invention relate to synthetically generating input data that may be used to train neural networks to detect boundaries of such obstructions, and to training such neural networks using a combination of synthetically and non-synthetically generated data. As disclosed herein, synthetic input data for the camera occlusion detection neural network may be generated by: a blocking image is controllably generated over a known background (e.g., a background of known color), a blocking boundary is detected by analysis of the image based on the known background, and the known background is replaced with various other background images (e.g., real world imagery) to generate a composite blocking image with a known blocking boundary. This technique may generate a more accurate composite occlusion image while retaining the ability to accurately detect occlusion boundaries, as compared to alternative techniques. The neural network may then be trained based on both synthetic blocking images with known blocking boundaries and non-synthetic blocking images (e.g., obtained from a production environment) with unknown blocking boundaries to generate a trained network that may generate the expected blocking boundaries for the images given the non-synthetic blocking images. Thereafter, the trained network may be applied (e.g., in production) to detect camera obstructions and estimated boundaries of such obstructions. More accurate detection of obstructions and boundaries may in turn lead to more accurate and safer use of camera data, such as by enabling more accurate and safe operation of autonomous vehicles.

As will be appreciated by those skilled in the art, accurate training of neural networks may often require large input data sets. In some cases, it may be difficult or impossible to obtain non-synthetic input data sufficient to accurately train the network. This may be particularly true for neural networks that are intended to cope with infrequent "edge case" scenarios. One example of such an edge condition may be a camera obstruction in an autonomous vehicle, such as when water or other matter (e.g., foliage, mud, sand, etc.) adheres to the lens of the vehicle's camera and prevents capture of the vehicle's environment. Such an example may pose a significant safety issue, particularly where the vehicle camera is used in whole or in part to control operation of the vehicle. While human operators may not be as effortless in recognizing and to what extent such obstructions, it is more difficult to do so programmatically. Typically, the camera image is analyzed as two-dimensional data (e.g., as a two-dimensional matrix of pixels), and it may be difficult from a programming standpoint to accurately distinguish between pixels that capture the obstacle and pictures that capture the vehicle environment. One method of training a computing device to programmatically distinguish a blocked image from an unblocked image, and detect the extent of such an obstruction, may be to train a neural network on inputs that include both blocked and unblocked images. However, the relative sparsity of the barrier image may inhibit accurate training (e.g., due to its nature as an edge condition).

One technique for generating input data synthetically may be to use a neural network framework such as a generate confrontational network (GAN). Such a framework may be trained (e.g., based on non-synthetic blocking images) to generate synthetic blocking images. However, this approach has a number of disadvantages. For example, the output of such a network may not be of sufficient quality to train a network for use in autonomous vehicles on public roads. Furthermore, even if such networks are capable of providing synthetic data of sufficient quality, such networks typically cannot accurately indicate which portions of an image are blocked or unblocked. However, the nature and extent of the barrier may be important in various contexts, such as for safe and accurate operation of the autonomous vehicle. For example, such vehicles may continue to operate in a slightly blocked condition (e.g., 1% to 2% blocked) during implementation of a responsive action (e.g., engaging a lens cleaning device), but may fail to operate as intended when there is a major obstruction (e.g., 75% or more blocked). Because the GAN-generated composite images may not include data regarding which portions of the images are blocked, these images may not be suitable for this application.

Another possible technique for generating composite data may be to crop the blocking image directly from one background image and superimpose the cropped blocking image onto the background image (e.g., of the vehicle environment). For example, a non-composite image including a barrier may be processed (e.g., by cutting out the background) to separate the barrier. The cropped barrier may then be superimposed over the non-blocking background image to generate a composite blocking image of the new background. Because the locations of the obstructions are known, the images may be associated with obstruction data that indicates those locations. In this manner, one non-composite image with blockage may be used to generate one or more composite images with blockage. However, this "copy and paste" technique also has drawbacks. In particular, such a composite image may not accurately reflect real-world obstructions due to interactions between the obstructions and the background that are not captured via the cropping and overlaying operations. By way of illustration, some barriers may be partially transparent and thus only partially obscure the sensor of the camera. This may be particularly true for the edges of the barrier. For example, even a completely opaque object may cause the camera's sensor to partially capture "feathering" around the object of the background environment. The crop and overlap operation will generally fail to capture such partial obstructions.

Embodiments of the present disclosure address these issues by providing for the synthetic generation of highly accurate input data representing a camera barrier. In particular, according to embodiments of the present disclosure, blocking objects (e.g., leaves, water, mud, etc.) may be captured against a known background, such as a "green screen" or other background of a known color range, etc. To facilitate capture, the blocking object may be attached to a transparent object such as a glass panel or a lens of a photographic camera, for example. Chroma keying may then be used to remove the known background, thereby producing an image that accurately captures the barrier (e.g., including the partially transparent portion of the barrier). Thereafter, the occlusion may be superimposed on the new background to generate a composite occlusion image. This mechanism provides similar benefits to the cropping and overlap techniques discussed above, such as the ability to determine which portions of the image represent the obstacle (e.g., in pixels) and which represent the environment, among other things. However, this mechanism avoids the disadvantages of the cropping and overlap technique. For example, the chromakeying mechanism enables capture of a partially transparent barrier, such as a translucent object or an object whose edges cause feathering. Thus, the composite image generation technique enables generation of a large amount of highly accurate input data.

In addition to the generation of synthetic occlusion images, embodiments of the present disclosure also relate to training a neural network to detect (e.g., at the pixel level) occluded portions of an image containing an occlusion. Typically, to generate a given type of output, a neural network must be trained with that type of input. Thus, in order to train the neural network to detect which portions of an image are blocked, it may be necessary to provide an input that includes both the image with the blockage and information identifying which portions of the image contain the blockage. However, providing this information for non-composite images may be difficult or cumbersome. For example, providing this information may typically require a human to manually view the individual images and specify which portions of the images are blocked at a given granularity. This process may require a significant amount of time and may have limited accuracy, particularly as the granularity increases. For example, a human may not be able to generate a large number of accurate pixel-level image annotations indicating which portions of an image are blocked. As described above, embodiments of the present disclosure may address this problem by providing for the generation of a composite occlusion image and a corresponding indication (sometimes referred to as an annotation) of which portions (e.g., which pixels) are occluded. However, it may not be desirable to train the neural network only on such a synthetic image. For example, there may be subtle differences between the synthetic and non-synthetic images that reduce the accuracy of the trained network during application to the non-synthetic input.

To address this issue, embodiments of the present disclosure provide a neural network architecture that can be trained using a combination of synthetic images and non-synthetic images containing occlusions as input data to generate highly accurate (e.g., pixel-level) annotations of the non-synthetic images. More specifically, this document describes a machine learning architecture in which both synthetic and non-synthetic images are input into a neural network to generate an indication of which portions of the image (e.g., in pixels) are blocked. The input synthetic image may be accompanied by annotations indicating which pixels are blocked (if any), and thus when training based on the synthetic image, the neural network may be trained based on the loss between indications generated by the neural network and these annotations. The input non-composite image may lack annotations to indicate which pixels are blocked, but instead be associated with a binary blocking label (e.g., blocked or unblocked). To train based on such binary labeled input data, the network may include a conversion function for converting the indication of which portions are blocked into binary indicators of whether a block is present. For example, the transfer function may be a threshold function for generating a binary "block" indicator when a threshold percentage (e.g., n%) of the image is blocked. For an input that includes binary labeled input data, the neural network may then be trained by processing the input data to generate indications of which portions are blocked, converting the indications to binary outputs via a conversion function, and comparing the binary outputs to the binary labels of the input. In this way, the network can be trained based on a combination of binary labeled data (e.g., non-synthesized data) and finely (granular) labeled data (e.g., synthesized data with pixel-level labeling). Thus, the model resulting from such training may be enabled to obtain new input data and determine annotations indicating which portions (e.g., which pixels) of the image are blocked. As a result, such a model may provide highly accurate obstruction detection in various scenarios, such as autonomous vehicles and the like.

As will be appreciated by those skilled in the art in light of the present disclosure, embodiments disclosed herein improve the ability of computing systems (such as computing devices included within or supporting the operation of autonomous vehicles, etc.) to detect and characterize obstructions of a camera. Such detection and characterization in turn provides for more accurate, safe and reliable operation of the device using the camera image as input. For example, the autonomous vehicle may be enabled to detect an obstruction on the camera and take corrective action commensurate with the extent of the detected obstruction. Furthermore, the presently disclosed embodiments address technical problems inherent within computing systems, particularly the difficulty of training a neural network with limited data, the difficulty of generating highly accurate synthetic data with granular (e.g., pixel-level) blocking annotations, and the difficulty of training a neural network to provide accurate and granular blocking annotations to non-synthetic images containing partial blocking. These technical problems are addressed by various technical solutions described herein, including generating a composite image using partially blocked images on a known background and applying chroma-keying techniques to apply the partial blocking to an additional background, and training a machine learning model on a combination of binary labeled (e.g., non-composite) images and more finely labeled (e.g., composite) images. Accordingly, the present disclosure generally embodies improvements in computer vision systems and computing systems.

The foregoing aspects and many of the attendant advantages of this disclosure will become more readily appreciated as the same become better understood by reference to the following description, when taken in conjunction with the accompanying drawings.

Overview of hardware

Fig. 1 shows an example of an autonomous vehicle 100 with autonomous capabilities.

As used herein, the term "autonomous capability" refers to a function, feature, or facility that enables a vehicle to operate partially or fully without real-time human intervention, including, but not limited to, fully autonomous vehicles, highly autonomous vehicles, and conditional autonomous vehicles.

As used herein, an Autonomous Vehicle (AV) is a vehicle with autonomous capabilities.

As used herein, "vehicle" includes a means of transportation of goods or people. Such as cars, buses, trains, airplanes, drones, trucks, boats, ships, submersibles, airships, etc. An unmanned car is an example of a vehicle.

As used herein, "trajectory" refers to a path or route that navigates an AV from a first spatiotemporal location to a second spatiotemporal location. In an embodiment, the first spatiotemporal location is referred to as an initial location or a starting location and the second spatiotemporal location is referred to as a destination, a final location, a target location, or a target location. In some examples, a track consists of one or more road segments (e.g., segments of a road), and each road segment consists of one or more blocks (e.g., a portion of a lane or intersection). In an embodiment, the spatio-temporal locations correspond to real-world locations. For example, the space-time location is a boarding or alighting location to allow people or cargo to board or disembark.

As used herein, a "sensor(s)" includes one or more hardware components for detecting information about the environment surrounding the sensor. Some hardware components may include sensing components (e.g., image sensors, biometric sensors), transmitting and/or receiving components (e.g., laser or radio frequency wave transmitters and receivers), electronic components (such as analog-to-digital converters and the like), data storage devices (such as RAM and/or non-volatile memory and the like), software or firmware components and data processing components (such as application specific integrated circuits and the like), microprocessors and/or microcontrollers.

As used herein, a "scene description" is a data structure (e.g., a list) or data stream that includes one or more classified or tagged objects detected by one or more sensors on an AV vehicle, or one or more classified or tagged objects provided by a source external to the AV.

As used herein, a "roadway" is a physical area that can be traversed by a vehicle and may correspond to a named aisle (e.g., a city street, an interstate highway, etc.) or may correspond to an unnamed aisle (e.g., a roadway within a house or office building, a segment of a parking lot, a segment of an empty parking lot, a dirt aisle in a rural area, etc.). Because some vehicles (e.g., four-wheel drive trucks, off-road vehicles (SUVs), etc.) are able to traverse a variety of physical areas not particularly suited for vehicle travel, a "road" may be any physical area that a municipality or other government or administrative authority has not formally defined as a passageway.

As used herein, a "lane" is a portion of a roadway that may be traversed by a vehicle, and may correspond to most or all of the space between lane markings, or only a portion of the space between lane markings (e.g., less than 50%). For example, a roadway with lane markings that are far apart may accommodate two or more vehicles such that one vehicle may pass another without crossing the lane markings, and thus may be interpreted as a lane being narrower than the space between the lane markings, or two lanes between lanes. In the absence of lane markings, the lane may also be interpreted. For example, lanes may be defined based on physical characteristics of the environment (e.g., rocks in rural areas and trees along thoroughfares).

"one or more" includes a function performed by one element, a function performed by multiple elements, for example, in a distributed manner, several functions performed by one element, several functions performed by several elements, or any combination thereof.

As used herein, an AV system refers to AV and to an array of hardware, software, stored data, and real-time generated data that support AV operations. In an embodiment, the AV system is incorporated within the AV. In an embodiment, the AV system is distributed across several sites. For example, some software of the AV system is implemented on a cloud computing environment similar to the cloud computing environment 200 described below with respect to fig. 2.

In general, this document describes techniques applicable to any vehicle having one or more autonomous capabilities, including fully autonomous vehicles, highly autonomous vehicles, and conditional autonomous vehicles, such as so-called class 5, class 4, and class 3 vehicles, respectively (see SAE International Standard J3016: classification and definition of terms related to automotive autonomous systems on roads, the entire contents of which are incorporated by reference into this document for more detailed information on the level of autonomy of the vehicle). The technology described in this document is also applicable to partly autonomous vehicles and driver-assisted vehicles, such as so-called class 2 and class 1 vehicles (see SAE international standard J3016: classification and definition of terms relating to automotive autonomous systems on roads). In embodiments, one or more of the class 1, class 2, class 3, class 4, and class 5 vehicle systems may automatically perform certain vehicle operations (e.g., steering, braking, and map usage) under certain operating conditions based on processing of sensor inputs. The technology described in this document may benefit any class of vehicles ranging from fully autonomous vehicles to vehicles operated by humans.

Referring to fig. 1, the AV system 120 operates the AV100 along a trajectory 198, through the environment 190 to a destination 199 (sometimes referred to as a final location) while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, riders, and other obstacles) and complying with road regulations (e.g., operating regulations or driving preferences).

In an embodiment, the AV system 120 comprises means 101 for receiving and operating an operation command from the calculation processor 146. In an embodiment, the calculation processor 146 is similar to the processor 304 described below with reference to fig. 3. Examples of devices 101 include a steering controller 102, a brake 103, a gear, an accelerator pedal or other acceleration control mechanism, a windshield wiper, a side door lock, a window controller, and a steering indicator.

In an embodiment, AV system 120 includes sensors 121 for measuring or inferring properties of the state or condition of AV100, such as the position, linear and angular velocities and accelerations, and heading (e.g., direction of the front of AV 100) of the AV. Examples of sensors 121 are GPS, inertial Measurement Units (IMU) that measure both linear acceleration and angular velocity of the vehicle, wheel speed sensors for measuring or estimating wheel slip rate, wheel brake pressure or torque sensors, engine torque or wheel torque sensors, and steering angle and angular velocity sensors.

In an embodiment, the sensors 121 further comprise sensors for sensing or measuring properties of the environment of the AV. Such as a monocular or stereo camera 122, lidar 123, radar, ultrasonic sensors, time-of-flight (TOF) depth sensors, rate sensors, temperature sensors, humidity sensors, and precipitation sensors for the visible, infrared, or thermal (or both) spectrum.

In an embodiment, the AV system 120 includes a data storage unit 142 and a memory 144 for storing machine instructions associated with a computation processor 146 or data collected by the sensors 121. In an embodiment, the data storage unit 142 is similar to the ROM 308 or the storage device 310 described below with respect to fig. 3. In an embodiment, memory 144 is similar to main memory 306 described below. In an embodiment, data storage unit 142 and memory 144 store historical, real-time, and/or predictive information about environment 190. In an embodiment, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In an embodiment, data related to the environment 190 is transmitted from the remote database 134 to the AV100 over a communication channel.

In an embodiment, the AV system 120 includes a communication device 140 for communicating to the AV100 attributes measured or inferred for the state and conditions of other vehicles, such as position, linear and angular velocity, linear and angular acceleration, and linear and angular heading. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices as well as devices for wireless communication over point-to-point or ad hoc (ad hoc) networks or both. In embodiments, communications devices 140 communicate across the electromagnetic spectrum (including radio and optical communications) or other media (e.g., air and acoustic media). The combination of vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) communications (and in some embodiments one or more other types of communications) is sometimes referred to as vehicle-to-everything (V2X) communications. V2X communications typically conform to one or more communication standards for communication with and between autonomous vehicles.

In an embodiment, the communication device 140 comprises a communication interface. Such as a wired, wireless, wiMAX, wiFi, bluetooth, satellite, cellular, optical, near field, infrared, or radio interface. The communication interface transmits data from the remote database 134 to the AV system 120. In an embodiment, remote database 134 is embedded in cloud computing environment 200 as described in fig. 2. The communication interface 140 transmits data collected from the sensors 121 or other data related to the operation of the AV100 to the remote database 134. In an embodiment, the communication interface 140 transmits teleoperation-related information to the AV 100. In some embodiments, the AV100 communicates with other remote (e.g., "cloud") servers 136.

In an embodiment, the remote database 134 also stores and transmits digital data (e.g., stores data such as road and street locations). These data are stored in memory 144 on AV100 or transmitted from remote database 134 to AV100 over a communications channel.

In an embodiment, the remote database 134 stores and transmits historical information (e.g., velocity and acceleration profiles) related to driving attributes of vehicles that have previously traveled along the trajectory 198 at similar times of the day. In one implementation, such data may be stored in memory 144 on AV100 or transmitted from remote database 134 to AV100 over a communication channel.

A computational processor 146 located on the AV100 algorithmically generates control actions based on both real-time sensor data and a priori information, allowing the AV system 120 to perform its autonomous driving capabilities.

In an embodiment, the AV system 120 includes a computer peripheral 132 coupled to a computing processor 146 for providing information and reminders to a user (e.g., an occupant or remote user) of the AV100 and receiving input from the user. In an embodiment, peripheral 132 is similar to display 312, input device 314, and cursor controller 316 discussed below with reference to fig. 3. The coupling is wireless or wired. Any two or more interface devices may be integrated into a single device.

FIG. 2 illustrates an example "cloud" computing environment. Cloud computing is a service delivery model for enabling convenient, on-demand access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) over a network. In a typical cloud computing system, one or more large cloud data centers house machines for delivering services provided by the cloud. Referring now to fig. 2, cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c interconnected by cloud 202. Data centers 204a, 204b, and 204c provide cloud computing services for computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to cloud 202.

The cloud computing environment 200 includes one or more cloud data centers. In general, a cloud data center (e.g., cloud data center 204a shown in fig. 2) refers to a physical arrangement of servers that make up a cloud (e.g., cloud 202 shown in fig. 2 or a particular portion of a cloud). For example, the servers are physically arranged in rooms, groups, rows, and racks in a cloud data center. A cloud data center has one or more areas that include one or more server rooms. There are one or more rows of servers per room, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, servers in a zone, room, rack, and/or row are arranged into groups based on physical infrastructure requirements (including electrical, energy, thermal, heat, and/or other requirements) of a data center facility. In an embodiment, the server node is similar to the computer system described in FIG. 3. Data center 204a has many computing systems distributed across multiple racks.

Cloud 202 includes cloud data centers 204a, 204b, and 204c and network resources (e.g., network devices, nodes, routers, switches, and network cables) for connecting cloud data centers 204a, 204b, and 204c and facilitating access to cloud computing services by computing systems 206 a-f. In an embodiment, the network represents any combination of one or more local networks, wide area networks, or internetworks coupled by wired or wireless links deployed using terrestrial or satellite connections. Data exchanged over a network is transmitted using a variety of network layer protocols, such as Internet Protocol (IP), multi-protocol label switching (MPLS), asynchronous Transfer Mode (ATM), frame Relay (Frame Relay), etc. Further, in embodiments where the network represents a combination of multiple sub-networks, a different network layer protocol is used on each underlying sub-network. In some embodiments, the network represents one or more interconnected internet networks (such as the public internet, etc.).

Computing systems 206a-f or cloud computing service consumers are connected to cloud 202 through network links and network adapters. In embodiments, computing systems 206a-f are implemented as a variety of computing devices, such as servers, desktops, laptops, tablets, smartphones, internet of things (IoT) devices, autonomous vehicles (including cars, drones, space shuttles, trains, buses, and the like), and consumer electronics. In embodiments, computing systems 206a-f are implemented in or as part of other systems.

Fig. 3 illustrates a computer system 300. In an implementation, the computer system 300 is a special purpose computing device. Special purpose computing devices are hardwired to perform the techniques, or include digital electronic devices such as one or more Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination. Such dedicated computing devices may also incorporate custom hardwired logic, ASICs or FPGAs with custom programming to accomplish these techniques. In various embodiments, the special purpose computing device is a desktop computer system, portable computer system, handheld device, network device, or any other device that contains hard-wired and/or program logic to implement these techniques.

In an embodiment, computer system 300 includes a bus 302 or other communication mechanism for communicating information, and a hardware processor 304 coupled with bus 302 for processing information. The hardware processor 304 is, for example, a general purpose microprocessor. Computer system 300 also includes a main memory 306, such as a Random Access Memory (RAM) or other dynamic storage device, coupled to bus 302 for storing information and instructions to be executed by processor 304. In one implementation, main memory 306 is used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 304. When stored in a non-transitory storage medium accessible to processor 304, these instructions cause computer system 300 to become a special-purpose machine that is customized to perform the operations specified in the instructions.

In an embodiment, computer system 300 further includes a Read Only Memory (ROM) 308 or other static storage device coupled to bus 302 for storing static information and instructions for processor 304. A storage device 310, such as a magnetic disk, optical disk, solid state drive, or three-dimensional cross-point memory, is provided and coupled to bus 302 to store information and instructions.

In an embodiment, computer system 300 is coupled via bus 302 to a display 312, such as a Cathode Ray Tube (CRT), liquid Crystal Display (LCD), plasma display, light Emitting Diode (LED) display, or Organic Light Emitting Diode (OLED) display for displaying information to a computer user. An input device 314, including alphanumeric and other keys, is coupled to bus 302 for communicating information and command selections to processor 304. Another type of user input device is cursor control 316, such as a mouse, a trackball, touch display, or cursor direction keys for communicating direction information and command selections to processor 304 and for controlling cursor movement on display 312. Such input devices typically have two degrees of freedom in two axes, a first axis (e.g., the x-axis) and a second axis (e.g., the y-axis), that allow the device to specify positions in a plane.

According to one embodiment, the techniques herein are performed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306. Such instructions are read into main memory 306 from another storage medium, such as storage device 310. Execution of the sequences of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

The term "storage medium" as used herein refers to any non-transitory medium that stores data and/or instructions that cause a machine to operate in a specific manner. Such storage media includes non-volatile media and/or volatile media. Non-volatile media includes, for example, optical disks, magnetic disks, solid state drives, or three-dimensional cross-point memories such as storage device 310. Volatile media includes dynamic memory, such as main memory 306. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with a hole pattern, a RAM, a PROM, and EPROM, a FLASH-EPROM, an NV-RAM, or any other memory chip or cartridge.

Storage media is distinct from but may be used in combination with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 302. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

In an embodiment, various forms of media are involved in carrying one or more sequences of one or more instructions to processor 304 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer loads the instructions into its dynamic memory and sends the instructions over a telephone line using a modem. A modem local to computer system 300 receives the data on the telephone line and uses an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector receives the data carried in the infra-red signal and appropriate circuitry places the data on bus 302. Bus 302 carries the data to main memory 306, from which main memory 306 processor 304 retrieves and executes the instructions. The instructions received by main memory 306 may optionally be stored on storage device 310 either before or after execution by processor 304.

Computer system 300 also includes a communication interface 318 coupled to bus 302. Communication interface 318 provides a two-way data communication coupling to a network link 320 that is connected to a local network 322. For example, communication interface 318 is an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 318 is a Local Area Network (LAN) card to provide a data communication connection to a compatible LAN. In some implementations, a wireless link is also implemented. In any such implementation, communication interface 318 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

Network link 320 typically provides data communication through one or more networks to other data devices. For example, network link 320 provides a connection through local network 322 to a host computer 324 or to a cloud data center or equipment operated by an Internet Service Provider (ISP) 326. ISP 326 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "internet" 328. Local network 322 and Internet 328 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 320 and through communication interface 318, which carry the digital data to and from computer system 300, are exemplary forms of transmission media. In an embodiment, network 320 comprises cloud 202 or a portion of cloud 202 as described above.

Computer system 300 sends messages and receives data, including program code, through the network(s), network link 320 and communication interface 318. In an embodiment, computer system 300 receives code for processing. The received code may be executed by processor 304 as it is received, and/or stored in storage device 310, or other non-volatile storage for later execution.

Autonomous vehicle architecture

Fig. 4 illustrates an example architecture 400 for an autonomous vehicle (e.g., AV100 shown in fig. 1). Architecture 400 includes a perception module 402 (sometimes referred to as a perception circuit), a planning module 404 (sometimes referred to as a planning circuit), a control module 406 (sometimes referred to as a control circuit), a positioning module 408 (sometimes referred to as a positioning circuit), and a database module 410 (sometimes referred to as a database circuit). Each module plays a role in the operation of the AV 100. Collectively, the modules 402, 404, 406, 408, and 410 may be part of the AV system 120 shown in fig. 1. In some embodiments, any of the modules 402, 404, 406, 408, and 410 are a combination of computer software (e.g., executable code stored on a computer-readable medium) and computer hardware (e.g., one or more microprocessors, microcontrollers, application specific integrated circuits [ ASICs ], hardware memory devices, other types of integrated circuits, other types of computer hardware, or a combination of any or all of these).

In use, the planning module 404 receives data representing a destination 412 and determines data representing a trajectory 414 (sometimes referred to as a route) that the AV100 can travel in order to reach (e.g., arrive at) the destination 412. In order for planning module 404 to determine data representing trajectory 414, planning module 404 receives data from perception module 402, positioning module 408, and database module 410.

The perception module 402 identifies nearby physical objects using, for example, one or more sensors 121 as also shown in fig. 1. The objects are classified (e.g., into types such as pedestrian, bicycle, automobile, traffic sign, etc.), and a scene description including the classified objects 416 is provided to the planning module 404.

The planning module 404 also receives data representing the AV location 418 from the positioning module 408. The positioning module 408 determines the AV location by using data from the sensors 121 and data (e.g., geographic data) from the database module 410 to calculate the location. For example, the positioning module 408 calculates the longitude and latitude of the AV using data from GNSS (global navigation satellite system) sensors and geographic data. In embodiments, the data used by the positioning module 408 includes high precision maps with lane geometry attributes, maps describing road network connection attributes, maps describing lane physics attributes such as traffic rate, traffic volume, number of vehicle and bicycle lanes, lane width, lane traffic direction, or lane marker types and locations, or combinations thereof, and maps describing spatial locations of road features such as crosswalks, traffic signs, or other travel signals of various types, and the like.

The control module 406 receives data representing the track 414 and data representing the AV location 418 and operates the control functions 420 a-420 c of the AV (e.g., steering, throttle, brake, ignition) in a manner that will cause the AV100 to travel the track 414 to the destination 412. For example, if the trajectory 414 includes a left turn, the control module 406 will operate the control functions 420 a-420 c as follows: the steering angle of the steering function will cause the AV100 to turn left and the throttle and brakes will cause the AV100 to pause and wait for a passing pedestrian or vehicle before making a turn.

Autonomous vehicle input

FIG. 5 shows examples of inputs 502a-502d (e.g., sensors 121 shown in FIG. 1) and outputs 504a-504d (e.g., sensor data) used by the perception module 402 (FIG. 4). One input 502a is a LiDAR (light detection and ranging) system (e.g., liDAR 123 shown in FIG. 1). LiDAR is a technology that uses light (e.g., a line of light such as infrared light) to obtain data about a physical object in its line of sight. The LiDAR system generates LiDAR data as output 504a. For example, liDAR data is a collection of 3D or 2D points (also referred to as point clouds) used to construct a representation of the environment 190.

The other input 502b is a RADAR system. RADAR is a technology that uses radio waves to obtain data about nearby physical objects. RADAR may obtain data related to objects that are not within a line of sight of the LiDAR system. The RADAR system 502b generates RADAR data as output 504b. For example, RADAR data is one or more radio frequency electromagnetic signals used to construct a representation of the environment 190.

Another input 502c is a camera system. Camera systems use one or more cameras (e.g., digital cameras using light sensors such as charge coupled devices [ CCDs ]) to acquire information about nearby physical objects. The camera system generates camera data as output 504c. The camera data is generally in the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). In some examples, the camera system has multiple independent cameras, for example for the purpose of stereoscopic imagery (stereo vision), which enables the camera system to perceive depth. Although the object perceived by the camera system is described herein as "nearby," this is with respect to AV. In use, the camera system may be configured to "see" objects that are far away (e.g., as far as 1 kilometer or more in front of the AV). Accordingly, the camera system may have features such as a sensor and a lens optimized for sensing a distant object.

Another input 502d is a Traffic Light Detection (TLD) system. TLD systems use one or more cameras to obtain information about traffic lights, street signs, and other physical objects that provide visual navigation information. The TLD system generates TLD data as output 504d. The TLD data often takes the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). TLD systems differ from systems that include a camera in that: TLD systems use cameras with a wide field of view (e.g., using a wide-angle lens or a fisheye lens) to obtain information about as many physical objects as possible that provide visual navigation information, so that the AV100 can access all relevant navigation information provided by these objects. For example, the viewing angle of a TLD system may be about 120 degrees or greater.

In some embodiments, the outputs 504a-504d are combined using sensor fusion techniques. Thus, the individual outputs 504a-504d are provided to other systems of the AV100 (e.g., to the planning module 404 as shown in fig. 4), or the combined outputs may be provided to other systems in the form of a single combined output or multiple combined outputs of the same type (e.g., using the same combining technique or combining the same output or both) or different types of single combined output or multiple combined outputs (e.g., using different individual combining techniques or combining different individual outputs or both). In some embodiments, early fusion techniques are used. Early fusion techniques were characterized by: the outputs are combined before one or more data processing steps are applied to the combined output. In some embodiments, post-fusion techniques are used. The later stage fusion technology is characterized in that: after applying one or more data processing steps to the individual outputs, the outputs are combined.

Sensor obstruction detection

As described above, one problem that may occur when relying on the output of a 2D sensor such as the camera 502c is the following obstruction: dust, water, fog, debris, or other matter adheres to the lens or housing of the camera or otherwise interferes with the sensor used to capture the 2D image of the environment surrounding the sensor. In the context of autonomous vehicles, such obstructions may represent a significant safety issue. Therefore, it is beneficial to detect and resolve such blockage. Furthermore, knowing the nature and extent of such obstructions may be beneficial in determining an appropriate corrective action. For example, a significant obstruction may require immediate stopping of vehicle operation, while an insignificant obstruction may cause other corrective actions (e.g., engaging a lens cleaning device, decelerating the vehicle, etc.) or be determined that the vehicle may continue safe operation despite the obstruction being insignificant.

Fig. 6A-6C illustrate examples of how debris can at least partially block the field of view of an optical sensor. Fig. 6A shows an exemplary image collected by the optical sensor of AV100 during a period in which the leaf partially blocks the field of view of the optical sensor. In fig. 6A, a leaf is stuck in a sensor window associated with an optical sensor. The blockage may end after the wind transition, or even in response to the car stopping, but it may still affect the ability of the AV100 to monitor its surroundings, regardless of how long the blockage lasts. The image in fig. 6A shows that the leaves are only slightly out of focus. This may be the case if the optical sensor has a very large depth of field or if the bending of the leaf prevents the leaf from being blown flat against the sensor window associated with the optical sensor.

Fig. 6B shows another example where road debris is very close to the optical sensor so that only the perimeter of these road debris can be clearly distinguished. This may be the case: road debris has a flat or flattened surface against the sensor window making it difficult to characterize the road debris. Typically, due to the large depth of field of the optical sensor, the object will only appear as blurred when in contact with the sensor window.

Fig. 6C shows another example of the road debris in the form of plastic bags. In this case, even though the plastic bag covers a large part of the field of view of the optical sensor, useful images can still be received by at least some of the covered portions of the image. For this reason, it may be beneficial for the occlusion detection system to characterize not only which parts of the image are occluded, but also whether any part of the occluded part is at least partially transparent, and still thereby capture relevant information.

As discussed above, while a human may be able to easily infer from the examples of fig. 6A-6C that an occlusion occurred, it may be more difficult for a computing device to do so. That is, the computing device may typically (e.g., without sufficient dedicated programming) be unable to distinguish between the occlusion examples of fig. 6A-6C and the camera image taken without occlusion. One way to enable a computing device to distinguish between examples with and without blocking is to utilize machine learning techniques. However, these techniques typically require a large amount of data to generate an accurate machine learning model. In practice, blocking may occur relatively infrequently, and thus it may be difficult to train such a model using only non-synthetic imagery.

Embodiments of the present disclosure address these issues by providing for the synthetic generation of highly accurate input data representing a camera barrier. In particular, according to embodiments of the present disclosure, blocking objects (e.g., leaves, water, dirt, etc.) may be captured against a known background, such as a "green screen" or other background of a known color range, etc. Fig. 7 illustrates an example system 700 for capturing an image of an obstructing object relative to such a background. In fig. 7, one of the blocking objects is a road fragment 704, and the background is a chroma key background 718. An image of the road debris 704 may be captured by the optical sensor 702. The location of the road debris 704 and/or the sensor 702 may vary, such as to capture images of various types of road debris 704 at different distances from the optical sensor 702, and so forth. In some embodiments, the distance may be marked by markings (indicia) 706-716 to assist in locating the road debris 704 at a consistent distance from the optical sensor 702. In one embodiment, the road debris 704 is suspended in front of the chroma-key background 718 by fishing line or other support structure that is unlikely to be captured by the optical sensor 702. The support structure may be the same or similar color as the chroma-key background 718 to make it easier to remove from the image. In another embodiment, the road debris 704 is attached to a transparent medium such as a window or directly to the sensor 702 (e.g., on the lens or housing of the sensor). Chroma-key background 718 may take many forms including a blue screen or a green screen. The chroma-key background 718 enables the extraction of the image of the road debris 704 from its background and makes it easier to perform an analysis of the transparency of the various portions of the imagery showing the road debris. For example, in the case where the road debris 704 takes the form of a plastic bag similar to the plastic bag depicted in fig. 6C, the color of the chroma-key background 718 enables the transparency of the plastic bag to be accurately determined by the intensity or amount of plastic bag illumination. Although only a single road debris 704 is depicted, it should be understood that in some embodiments, multiple road debris 704 may be imaged at once to illustrate a configuration in which multiple objects are blocking the field of view of the optical sensor. While road debris 704 is one example of an obstacle, other scenarios are possible. For example, a transparent window showing water droplets, mud splash, condensation, or the like may be used in the configuration of fig. 7 in addition to or in place of the road debris 704.

Fig. 7 also shows how the system 700 may include one or more light sources 720, where the light sources 720 may be moved relative to the road debris 704 and/or the sensor 702. This enables the system 700 to more accurately model the illumination of road debris in more general situations. Controlling the illumination variations may help to provide a wider variety of appearances for road fragments, given that illumination may have a significant impact on the appearance of the road fragments. For example, road debris in the case of backlighting may have very different appearances and/or transparencies depending on the illumination source/location. The back-lit plastic bag may have different transparency according to illumination. Illumination may also be used to simulate the effects of the sun present within the field of view of the optical sensor.

Fig. 8 visually illustrates a process for combining data extracted from a partial occlusion image 802 captured by the system 700 with a separate background image 808 (e.g., captured by a sensor of a vehicle without an occlusion) to generate synthetic input data suitable for training a neural network. The process of FIG. 8 may be implemented, for example, by computing system 300 of FIG. 3.

In fig. 8, an image 802 includes imagery of road debris on a solid background 803 (e.g., corresponding to a chroma-key background 718). The image of the road debris can be extracted from the monochrome background 803 by performing a chroma-key operation. Various parameters for the chromakeying operation may be set according to the details of image 802. For example, a threshold transparency value for separating road debris from the background may be set to a sufficiently high value (e.g., 90%, 95%, 97%, 98%, 99%, etc.) to accurately remove the background 803. The chroma keying operation may produce a mask 804 that shows the blocked portion of the field of view of the optical sensor in white (corresponding to the road debris depicted in the image) and the unblocked portion of the field of view of the optical sensor in black. Mask 804 may take the form of an RGB or binary mask. The chromakeying operation also enables a good definition of the transition between the white and black areas, which shows a gradual change in transparency of the chip at its edges. The chroma-keying operation may also enable the creation of a texture layer 806 representing the portion of the image 802 that constitutes a road fragment. Thus, the texture layer 806 may be used to fill the white portions of the mask 804 to produce an image that includes road debris but excludes the background color 803. It should be noted that the checkered portion of the texture layer 806 represents a transparent portion of the texture layer 806. In some embodiments, the extracted pixels of the road debris object may be increased such that the perimeter of the extracted pixels is placed outside the white area of the mask 804. Increasing the size of the road fragments in the texture layer 806 may be done to help prevent the inclusion of any colors that form edges along the perimeter of the road fragment object caused by the chroma-key background. In some embodiments, the image of the road debris may be rotated by different amounts before or after the background color 803 is removed to provide more variation in the data set before the chromakeying operation is performed. For example, the image 802 may be radially offset by 1 to 15 degrees to produce a large number of block images from a single image. In some embodiments, where multiple images of road fragments are to be added to a composite image, different road fragments may be rotated by different amounts.

The texture layer 806 may be combined with the mask 804 and overlaid on the background image 808 to produce a composite training image 810. The background image 808 may represent a suitable background that may otherwise be captured in the non-composite occlusion image. For example, background image 808 may be captured by an optical sensor on AV100 and thereby represent the environment of AV 100. The background image 808 may include imagery of streets, highways, and/or traffic intersections. To increase the accuracy of the composite image, the image sensor used to capture the background image 808 may be the same as or similar to the optical sensor 702 used to capture road debris or blockage items.

In some embodiments, the mask 804 may be stored as metadata with the composite training image 810. For example, since the mask 804 indicates which pixels of the resulting composite image 810 are blocked by road fragments and which pixels are not, the mask 804 may be used as a pixel level blocking annotation. In some embodiments, image-level blocking annotations may be created as part of the generation of the composite image. The image level block callout is a binary value that indicates whether the image 810 constitutes a "block" image. In one embodiment, the binary value indicates whether the field of view is blocked by an amount exceeding a predetermined threshold. As discussed below, the blocking threshold may be set according to operational requirements for training the machine learning model. For example, the threshold may be set in a manner that attempts to match a human operator's assessment of whether the image includes significant obstruction. Significant blocking, in turn, may depend at least in part on the use of the image. For example, the threshold may be lower for optical sensors that are more critical to autonomous navigation of AV100, since more minor obstructions may have a greater impact on the operation of AV 100. In one embodiment, the threshold is set to one or more of 50%, 60%, 70%, 80%, and 90%.

Although fig. 8 depicts the generation of a single composite image 810, the process of fig. 8 may be repeated multiple times to generate a sufficient number of composite images 810 for accurate training of the machine learning model. For example, the apparatus 300 may be provided or configured to generate a variety of background images 808, such as via extraction of such images from sensor data during operation of the AV 100. Similarly, the apparatus 300 may be provided with a plurality of images 802 comprising blocking objects on a known background 803. Each different combination of images 802 and 808 may produce a different composite image 810, thereby enabling a large number of composite images 810 to be generated quickly. For example, the combination of 100 images 802 with blocking objects and 10,000 background images 808 would produce one million different composite images 810. Programmed alterations of the image 802 (such as blocking rotation of the object to multiple orientations, etc.) will also increase the number. Thus, the process of fig. 8 may enable preparation for generating a large number of composite images 810, thereby enabling training of highly accurate machine learning models.

As described above, while the process of fig. 8 may provide highly accurate synthetic image generation, it may not be desirable in all cases to train a machine learning model based solely on synthetic images. For example, such training may result in overfitting, thereby enabling the machine learning model to identify distinctiveness (pedigree) in the synthetic image that is not present in the non-synthetic image, thereby enabling highly accurate detection of obstructions in the synthetic image without accurately detecting obstructions in the non-synthetic image. This is an undesirable result since the goal of the model may be to detect occlusions in the non-synthetic image.

To address this issue, embodiments of the present disclosure may provide for training of machine learning models based on both synthetic and non-synthetic images, thereby avoiding overfitting of the synthetic images without disadvantageously lacking data resulting from training without synthetic data. In one embodiment, a neural network machine learning model is used and trained to detect which portions (e.g., pixels) of an image are blocking and which portions are not (e.g., reflect the environment of the sensor other than blocking). In order to train the neural network to detect which parts of an image are blocked, it may be necessary to provide an input that includes both the image with the blockage and information identifying which parts of the image contain the blockage. As described above, the processes disclosed herein may enable such information to be quickly generated for a composite image. However, providing this information for non-composite images may be difficult or cumbersome. For example, providing this information may typically require a human to manually view the individual images and specify which portions of the images are blocked at a given granularity. This process may require a significant amount of time and may have limited accuracy, particularly as the granularity increases. Thus, a human-classified image, such as a non-synthesized image, may have less blocking information than synthesized information granularity, where the blocking information may be determined directly during synthesis. For example, the human-classified image may have a binary obstruction indicator (e.g., the presence or absence of obstruction) or a low granularity indicator (e.g., the presence of obstruction in a given quadrant) relative to the composite image.

To address this issue, embodiments of the present disclosure provide a neural network architecture that can be trained using a combination of images with high granularity blocking information (e.g., pixel-level by pixel) and images with low granularity blocking information (e.g., binary blocking indicators) as input data to generate highly accurate (e.g., pixel-level) annotations containing blocked non-synthetic images. An example of such an architecture is shown in fig. 9 as a neural network training architecture 900. The architecture 900 may be implemented, for example, on the apparatus 300 described in fig. 3. All or a portion of architecture 900 may also be implemented on an autonomous vehicle (e.g., via computing processor 146), such as AV100 of fig. 1. For example, the architecture 900 may be implemented on a device 300 external to the AV100 for training purposes, and a portion of the architecture 900 (such as the box surrounded by the dashed line 914, etc.) may be implemented within the AV100 after training to make inferences about non-synthetic input data (e.g., to control operations of the AV 100).

As shown in fig. 9, the neural network training architecture 900 includes a neural network 902, the neural network 902 configured to obtain data corresponding to an image 901 as input, process the data according to the neural network 902, and output pixel-level occlusion data 904 for the input data. The image 901 may be, for example, a composite image generated via the process of fig. 8. Alternatively, the image 901 may be a non-synthetic image, such as a non-synthetic image captured during operation of the AV 100. The processing within the neural network 902 may include various operations to transform the input into corresponding pixel-level barrier data 904. For example, the neural network 902 may be a convolutional neural network, in which the image data is passed through one or more convolutions to extract features from the image data. The neural network 902 may also include one or more hidden layers (e.g., fully connected layers) in which features of the image data are multiplied by weights of various nodes within the hidden layers to potentially activate the nodes, where the hidden layers are connected to an output layer that indicates, for example, whether various pixels correspond to a blockage.

Initially, the neural network 902 may be untrained. As such, the output pixel level blocking data 904 generated by the architecture 900 in its initial state may have low accuracy (e.g., substantially equal to chance). Thus, an image 901 with known occlusion information may be passed through the network 902 to train the network 902, enabling the characteristics of the network (e.g., convolution of hidden layers and/or weights) to be set in a manner that matches the output of the network 902 (pixel-level occlusion data 904) to the known occlusion information.

In the case of an image 901 with known pixel level blocking data 907, training may occur directly from such known pixel level blocking data 907. By way of illustration, where the image 901 is a composite image generated by the process of fig. 8, the image 901 may be accompanied by known pixel level blocking data 907 (e.g., corresponding to the mask 804) that indicates, in units of pixels, whether the pixels correspond to a barrier (and potentially the transparency of such a barrier). Thus, as shown in fig. 9 as a comparison 906 (which may, for example, represent a loss function), the output pixel level blocking data 904 generated by the neural network 902 for the image 901 may be compared to the known pixel level blocking data 907 to adjust the neural network 902 (e.g., via back propagation) so that the output pixel level blocking data 904 more closely matches the known pixel level blocking data 907.

As discussed above, training based solely on synthetic images may not be desirable and may result in drawbacks such as overfitting. Further, non-synthetic data may not include annotations of the same granularity as synthetic data. For example, such non-synthetic data may have only known binary barrier data 908. Thus, the architecture 900 also enables training based on image data without known pixel-level blocking data 907 (such as non-synthetic image data that is manually tagged as including or not including blocking, etc.). In architecture 900, data representing an image 901 without corresponding known pixel-level blocking data 907 is processed via a neural network 902 in the same manner as an image with known pixel-level blocking data 907, thereby generating pixel-level blocking data 904. Comparison 906 may not be feasible because image 901 in this example is assumed to have no known pixel-level blocking data 907. Instead, the pixel-level barrier data 904 output by the neural network 902 is converted to output binary barrier data 910 via a conversion function. By way of illustration, the transfer function may represent thresholding such that the output binary barrier data 910 is true if the output pixel-level barrier data 904 satisfies the threshold. The threshold may be set to an appropriate value (e.g., a value commensurate with an expected threshold of a human operator who manually generated the known binary blocking data 908). Thus, the output binary occlusion data 910 of the neural network 902 may represent an estimate of whether the network 902 has tagged the input image 901 to be occluded to a human operator. Thereafter, the output binary barrier data 910 is compared to the known binary barrier data 908 at a comparison 912, which may, for example, represent a loss function of the network 902. The result of this comparison 912 is then used to modify the network 902 such that the boot output binary barrier data 910 matches the known binary barrier data 908, thereby training the network 902.

In practice, training of the neural network 902 may occur by passing a plurality of images 901 through the architecture 900, where the plurality of images 901 includes both images with known pixel-level blocking data 907 and images 901 with known binary blocking data 908. In this manner, a trained network 902 is generated. Thereafter, during inference, the elements indicated by block 914 may be used to generate output pixel level blocking data 904 from new non-training data corresponding to an image 901, where the image 901 may lack both known pixel level blocking data 907 and known binary blocking data 908. However, the network 902 may enable devices (such as processors within the AV 100) to generate output pixel level blocking data 904 for such input images 901. Thus, a device implementing network 902 may replicate a human's subjective evaluation via an objective evaluation to determine which portions of an image are blocked and potentially to what extent the various portions are blocked. This in turn can enable more accurate and secure operation of devices (such as AV 100) that rely on sensor data that generate image 901.

Example synthetic partial blocking training data set Generation routine

FIG. 10 is a flow diagram depicting an example routine 1000 for generation of a synthetic partial occlusion training data set. The routine 1000 may be implemented, for example, by the computer systems 206a-206f of FIG. 2.

The routine 1000 begins at block 1002, where the computer systems 206a-206f obtain data representing a partially occluded image against a chroma-key background. The image may illustratively be captured by a camera system, such as system 700 of fig. 7. Each image may depict a partial obstruction (e.g., road debris, water, mud, condensation, etc.) in front of one or more chroma-key backgrounds (such as green screens, etc.). The size, shape, orientation and type of the partial obstruction may vary between images. For example, the size, shape, orientation, and type of obstructions within the obtained data may be selected to provide a representative sample of partial obstructions that may be experienced during operation of a target device, such as an autonomous vehicle. The number of obstructions may vary between images. For example, some images may depict a single item that constitutes a barrier, while other images may depict multiple items or different types of barriers in different portions of the image. The blocked lighting within the image may also vary in intensity, location, and color to simulate the appearance of the obstruction during different times of the day and/or during particular types of weather. In some embodiments, the barrier may be subjected to wind forces during image capture to simulate movement of the barrier. Movement of the obstruction may cause motion blur in the image, which may produce different transparency effects in the resulting image. Further, the chroma-key background may vary between images.

At block 1004, the computer systems 206a-206f perform a chroma keying operation to extract the shadow of the barrier from the image. The chroma-keying operation is illustratively used to remove all colors corresponding to the relevant chroma-key background for each image, leaving only the imagery of the blocking item. Thus, the chroma-keying operation enables the computer systems 206a-206f to distinguish between pixels that constitute barriers (pixels that are not removed by the chroma-keying operation) and pixels that constitute the background.

In some embodiments, the chromakeying operation may take into account the partial transparency of the barrier. For example, a translucent barrier may partially but not completely block the chroma-key background. Thus, the chroma keying operation may set a partial transparency to pixels representing partially transparent barriers. In one example, the chroma keying operation assumes that the pixel color in the partially blocked image is a weighted average of the chroma-key background color and the blocking color, and thus sets the color and transparency value of the pixel based on the difference between the pixel color, the blocking color, and the background color (e.g., such that the background color is removed and the pixel assumes that the color of the barrier has a transparency value determined according to the degree to which the pixel color matches the chroma-key background color relative to the blocking color).

At block 1006, the computer systems 206a-206f superimpose the extracted imagery of the obstruction on one or more background images to generate a composite partial obstruction image. In some embodiments, the extracted imagery is used to create texture. The texture may then be applied to the areas of the image that were identified in the chroma keying operation as corresponding to the barrier. In some embodiments, the size/scale and/or orientation of the obstacle may be changed prior to superimposing the image of the obstacle over the background image to create a wider variety of composite partial obstacle images. For example, the barrier may be made larger to constitute a larger barrier, or smaller to constitute a smaller barrier. The extracted individual obstacle images may be superimposed on the background image multiple times, for example, in different sizes or orientations. Further, the extracted individual obstacle images may be superimposed on a plurality of different background images. In this way, a wide variety of composite partial block images can be generated from a small number of captured partial block images.

Additionally, at block 1008, the computer systems 206a-206f generate annotation data for each of the synthetic partial block images that distinguishes the portion of each of the synthetic partial block images that constitutes the obstruction from the portion representing the background image. By way of illustration, the annotation data can identify individual pixels of the respective composite barrier image (e.g., those pixels that were not removed during the chroma keying operation) that correspond to the imagery extracted at block 1004. Conversely, the annotation data may identify pixels corresponding to the background image as constituting the background rather than the obstacle. In some instances, the annotation data can also indicate the transparency of the barrier, such as by associating an opacity value with each pixel identified as a barrier. The annotation data may thus provide high-particle-size markers for each composite partial block image, as described below.

At block 1012, the computer systems 206a-206f train the machine learning model using the annotation data and the composite portion blocking images. For example, the composite partial occlusion image may be passed through a convolutional neural network to train the network to identify partial occlusions within the imagery. As described above, training a machine learning model to produce accurate results may require a large amount of data. Thus, the ability of the routine 1000 to generate a wide variety of composite partial occlusion images from a limited set of captured partial occlusion images may be beneficial in training.

As discussed above, in some examples, it may not be desirable to train a machine learning model based solely on synthetic data. For example, the model may be overfit to the synthesized data, such as by learning to distinguish between synthesized and non-synthesized images. Such overfitting may make the model inaccurate during use in a non-synthetic environment (which may be the desired deployment environment). Thus, in some embodiments, training at block 1012 may include training the model on a combination of synthetic data (such as synthetic data generated during implementation of routine 1000, etc.) and non-synthetic data (such as manually labeled barrier data, etc.). In some such instances, non-synthetic data may be associated with lower granularity annotations than synthetic data. For example, rather than having pixel-level labeling, non-synthesized data may be manually classified with binary labeling or other non-pixel-level labeling. One example routine 1100 for training a neural network with training data having multiple levels of granularity is shown in FIG. 11. Accordingly, an implementation of block 1012 may include an implementation of routine 1100 of FIG. 11.

As described above, fig. 11 depicts an example routine 1100 for training a neural network (such as a convolutional neural network, etc.) with training data having multiple levels of granularity. Routine 1100 may be implemented, for example, by computer systems 206a-206f of FIG. 2. By way of illustration, routine 1100 may be used to train a model that enables an autonomous vehicle to identify portions of two-dimensional sensor data that make up obstructions to the sensor, thereby enabling more accurate and safer operation of the vehicle.

Routine 1100 begins at block 1102, where the computer systems 206a-206f obtain training data that includes both data with high granularity flags and data with low granularity flags. The data may, for example, represent a 2D image such as a camera image. High granularity flags may indicate that a particular portion of the imagery (if any) constitutes a feature (such as a blockage, etc.) that the network is to be trained to distinguish. While blocking is one example of a feature that a neural network may be trained to distinguish, other features are possible. For example, routine 1100 may be used to train a model to distinguish the presence or absence of other depictions within an image (such as pedestrians, object types, etc.). Low granularity flags may similarly indicate whether features are present in the respective image, but do so at a lower granularity than high granularity flags. In one embodiment, high granularity labels are pixel level labels that indicate which pixels of an image correspond to features (or conversely, which pixels do not correspond to features), while low granularity labels do not provide pixel level labels, but rather provide lower granularity labels, such as region labels (e.g., which half of an image, which quartile(s), etc., comprises a shadow) or binary labels (e.g., indicating the presence or absence of a feature within an image), etc. In another embodiment, the high granularity marking is a region marking and the low granularity marking is a binary marking.

Thereafter, the computer systems 206a-206f train the neural network for high granularity output, such that the network, once trained, can provide output indicating which portions of a given input constitute features that the network is trained to detect, with granularity equal to the granularity of training data having high granularity labels. For example, where a high granularity portion of the input data set provides blocked pixel-level labeling, the network may be trained to output the blocked pixel-level labeling given an input image that includes some barriers.

At block 1104, training the neural network illustratively includes: items from the training data set are iteratively fed through the network, and the weights of the network are updated based on a comparison between the output of the network and the labels of the individual data items. As shown at block 1106, for each item from the training data set, routine 1100 then varies depending on whether the item is associated with a high granularity tag or a low granularity tag.

Where the item is associated with a high granularity flag, routine 1100 proceeds to block 1108, where the neural network is updated based on a comparison of the output of the network to the high granularity flag. For example, the systems 206a-206f may implement back propagation based on the difference between the output of the network corresponding to the predicted value of the high granularity marker and the actual value of the high granularity marker, thereby updating the weight of the network. Through multiple iterations, the network can thus be trained to accurately predict the values of high-particle-size markers.

In the event that the item is associated with a low granularity marker, routine 1100 proceeds to block 1110, where the output of the network is converted from high granularity to low granularity. To implement the transformation, the computer systems 206a-206f may deliver a high granularity output through a transformation function that may be customized (tailor) for particular data in the training data set. For example, where the low granularity data is binary data, the conversion function may determine whether a threshold amount of output is indicative of a given feature (e.g., whether a threshold number of pixels, regions, etc. constitute a block), and output a "true" value when the threshold is met, or conversely, a "false" value when the threshold is not met. Where low granularity data is a region indication of a feature, such as by outputting a "presence" indicator of a feature of a region when a threshold of the region is indicated as possessing at least a feature of a high granularity output, the transfer function may similarly evaluate a portion of the output corresponding to the region relative to the threshold. In one embodiment, the transfer function itself may be a machine learning model. For example, a transfer function machine learning model may be initialized with a set of weights to serve as a simple thresholding function while training a neural network for generating a high granularity output indicative of a given feature. When sufficient accuracy is achieved at the neural network, the weights of the neural network may be kept constant during training of the conversion function model. In one example, the neural network and the transfer function machine learning model (which may itself be a neural network) may be trained simultaneously, such as by iteratively keeping one of the two models constant at each iteration while training the other model. Thus, the high granularity output of the network is converted to a lower granularity, matching the granularity of the item's tokens.

Thereafter, at block 1112, the network is updated based on the comparison of the output of the converted network to the low granularity flag. For example, the systems 206a-206f may implement back propagation based on the difference between the converted output of the network corresponding to the predicted value of the low granularity marker and the actual value of the low granularity marker, thereby updating the weight of the network. Through multiple iterations, the network can thus be trained to accurately predict the values of high granularity tokens, which when transformed match the low granularity tokens assigned to the item.

Routine 1100 then varies depending on whether there is more data within the training data set, as shown in block 1114. If so, the routine 1100 returns to block 1104, where in block 1104 the weights of the network are updated using additional terms from the training data set as described above. If no further training data exists in the dataset, routine 1100 proceeds to block 1116 where the trained model may be deployed, output, or transmitted to the destination computing device for use during inference. For example, a model trained to provide pixel-level blockage indications of 2D imagery (e.g., implemented by the processor 146 of fig. 1 based on sensor data obtained from the camera 112) may be deployed, output, or transmitted to the autonomous vehicle to enable the vehicle to accurately detect the presence and extent of blockage of the sensor, and if such blockage occurs, take appropriate corrective action. Such corrective action may include, for example, cleaning the sensor (e.g., via a wiper mechanism or the like), notifying the operator, or performing minimum risk maneuvers such as decelerating and leaving the lane, etc. Thus, the neural network trained via routine 1100 may provide safer and more accurate use of sensor data in an autonomous vehicle or other context.

Various example embodiments of the present disclosure may be described by the following clauses:

clause 1. A computer-implemented method, implemented by one or more hardware processors, the method comprising:

obtaining, using the one or more hardware processors, data representing partial block images, each partial block image depicting a block in front of a chroma-key background;

generating, using the one or more hardware processors, a training data set comprising a plurality of synthetic partial occlusion images, wherein generating the training data set comprises: for each of the composite partial block images,

performing a chroma keying operation to extract an image of a barrier from a partially blocked image of the partially blocked images,

superimposing the extracted image of the obstacle on the background image to generate the composite partial obstacle image, an

Generating annotation data for the composite portion block image, the annotation data for distinguishing a portion of the composite portion block image representing imagery of the extracted obstacle from a portion of the composite portion block image representing the background image; and

training, using the one or more hardware processors, a neural network with the plurality of synthetic partial block images to identify portions of the input image data corresponding to data from a sensor of an autonomous vehicle that represent obstructions on the sensor of the autonomous vehicle.

Clause 2. The computer-implemented method of clause 1, wherein the annotation data identifies pixels in the background image that are blocked by the superimposed image of the extracted barrier.

Clause 3. The computer-implemented method of clause 1 or 2, wherein the chroma-key background is a green screen.

Clause 4. The computer-implemented method of any of clauses 1-3, wherein the obstruction is a substance attached to a lens of an imaging device used to capture the partially obstructed image.

Clause 5. The computer-implemented method of any one of clauses 1-4, wherein the neural network is a convolutional neural network.

Clause 6. The computer-implemented method of any one of clauses 1-5, further comprising: during navigation of a roadway by a vehicle carrying a vehicle-mounted camera, a background image is randomly selected from a plurality of background images captured by the vehicle-mounted camera.

Clause 7. The computer-implemented method of clause 6, wherein the camera mounted to the vehicle is substantially identical to the camera used to capture the partial occlusion image.

Clause 8. The computer-implemented method of any of clauses 1-7, wherein performing the chroma keying operation comprises: the extracted portion of the image of the barrier is associated with a transparency value.

Clause 9. The computer-implemented method of any one of clauses 1-8, wherein generating the training data set comprises: rotating an image of the extracted obstacle for at least one of the partially obstructed images.

Clause 10. The computer-implemented method of any of clauses 1-9,

wherein for at least one of the partial occlusion images a composite partial occlusion image, the image of the extracted occlusion depicts two occlusion portions, an

Wherein generating the training data set comprises: blocking the image for the at least one synthesized portion,

rotating a first blocked portion of the extracted image by a first amount; and

rotating a second blocked portion of the extracted image by a second amount different from the first amount.

Clause 11. The computer-implemented method of clause 10, further comprising: the first blocked portion of the extracted blocked item's imagery is resized by a first amount and the resolution of the second subset of the extracted blocked item's imagery is resized by a second amount prior to superimposing the extracted imagery on a different background image.

Clause 12. The computer-implemented method of clause 1, wherein performing the chroma-keying operation comprises: any image having a transparency value below the threshold is extracted.

Clause 13. The computer-implemented method of clause 12, wherein the threshold is at least 90%.

Item 14. The computer-implemented method of item 1, wherein generating the training data set for at least one of the partial block images comprises: superimposing the extracted image of the obstacle and an image of a second obstacle on the background image to generate the composite partial obstacle image.

Clause 15. The computer-implemented method of clause 14, further comprising: performing a chroma keying operation to extract imagery of a second barrier from at least one of the partially blocked images.

Clause 16. The computer-implemented method of any of clauses 1-15, wherein the partial obstruction image comprises at least two images depicting a given obstruction at different distances from a device used to capture the at least two images.

Clause 17. The computer-implemented method of any of clauses 1-16, wherein the first subset of the partially blocked images is captured at a first illumination level and the second subset of the partially blocked images is captured at a second illumination level different from the first illumination level.

Clause 18. The computer-implemented method of clause 17, wherein the light source for illuminating the barrier is located at a first location to create the first illumination level and a second location different from the first location to create the second illumination level.

Clause 19. The computer-implemented method of clause 18, wherein the amount of light emitted by the light source is greater at the first light level than at the second light level.

Clause 20. The computer-implemented method of clause 18, wherein the light source is a directional light source, the directional light source being positioned in a first direction and emitting a first amount of light to create the first illumination level, and wherein the light source is positioned in a second direction and emitting a second amount of light to create the second illumination level.

Clause 21. A system, comprising:

a data storage device comprising computer executable instructions; and

a processor configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to:

obtaining data representing partial block images, each partial block image depicting a block in front of a chroma-key background;

generating a training data set comprising a plurality of synthetic partial occlusion images by at least: for each of the composite partial block images,

performing a chroma keying operation to extract a shadow of the obstruction from a partially obstructed view of the partially obstructed view,

superimposing the extracted image of the obstacle on the background image to generate the composite partial obstacle image, an

Generating annotation data for the composite portion block image, the annotation data for distinguishing a portion of the composite portion block image representing imagery of the extracted obstacle from a portion of the composite portion block image representing the background image; and

training a neural network with the plurality of synthetic partial obstruction images to identify portions of the input image data corresponding to data from a sensor of an autonomous vehicle that represent obstructions on the sensor of the autonomous vehicle.

One or more non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system comprising a processor, cause the computing system to:

obtaining data representing partial block images, each partial block image depicting a block in front of a chroma-key background;

generating a training data set comprising a plurality of synthetic partial occlusion images by at least: for each of the composite partial block images,

performing a chroma keying operation to extract an image of a barrier from a partially blocked image of the partially blocked images,

superimposing the extracted image of the obstacle on the background image to generate the composite partial obstacle image, an

Generating annotation data for the composite portion block image, the annotation data for distinguishing a portion of the composite portion block image representing imagery of the extracted obstacle from a portion of the composite portion block image representing the background image; and

training a neural network with the plurality of synthetic partial obstruction images to identify portions of the input image data corresponding to data from a sensor of an autonomous vehicle that represent obstructions on the sensor of the autonomous vehicle.

Various additional example embodiments of the present disclosure may be described by the following additional clauses:

clause 1. A computer-implemented method, implemented by one or more hardware processors, the method comprising:

training, using the one or more hardware processors, a neural network based on a training data set to output data of a first granularity that is higher than a second granularity, the training data set including a first subset of items labeled at the first granularity and a second subset of items labeled at the second granularity that is lower than the first granularity, wherein training the neural network comprises:

for items within a first subset of items labeled at the first granularity higher than the second granularity:

passing items labeled at the first granularity through the neural network to generate an output corresponding to items labeled at the first granularity, an

Updating weights of the neural network based on a comparison between the outputs corresponding to items labeled at the first granularity and labels of items corresponding to the first granularity;

for items within a second subset of items labeled at the second granularity lower than the first granularity:

passing items labeled at the second granularity through the neural network to generate outputs corresponding to items labeled at the second granularity,

passing the output corresponding to the items marked at the second granularity through a conversion function to convert the output corresponding to the items marked at the second granularity to a converted output of the second granularity, an

Updating weights of the neural network based on a comparison between the converted output of the second granularity and labels of items corresponding to the second granularity.

Clause 2. The computer-implemented method of clause 1, wherein the neural network is a convolutional neural network.

Clause 3. The computer-implemented method of clause 1 or 2, wherein the first subset of items labeled at the first granularity is an image with a pixel-level label identifying a location of a feature within the image of the first subset.

The computer-implemented method of clause 4. The method of clause 3, wherein the second subset of items labeled at the second granularity are images having at least one of non-pixel-level region labels for identifying a location of a feature within the images of the second subset and binary labels for identifying whether a feature is present within the images of the second subset.

Clause 5. The computer-implemented method of clause 1 or 2, wherein the first subset of items labeled at the first granularity is images with non-pixel-level region labeling, and wherein the second subset of items labeled at the second granularity is images with binary labeling identifying a location of a feature within the images of the second subset, the binary labeling identifying whether a feature is present within the images of the second subset.

Clause 6. The computer-implemented method of any of clauses 3-5, wherein the feature is a sensor barrier.

Clause 7. The computer-implemented method of any of clauses 1-6, wherein the transfer function is a thresholding function for converting tokens of the first granularity to tokens of the second granularity.

Clause 8. The computer-implemented method of clause 7, wherein the transfer function is a machine learning model.

Clause 9. The computer-implemented method of clause 8, wherein the machine learning model is a second neural network.

Clause 10. The computer-implemented method of clause 8 or 9, wherein the machine learning model is trained concurrently with the neural network.

Clause 11. The computer-implemented method of any of clauses 1-10, wherein the second subset of items labeled at the second granularity lower than the first granularity comprises manually labeled items.

Clause 12. The computer-implemented method of any of clauses 1-11, wherein the first subset of items labeled at the first granularity that is higher than the second granularity comprises programmatically-generated synthetic items.

Clause 13. The computer-implemented method of clause 12, further comprising: a programmatically generated synthetic term is generated,

wherein generating the programmatically-generated synthetic term comprises: for each of the programmatically-generated synthetic terms:

performing chroma keying operation to extract characteristic image from partial characteristic image,

superimposing the imagery of the extracted features on the background image to generate a programmatically generated composite term, an

Generating annotation data for the programmatically-generated synthetic item for distinguishing portions of the imagery of the programmatically-generated synthetic item representing the extracted features from portions of the programmatically-generated synthetic item representing the background image.

Clause 14. The computer-implemented method of any of clauses 1-13, further comprising: deploying the neural network after training to infer for input and provide output of the first granularity.

Clause 15. The computer-implemented method of clause 14, wherein deploying the neural network after training to infer for input and provide the first granularity of output comprises: deploying the neural network to an autonomous vehicle.

Clause 16. The computer-implemented method of clause 15, wherein deploying the neural network to an autonomous vehicle enables the autonomous vehicle to identify a portion of the input image data corresponding to data from a sensor of the autonomous vehicle that represents an obstacle on the sensor of the autonomous vehicle.

Clause 17. A system, comprising:

a data storage device comprising computer executable instructions; and

a processor configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to:

training a neural network based on a training data set to output data at a first granularity that is higher than a second granularity, the training data set including a first subset of items labeled at the first granularity and a second subset of items labeled at the second granularity that is lower than the first granularity, wherein training the neural network comprises:

for items within a first subset of items labeled at the first granularity that is higher than the second granularity:

passing items labeled at the first granularity through the neural network to generate an output corresponding to items labeled at the first granularity, an

Updating weights of the neural network based on a comparison between the output corresponding to items labeled at the first granularity and labels of items corresponding to the first granularity;

for items within a second subset of items labeled at the second granularity lower than the first granularity:

passing items labeled at the second granularity through the neural network to generate outputs corresponding to items labeled at the second granularity,

passing outputs corresponding to items marked at the second granularity through a conversion function to convert outputs corresponding to items marked at the second granularity to converted outputs of the second granularity, an

Updating weights of the neural network based on a comparison between the converted output of the second granularity and labels of items corresponding to the second granularity; and

storing the trained neural network for subsequent deployment to infer for input and provide output of the first granularity.

Clause 18. The system of clause 17, wherein the first subset of items labeled at the first granularity is an image with a pixel-level label identifying a location of a feature within the image of the first subset.

Clause 19. The system of clause 18, wherein the feature is a sensor barrier.

The system of any of clauses 17-19, wherein deploying the neural network after training to infer for input and provide output at the first granularity comprises: deploying the neural network to an autonomous vehicle, wherein deploying the neural network to an autonomous vehicle enables the autonomous vehicle to identify a portion of the input image data corresponding to data from a sensor of the autonomous vehicle that represents a sensor barrier on the sensor of the autonomous vehicle.

Clause 21 one or more non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system comprising a processor, cause the computing system to:

training a neural network based on a training data set to output data at a first granularity that is higher than a second granularity, the training data set including a first subset of items labeled at the first granularity and a second subset of items labeled at the second granularity that is lower than the first granularity, wherein training the neural network comprises:

for items within a first subset of items labeled at the first granularity higher than the second granularity:

passing items labeled at the first granularity through the neural network to generate an output corresponding to items labeled at the first granularity, an

Updating weights of the neural network based on a comparison between the outputs corresponding to items labeled at the first granularity and labels of items corresponding to the first granularity;

for items within a second subset of items labeled at the second granularity lower than the first granularity:

passing items labeled at the second granularity through the neural network to generate outputs corresponding to items labeled at the second granularity,

passing outputs corresponding to items marked at the second granularity through a conversion function to convert outputs corresponding to items marked at the second granularity to converted outputs of the second granularity, an

Updating weights of the neural network based on a comparison between the converted output of the second granularity and labels of items corresponding to the second granularity; and

storing the trained neural network for subsequent deployment to infer against inputs and provide outputs of the first granularity.

In the previous description, embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the claims, and what is intended by the applicants to be the scope of the claims, is the literal and equivalent scope of the claims, including any subsequent correction, that issue from this application in the specific form in which the claims issue. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Additionally, when the term "further comprising" is used in the preceding description or the appended claims, the following of the phrase may be additional steps or entities, or sub-steps/sub-entities of previously described steps or entities.

Claims (22)

1. A computer-implemented method implemented by one or more hardware processors, the method comprising:

obtaining, using the one or more hardware processors, data representing partial block images, each partial block image depicting a block in front of a chroma-key background;

generating, using the one or more hardware processors, a training data set comprising a plurality of synthetic partial occlusion images, wherein generating the training data set comprises: for each of the composite partial block images,

performing a chroma keying operation to extract an image of a barrier from a partially blocked image of the partially blocked images,

superimposing the extracted image of the obstruction on a background image to generate the composite partial obstruction image, an

Generating annotation data for the composite partially blocked image, the annotation data for distinguishing a portion of the composite partially blocked image representing imagery of the extracted barrier from a portion of the composite partially blocked image representing the background image; and

training, using the one or more hardware processors, a neural network with the plurality of synthetic partial obstruction images to identify portions of the input image data corresponding to data from sensors of an autonomous vehicle that represent obstructions on the sensors of the autonomous vehicle.

2. The computer-implemented method of claim 1, wherein the annotation data identifies pixels in the background image that are blocked by the superimposed imagery of the extracted barrier.

3. The computer-implemented method of claim 1 or 2, wherein the chroma-key background is a green screen.

4. A computer-implemented method as in any of claims 1-3, wherein the obstruction is a substance that adheres to a lens of a camera used to capture the partially obstructed image.

5. The computer-implemented method of any of claims 1-4, wherein the neural network is a convolutional neural network.

6. The computer-implemented method of any of claims 1-5, further comprising: during navigation of a roadway by a vehicle carrying a vehicle-mounted camera, a background image is randomly selected from a plurality of background images captured by the vehicle-mounted camera.

7. The computer-implemented method of claim 6, wherein the camera mounted to the vehicle is substantially identical to the camera used to capture the partial occlusion image.

8. The computer-implemented method of any of claims 1-7, wherein performing the chroma keying operation comprises: the extracted portion of the image of the barrier is associated with a transparency value.

9. The computer-implemented method of any of claims 1-8, wherein generating the training data set comprises: rotating the imagery of the extracted obstruction for at least one of the partially obstructed images.

10. The computer-implemented method of any of claims 1-9,

wherein for at least one of the partial occlusion images a composite partial occlusion image, the image of the extracted occlusion depicts two occlusion portions, an

Wherein generating the training data set comprises: blocking the image for the at least one synthesized portion,

rotating a first blocked portion of the extracted image by a first amount; and

rotating a second blocked portion of the extracted image by a second amount different from the first amount.

11. The computer-implemented method of claim 10, further comprising: the first blocked portion of the extracted blocked item's imagery is resized by a first amount and the resolution of the second subset of the extracted blocked item's imagery is resized by a second amount prior to superimposing the extracted imagery on a different background image.

12. The computer-implemented method of claim 1, wherein performing the chroma keying operation comprises: any image with a transparency value below the threshold is extracted.

13. The computer-implemented method of claim 12, wherein the threshold is at least 90%.

14. The computer-implemented method of claim 1, wherein generating the training data set for at least one of the partial block images that is a composite partial block image comprises: superimposing the extracted image of the obstacle and an image of a second obstacle on the background image to generate the composite partial obstacle image.

15. The computer-implemented method of claim 14, further comprising: performing a chroma keying operation to extract imagery of the second barrier from at least one of the partially blocked imagery.

16. The computer-implemented method of any of claims 1-15, wherein the partial obstruction image comprises at least two images depicting a given obstruction at different distances from a device used to capture the at least two images.

17. The computer-implemented method of any of claims 1-16, wherein a first subset of the partially blocked images is captured at a first illumination level and a second subset of the partially blocked images is captured at a second illumination level different from the first illumination level.

18. The computer-implemented method of claim 17, wherein a light source for illuminating a barrier is located at a first location to create the first illumination level and a second location different from the first location to create the second illumination level.

19. The computer-implemented method of claim 18, wherein an amount of light emitted by the light source is greater at the first illumination level than at the second illumination level.

20. The computer-implemented method of claim 18, wherein the light source is a directional light source positioned in a first direction and emitting a first amount of light to create the first illumination level, and wherein the light source is positioned in a second direction and emitting a second amount of light to create the second illumination level.

21. A system for a vehicle, comprising:

a data storage device comprising computer executable instructions; and

a processor configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to:

obtaining data representing partial block images, each partial block image depicting a blockage in front of a chroma-key background;

generating a training data set comprising a plurality of synthetic partial occlusion images by at least: for each of the composite partial block images,

performing a chroma keying operation to extract an image of a barrier from a partially blocked image of the partially blocked images,

superimposing the extracted image of the obstacle on the background image to generate the composite partial obstacle image, an

Generating annotation data for the composite partially blocked image, the annotation data for distinguishing a portion of the composite partially blocked image representing imagery of the extracted barrier from a portion of the composite partially blocked image representing the background image; and

training a neural network with the plurality of synthetic partial obstruction images to identify portions of the input image data corresponding to data from a sensor of an autonomous vehicle that represent obstructions on the sensor of the autonomous vehicle.

22. One or more non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system comprising a processor, cause the computing system to:

obtaining data representing partial block images, each partial block image depicting a blockage in front of a chroma-key background;

generating a training data set comprising a plurality of synthetic partial occlusion images by at least: for each of the composite partial block images,

performing a chroma keying operation to extract a shadow of the obstruction from a partially obstructed view of the partially obstructed view,

superimposing the extracted image of the obstacle on the background image to generate the composite partial obstacle image, an

Generating annotation data for the composite portion block image, the annotation data for distinguishing a portion of the composite portion block image representing imagery of the extracted obstacle from a portion of the composite portion block image representing the background image; and

training a neural network with the plurality of synthetic partial obstruction images to identify portions of the input image data corresponding to data from sensors of an autonomous vehicle that represent obstructions on the sensors of the autonomous vehicle.

Images