US20170359561A1 - Disparity mapping for an autonomous vehicle - Google Patents
Disparity mapping for an autonomous vehicle Download PDFInfo
- Publication number
- US20170359561A1 US20170359561A1 US15/176,561 US201615176561A US2017359561A1 US 20170359561 A1 US20170359561 A1 US 20170359561A1 US 201615176561 A US201615176561 A US 201615176561A US 2017359561 A1 US2017359561 A1 US 2017359561A1
- Authority
- US
- United States
- Prior art keywords
- image
- disparity
- data
- pixel
- vehicle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000013507 mapping Methods 0.000 title abstract description 23
- 238000000034 method Methods 0.000 claims description 26
- 230000033001 locomotion Effects 0.000 claims description 15
- 238000004422 calculation algorithm Methods 0.000 claims description 14
- 238000005266 casting Methods 0.000 claims description 9
- 238000003384 imaging method Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 description 33
- 230000003287 optical effect Effects 0.000 description 24
- 230000009471 action Effects 0.000 description 17
- 239000013598 vector Substances 0.000 description 9
- 238000004891 communication Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 230000001133 acceleration Effects 0.000 description 6
- 230000004044 response Effects 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 5
- 230000003068 static effect Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000004807 localization Effects 0.000 description 3
- 239000013589 supplement Substances 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 238000010845 search algorithm Methods 0.000 description 2
- 241000282412 Homo Species 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 230000003466 anti-cipated effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 229920001690 polydopamine Polymers 0.000 description 1
- 230000002250 progressing effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000009012 visual motion Effects 0.000 description 1
Images
Classifications
-
- H04N13/0022—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/128—Adjusting depth or disparity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/593—Depth or shape recovery from multiple images from stereo images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30261—Obstacle
-
- H04N13/0239—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0081—Depth or disparity estimation from stereoscopic image signals
Definitions
- AVs Autonomous vehicles
- AVs may require continuous sensor data processing in order to operate through road traffic on public roads in order to match or even surpass human capabilities.
- AVs can be equipped with many kinds of sensors, including stereoscopic cameras, but processing images from a stereoscopic camera in real-time with enough fidelity to properly identify and classify obstacles is a challenge.
- stereo vision images are captured from a pair of cameras or lenses of a camera that are slightly displaced relative to each other. This positional difference is known as horizontal disparity and allows a stereo camera to perceive and calculate depth, or the distance from the camera to objects in a scene.
- stereoscopic imaging is mostly fulfilled by utilizing a parallax effect.
- a two-view stereoscopic video is a video generated by utilizing such an effect and each frame of the video includes an image for a left eye and another image for a right eye.
- the depth information of objects in the frame can be obtained by processing the two-view stereoscopic video.
- the depth information for all pixels of the image makes up a disparity map.
- Optical flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer (an eye or a camera) and the scene.
- the optical flow methods try to calculate the motion, for each pixel or voxel position, between two image frames which are taken at separate times.
- An optical flow sensor is a vision sensor capable of measuring optical flow or visual motion and outputting a measurement based on optical flow.
- FIG. 1 illustrates an example control system for operating an autonomous vehicle
- FIG. 2 illustrates an example autonomous vehicle including an improved disparity mapping and object classification system
- FIG. 3 illustrates an example method of object classification in accordance with one or more embodiments
- FIG. 4 illustrates an example method of disparity mapping in accordance with one or more embodiments.
- FIG. 5 is a block diagram illustrating a computer system upon which examples described herein may be implemented.
- An improved disparity mapping system that enables an autonomous vehicle (AV) to efficiently calculate disparity maps from images taken by a stereo camera mounted on the AV.
- the disparity mapping system utilizes mapping resource data and previously recorded sub-maps that contain surface data for a given region and compares this sub-map data with the images taken in order to improve disparity map calculations, both in terms of accuracy and speed.
- a classifier can then use these disparity maps along with optical flow images to classify objects and assist the AV in maneuvering through road traffic to a particular destination.
- the disparity mapping system can utilize a sub-map that includes recorded 3D LIDAR data and 3D stereo data of the current route traveled by the AV.
- the system can continuously compare real-time sensor data to the pre-recorded data of the current sub-map to help classify potential hazards, such as pedestrians, other vehicles, bicyclists, etc. Accordingly, an AV control system can use these classifications to better avoid collisions with dangerous hazards.
- Relative depth information for objects in a scene can be obtained in the form of a disparity map, which encodes the difference in coordinates of corresponding pixels between two images taken from a stereoscopic camera.
- the values in a disparity map are inversely proportional to the scene depth at the corresponding pixel location.
- the two images are captured from a pair of lenses that are slightly displaced relative to each other. This positional difference is known as horizontal disparity and allows a stereo camera to perceive and calculate depth, or the distance from the camera to objects in a scene.
- the depth information for all pixels in the images is obtained by processing the two images to construct a disparity map.
- Processing the images from the stereo camera involves finding a set of points in one image which can be identified as the same set of points in the other image. To do this, points or features in one image are matched with the corresponding points or features in the other image to create a disparity map.
- This processing is known as the correspondence problem, and solving it can be very time-consuming and processor intensive for a computing device.
- the calculation load is heavy since each individual frame has to be computed to obtain a corresponding disparity map.
- the full computation of a disparity map can take a few minutes using conventional algorithms.
- a set of imaging devices such as a stereoscopic camera acquires a first image and a second image of a scene.
- a system generates baseline disparity data from a location and orientation of the stereoscopic camera and three-dimensional environment data for the scene. Using the first image, second image, and baseline disparity data, the system can then generate a disparity map for the scene.
- the system compares, for each pixel in the first image, the pixel to the baseline disparity data to determine a likely location in the second image for a matching pixel that corresponds to the pixel in the first image, wherein the pixel in the first image and the matching pixel in the second image correspond to an object in the scene.
- the likely locations in the second image can be used for at least some of the pixels in the first image to reduce a search space when locating the matching pixels in the second image.
- generating the baseline disparity data uses a ray casting algorithm to render the 3D environment data into a 2D image.
- the 3D environment data is ground-based data corresponding to a location of the stereoscopic camera that is compiled from a fleet of autonomous vehicles.
- One or more examples described herein provide that methods, techniques, and actions performed by a computing device are performed programmatically, or as a computer-implemented method.
- Programmatically means through the use of code or computer-executable instructions. These instructions can be stored in one or more memory resources of the computing device.
- a programmatically performed step may or may not be automatic.
- a programmatic module, engine, or component can include a program, a sub-routine, a portion of a program, or a software component or a hardware component capable of performing one or more stated tasks or functions.
- a module or component can exist on a hardware component independently of other modules or components. Alternatively, a module or component can be a shared element or process of other modules, programs or machines.
- computing devices including processing and memory resources.
- one or more examples described herein may be implemented, in whole or in part, on computing devices such as servers, desktop computers, cellular or smartphones, personal digital assistants (e.g., PDAs), laptop computers, printers, digital picture frames, network equipment (e.g., routers) and tablet devices.
- PDAs personal digital assistants
- Memory, processing, and network resources may all be used in connection with the establishment, use, or performance of any example described herein (including with the performance of any method or with the implementation of any system).
- one or more examples described herein may be implemented through the use of instructions that are executable by one or more processors. These instructions may be carried on a computer-readable medium.
- Machines shown or described with figures below provide examples of processing resources and computer-readable mediums on which instructions for implementing examples disclosed herein can be carried and/or executed.
- the numerous machines shown with examples of the invention include processors and various forms of memory for holding data and instructions.
- Examples of computer-readable mediums include permanent memory storage devices, such as hard drives on personal computers or servers.
- Other examples of computer storage mediums include portable storage units, such as CD or DVD units, flash memory (such as carried on smartphones, multifunctional devices or tablets), and magnetic memory.
- Computers, terminals, network enabled devices are all examples of machines and devices that utilize processors, memory, and instructions stored on computer-readable mediums. Additionally, examples may be implemented in the form of computer-programs, or a computer usable carrier medium capable of carrying such a program.
- An AV refers to any vehicle which is operated in a state of automation with respect to steering and propulsion. Different levels of autonomy may exist with respect to AVs. For example, some vehicles may enable automation in limited scenarios, such as on highways, provided that drivers are present in the vehicle. More advanced AVs drive without any human assistance from within or external to the vehicle. Such vehicles often are required to make advance determinations regarding how the vehicle is behave given challenging surroundings of the vehicle environment.
- FIG. 1 illustrates an example control system for operating an autonomous vehicle.
- a control system 100 can be used to autonomously operate an AV 10 in a given geographic region for a variety of purposes, including transport services (e.g., transport of humans, delivery services, etc.).
- transport services e.g., transport of humans, delivery services, etc.
- an autonomously driven vehicle can operate without human control.
- an autonomously driven vehicle can steer, accelerate, shift, brake, and operate lighting components.
- an autonomous-capable vehicle can be operated either autonomously or manually.
- control system 100 can utilize specific sensor resources in order to intelligently operate the vehicle 10 in most common driving situations.
- the control system 100 can operate the vehicle 10 by autonomously steering, accelerating, and braking the vehicle 10 as the vehicle progresses to a destination.
- the control system 100 can perform vehicle control actions (e.g., braking, steering, accelerating) and route planning using sensor information, as well as other inputs (e.g., transmissions from remote or local human operators, network communication from other vehicles, etc.).
- the control system 100 includes a computer or processing system which operates to process sensor data 99 that is obtained on the vehicle with respect to a road segment upon which the vehicle 10 operates.
- the sensor data 99 can be used to determine actions which are to be performed by the vehicle 10 in order for the vehicle 10 to continue on a route to a destination.
- the control system 100 can include other functionality, such as wireless communication capabilities, to send and/or receive wireless communications with one or more remote sources.
- the control system 100 can issue instructions and data, shown as commands 85 , which programmatically controls various electromechanical interfaces of the vehicle 10 .
- the commands 85 can serve to control operational aspects of the vehicle 10 , including propulsion, braking, steering, and auxiliary behavior (e.g., turning lights on).
- the AV 10 can be equipped with multiple types of sensors 101 and 103 , which combine to provide a computerized perception of the space and environment surrounding the vehicle 10 .
- the control system 100 can operate within the AV 10 to receive sensor data 99 from the collection of sensors 101 and 103 , and to control various electromechanical interfaces for operating the vehicle on roadways.
- the sensors 101 and 103 operate to collectively obtain a complete sensor view of the vehicle 10 , and further to obtain situational information proximate to the vehicle 10 , including any potential hazards in a forward operational direction of the vehicle 10 .
- the sensors can include proximity or touch sensors, remote detection sensors such as provided by radar or LIDAR, a stereo camera 105 (stereoscopic pairs of cameras or depth perception cameras), and/or sonar sensors.
- Each of the sensors 101 and 103 and stereo camera 105 can communicate with the control system 100 utilizing a corresponding sensor interface 110 , 112 or camera interface 114 .
- Each of the interfaces 110 , 112 , 114 can include, for example, hardware and/or other logical components which are coupled or otherwise provided with the respective sensor.
- camera interface 114 can connect to a video camera and/or stereoscopic camera 105 which continually generates image data of an environment of the vehicle 10 .
- the stereo camera 105 can include a pair of imagers, each of which is mounted to a rigid housing structure that maintains the alignment of the imagers on a common plane when the vehicle is in motion.
- the interfaces 110 , 112 , 114 can include a dedicated processing resource, such as provided with a field programmable gate array (“FPGA”) which can, for example, receive and/or process raw image data from the camera sensor.
- FPGA field programmable gate array
- the interfaces 110 , 112 , 114 can include logic, such as provided with hardware and/or programming, to process sensor data 99 from a respective sensor 101 or 103 .
- the processed sensor data 99 can be outputted as sensor data 111 .
- Camera interface 114 can process raw image data from stereo camera 105 into images 113 for the control system 100 .
- the control system 100 can also include logic for processing raw or pre-processed sensor data 99 and images 113 .
- the vehicle interface subsystem 90 can include or control multiple interfaces to control mechanisms of the vehicle 10 .
- the vehicle interface subsystem 90 can include a propulsion interface 92 to electrically (or through programming) control a propulsion component (e.g., an accelerator pedal), a steering interface 94 for a steering mechanism, a braking interface 96 for a braking component, and a lighting/auxiliary interface 98 for exterior lights of the vehicle.
- the vehicle interface subsystem 90 and/or the control system 100 can include one or more controllers 84 which can receive one or more commands 85 from the control system 100 .
- the commands 85 can include route information 87 and one or more operational parameters 89 which specify an operational state of the vehicle 10 (e.g., desired speed and pose, acceleration, etc.).
- the controller(s) 84 can generate control signals 119 in response to receiving the commands 85 for one or more of the vehicle interfaces 92 , 94 , 96 , 98 .
- the controllers 84 can use the commands 85 as input to control propulsion, steering, braking, and/or other vehicle behavior while the AV 10 follows a current route.
- the controller(s) 84 can continuously adjust and alter the movement of the vehicle 10 in response to receiving a corresponding set of commands 85 from the control system 100 . Absent events or conditions which affect the confidence of the vehicle 10 in safely progressing along the route, the control system 100 can generate additional commands 85 from which the controller(s) 84 can generate various vehicle control signals 119 for the different interfaces of the vehicle interface subsystem 90 .
- the commands 85 can specify actions to be performed by the vehicle 10 .
- the actions can correlate to one or multiple vehicle control mechanisms (e.g., steering mechanism, brakes, etc.).
- the commands 85 can specify the actions, along with attributes such as magnitude, duration, directionality, or other operational characteristic of the vehicle 10 .
- the commands 85 generated from the control system 100 can specify a relative location of a road segment which the AV 10 is to occupy while in motion (e.g., change lanes, move into a center divider or towards shoulder, turn vehicle, etc.).
- the commands 85 can specify a speed, a change in acceleration (or deceleration) from braking or accelerating, a turning action, or a state change of exterior lighting or other components.
- the controllers 84 can translate the commands 85 into control signals 119 for a corresponding interface of the vehicle interface subsystem 90 .
- the control signals 119 can take the form of electrical signals which correlate to the specified vehicle action by virtue of electrical characteristics that have attributes for magnitude, duration, frequency or pulse, or other electrical characteristics.
- the control system 100 can include a route planner 122 , optical flow unit 121 , disparity mapper 126 , classifier 127 , event logic 124 , and a vehicle control 128 .
- the vehicle control 128 represents logic that converts alerts of event logic 124 (“event alert 135 ”) into commands 85 that specify a set of vehicle actions.
- the route planner 122 can select one or more route segments that collectively form a path of travel for the AV 10 when the vehicle 10 is on a current trip (e.g., servicing a pick-up request).
- the route planner 122 can specify route segments 131 of a planned vehicle path which defines turn by turn directions for the vehicle 10 at any given time during the trip.
- the route planner 122 may utilize the sensor interface 110 to receive GPS information as sensor data 111 .
- the vehicle control 128 can process route updates from the route planner 122 as commands 85 to progress along a path or route using default driving rules and actions (e.g., moderate steering and speed).
- control system 100 includes an optical flow unit 121 and disparity mapper 126 to monitor the situational environment of the AV 10 continuously in order to dynamically calculate disparity maps and optical flow images as the AV 10 travels along a current route.
- the external entity can be a pedestrian or group of pedestrians, a human-driven vehicle, a bicyclist, and the like.
- the sensor data 111 captured by the sensors 101 and 103 and images 113 from the camera interface 114 can be processed by an on-board optical flow unit 121 and disparity mapper 126 .
- Optical flow unit 121 and disparity mapper 126 can utilize mapping resource data and previously recorded sub-maps that contain surface data for a given region.
- Disparity mapper 126 can compare this sub-map data with the images 113 taken from stereo camera 105 in order to improve disparity map calculations, both in terms of accuracy and speed.
- Classifier 127 can then use these maps and optical flow images to create object classifications 133 to assist the AV 10 in maneuvering through road traffic to a particular destination.
- the disparity mapper 126 can utilize a current sub-map that includes recorded 3D LIDAR data and 3D stereo data of the current route traveled by the AV 10 .
- the disparity mapper 126 can continuously compare the sensor data 111 to the 3D LIDAR data and stereo data of the current sub-map to help classifier 127 identify potential hazards, such as pedestrians, other vehicles, bicyclists, etc. Accordingly, classifier 127 can generate object classifications 133 for event logic 124 .
- the event logic 124 can refer to the object classifications 133 in determining whether to trigger a response to a detected event.
- a detected event can correspond to a roadway condition or obstacle which, when detected, poses a potential hazard or threat of collision to the vehicle 10 .
- a detected event can include an object in the road segment, heavy traffic ahead, and/or wetness or other environment conditions on the road segment.
- the event logic 124 can use sensor data 111 and images 113 from cameras, LIDAR, radar, sonar, or various other image or sensor component sets in order to detect the presence of such events as described.
- the event logic 124 can detect potholes, debris, objects projected to be on a collision trajectory, and the like.
- the event logic 124 can detect events which enable the control system 100 to make evasive actions or plan for any potential threats.
- the event logic 124 can signal an event alert 135 that classifies the event and indicates the type of avoidance action to be performed. Additionally, the control system 100 can determine whether an event corresponds to a potential incident with a human driven vehicle, a pedestrian, or other human entity external to the AV 10 . An event can be scored or classified between a range of likely harmless (e.g., small debris in roadway) to very harmful (e.g., vehicle crash may be imminent) from the sensor data 111 and object classifications 133 . In turn, the vehicle control 128 can determine a response based on the score or classification. Such response can correspond to an event avoidance action 145 , or an action that the vehicle 10 can perform to maneuver the vehicle 10 based on the detected event and its score or classification.
- the vehicle response can include a slight or sharp vehicle maneuvering for avoidance using a steering control mechanism and/or braking component.
- the event avoidance action 145 can be signaled through the commands 85 for controllers 84 of the vehicle interface subsystem 90 .
- event logic 124 can signal an event alert 135 to cause the vehicle control 128 to generate commands 85 that correspond to an event avoidance action 145 .
- event logic 124 can signal an event alert 135 to avoid the collision.
- the event alert 135 can indicate (i) a classification of the event (e.g., “serious” and/or “immediate”), (ii) information about the event, such as the type of object that generated the event alert 135 , and/or information indicating a type of action the vehicle 10 should take (e.g., location of object relative to path of vehicle, size or type of object, etc.).
- a classification of the event e.g., “serious” and/or “immediate”
- information about the event such as the type of object that generated the event alert 135
- information indicating a type of action the vehicle 10 should take e.g., location of object relative to path of vehicle, size or type of object, etc.
- FIG. 2 illustrates an example autonomous vehicle including an improved disparity mapping and object classification system.
- the AV 200 shown in FIG. 2 can include some or all aspects and functionality of the autonomous vehicle 10 described with respect to FIG. 1 .
- the AV 200 can include a sensor array 205 that can provide sensor data 207 to an on-board data processing system 210 .
- the sensor array 205 can include any number of active or passive sensors that continuously detect a situational environment of the AV 200 .
- the sensor array 205 can include a number of camera sensors (e.g., stereo camera 206 ), LIDAR sensor(s), proximity sensors, radar, and the like.
- the data processing system 210 can utilize the sensor data 207 and images 208 to detect the situational conditions of the AV 200 as the AV 200 travels along a current route. For example, the data processing system 210 can identify potential obstacles or road hazards, such as pedestrians, bicyclists, objects on the road, road cones, road signs, animals, etc., which classifier 235 can classify in order to enable an AV control system 220 to react accordingly.
- potential obstacles or road hazards such as pedestrians, bicyclists, objects on the road, road cones, road signs, animals, etc.
- the AV 200 can further include a database 230 that includes sub-maps 231 for the given region in which the AV 200 operates.
- the sub-maps 231 can comprise detailed road data previously recorded by a recording vehicle using sensor equipment, such as LIDAR, stereo camera, and/or radar equipment.
- sensor equipment such as LIDAR, stereo camera, and/or radar equipment.
- several or all AVs in the fleet can include this sensor equipment to record updated sub-maps 231 along traveled routes and submit the updated sub-maps 231 to the backend system 290 , which can transmit the updated sub-maps 231 to the other AVs in the fleet for storage.
- the sub-maps 231 can comprise ground-based, three-dimensional (3D) environment data along various routes throughout the given region (e.g., a city).
- the on-board data processing system 210 can provide continuous processed data 214 to the AV control system 220 to respond to point-to-point activity in the AV's 200 surroundings.
- the processed data 214 can comprise comparisons between the actual sensor data 207 —which represents an operational environment of the AV 200 , and which is continuously collected by the sensor array 205 —and the stored sub-maps 231 (e.g., LIDAR-based sub-maps).
- the data processing system 210 is programmed with machine learning capabilities to enable the AV 200 to identify and respond to conditions, events, or potential hazards.
- the on-board data processing system 210 can continuously compare sensor data 207 to stored sub-maps 231 in order to perform a localization to continuously determine a location and orientation of the AV 200 within the given region. Localization of the AV 200 is necessary in order to make the AV 200 self-aware of its instant location and orientation in comparison to the stored sub-maps 231 in order to maneuver the AV 200 on surface streets through traffic and identify and respond to potential hazards, such as pedestrians, or local conditions, such as weather or traffic.
- the data processing system 210 can compare the sensor data 207 from the sensor array 205 with a current sub-map 238 from the sub-maps 231 to identify obstacles and potential road hazards in real time.
- a disparity mapper 211 and optical flow unit 212 which can be part of the data processing system 210 , process the sensor data 207 , images 208 from the stereo camera 206 , and the current sub-map 238 to create image maps 218 (e.g., disparity maps and optical flow images).
- Classifier 235 can then provide object classifications 213 —identifying obstacles and road hazards—to the AV control system 220 , which can react accordingly by operating the steering, braking, and acceleration systems 225 of the AV 200 to perform low level maneuvering.
- the disparity mapper 211 can use the 3D environment data from the current sub-map 238 to generate a baseline disparity image that represents distances from the stereo camera 206 to known features of the environment. These features generally include terrain features, buildings, and other static, non-moving objects such as signs and trees.
- the disparity mapper 211 can generate the baseline disparity image through a ray casting algorithm that renders the three-dimensional environment into a two-dimensional image. Ray casting traces rays from a point corresponding to one of the stereo camera lenses, one ray for each pixel of the image sensor resolution, and finds the distance to the closest object blocking the path of that ray.
- the distances for all the rays cast make up a set of data that represents the baseline disparity image.
- the baseline disparity image should represent an accurate map of the distances from the stereo camera 206 to features and objects in the scene.
- the disparity mapper 211 finds a set of points in one image which can be identified as the same points in the other image in order to create a disparity map. To do this, points (i.e., pixels) or features in one image are matched with the corresponding points or features in the other image, which can be done using standard algorithms comparing colors, lighting, etc.
- points i.e., pixels
- features in one image are matched with the corresponding points or features in the other image, which can be done using standard algorithms comparing colors, lighting, etc.
- the construction of these disparity maps is computationally expensive because the search space for matching pixels between the two images is large, especially for high resolution images needed to accurately identify and classify objects in a scene.
- the disparity mapper 211 can use a combination of the baseline disparity image generated from the current sub-map 238 and the pair of images 208 acquired from the stereo camera 206 to narrow the search spaces.
- the disparity mapper 211 can efficiently output a disparity map of the scene that represents the distances from the stereo camera 206 to features and objects in the scene, including both existing features from the current sub-map 238 and new features and objects present in the scene that are not part of the current sub-map 238 data.
- the disparity mapper 211 matches pixels from the baseline disparity image to pixels in one of the stereo camera images. For example, disparity data corresponding to a pixel in the upper left corner of the left stereo camera image is taken from the baseline disparity image.
- the disparity data should be roughly equal (within a reasonable margin of error to account for map inaccuracies) to the disparity between pixels in the left and right stereo images taken of the scene.
- the disparity mapper 211 can then apply the baseline disparity image data to each pixel in the left image to determine a likely location of the corresponding pixel in the right image and reduce the search space of the correspondence algorithm.
- the disparity mapper 211 can output the generated disparity map (as image maps 218 ) for classifier 235 to use in classifying objects in the scene.
- an optical flow unit 212 can use the apparent motion of features in the field of view of the moving stereo camera 206 to supplement or replace the baseline disparity image generated from the 3D environment data. From either of the lenses of the stereo camera 206 , a map of optical flow vectors can be calculated between a previous frame and a current frame. The optical flow unit 212 can use these vectors to improve the correspondence search algorithm.
- image maps 218 can include images of optical flow vectors that classifier 235 can use to improve object classifications 213 .
- the classifier 235 can also monitor situational data 217 from the data processing system 210 to identify potential areas of conflict. For example, the classifier 235 can monitor forward directional stereoscopic camera data or LIDAR data to identify areas of concern. In one example, the classifier 235 can utilize the current sub-map 238 to identify features along the current route traveled (e.g., as indicated by the route data 232 ), such as traffic signals, intersections, road signs, crosswalks, bicycle lanes, parking areas, and the like. As the AV 200 approaches such features or areas, the classifier 235 can monitor the forward situational data 217 to identify any external entities that may conflict with the operational flow of the AV 200 , such as pedestrians near a crosswalk or another vehicle approaching an intersection.
- object classifications 213 provided to the AV control system 220 can indicate low level occurrences, such as obstacles and potential hazards, to which the AV control system 220 can make decisions and react.
- object classifications 213 can indicate a pedestrian crossing the road, traffic signals, stop signs, other vehicles, road conditions, traffic conditions, bicycle lanes, crosswalks, pedestrian activity (e.g., a crowded adjacent sidewalk), and the like.
- the AV control system 220 can respond to different types of objects by generating control commands 221 to reactively operate the steering, braking, and acceleration systems 225 accordingly.
- the AV control system 220 can receive a destination 219 from, for example, an interface system 215 of the AV 200 .
- the interface system 215 can include any number of touch-screens, voice sensors, mapping resources, etc., that enable a passenger 239 to provide a passenger input 241 indicating the destination 219 .
- the passenger 239 can type the destination 219 into a mapping engine 275 of the AV 200 , or can speak the destination 219 into the interface system 215 .
- the interface system 215 can include a wireless communication module that can connect the AV 200 to a network 280 to communicate with a backend transport arrangement system 290 to receive invitations 282 to service a pick-up or drop-off request.
- Such invitations 282 can include the destination 219 (e.g., a pick-up location), and can be received by the AV 200 as a communication over the network 280 from the backend transport arrangement system 290 .
- the backend transport arrangement system 290 can manage routes and/or facilitate transportation for users using a fleet of autonomous vehicles throughout a given region.
- the backend transport arrangement system 290 can be operative to facilitate passenger pick-ups and drop-offs to generally service pick-up requests, facilitate delivery such as packages or food, and the like.
- the AV control system 220 can utilize the mapping engine 275 to receive route data 232 indicating a route to the destination 219 .
- the mapping engine 275 can also generate map content 226 dynamically indicating the route traveled to the destination 219 .
- the route data 232 and/or map content 226 can be utilized by the AV control system 220 to maneuver the AV 200 to the destination 219 along the selected route.
- the AV control system 220 can dynamically generate control commands 221 for the autonomous vehicle's steering, braking, and acceleration system 225 to actively drive the AV 200 to the destination 219 along the selected route.
- the map content 226 showing the current route traveled can be streamed to the interior interface system 215 so that the passenger(s) 239 can view the route and route progress in real time.
- FIG. 3 illustrates an example method of object classification in accordance with one or more embodiments.
- FIG. 4 illustrates an example method of disparity mapping in accordance with one or more embodiments. While operations of these example implementations are described below as being performed by specific components, modules or systems of the AV 200 , it will be appreciated that these operations need not necessarily be performed by the specific components identified, and could be performed by a variety of components and modules, potentially distributed over a number of machines. Accordingly, references may be made to elements of AV 200 for the purpose of illustrating suitable components or elements for performing a step or sub step being described. Alternatively, at least certain ones of the variety of components and modules described in AV 200 can be arranged within a single hardware, software, or firmware component. It will also be appreciated that some of the steps of this method may be performed in parallel or in a different order than illustrated.
- a vehicle can obtain sensor data for the environment through, for example, proximity or touch sensors, remote detection sensors such as provided by radar or LIDAR, a stereo camera, and/or sonar sensors as described with respect to FIGS. 1 and 2 ( 310 ).
- the vehicle can additionally obtain known data for the environment from previously recorded mapping resource data (i.e., sub-maps) that contain surface data for a given region.
- mapping resource data i.e., sub-maps
- the vehicle can compare this sub-map data with the sensor data for the environment ( 320 ).
- the vehicle can then use the comparisons, including disparity maps and optical flow images, to create object classifications to assist the vehicle in maneuvering through road traffic to a particular destination ( 330 ).
- a disparity mapper can utilize a current sub-map that includes recorded 3D LIDAR data and 3D stereo data of the current route traveled by the vehicle.
- the disparity mapper can continuously compare real-time sensor data to the data in the current sub-map to help a classifier identify potential hazards, such as pedestrians, other vehicles, bicyclists, etc.
- vehicle sensors can determine the location and orientation of the vehicle and its stereo camera.
- the sensors can determine latitude and longitude coordinates of the vehicle and a direction of travel, which can be further refined to identify the stereo camera's location in the world.
- the vehicle's data processing system can retrieve sub-maps stored in a database of the vehicle or accessed remotely from the backend system via a network ( 410 ).
- the data processing system can use the 3D environment data stored in these sub-maps to perform localization and pose operations to determine a current location and orientation of the vehicle in relation to a given region (e.g., a city).
- a given region e.g., a city
- the data processing system Given the location of the stereo camera as it is disposed on the vehicle, the data processing system can further determine the precise location and orientation of the stereo camera in the world ( 412 ).
- a disparity mapper can use the 3D environment data to generate a baseline disparity image that represents distances from the stereo camera to known features of the environment ( 414 ).
- These features generally include terrain features, buildings, and other static, non-moving objects such as signs and trees.
- the disparity mapper can generate the baseline disparity image through a ray casting algorithm that renders the three-dimensional environment into a two-dimensional image. Ray casting traces rays from a point corresponding to one of the stereo camera lenses, one ray for each pixel of the image sensor resolution, and finds the distance to the closest object blocking the path of that ray. The distances for all the rays cast make up a set of data that represents the baseline disparity image. In an environment with no new features or objects, such as pedestrians or other vehicles, that are not included in the 3D environment data, the baseline disparity image should represent an accurate map of the distances from the stereo camera to features and objects in the scene.
- the disparity mapper can generate a baseline disparity image for each camera lens in the disparity mapping system using an alternate ray casting algorithm.
- the disparity mapper identifies 3D points in the 3D environment data that are visible to one or more camera lenses, and for each 3D point identified, the disparity mapper projects a ray from the 3D point to points representing each camera sensor in the disparity mapping system.
- the disparity mapper can then match each ray to 2D image coordinates for a baseline disparity image for that camera lens.
- the stereo camera takes a simultaneous pair of images of the scene, one for each lens of the stereo camera ( 420 ).
- the data processing system can remove distortions from the images and perform image rectification to reduce the pixel correspondence search space ( 422 ).
- finding correspondences between pixels in the two images requires a search in two dimensions.
- the search is simplified to one dimension—a horizontal line parallel to the line between the lenses. Since it may be impractical to maintain perfect alignment between the lenses, the disparity mapper can perform image rectification to achieve similar results.
- Rectification can be performed using a variety of algorithms, such as planar rectification, cylindrical rectification, and polar rectification so that the rectified images have epipolar lines parallel to the horizontal axis and corresponding points with identical vertical coordinates.
- the disparity mapper can create the baseline disparity image in order to take the rectification into account for performance gains.
- the disparity mapper finds a set of points in one image which can be identified as the same points in the other image in order to create a disparity map. To do this, points (i.e., pixels) or features in one image are matched with the corresponding points or features in the other image, which can be done using standard algorithms comparing colors, lighting, etc.
- points i.e., pixels
- features in one image are matched with the corresponding points or features in the other image, which can be done using standard algorithms comparing colors, lighting, etc.
- the construction of these disparity maps is computationally expensive because the search space for matching pixels between the two images is large, even when the images are rectified.
- the disparity mapper can use a combination of the baseline disparity image from the 3D environment data and the pair of images acquired from the stereo camera to narrow the search spaces. In doing so, the disparity mapper can efficiently output a disparity map of the scene that represents the distances from the stereo camera to features and objects in the scene, including both existing features from the 3D environment data and new features and objects present in the scene that are not part of the 3D environment data. In some implementations, the disparity mapper iterates through the pixels in the baseline disparity image and compares the pixel data in the baseline disparity image to its corresponding pixel in the same 2D location in one of the stereo camera images (typically the left image, but the right image can be used instead) ( 430 ).
- disparity data corresponding to a pixel in the upper left corner of the left stereo camera image is taken from the baseline disparity image. Assuming that no new feature or object is present in the scene that is not included in the 3D environment data, the disparity data should be roughly equal (within a reasonable margin of error to account for map inaccuracies) to the disparity between pixels in the left and right stereo images taken of the scene.
- the disparity mapper can apply the baseline disparity image data to each pixel in the left image to determine a likely location of the corresponding pixel in the right image ( 432 ).
- This baseline disparity image data contains a disparity value for each pixel that can be added to or subtracted from the 2D coordinates of the pixel to find the likely 2D coordinates for the corresponding pixel in the right image ( 434 ).
- the coordinate search may be performed on a single dimension.
- the disparity mapper starts the search for the corresponding pixel in the right image at a large coordinate distance from where the left pixel is located in the left image.
- the disparity mapper starts the search for the corresponding pixel in the right image at a small coordinate distance from where the left pixel is located in the left image.
- This likely location used for the start of the correspondence search should be the correct corresponding pixel in conditions where the map is perfect and there are no changes from the environment recorded in the 3D environment data to the actual scene in the present. Therefore, given images of a scene that mostly matches the 3D environment data, the disparity mapper can accurately locate corresponding pixels in the right image without further searching.
- the disparity mapper may still reduce the search space using the determined likely location of the corresponding pixel. For example, the disparity mapper can search the pixel at the likely location and any pixels at a programmed distance around the likely location to find the most probable corresponding pixel.
- disparity maps calculated for previous images from the stereo camera can be used to supplement or replace the baseline disparity image generated from the 3D environment data.
- a vehicle equipped with a stereo camera may collect images fast enough such that the camera location does not change significantly from frame to frame and the background remains mostly static. Therefore, the prior disparity map generated for a previous frame can be useful to initialize the subsequent map. Only near range discontinuities and moving objects should result in large disparity differences. Everywhere else, the disparity values should be similar.
- the disparity mapper can incorporate information about the stereo camera's movement, which can be taken from other sensors on a vehicle, to modify the prior disparity map such that it represents a more accurate baseline disparity image for the stereo camera's current position.
- the disparity mapper can adjust the previous disparity map to take that one meter movement into account in generating a baseline disparity map.
- the disparity mapper can then use the generated baseline disparity map to reduce the search space for corresponding pixels between the current pair of stereo camera images.
- an optical flow unit can use the apparent motion of features in the field of view of a moving stereo camera to supplement or replace the baseline disparity image generated from the 3D environment data. From either of the lenses of the stereo camera, a map of optical flow vectors can be calculated between a previous frame and a current frame. The optical flow unit can use these vectors to improve the correspondence search algorithm in similar ways to using previously calculated disparity maps. For example, given the motion vector of a pixel in the left image from the stereo camera, the motion vector of a corresponding pixel in the right image should be similar after accounting for the different perspective of the right lens of the stereo camera.
- the disparity mapper can proceed to match remaining pixels ( 436 ). In some implementations, the disparity mapper can perform matching on multiple pixels simultaneously, either through separate processes or by grouping pixels together. Once corresponding pixels are found for each of the pixels in the left image, the disparity mapper can output the generated disparity map for other systems to use ( 438 ). For example, an object classifier can use the generated disparity map to classify objects in the scene based on the disparity map and other data.
- FIG. 5 is a block diagram illustrating a computer system upon which examples described herein may be implemented.
- the data processing system 210 and classifier 235 shown and described in FIG. 2 may be implemented on the computer system 500 of FIG. 5 .
- the computer system 500 can be implemented using one or more processors 504 , and one or more memory resources 506 .
- the computer system 500 may be implemented within an autonomous vehicle with software and hardware resources such as described with examples of FIGS. 1 and 2 .
- the computer system 500 can be distributed spatially into various regions of the autonomous vehicle, with various aspects integrated with other components of the autonomous vehicle itself.
- the processors 504 and/or memory resources 506 can be provided in the trunk of the autonomous vehicle.
- the various processing resources 504 of the computer system 500 can also execute disparity mapping instructions 512 using microprocessors or integrated circuits.
- the disparity mapping instructions 512 can be executed by the processing resources 504 or using field-programmable gate arrays (FPGAs).
- FPGAs field-programmable gate arrays
- the computer system 500 can include a local communication interface 550 (or series of local links) to vehicle interfaces and other resources of the autonomous vehicle (e.g., the computer stack drives).
- the communication interface 550 provides a data bus or other local links to electro-mechanical interfaces of the vehicle, such as wireless or wired links to the AV control system 220 .
- the memory resources 506 can include, for example, main memory, a read-only memory (ROM), storage device, and cache resources.
- the main memory of memory resources 506 can include random access memory (RAM) or other dynamic storage device, for storing information and instructions which are executable by the processors 504 .
- the processors 504 can execute instructions for processing information stored with the main memory of the memory resources 506 .
- the main memory 506 can also store temporary variables or other intermediate information which can be used during execution of instructions by one or more of the processors 504 .
- the memory resources 506 can also include ROM or other static storage device for storing static information and instructions for one or more of the processors 504 .
- the memory resources 506 can also include other forms of memory devices and components, such as a magnetic disk or optical disk, for purpose of storing information and instructions for use by one or more of the processors 504 .
- the memory 506 may store a plurality of software instructions including, for example, disparity mapping instructions 512 .
- the disparity mapping instructions 512 may be executed by one or more of the processors 504 in order to implement functionality such as described with respect to the disparity mapper 211 , optical flow unit 212 , and classifier 235 of FIG. 2 .
- the computer system 500 can receive sensor data 562 over the communication interface 550 from various AV subsystems 560 (e.g., the AV control system 220 or data processing system 210 ).
- the processing resources 504 can monitor the sensor data 562 and generate object classifications that the AV control system 220 can use to send commands to the output systems 520 of the AV 200 in accordance with examples described herein.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Traffic Control Systems (AREA)
Abstract
Description
- Autonomous vehicles (AVs) may require continuous sensor data processing in order to operate through road traffic on public roads in order to match or even surpass human capabilities. AVs can be equipped with many kinds of sensors, including stereoscopic cameras, but processing images from a stereoscopic camera in real-time with enough fidelity to properly identify and classify obstacles is a challenge.
- In stereo vision, images are captured from a pair of cameras or lenses of a camera that are slightly displaced relative to each other. This positional difference is known as horizontal disparity and allows a stereo camera to perceive and calculate depth, or the distance from the camera to objects in a scene. At present, stereoscopic imaging is mostly fulfilled by utilizing a parallax effect. By providing a left image for a left eye and a right image for a right eye, it is possible to convey a 3D impression to a viewer when the viewer is watching the images at an appropriate viewing angle. A two-view stereoscopic video is a video generated by utilizing such an effect and each frame of the video includes an image for a left eye and another image for a right eye. The depth information of objects in the frame can be obtained by processing the two-view stereoscopic video. The depth information for all pixels of the image makes up a disparity map.
- Optical flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer (an eye or a camera) and the scene. The optical flow methods try to calculate the motion, for each pixel or voxel position, between two image frames which are taken at separate times. An optical flow sensor is a vision sensor capable of measuring optical flow or visual motion and outputting a measurement based on optical flow.
- The disclosure herein is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements, and in which:
-
FIG. 1 illustrates an example control system for operating an autonomous vehicle; -
FIG. 2 illustrates an example autonomous vehicle including an improved disparity mapping and object classification system; -
FIG. 3 illustrates an example method of object classification in accordance with one or more embodiments; -
FIG. 4 illustrates an example method of disparity mapping in accordance with one or more embodiments; and -
FIG. 5 is a block diagram illustrating a computer system upon which examples described herein may be implemented. - An improved disparity mapping system is disclosed that enables an autonomous vehicle (AV) to efficiently calculate disparity maps from images taken by a stereo camera mounted on the AV. The disparity mapping system utilizes mapping resource data and previously recorded sub-maps that contain surface data for a given region and compares this sub-map data with the images taken in order to improve disparity map calculations, both in terms of accuracy and speed. A classifier can then use these disparity maps along with optical flow images to classify objects and assist the AV in maneuvering through road traffic to a particular destination. For example, the disparity mapping system can utilize a sub-map that includes recorded 3D LIDAR data and 3D stereo data of the current route traveled by the AV. The system can continuously compare real-time sensor data to the pre-recorded data of the current sub-map to help classify potential hazards, such as pedestrians, other vehicles, bicyclists, etc. Accordingly, an AV control system can use these classifications to better avoid collisions with dangerous hazards.
- Relative depth information for objects in a scene can be obtained in the form of a disparity map, which encodes the difference in coordinates of corresponding pixels between two images taken from a stereoscopic camera. The values in a disparity map are inversely proportional to the scene depth at the corresponding pixel location. The two images are captured from a pair of lenses that are slightly displaced relative to each other. This positional difference is known as horizontal disparity and allows a stereo camera to perceive and calculate depth, or the distance from the camera to objects in a scene. The depth information for all pixels in the images is obtained by processing the two images to construct a disparity map.
- Processing the images from the stereo camera involves finding a set of points in one image which can be identified as the same set of points in the other image. To do this, points or features in one image are matched with the corresponding points or features in the other image to create a disparity map. This processing is known as the correspondence problem, and solving it can be very time-consuming and processor intensive for a computing device. When processing stereoscopic images in real-time, the calculation load is heavy since each individual frame has to be computed to obtain a corresponding disparity map. For a two-view stereoscopic camera having a left image and a right image of with sufficient resolution to identify objects in a scene, the full computation of a disparity map can take a few minutes using conventional algorithms.
- Other algorithms for computing disparity maps can perform the calculations in less time, but at a cost of accuracy. Considering the need to properly identify depth and classify objects in an AV, a loss of precision in computing the disparity maps can be an unacceptable trade-off. Moreover, even the fastest conventional algorithms take several seconds to compute the disparity map for a scene, which is too slow for real-time processing in an AV travelling down a street. Therefore, how to improve the efficiency of the disparity map calculation while maintaining the accuracy of the disparity map is an important issue in computer vision.
- In some examples, a set of imaging devices such as a stereoscopic camera acquires a first image and a second image of a scene. A system generates baseline disparity data from a location and orientation of the stereoscopic camera and three-dimensional environment data for the scene. Using the first image, second image, and baseline disparity data, the system can then generate a disparity map for the scene.
- According to one aspect, the system compares, for each pixel in the first image, the pixel to the baseline disparity data to determine a likely location in the second image for a matching pixel that corresponds to the pixel in the first image, wherein the pixel in the first image and the matching pixel in the second image correspond to an object in the scene. When generating the disparity map, the likely locations in the second image can be used for at least some of the pixels in the first image to reduce a search space when locating the matching pixels in the second image.
- In some aspects, generating the baseline disparity data uses a ray casting algorithm to render the 3D environment data into a 2D image.
- In some aspects, the 3D environment data is ground-based data corresponding to a location of the stereoscopic camera that is compiled from a fleet of autonomous vehicles.
- One or more examples described herein provide that methods, techniques, and actions performed by a computing device are performed programmatically, or as a computer-implemented method. Programmatically, as used herein, means through the use of code or computer-executable instructions. These instructions can be stored in one or more memory resources of the computing device. A programmatically performed step may or may not be automatic.
- One or more examples described herein can be implemented using programmatic modules, engines, or components. A programmatic module, engine, or component can include a program, a sub-routine, a portion of a program, or a software component or a hardware component capable of performing one or more stated tasks or functions. As used herein, a module or component can exist on a hardware component independently of other modules or components. Alternatively, a module or component can be a shared element or process of other modules, programs or machines.
- Some examples described herein can generally require the use of computing devices, including processing and memory resources. For example, one or more examples described herein may be implemented, in whole or in part, on computing devices such as servers, desktop computers, cellular or smartphones, personal digital assistants (e.g., PDAs), laptop computers, printers, digital picture frames, network equipment (e.g., routers) and tablet devices. Memory, processing, and network resources may all be used in connection with the establishment, use, or performance of any example described herein (including with the performance of any method or with the implementation of any system).
- Furthermore, one or more examples described herein may be implemented through the use of instructions that are executable by one or more processors. These instructions may be carried on a computer-readable medium. Machines shown or described with figures below provide examples of processing resources and computer-readable mediums on which instructions for implementing examples disclosed herein can be carried and/or executed. In particular, the numerous machines shown with examples of the invention include processors and various forms of memory for holding data and instructions. Examples of computer-readable mediums include permanent memory storage devices, such as hard drives on personal computers or servers. Other examples of computer storage mediums include portable storage units, such as CD or DVD units, flash memory (such as carried on smartphones, multifunctional devices or tablets), and magnetic memory. Computers, terminals, network enabled devices (e.g., mobile devices, such as cell phones) are all examples of machines and devices that utilize processors, memory, and instructions stored on computer-readable mediums. Additionally, examples may be implemented in the form of computer-programs, or a computer usable carrier medium capable of carrying such a program.
- Numerous examples are referenced herein in context of an autonomous vehicle (AV). An AV refers to any vehicle which is operated in a state of automation with respect to steering and propulsion. Different levels of autonomy may exist with respect to AVs. For example, some vehicles may enable automation in limited scenarios, such as on highways, provided that drivers are present in the vehicle. More advanced AVs drive without any human assistance from within or external to the vehicle. Such vehicles often are required to make advance determinations regarding how the vehicle is behave given challenging surroundings of the vehicle environment.
- System Description
-
FIG. 1 illustrates an example control system for operating an autonomous vehicle. In an example ofFIG. 1 , acontrol system 100 can be used to autonomously operate an AV 10 in a given geographic region for a variety of purposes, including transport services (e.g., transport of humans, delivery services, etc.). In examples described, an autonomously driven vehicle can operate without human control. For example, in the context of automobiles, an autonomously driven vehicle can steer, accelerate, shift, brake, and operate lighting components. Some variations also recognize that an autonomous-capable vehicle can be operated either autonomously or manually. - In one implementation, the
control system 100 can utilize specific sensor resources in order to intelligently operate the vehicle 10 in most common driving situations. For example, thecontrol system 100 can operate the vehicle 10 by autonomously steering, accelerating, and braking the vehicle 10 as the vehicle progresses to a destination. Thecontrol system 100 can perform vehicle control actions (e.g., braking, steering, accelerating) and route planning using sensor information, as well as other inputs (e.g., transmissions from remote or local human operators, network communication from other vehicles, etc.). - In an example of
FIG. 1 , thecontrol system 100 includes a computer or processing system which operates to processsensor data 99 that is obtained on the vehicle with respect to a road segment upon which the vehicle 10 operates. Thesensor data 99 can be used to determine actions which are to be performed by the vehicle 10 in order for the vehicle 10 to continue on a route to a destination. In some variations, thecontrol system 100 can include other functionality, such as wireless communication capabilities, to send and/or receive wireless communications with one or more remote sources. In controlling the vehicle 10, thecontrol system 100 can issue instructions and data, shown ascommands 85, which programmatically controls various electromechanical interfaces of the vehicle 10. Thecommands 85 can serve to control operational aspects of the vehicle 10, including propulsion, braking, steering, and auxiliary behavior (e.g., turning lights on). - The AV 10 can be equipped with multiple types of
sensors control system 100 can operate within the AV 10 to receivesensor data 99 from the collection ofsensors - In more detail, the
sensors - Each of the
sensors stereo camera 105 can communicate with thecontrol system 100 utilizing acorresponding sensor interface camera interface 114. Each of theinterfaces camera interface 114 can connect to a video camera and/orstereoscopic camera 105 which continually generates image data of an environment of the vehicle 10. Thestereo camera 105 can include a pair of imagers, each of which is mounted to a rigid housing structure that maintains the alignment of the imagers on a common plane when the vehicle is in motion. As an addition or alternative, theinterfaces - In some examples, the
interfaces sensor data 99 from arespective sensor sensor data 99 can be outputted assensor data 111.Camera interface 114 can process raw image data fromstereo camera 105 intoimages 113 for thecontrol system 100. As an addition or variation, thecontrol system 100 can also include logic for processing raw orpre-processed sensor data 99 andimages 113. - According to one implementation, the
vehicle interface subsystem 90 can include or control multiple interfaces to control mechanisms of the vehicle 10. Thevehicle interface subsystem 90 can include apropulsion interface 92 to electrically (or through programming) control a propulsion component (e.g., an accelerator pedal), asteering interface 94 for a steering mechanism, abraking interface 96 for a braking component, and a lighting/auxiliary interface 98 for exterior lights of the vehicle. Thevehicle interface subsystem 90 and/or thecontrol system 100 can include one ormore controllers 84 which can receive one ormore commands 85 from thecontrol system 100. Thecommands 85 can includeroute information 87 and one or moreoperational parameters 89 which specify an operational state of the vehicle 10 (e.g., desired speed and pose, acceleration, etc.). - The controller(s) 84 can generate
control signals 119 in response to receiving thecommands 85 for one or more of the vehicle interfaces 92, 94, 96, 98. Thecontrollers 84 can use thecommands 85 as input to control propulsion, steering, braking, and/or other vehicle behavior while the AV 10 follows a current route. Thus, while the vehicle 10 is actively driven along the current route, the controller(s) 84 can continuously adjust and alter the movement of the vehicle 10 in response to receiving a corresponding set ofcommands 85 from thecontrol system 100. Absent events or conditions which affect the confidence of the vehicle 10 in safely progressing along the route, thecontrol system 100 can generateadditional commands 85 from which the controller(s) 84 can generate various vehicle control signals 119 for the different interfaces of thevehicle interface subsystem 90. - According to examples, the
commands 85 can specify actions to be performed by the vehicle 10. The actions can correlate to one or multiple vehicle control mechanisms (e.g., steering mechanism, brakes, etc.). Thecommands 85 can specify the actions, along with attributes such as magnitude, duration, directionality, or other operational characteristic of the vehicle 10. By way of example, thecommands 85 generated from thecontrol system 100 can specify a relative location of a road segment which the AV 10 is to occupy while in motion (e.g., change lanes, move into a center divider or towards shoulder, turn vehicle, etc.). As other examples, thecommands 85 can specify a speed, a change in acceleration (or deceleration) from braking or accelerating, a turning action, or a state change of exterior lighting or other components. Thecontrollers 84 can translate thecommands 85 intocontrol signals 119 for a corresponding interface of thevehicle interface subsystem 90. The control signals 119 can take the form of electrical signals which correlate to the specified vehicle action by virtue of electrical characteristics that have attributes for magnitude, duration, frequency or pulse, or other electrical characteristics. - In an example of
FIG. 1 , thecontrol system 100 can include aroute planner 122,optical flow unit 121,disparity mapper 126,classifier 127,event logic 124, and avehicle control 128. Thevehicle control 128 represents logic that converts alerts of event logic 124 (“event alert 135”) intocommands 85 that specify a set of vehicle actions. - Additionally, the
route planner 122 can select one or more route segments that collectively form a path of travel for the AV 10 when the vehicle 10 is on a current trip (e.g., servicing a pick-up request). In one implementation, theroute planner 122 can specifyroute segments 131 of a planned vehicle path which defines turn by turn directions for the vehicle 10 at any given time during the trip. Theroute planner 122 may utilize thesensor interface 110 to receive GPS information assensor data 111. Thevehicle control 128 can process route updates from theroute planner 122 ascommands 85 to progress along a path or route using default driving rules and actions (e.g., moderate steering and speed). - According to examples described herein, the
control system 100 includes anoptical flow unit 121 anddisparity mapper 126 to monitor the situational environment of the AV 10 continuously in order to dynamically calculate disparity maps and optical flow images as the AV 10 travels along a current route. The external entity can be a pedestrian or group of pedestrians, a human-driven vehicle, a bicyclist, and the like. - The
sensor data 111 captured by thesensors images 113 from thecamera interface 114 can be processed by an on-boardoptical flow unit 121 anddisparity mapper 126.Optical flow unit 121 anddisparity mapper 126 can utilize mapping resource data and previously recorded sub-maps that contain surface data for a given region.Disparity mapper 126 can compare this sub-map data with theimages 113 taken fromstereo camera 105 in order to improve disparity map calculations, both in terms of accuracy and speed.Classifier 127 can then use these maps and optical flow images to createobject classifications 133 to assist the AV 10 in maneuvering through road traffic to a particular destination. For example, thedisparity mapper 126 can utilize a current sub-map that includes recorded 3D LIDAR data and 3D stereo data of the current route traveled by the AV 10. Thedisparity mapper 126 can continuously compare thesensor data 111 to the 3D LIDAR data and stereo data of the current sub-map to help classifier 127 identify potential hazards, such as pedestrians, other vehicles, bicyclists, etc. Accordingly,classifier 127 can generateobject classifications 133 forevent logic 124. - In certain implementations, the
event logic 124 can refer to theobject classifications 133 in determining whether to trigger a response to a detected event. A detected event can correspond to a roadway condition or obstacle which, when detected, poses a potential hazard or threat of collision to the vehicle 10. By way of example, a detected event can include an object in the road segment, heavy traffic ahead, and/or wetness or other environment conditions on the road segment. Theevent logic 124 can usesensor data 111 andimages 113 from cameras, LIDAR, radar, sonar, or various other image or sensor component sets in order to detect the presence of such events as described. For example, theevent logic 124 can detect potholes, debris, objects projected to be on a collision trajectory, and the like. Thus, theevent logic 124 can detect events which enable thecontrol system 100 to make evasive actions or plan for any potential threats. - When events are detected, the
event logic 124 can signal anevent alert 135 that classifies the event and indicates the type of avoidance action to be performed. Additionally, thecontrol system 100 can determine whether an event corresponds to a potential incident with a human driven vehicle, a pedestrian, or other human entity external to the AV 10. An event can be scored or classified between a range of likely harmless (e.g., small debris in roadway) to very harmful (e.g., vehicle crash may be imminent) from thesensor data 111 and objectclassifications 133. In turn, thevehicle control 128 can determine a response based on the score or classification. Such response can correspond to anevent avoidance action 145, or an action that the vehicle 10 can perform to maneuver the vehicle 10 based on the detected event and its score or classification. By way of example, the vehicle response can include a slight or sharp vehicle maneuvering for avoidance using a steering control mechanism and/or braking component. Theevent avoidance action 145 can be signaled through thecommands 85 forcontrollers 84 of thevehicle interface subsystem 90. - When an anticipated dynamic object with a particular classification moves into a position of likely collision or interference, some examples provide that
event logic 124 can signal anevent alert 135 to cause thevehicle control 128 to generatecommands 85 that correspond to anevent avoidance action 145. For example, in the event of a bicycle crash in which the bicycle (or bicyclist) falls into the path of the vehicle 10,event logic 124 can signal anevent alert 135 to avoid the collision. Theevent alert 135 can indicate (i) a classification of the event (e.g., “serious” and/or “immediate”), (ii) information about the event, such as the type of object that generated theevent alert 135, and/or information indicating a type of action the vehicle 10 should take (e.g., location of object relative to path of vehicle, size or type of object, etc.). -
FIG. 2 illustrates an example autonomous vehicle including an improved disparity mapping and object classification system. TheAV 200 shown inFIG. 2 can include some or all aspects and functionality of the autonomous vehicle 10 described with respect toFIG. 1 . Referring toFIG. 2 , theAV 200 can include asensor array 205 that can providesensor data 207 to an on-boarddata processing system 210. As described herein, thesensor array 205 can include any number of active or passive sensors that continuously detect a situational environment of theAV 200. For example, thesensor array 205 can include a number of camera sensors (e.g., stereo camera 206), LIDAR sensor(s), proximity sensors, radar, and the like. Thedata processing system 210 can utilize thesensor data 207 andimages 208 to detect the situational conditions of theAV 200 as theAV 200 travels along a current route. For example, thedata processing system 210 can identify potential obstacles or road hazards, such as pedestrians, bicyclists, objects on the road, road cones, road signs, animals, etc., which classifier 235 can classify in order to enable anAV control system 220 to react accordingly. - The
AV 200 can further include adatabase 230 that includessub-maps 231 for the given region in which theAV 200 operates. The sub-maps 231 can comprise detailed road data previously recorded by a recording vehicle using sensor equipment, such as LIDAR, stereo camera, and/or radar equipment. In some aspects, several or all AVs in the fleet can include this sensor equipment to record updatedsub-maps 231 along traveled routes and submit the updatedsub-maps 231 to thebackend system 290, which can transmit the updatedsub-maps 231 to the other AVs in the fleet for storage. Accordingly, the sub-maps 231 can comprise ground-based, three-dimensional (3D) environment data along various routes throughout the given region (e.g., a city). - In many aspects, the on-board
data processing system 210 can provide continuous processeddata 214 to theAV control system 220 to respond to point-to-point activity in the AV's 200 surroundings. The processeddata 214 can comprise comparisons between theactual sensor data 207—which represents an operational environment of theAV 200, and which is continuously collected by thesensor array 205—and the stored sub-maps 231 (e.g., LIDAR-based sub-maps). In certain examples, thedata processing system 210 is programmed with machine learning capabilities to enable theAV 200 to identify and respond to conditions, events, or potential hazards. In variations, the on-boarddata processing system 210 can continuously comparesensor data 207 to stored sub-maps 231 in order to perform a localization to continuously determine a location and orientation of theAV 200 within the given region. Localization of theAV 200 is necessary in order to make theAV 200 self-aware of its instant location and orientation in comparison to the stored sub-maps 231 in order to maneuver theAV 200 on surface streets through traffic and identify and respond to potential hazards, such as pedestrians, or local conditions, such as weather or traffic. - The
data processing system 210 can compare thesensor data 207 from thesensor array 205 with a current sub-map 238 from the sub-maps 231 to identify obstacles and potential road hazards in real time. In some aspects, adisparity mapper 211 and optical flow unit 212, which can be part of thedata processing system 210, process thesensor data 207,images 208 from thestereo camera 206, and thecurrent sub-map 238 to create image maps 218 (e.g., disparity maps and optical flow images).Classifier 235 can then provideobject classifications 213—identifying obstacles and road hazards—to theAV control system 220, which can react accordingly by operating the steering, braking, andacceleration systems 225 of theAV 200 to perform low level maneuvering. - Applying knowledge of the location and orientation of
stereo camera 206 determined fromsensor data 207, thedisparity mapper 211 can use the 3D environment data from thecurrent sub-map 238 to generate a baseline disparity image that represents distances from thestereo camera 206 to known features of the environment. These features generally include terrain features, buildings, and other static, non-moving objects such as signs and trees. In some implementations, thedisparity mapper 211 can generate the baseline disparity image through a ray casting algorithm that renders the three-dimensional environment into a two-dimensional image. Ray casting traces rays from a point corresponding to one of the stereo camera lenses, one ray for each pixel of the image sensor resolution, and finds the distance to the closest object blocking the path of that ray. The distances for all the rays cast make up a set of data that represents the baseline disparity image. In an environment with no new features or objects that are not included in the 3D environment data (e.g., pedestrians or other vehicles), the baseline disparity image should represent an accurate map of the distances from thestereo camera 206 to features and objects in the scene. - Using a pair of
images 208 taken simultaneously of the scene, thedisparity mapper 211 finds a set of points in one image which can be identified as the same points in the other image in order to create a disparity map. To do this, points (i.e., pixels) or features in one image are matched with the corresponding points or features in the other image, which can be done using standard algorithms comparing colors, lighting, etc. However, the construction of these disparity maps is computationally expensive because the search space for matching pixels between the two images is large, especially for high resolution images needed to accurately identify and classify objects in a scene. - Therefore, the
disparity mapper 211 can use a combination of the baseline disparity image generated from thecurrent sub-map 238 and the pair ofimages 208 acquired from thestereo camera 206 to narrow the search spaces. Thedisparity mapper 211 can efficiently output a disparity map of the scene that represents the distances from thestereo camera 206 to features and objects in the scene, including both existing features from thecurrent sub-map 238 and new features and objects present in the scene that are not part of thecurrent sub-map 238 data. In some implementations, thedisparity mapper 211 matches pixels from the baseline disparity image to pixels in one of the stereo camera images. For example, disparity data corresponding to a pixel in the upper left corner of the left stereo camera image is taken from the baseline disparity image. Assuming that no new feature or object is present in the scene that is not included in thecurrent sub-map 238, the disparity data should be roughly equal (within a reasonable margin of error to account for map inaccuracies) to the disparity between pixels in the left and right stereo images taken of the scene. Thedisparity mapper 211 can then apply the baseline disparity image data to each pixel in the left image to determine a likely location of the corresponding pixel in the right image and reduce the search space of the correspondence algorithm. - Once corresponding pixels are found for each of the pixels in the left image, the
disparity mapper 211 can output the generated disparity map (as image maps 218) forclassifier 235 to use in classifying objects in the scene. In some aspects, an optical flow unit 212 can use the apparent motion of features in the field of view of the movingstereo camera 206 to supplement or replace the baseline disparity image generated from the 3D environment data. From either of the lenses of thestereo camera 206, a map of optical flow vectors can be calculated between a previous frame and a current frame. The optical flow unit 212 can use these vectors to improve the correspondence search algorithm. For example, given the motion vector of a pixel in the left image from thestereo camera 206, the motion vector of a corresponding pixel in the right image should be similar after accounting for the different perspective of the right lens of thestereo camera 206. Furthermore, image maps 218 can include images of optical flow vectors that classifier 235 can use to improveobject classifications 213. - In accordance with aspects disclosed, the
classifier 235 can also monitorsituational data 217 from thedata processing system 210 to identify potential areas of conflict. For example, theclassifier 235 can monitor forward directional stereoscopic camera data or LIDAR data to identify areas of concern. In one example, theclassifier 235 can utilize thecurrent sub-map 238 to identify features along the current route traveled (e.g., as indicated by the route data 232), such as traffic signals, intersections, road signs, crosswalks, bicycle lanes, parking areas, and the like. As theAV 200 approaches such features or areas, theclassifier 235 can monitor the forwardsituational data 217 to identify any external entities that may conflict with the operational flow of theAV 200, such as pedestrians near a crosswalk or another vehicle approaching an intersection. - In many examples, while the
AV control system 220 operates the steering, braking, andacceleration systems 225 along the current route on a high level,object classifications 213 provided to theAV control system 220 can indicate low level occurrences, such as obstacles and potential hazards, to which theAV control system 220 can make decisions and react. For example, objectclassifications 213 can indicate a pedestrian crossing the road, traffic signals, stop signs, other vehicles, road conditions, traffic conditions, bicycle lanes, crosswalks, pedestrian activity (e.g., a crowded adjacent sidewalk), and the like. TheAV control system 220 can respond to different types of objects by generating control commands 221 to reactively operate the steering, braking, andacceleration systems 225 accordingly. - In many implementations, the
AV control system 220 can receive adestination 219 from, for example, aninterface system 215 of theAV 200. Theinterface system 215 can include any number of touch-screens, voice sensors, mapping resources, etc., that enable apassenger 239 to provide apassenger input 241 indicating thedestination 219. For example, thepassenger 239 can type thedestination 219 into amapping engine 275 of theAV 200, or can speak thedestination 219 into theinterface system 215. Additionally or alternatively, theinterface system 215 can include a wireless communication module that can connect theAV 200 to anetwork 280 to communicate with a backendtransport arrangement system 290 to receiveinvitations 282 to service a pick-up or drop-off request.Such invitations 282 can include the destination 219 (e.g., a pick-up location), and can be received by theAV 200 as a communication over thenetwork 280 from the backendtransport arrangement system 290. In many aspects, the backendtransport arrangement system 290 can manage routes and/or facilitate transportation for users using a fleet of autonomous vehicles throughout a given region. The backendtransport arrangement system 290 can be operative to facilitate passenger pick-ups and drop-offs to generally service pick-up requests, facilitate delivery such as packages or food, and the like. - Based on the destination 219 (e.g., a pick-up location), the
AV control system 220 can utilize themapping engine 275 to receiveroute data 232 indicating a route to thedestination 219. In variations, themapping engine 275 can also generatemap content 226 dynamically indicating the route traveled to thedestination 219. Theroute data 232 and/ormap content 226 can be utilized by theAV control system 220 to maneuver theAV 200 to thedestination 219 along the selected route. For example, theAV control system 220 can dynamically generate control commands 221 for the autonomous vehicle's steering, braking, andacceleration system 225 to actively drive theAV 200 to thedestination 219 along the selected route. Optionally, themap content 226 showing the current route traveled can be streamed to theinterior interface system 215 so that the passenger(s) 239 can view the route and route progress in real time. - Methodology
-
FIG. 3 illustrates an example method of object classification in accordance with one or more embodiments.FIG. 4 illustrates an example method of disparity mapping in accordance with one or more embodiments. While operations of these example implementations are described below as being performed by specific components, modules or systems of theAV 200, it will be appreciated that these operations need not necessarily be performed by the specific components identified, and could be performed by a variety of components and modules, potentially distributed over a number of machines. Accordingly, references may be made to elements ofAV 200 for the purpose of illustrating suitable components or elements for performing a step or sub step being described. Alternatively, at least certain ones of the variety of components and modules described inAV 200 can be arranged within a single hardware, software, or firmware component. It will also be appreciated that some of the steps of this method may be performed in parallel or in a different order than illustrated. - Referring to
FIG. 3 , a vehicle can obtain sensor data for the environment through, for example, proximity or touch sensors, remote detection sensors such as provided by radar or LIDAR, a stereo camera, and/or sonar sensors as described with respect toFIGS. 1 and 2 (310). The vehicle can additionally obtain known data for the environment from previously recorded mapping resource data (i.e., sub-maps) that contain surface data for a given region. The vehicle can compare this sub-map data with the sensor data for the environment (320). The vehicle can then use the comparisons, including disparity maps and optical flow images, to create object classifications to assist the vehicle in maneuvering through road traffic to a particular destination (330). For example, a disparity mapper can utilize a current sub-map that includes recorded 3D LIDAR data and 3D stereo data of the current route traveled by the vehicle. The disparity mapper can continuously compare real-time sensor data to the data in the current sub-map to help a classifier identify potential hazards, such as pedestrians, other vehicles, bicyclists, etc. - Referring to
FIG. 4 , as the vehicle travels along a route, vehicle sensors can determine the location and orientation of the vehicle and its stereo camera. The sensors can determine latitude and longitude coordinates of the vehicle and a direction of travel, which can be further refined to identify the stereo camera's location in the world. For example, the vehicle's data processing system can retrieve sub-maps stored in a database of the vehicle or accessed remotely from the backend system via a network (410). The data processing system can use the 3D environment data stored in these sub-maps to perform localization and pose operations to determine a current location and orientation of the vehicle in relation to a given region (e.g., a city). Given the location of the stereo camera as it is disposed on the vehicle, the data processing system can further determine the precise location and orientation of the stereo camera in the world (412). - Applying knowledge of the location and orientation of the vehicle's stereo camera, a disparity mapper can use the 3D environment data to generate a baseline disparity image that represents distances from the stereo camera to known features of the environment (414). These features generally include terrain features, buildings, and other static, non-moving objects such as signs and trees.
- In some implementations, the disparity mapper can generate the baseline disparity image through a ray casting algorithm that renders the three-dimensional environment into a two-dimensional image. Ray casting traces rays from a point corresponding to one of the stereo camera lenses, one ray for each pixel of the image sensor resolution, and finds the distance to the closest object blocking the path of that ray. The distances for all the rays cast make up a set of data that represents the baseline disparity image. In an environment with no new features or objects, such as pedestrians or other vehicles, that are not included in the 3D environment data, the baseline disparity image should represent an accurate map of the distances from the stereo camera to features and objects in the scene.
- In another implementation, the disparity mapper can generate a baseline disparity image for each camera lens in the disparity mapping system using an alternate ray casting algorithm. The disparity mapper identifies 3D points in the 3D environment data that are visible to one or more camera lenses, and for each 3D point identified, the disparity mapper projects a ray from the 3D point to points representing each camera sensor in the disparity mapping system. The disparity mapper can then match each ray to 2D image coordinates for a baseline disparity image for that camera lens.
- In order for the disparity mapper to create a disparity map for a scene that accounts for new and unexpected features and objects, the stereo camera takes a simultaneous pair of images of the scene, one for each lens of the stereo camera (420). In some implementations, the data processing system can remove distortions from the images and perform image rectification to reduce the pixel correspondence search space (422). In most camera configurations, finding correspondences between pixels in the two images requires a search in two dimensions. However, if the two lenses of a stereo camera are aligned correctly to be coplanar, the search is simplified to one dimension—a horizontal line parallel to the line between the lenses. Since it may be impractical to maintain perfect alignment between the lenses, the disparity mapper can perform image rectification to achieve similar results. Rectification can be performed using a variety of algorithms, such as planar rectification, cylindrical rectification, and polar rectification so that the rectified images have epipolar lines parallel to the horizontal axis and corresponding points with identical vertical coordinates. In implementations where rectification is performed, the disparity mapper can create the baseline disparity image in order to take the rectification into account for performance gains.
- With a left and right image, horizontally displaced, taken simultaneously of the scene, the disparity mapper finds a set of points in one image which can be identified as the same points in the other image in order to create a disparity map. To do this, points (i.e., pixels) or features in one image are matched with the corresponding points or features in the other image, which can be done using standard algorithms comparing colors, lighting, etc. However, the construction of these disparity maps is computationally expensive because the search space for matching pixels between the two images is large, even when the images are rectified.
- The disparity mapper can use a combination of the baseline disparity image from the 3D environment data and the pair of images acquired from the stereo camera to narrow the search spaces. In doing so, the disparity mapper can efficiently output a disparity map of the scene that represents the distances from the stereo camera to features and objects in the scene, including both existing features from the 3D environment data and new features and objects present in the scene that are not part of the 3D environment data. In some implementations, the disparity mapper iterates through the pixels in the baseline disparity image and compares the pixel data in the baseline disparity image to its corresponding pixel in the same 2D location in one of the stereo camera images (typically the left image, but the right image can be used instead) (430). For example, disparity data corresponding to a pixel in the upper left corner of the left stereo camera image is taken from the baseline disparity image. Assuming that no new feature or object is present in the scene that is not included in the 3D environment data, the disparity data should be roughly equal (within a reasonable margin of error to account for map inaccuracies) to the disparity between pixels in the left and right stereo images taken of the scene.
- Therefore, the disparity mapper can apply the baseline disparity image data to each pixel in the left image to determine a likely location of the corresponding pixel in the right image (432). This baseline disparity image data contains a disparity value for each pixel that can be added to or subtracted from the 2D coordinates of the pixel to find the likely 2D coordinates for the corresponding pixel in the right image (434). Depending on rectification steps taken for the stereo camera images, the coordinate search may be performed on a single dimension.
- For a given pixel in the left image, if the baseline disparity image data indicates that the pixel should have a large disparity with its corresponding pixel in the right image (which would mean there is a nearby feature or object in the 3D environment data in the space that pixel represents), the disparity mapper starts the search for the corresponding pixel in the right image at a large coordinate distance from where the left pixel is located in the left image. If the baseline disparity image data indicates that the pixel should have a small disparity with its corresponding pixel in the right image (which would mean there are no nearby features or objects in the 3D environment data in the space that pixel represents), the disparity mapper starts the search for the corresponding pixel in the right image at a small coordinate distance from where the left pixel is located in the left image. This likely location used for the start of the correspondence search should be the correct corresponding pixel in conditions where the map is perfect and there are no changes from the environment recorded in the 3D environment data to the actual scene in the present. Therefore, given images of a scene that mostly matches the 3D environment data, the disparity mapper can accurately locate corresponding pixels in the right image without further searching. Where portions of the images of the scene do not match the 3D environment data (e.g., there is a moving object such as a pedestrian or vehicle in the scene), the disparity mapper may still reduce the search space using the determined likely location of the corresponding pixel. For example, the disparity mapper can search the pixel at the likely location and any pixels at a programmed distance around the likely location to find the most probable corresponding pixel.
- In some aspects, disparity maps calculated for previous images from the stereo camera can be used to supplement or replace the baseline disparity image generated from the 3D environment data. A vehicle equipped with a stereo camera may collect images fast enough such that the camera location does not change significantly from frame to frame and the background remains mostly static. Therefore, the prior disparity map generated for a previous frame can be useful to initialize the subsequent map. Only near range discontinuities and moving objects should result in large disparity differences. Everywhere else, the disparity values should be similar. In addition, the disparity mapper can incorporate information about the stereo camera's movement, which can be taken from other sensors on a vehicle, to modify the prior disparity map such that it represents a more accurate baseline disparity image for the stereo camera's current position. For example, if sensors on the vehicle indicate that the vehicle, and therefore the stereo camera, has travelled one meter forward since the last frame was taken and previous disparity map calculated, the disparity mapper can adjust the previous disparity map to take that one meter movement into account in generating a baseline disparity map. The disparity mapper can then use the generated baseline disparity map to reduce the search space for corresponding pixels between the current pair of stereo camera images.
- In another aspect, an optical flow unit can use the apparent motion of features in the field of view of a moving stereo camera to supplement or replace the baseline disparity image generated from the 3D environment data. From either of the lenses of the stereo camera, a map of optical flow vectors can be calculated between a previous frame and a current frame. The optical flow unit can use these vectors to improve the correspondence search algorithm in similar ways to using previously calculated disparity maps. For example, given the motion vector of a pixel in the left image from the stereo camera, the motion vector of a corresponding pixel in the right image should be similar after accounting for the different perspective of the right lens of the stereo camera.
- After a corresponding pixel is found in the right image, the disparity mapper can proceed to match remaining pixels (436). In some implementations, the disparity mapper can perform matching on multiple pixels simultaneously, either through separate processes or by grouping pixels together. Once corresponding pixels are found for each of the pixels in the left image, the disparity mapper can output the generated disparity map for other systems to use (438). For example, an object classifier can use the generated disparity map to classify objects in the scene based on the disparity map and other data.
- Hardware Diagram
-
FIG. 5 is a block diagram illustrating a computer system upon which examples described herein may be implemented. For example, thedata processing system 210 andclassifier 235 shown and described inFIG. 2 may be implemented on thecomputer system 500 ofFIG. 5 . Thecomputer system 500 can be implemented using one ormore processors 504, and one ormore memory resources 506. - According to some examples, the
computer system 500 may be implemented within an autonomous vehicle with software and hardware resources such as described with examples ofFIGS. 1 and 2 . In an example shown, thecomputer system 500 can be distributed spatially into various regions of the autonomous vehicle, with various aspects integrated with other components of the autonomous vehicle itself. For example, theprocessors 504 and/ormemory resources 506 can be provided in the trunk of the autonomous vehicle. Thevarious processing resources 504 of thecomputer system 500 can also executedisparity mapping instructions 512 using microprocessors or integrated circuits. In some examples, thedisparity mapping instructions 512 can be executed by theprocessing resources 504 or using field-programmable gate arrays (FPGAs). - In an example of
FIG. 5 , thecomputer system 500 can include a local communication interface 550 (or series of local links) to vehicle interfaces and other resources of the autonomous vehicle (e.g., the computer stack drives). In one implementation, thecommunication interface 550 provides a data bus or other local links to electro-mechanical interfaces of the vehicle, such as wireless or wired links to theAV control system 220. - The
memory resources 506 can include, for example, main memory, a read-only memory (ROM), storage device, and cache resources. The main memory ofmemory resources 506 can include random access memory (RAM) or other dynamic storage device, for storing information and instructions which are executable by theprocessors 504. Theprocessors 504 can execute instructions for processing information stored with the main memory of thememory resources 506. Themain memory 506 can also store temporary variables or other intermediate information which can be used during execution of instructions by one or more of theprocessors 504. Thememory resources 506 can also include ROM or other static storage device for storing static information and instructions for one or more of theprocessors 504. Thememory resources 506 can also include other forms of memory devices and components, such as a magnetic disk or optical disk, for purpose of storing information and instructions for use by one or more of theprocessors 504. - According to some examples, the
memory 506 may store a plurality of software instructions including, for example,disparity mapping instructions 512. Thedisparity mapping instructions 512 may be executed by one or more of theprocessors 504 in order to implement functionality such as described with respect to thedisparity mapper 211, optical flow unit 212, andclassifier 235 ofFIG. 2 . - In certain examples, the
computer system 500 can receivesensor data 562 over thecommunication interface 550 from various AV subsystems 560 (e.g., theAV control system 220 or data processing system 210). In executing thedisparity mapping instructions 512, theprocessing resources 504 can monitor thesensor data 562 and generate object classifications that theAV control system 220 can use to send commands to theoutput systems 520 of theAV 200 in accordance with examples described herein. - It is contemplated for examples described herein to extend to individual elements and concepts described herein, independently of other concepts, ideas or systems, as well as for examples to include combinations of elements recited anywhere in this application. Although examples are described in detail herein with reference to the accompanying drawings, it is to be understood that the concepts are not limited to those precise examples. As such, many modifications and variations will be apparent to practitioners skilled in this art. Accordingly, it is intended that the scope of the concepts be defined by the following claims and their equivalents. Furthermore, it is contemplated that a particular feature described either individually or as part of an example can be combined with other individually described features, or parts of other examples, even if the other features and examples make no mentioned of the particular feature. Thus, the absence of describing combinations should not preclude claiming rights to such combinations.
Claims (18)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/176,561 US20170359561A1 (en) | 2016-06-08 | 2016-06-08 | Disparity mapping for an autonomous vehicle |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/176,561 US20170359561A1 (en) | 2016-06-08 | 2016-06-08 | Disparity mapping for an autonomous vehicle |
Publications (1)
Publication Number | Publication Date |
---|---|
US20170359561A1 true US20170359561A1 (en) | 2017-12-14 |
Family
ID=60573248
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/176,561 Abandoned US20170359561A1 (en) | 2016-06-08 | 2016-06-08 | Disparity mapping for an autonomous vehicle |
Country Status (1)
Country | Link |
---|---|
US (1) | US20170359561A1 (en) |
Cited By (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180267170A1 (en) * | 2017-03-17 | 2018-09-20 | Sharp Kabushiki Kaisha | Obstacle detection device, traveling apparatus, obstacle detection system, and obstacle detection method |
US20190057263A1 (en) * | 2017-08-21 | 2019-02-21 | 2236008 Ontario Inc. | Automated driving system that merges heterogenous sensor data |
US10244164B1 (en) * | 2017-09-11 | 2019-03-26 | Qualcomm Incorporated | Systems and methods for image stitching |
US20190232964A1 (en) * | 2018-01-30 | 2019-08-01 | Toyota Motor Engineering & Manufacturing North America, Inc. | Fusion of front vehicle sensor data for detection and ranging of preceding objects |
US20190258737A1 (en) * | 2018-02-20 | 2019-08-22 | Zoox, Inc. | Creating clean maps including semantic information |
US10663303B2 (en) * | 2017-06-12 | 2020-05-26 | Panasonic Intellectual Property Management Co., Ltd. | System and method for dynamically authenticating map data using blockchains |
US10832414B2 (en) | 2017-11-21 | 2020-11-10 | Zoox, Inc. | Sensor data segmentation |
US10853960B2 (en) * | 2017-09-14 | 2020-12-01 | Samsung Electronics Co., Ltd. | Stereo matching method and apparatus |
CN112166058A (en) * | 2018-05-18 | 2021-01-01 | 雷诺股份公司 | Adaptive cruise control for motor vehicles |
US10964034B1 (en) * | 2019-10-30 | 2021-03-30 | Nvidia Corporation | Vertical disparity detection in stereoscopic images from optical flow data |
US10967862B2 (en) | 2017-11-07 | 2021-04-06 | Uatc, Llc | Road anomaly detection for autonomous vehicle |
US11010924B2 (en) * | 2016-10-12 | 2021-05-18 | Hangzhou Hikvision Digital Technology Co., Ltd. | Method and device for determining external parameter of stereoscopic camera |
WO2021096629A1 (en) * | 2019-11-15 | 2021-05-20 | Argo AI, LLC | Geometry-aware instance segmentation in stereo image capture processes |
US11050932B2 (en) * | 2019-03-01 | 2021-06-29 | Texas Instruments Incorporated | Using real time ray tracing for lens remapping |
US11288860B2 (en) * | 2017-08-29 | 2022-03-29 | Sony Corporation | Information processing apparatus, information processing method, program, and movable object |
Citations (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5309522A (en) * | 1992-06-30 | 1994-05-03 | Environmental Research Institute Of Michigan | Stereoscopic determination of terrain elevation |
US20010043738A1 (en) * | 2000-03-07 | 2001-11-22 | Sawhney Harpreet Singh | Method of pose estimation and model refinement for video representation of a three dimensional scene |
US20050190972A1 (en) * | 2004-02-11 | 2005-09-01 | Thomas Graham A. | System and method for position determination |
US20060221072A1 (en) * | 2005-02-11 | 2006-10-05 | Se Shuen Y S | 3D imaging system |
US20070031064A1 (en) * | 2004-06-10 | 2007-02-08 | Wenyi Zhao | Method and apparatus for aligning video to three-dimensional point clouds |
US20100182406A1 (en) * | 2007-07-12 | 2010-07-22 | Benitez Ana B | System and method for three-dimensional object reconstruction from two-dimensional images |
US20110222757A1 (en) * | 2010-03-10 | 2011-09-15 | Gbo 3D Technology Pte. Ltd. | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
US8447098B1 (en) * | 2010-08-20 | 2013-05-21 | Adobe Systems Incorporated | Model-based stereo matching |
US20140172296A1 (en) * | 2012-07-30 | 2014-06-19 | Aleksandr Shtukater | Systems and methods for navigation |
US20140267633A1 (en) * | 2013-03-15 | 2014-09-18 | Pelican Imaging Corporation | Systems and Methods for Stereo Imaging with Camera Arrays |
US20150109415A1 (en) * | 2013-10-17 | 2015-04-23 | Samsung Electronics Co., Ltd. | System and method for reconstructing 3d model |
US20150269737A1 (en) * | 2014-03-24 | 2015-09-24 | Hong Kong Applied Science & Technology Research Institute Company Limited | Multi-View Synthesis in Real-Time With Fallback to 2D from 3D to Reduce Flicker in Low or Unstable Stereo-Matching Image Regions |
US20150312552A1 (en) * | 2014-04-29 | 2015-10-29 | Etron Technology, Inc. | Portable three-dimensional scanner and method of generating a three-dimensional scan result corresponding to an object |
US9369689B1 (en) * | 2015-02-24 | 2016-06-14 | HypeVR | Lidar stereo fusion live action 3D model video reconstruction for six degrees of freedom 360° volumetric virtual reality video |
US20160212409A1 (en) * | 2015-01-21 | 2016-07-21 | Nextvr Inc. | Methods and apparatus for environmental measurements and/or stereoscopic image capture |
US20160224856A1 (en) * | 2015-01-29 | 2016-08-04 | Qualcomm Incorporated | Occlusion handling for computer vision |
US20170256040A1 (en) * | 2014-08-31 | 2017-09-07 | Brightway Vision Ltd. | Self-Image Augmentation |
US20170344004A1 (en) * | 2016-05-27 | 2017-11-30 | Cnh Industrial America Llc | Dynamic in-cabin autonomous vehicle control systems |
US20170366795A1 (en) * | 2016-06-17 | 2017-12-21 | Altek Semiconductor Corp. | Stereo image generating method and electronic apparatus utilizing the method |
US20180007345A1 (en) * | 2015-01-16 | 2018-01-04 | Imra Europe S.A.S. | Self-rectification of stereo camera |
US20190011921A1 (en) * | 2015-09-15 | 2019-01-10 | SZ DJI Technology Co., Ltd. | Systems and methods for uav interactive instructions and control |
-
2016
- 2016-06-08 US US15/176,561 patent/US20170359561A1/en not_active Abandoned
Patent Citations (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5309522A (en) * | 1992-06-30 | 1994-05-03 | Environmental Research Institute Of Michigan | Stereoscopic determination of terrain elevation |
US20010043738A1 (en) * | 2000-03-07 | 2001-11-22 | Sawhney Harpreet Singh | Method of pose estimation and model refinement for video representation of a three dimensional scene |
US20050190972A1 (en) * | 2004-02-11 | 2005-09-01 | Thomas Graham A. | System and method for position determination |
US20070031064A1 (en) * | 2004-06-10 | 2007-02-08 | Wenyi Zhao | Method and apparatus for aligning video to three-dimensional point clouds |
US20060221072A1 (en) * | 2005-02-11 | 2006-10-05 | Se Shuen Y S | 3D imaging system |
US20100182406A1 (en) * | 2007-07-12 | 2010-07-22 | Benitez Ana B | System and method for three-dimensional object reconstruction from two-dimensional images |
US20110222757A1 (en) * | 2010-03-10 | 2011-09-15 | Gbo 3D Technology Pte. Ltd. | Systems and methods for 2D image and spatial data capture for 3D stereo imaging |
US8447098B1 (en) * | 2010-08-20 | 2013-05-21 | Adobe Systems Incorporated | Model-based stereo matching |
US20140172296A1 (en) * | 2012-07-30 | 2014-06-19 | Aleksandr Shtukater | Systems and methods for navigation |
US20140267633A1 (en) * | 2013-03-15 | 2014-09-18 | Pelican Imaging Corporation | Systems and Methods for Stereo Imaging with Camera Arrays |
US20150109415A1 (en) * | 2013-10-17 | 2015-04-23 | Samsung Electronics Co., Ltd. | System and method for reconstructing 3d model |
US20150269737A1 (en) * | 2014-03-24 | 2015-09-24 | Hong Kong Applied Science & Technology Research Institute Company Limited | Multi-View Synthesis in Real-Time With Fallback to 2D from 3D to Reduce Flicker in Low or Unstable Stereo-Matching Image Regions |
US20150312552A1 (en) * | 2014-04-29 | 2015-10-29 | Etron Technology, Inc. | Portable three-dimensional scanner and method of generating a three-dimensional scan result corresponding to an object |
US20170256040A1 (en) * | 2014-08-31 | 2017-09-07 | Brightway Vision Ltd. | Self-Image Augmentation |
US20180007345A1 (en) * | 2015-01-16 | 2018-01-04 | Imra Europe S.A.S. | Self-rectification of stereo camera |
US20160212409A1 (en) * | 2015-01-21 | 2016-07-21 | Nextvr Inc. | Methods and apparatus for environmental measurements and/or stereoscopic image capture |
US20160224856A1 (en) * | 2015-01-29 | 2016-08-04 | Qualcomm Incorporated | Occlusion handling for computer vision |
US9369689B1 (en) * | 2015-02-24 | 2016-06-14 | HypeVR | Lidar stereo fusion live action 3D model video reconstruction for six degrees of freedom 360° volumetric virtual reality video |
US20190011921A1 (en) * | 2015-09-15 | 2019-01-10 | SZ DJI Technology Co., Ltd. | Systems and methods for uav interactive instructions and control |
US20170344004A1 (en) * | 2016-05-27 | 2017-11-30 | Cnh Industrial America Llc | Dynamic in-cabin autonomous vehicle control systems |
US20170366795A1 (en) * | 2016-06-17 | 2017-12-21 | Altek Semiconductor Corp. | Stereo image generating method and electronic apparatus utilizing the method |
Cited By (27)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11010924B2 (en) * | 2016-10-12 | 2021-05-18 | Hangzhou Hikvision Digital Technology Co., Ltd. | Method and device for determining external parameter of stereoscopic camera |
US10830893B2 (en) * | 2017-03-17 | 2020-11-10 | Sharp Kabushiki Kaisha | Obstacle detection device, traveling apparatus, obstacle detection system, and obstacle detection method |
US20180267170A1 (en) * | 2017-03-17 | 2018-09-20 | Sharp Kabushiki Kaisha | Obstacle detection device, traveling apparatus, obstacle detection system, and obstacle detection method |
US10663303B2 (en) * | 2017-06-12 | 2020-05-26 | Panasonic Intellectual Property Management Co., Ltd. | System and method for dynamically authenticating map data using blockchains |
US20190057263A1 (en) * | 2017-08-21 | 2019-02-21 | 2236008 Ontario Inc. | Automated driving system that merges heterogenous sensor data |
US10599931B2 (en) * | 2017-08-21 | 2020-03-24 | 2236008 Ontario Inc. | Automated driving system that merges heterogenous sensor data |
US11288860B2 (en) * | 2017-08-29 | 2022-03-29 | Sony Corporation | Information processing apparatus, information processing method, program, and movable object |
US10244164B1 (en) * | 2017-09-11 | 2019-03-26 | Qualcomm Incorporated | Systems and methods for image stitching |
US10853960B2 (en) * | 2017-09-14 | 2020-12-01 | Samsung Electronics Co., Ltd. | Stereo matching method and apparatus |
US10967862B2 (en) | 2017-11-07 | 2021-04-06 | Uatc, Llc | Road anomaly detection for autonomous vehicle |
US11731627B2 (en) | 2017-11-07 | 2023-08-22 | Uatc, Llc | Road anomaly detection for autonomous vehicle |
US11798169B2 (en) | 2017-11-21 | 2023-10-24 | Zoox, Inc. | Sensor data segmentation |
US10832414B2 (en) | 2017-11-21 | 2020-11-10 | Zoox, Inc. | Sensor data segmentation |
US11475573B2 (en) | 2017-11-21 | 2022-10-18 | Zoox, Inc. | Sensor data segmentation |
US11091162B2 (en) * | 2018-01-30 | 2021-08-17 | Toyota Motor Engineering & Manufacturing North America, Inc. | Fusion of front vehicle sensor data for detection and ranging of preceding objects |
US20190232964A1 (en) * | 2018-01-30 | 2019-08-01 | Toyota Motor Engineering & Manufacturing North America, Inc. | Fusion of front vehicle sensor data for detection and ranging of preceding objects |
US11157527B2 (en) * | 2018-02-20 | 2021-10-26 | Zoox, Inc. | Creating clean maps including semantic information |
US20190258737A1 (en) * | 2018-02-20 | 2019-08-22 | Zoox, Inc. | Creating clean maps including semantic information |
US11524682B2 (en) * | 2018-05-18 | 2022-12-13 | Renault S.A.S | Adaptive cruise control for motor vehicle |
CN112166058A (en) * | 2018-05-18 | 2021-01-01 | 雷诺股份公司 | Adaptive cruise control for motor vehicles |
US11303807B2 (en) | 2019-03-01 | 2022-04-12 | Texas Instruments Incorporated | Using real time ray tracing for lens remapping |
US11050932B2 (en) * | 2019-03-01 | 2021-06-29 | Texas Instruments Incorporated | Using real time ray tracing for lens remapping |
US10964034B1 (en) * | 2019-10-30 | 2021-03-30 | Nvidia Corporation | Vertical disparity detection in stereoscopic images from optical flow data |
US11810308B2 (en) * | 2019-10-30 | 2023-11-07 | Nvidia Corporation | Vertical disparity detection in stereoscopic images using a deep neural network |
US11120280B2 (en) | 2019-11-15 | 2021-09-14 | Argo AI, LLC | Geometry-aware instance segmentation in stereo image capture processes |
US11669972B2 (en) | 2019-11-15 | 2023-06-06 | Argo AI, LLC | Geometry-aware instance segmentation in stereo image capture processes |
WO2021096629A1 (en) * | 2019-11-15 | 2021-05-20 | Argo AI, LLC | Geometry-aware instance segmentation in stereo image capture processes |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10489686B2 (en) | Object detection for an autonomous vehicle | |
US20170359561A1 (en) | Disparity mapping for an autonomous vehicle | |
US11462022B2 (en) | Traffic signal analysis system | |
US10846817B2 (en) | Systems and methods for registering 3D data with 2D image data | |
CN109863513B (en) | Neural network system for autonomous vehicle control | |
CN110377026B (en) | Information processing apparatus, storage medium, and information processing method | |
US20180273031A1 (en) | Travel Control Method and Travel Control Apparatus | |
EP3825958B1 (en) | A new way to generate tight 2d bounding boxes for autonomous driving labeling | |
US20230351773A1 (en) | Systems and methods for detecting traffic lights corresponding to a driving lane | |
US11948381B2 (en) | Determining object mobility parameters using an object sequence | |
EP4141482A1 (en) | Systems and methods for validating camera calibration in real-time | |
KR102631148B1 (en) | Automatically detecting traffic signals using sensor data | |
US12025752B2 (en) | Systems and methods for detecting erroneous LIDAR data | |
US20230417885A1 (en) | Systems and methods for detecting erroneous lidar data | |
US20240096109A1 (en) | Automatic lane marking extraction and classification from lidar scans | |
US20230382427A1 (en) | Motion prediction in an autonomous vehicle using fused synthetic and camera images | |
US12033338B2 (en) | Point cloud alignment systems for generating high definition maps for vehicle navigation | |
US20240054660A1 (en) | Point cloud alignment systems for generating high definition maps for vehicle navigation | |
US20240126268A1 (en) | Track refinement networks | |
US20230237679A1 (en) | Aligning geodata graph over electronic maps | |
US20230237813A1 (en) | Semantic annotation of sensor data with overlapping physical features | |
WO2024081226A1 (en) | Track refinement networks |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: UBER TECHNOLOGIES, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:VALLESPI-GONZALEZ, CARLOS;REEL/FRAME:039033/0437 Effective date: 20160628 |
|
AS | Assignment |
Owner name: APPARATE INTERNATIONAL C.V., BERMUDA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:UBER TECHNOLOGIES, INC.;REEL/FRAME:040541/0940 Effective date: 20160920 |
|
AS | Assignment |
Owner name: UBER TECHNOLOGIES, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:APPARATE INTERNATIONAL C.V.;REEL/FRAME:040543/0985 Effective date: 20161026 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
AS | Assignment |
Owner name: UTAC, LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:UBER TECHNOLOGIES, INC.;REEL/FRAME:050912/0757 Effective date: 20190701 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
AS | Assignment |
Owner name: UATC, LLC, CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA PREVIOUSLY RECORDED ON REEL 050912 FRAME 0757. ASSIGNOR(S) HEREBY CONFIRMS THE RECEIVING PARTY DATA/ASSIGNEE SHOULD BE UATC, LLC;ASSIGNOR:UBER TECHNOLOGIES, INC.;REEL/FRAME:052084/0590 Effective date: 20190701 |
|
AS | Assignment |
Owner name: UATC, LLC, CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE CORRECT ASSIGNEE NAME PREVIOUSLY RECORDED AT REEL: 050912 FRAME: 0757. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:UBER TECHNOLOGIES, INC.;REEL/FRAME:052133/0436 Effective date: 20190701 |
|
STCV | Information on status: appeal procedure |
Free format text: NOTICE OF APPEAL FILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: AURORA OPERATIONS, INC., PENNSYLVANIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:UATC, LLC;REEL/FRAME:067733/0001 Effective date: 20240321 |