JP2023521765A - Method and system for providing depth maps with confidence estimates - Google Patents

Abstract

An automated vehicle assistance system is provided for supervised or unsupervised vehicle movement. The system includes a control system and a first sensor system. A first sensor system may receive first image data of a scene and may output a first disparity map and a first confidence map based on the first image data. The control system may output a video stream based on the first disparity map and the first confidence map. The vehicle assistance system may also include a second sensor system that receives second image data of at least a portion of the scene and outputs a second confidence map based on the second image data. A video stream may include superframes, each superframe including a 2D image of a scene, a depth map corresponding to the 2D image, and a certainty map corresponding to the depth map.

Description

The technology of the present invention relates to stereo vision systems. Specifically, the technique of the present invention captures an image of a scene and uses two-dimensional (“2D”) image data, depth estimate data related to distances to objects in the scene, and robustness of the depth estimate data. Stereo vision systems (eg, stereo camera systems) that output confidence data related to sexuality levels.

Stereo vision systems typically use to estimate distance by measuring the disparity or parallax between matched pixels in the image captured by the left camera and the image captured by the right camera. Two cameras (eg, left camera and right camera) are used. For example, US Pat. No. 6,200,300 outputs a disparity map, a 2D matrix containing pixel shift data, corresponding to a rectified image captured by one of the cameras (often the left camera), a stereo A vision system and method are disclosed. Since the depth of a pixel is inversely proportional to the disparity of the pixel, an estimate of the depth for each pixel in the image, corresponding to the distance from the camera to the object imaged by the pixel, can be easily calculated from the disparity map. . As such, the terms "depth map" and "parallax map" are sometimes used interchangeably herein as they provide nearly identical information about the scene of an image.

A problem with existing stereo vision systems is that depth estimates typically have no indication of how reliable (or unreliable) the estimate is on a pixel-by-pixel or even frame-by-frame basis. is to be provided to In conventional driver assistance systems, automated decisions to stop or steer a vehicle are based on information from a number of sensors (e.g., radar, lidar, cameras, etc.). Information can also be done without knowledge of its reliability. This results in the driver assistance system overcompensating for the uncertainties of conflicting information from the sensors by performing unnecessary procedures (e.g. driving very slowly, braking too frequently, etc.). which can reduce passenger comfort, or result in driver assistance systems undercompensating for uncertainties in conflicting information by arbitrarily choosing one sensor over another. and may reduce passenger risk. As will be appreciated, decisions made by driver assistance systems can have a significant impact on passenger safety and, at times, can be critical and life-threatening decisions. For example, when a vehicle is traveling at cruising speeds typical of highway driving, accurate knowledge of the distance to the object will allow it to collide with the object while maintaining or nearly maintaining cruising speed. To avoid this, how the driver assistance system controls the vehicle can be essential.

U.S. Pat. No. 8,208,716

Autonomous vehicles and advanced driver assistance systems acquire information about the vehicle's surroundings to determine how the vehicle's control system steers the vehicle and adjusts the vehicle's speed (e.g., how various types of sensors (such as information from lidar, radar, cameras, ultrasound, stereo cameras, etc.). As will be appreciated, a vehicle's control system may include dozens of electronic control modules or units ("ECUs"), and in some cases over 100 ECUs, each ECU controlling the operation of the vehicle. (eg, speed control ECU, brake control ECU, transmission control ECU, engine control ECU, battery management ECU, etc.).

The inventors have recognized that it is important to know the confidence or level of certainty of the measurements from each of the sensors in order to best fuse or combine data from different types of sensors. and recognized. In some aspects of the present technology, high-resolution depth information, indicating distances to features appearing in pairs of images, may be determined through stereo matching of pairs of images. Stereo matching may be performed to provide range certainty information based on any one or any combination of disparity features, prior images, cost curves, and local characteristics. Range certainty information can be used to minimize or eliminate problems in sensor fusion by allowing the reliability of depth information to be graded, thus allowing decisions made using depth information to be can increase the degree of safety of

According to aspects of the present technology, an automated vehicle assistance system for supervised or unsupervised vehicle travel is provided. A system receives first image data of a vehicle control system and a scene, comprising a computer processor and a memory coupled to the computer processor, and generates a first disparity map based on the first image data. and a first sensor system configured to output a first confidence map. The vehicle control system receives the first disparity map and the first confidence map from the first sensor system and produces a video stream comprising the first disparity map and the first confidence map. output.

In some embodiments of this aspect, the first confidence map may be encoded to be part of the first disparity map in the video stream. In some embodiments, the disparity map may consist of disparity data for each of the pixels and the confidence map may consist of confidence data for each of the pixels. In some embodiments, the first image data may consist of data for a left and right two-dimensional (2D) first image, and the first sensor system may extract left and right (2D) first images from the first image data. 2D) rectified first image and a first cost volume map, wherein the first sensor system receives the 2D rectified first image and the first cost volume map; and the first cost volume map to generate a first confidence map. In some embodiments, the first sensor system uses one of a uniqueness value determined from a semi-global matching (SGM) algorithm and an image texture metric determined from a Sobel operation on the first image data, or Based on both, it may be configured to generate a first belief map.

In some embodiments of this aspect, the vehicle assistance system receives second image data of at least a portion of the scene and outputs a second confidence map based on the second image data. It may further comprise a second sensor system configured. The vehicle control system receives a second confidence map from the second sensor system and outputs the video stream as a sequence of superframes, each superframe being a first disparity map. and outputting information based on the first belief map and the second belief map. In some embodiments, the vehicle control system may be configured to output control signals to the vehicle's electronic control unit (ECU) based on the information in the video stream. In some embodiments, the first sensor system is configured to process the first image data to generate a first disparity map and a first confidence map. can be one sensor module and the second sensor system can be a second sensor module configured to process the second image data to generate a second belief map; A first sensor module and a second sensor module may be stored in memory and a computer processor may be configured to perform the first sensor module and the second sensor module. In some embodiments, the video stream comprises at least one superframe consisting of a first disparity map, a first confidence map, a first disparity map and a second confidence map. at least one superframe consisting of In some embodiments, the video stream may consist of at least one superframe consisting of a portion of the first belief map and a portion of the second belief map. In some embodiments, the first image data may consist of stereo vision data and the second image data may consist of lidar data.

In some embodiments of this aspect, the vehicle assistance system receives third image data of at least a portion of the scene and outputs a third confidence map based on the third image data. It may further comprise a configured third sensor system. The third image data consists of radar data or acoustic data.

In some embodiments of this aspect, each superframe of the video stream may consist of a two-dimensional (2D) image of the scene, a depth map of the scene, and a certainty map of the scene. In some embodiments, the scene certainty map may consist of a first confidence map, or a second confidence map, or a combination of the first and second confidence maps. . In some embodiments, the depth map of the scene may consist of a first disparity map modulated with image data corresponding to a 2D image of the scene, and the certainty map of the scene may consist of the 2D image of the scene. or a second belief map, or a combination of the first and second belief maps, modulated with image data corresponding to . In some embodiments, the pixels of the 2D image of the scene, the pixels of the depth map of the scene, and the pixels of the certainty map of the scene may be temporally and spatially matched. In some embodiments, the vehicle control system combines disparity information from the first disparity map and beliefs from the first and second belief maps to reduce the data size of the video stream. It may be configured to encode degree information.

In some embodiments of this aspect, the vehicle assistance system may further comprise a pair of cameras configured to be mounted on the vehicle. The camera may be configured to provide first image data to the first sensor system.

In some embodiments of this aspect, the video stream may consist of two-dimensional (2D) color images, each 2D color image consisting of a plurality of pixels, and the alpha channel transparency of each pixel determines the confidence about the pixel. Proportional to degrees. In some embodiments, the color of the 2D color image may indicate depth range.

According to another aspect of the present technology, a non-supervised vehicle assistance system having code stored thereon which, when performed by a computer processor, causes the computer processor to perform a method of an automotive vehicle assistance system for supervised or unsupervised vehicle movement. A temporary computer-readable storage medium is provided. The method comprises a computer processor obtaining a first disparity map and a first confidence map, the first disparity map and the first confidence map being first image data of a scene and the computer processor outputting a video stream comprising the first disparity map and the first confidence map.

In some embodiments of this aspect, outputting the video stream may comprise the computer processor encoding the first confidence map to become part of the first disparity map. In some embodiments, the first image data may consist of a plurality of pixels, the disparity map may consist of disparity data for each of the pixels, and the confidence map may consist of confidence data for each of the pixels. can consist of In some embodiments, the method comprises the steps of: a computer processor obtaining a second confidence map corresponding to second image data of at least a portion of a scene; wherein each superframe consists of information based on the first disparity map, the first confidence map, and the second confidence map . In some embodiments, the method may further comprise the computer processor outputting a control signal to an electronic control unit (ECU) of the vehicle based on the information in the video stream. In some embodiments, the method comprises the steps of: a computer processor processing the first image data to obtain a first disparity map and a first confidence map; processing the second image data to obtain a second confidence map.

In some embodiments of this aspect, the step of outputting the video stream includes the computer processor preparing at least one superframe to consist of the first disparity map and the first confidence map. and a computer processor preparing at least one superframe to consist of a first disparity map and a second confidence map. In some embodiments, the step of outputting the video stream includes the computer processor preparing at least one superframe consisting of a portion of the first belief map and a portion of the second belief map. It can further comprise a step. In some embodiments, the first image data may consist of stereo vision data and the second image data may consist of lidar data, or radar data, or acoustic data.

In some embodiments of this aspect, the step of outputting the video stream includes: Preparing each superframe of the video stream. In some embodiments, the step of preparing each superframe by a computer processor includes combining pixels of a 2D image of the scene, pixels of a depth map of the scene, and pixels of a certainty map of the scene into temporal and It may consist of a spatial matching step. In some embodiments, the step of outputting the video stream includes the computer processor disparity information from the first disparity map and the first confidence map and the second confidence map to reduce the data size of the video stream. encoding confidence information from the confidence map of the . In some embodiments, the step of outputting the video stream comprises: each 2D color image consisting of a plurality of pixels, an alpha channel transparency for each pixel proportional to a confidence value for the pixel; Color may consist of preparing a two-dimensional (2D) color image to indicate the depth range.

According to another aspect of the technology, a stereo vision system is provided. The system is a stereo camera system configured to capture a sequence of image pairs, each pair of images consisting of a first image and a second image captured simultaneously. a camera system and a computer processor programmed to receive a stream of image data from the stereo camera system, the image data corresponding to a sequence of pairs of images. , a computer processor; A computer processor rectifies the first image and the second image to generate a two-dimensional (2D) pixel map of matched pixels for each of the pair of images; and for each pixel of the pixmap a confidence value for the depth value. The computer processor may also be programmed to issue the control signal when at least one of the confidence values indicates an image anomaly.

In some embodiments of this aspect, the image anomaly may correspond to one or more pixels in the portion of the confidence map that have confidence values below a predetermined threshold. In some embodiments, the image anomaly is one or more pixels of the portion of the confidence map that have confidence values below a predetermined threshold for two or more consecutive pairs of images of the sequence. can correspond to In some embodiments, an image anomaly may consist of multiple pixels in contiguous regions of the confidence map. In some embodiments, the control signal may be configured to cause an audible tone, which may be a pre-recorded message. In some embodiments, the control signal may be issued to the vehicle's engine control module. In some embodiments, for each pixel of the pixmap, the confidence value is the presence or absence of an edge at the pixel, the illumination level of the pixel, and the texture of the first and second images from which the pixmap was generated. can be determined based on the value.

In some embodiments of this aspect, the computer processor may be programmed to output a sequence of superframes corresponding to the sequence of image pairs, each superframe being a 2D image and a 2D image. and the corresponding confidence map. In some embodiments, the 2D image can be the first image or the second image. In some embodiments, the computer processor outputs the sequence of superframes as a display signal that causes the display to show a 2D image and a visual confidence indicator corresponding to the confidence map. can be programmed to The display signal may cause the confidence indicator to be displayed pixel by pixel as the transparency of each pixel of the 2D image. In some embodiments, each superframe may consist of a 2D image, a confidence map, and a disparity map corresponding to the 2D image.

According to another aspect of the present technology, there is provided a non-transitory computer-readable storage medium having code stored thereon which, when executed by a computer processor, causes the computer processor to perform a method of a stereo vision system. The method comprises the steps of: a computer processor receiving a stream of image data from a stereo camera system, the image data corresponding to a sequence of image pairs, each pair of images captured simultaneously; and a second image, and for each pair of images, the computer processor generates a two-dimensional (2D) pixel map of matched pixels from the first image and , the second image; for each pixel of the pixmap, determining a depth value; for each pixel of the pixmap, determining a confidence value for the depth value; and the computer processor issuing a control signal when at least one of the maps indicates an image anomaly.

In some embodiments of this aspect, the image anomaly may correspond to one or more pixels in the portion of the confidence map that have confidence values below a predetermined threshold. In some embodiments, an image anomaly is one or more pixels in a portion of the confidence map that have confidence values below a predetermined threshold for two or more consecutive pairs of images of the sequence. can correspond to In some embodiments, an image anomaly may consist of multiple pixels in contiguous regions of the confidence map.

In some embodiments of this aspect, the control signal may be configured to cause an audible sound. For example, the audible tone can be a pre-recorded message. In some embodiments, the control signal may be issued to the vehicle's engine control module. In some embodiments, for each pixel in the pixmap, the confidence value is the presence or absence of an edge at the pixel, the illumination level of the pixel, and the texture of the first and second images from which the pixmap was generated. can be determined based on the value.

In some embodiments of this aspect, the method comprises the step of the computer processor outputting a sequence of superframes corresponding to the sequence of image pairs, each superframe being a 2D image and a 2D It may further comprise a process comprising a disparity map corresponding to the image and a confidence map corresponding to the 2D image. In some embodiments, the 2D image can be the first image or the second image. In some embodiments, outputting the sequence of superframes includes outputting a display signal that causes the display to show a 2D image and a visual confidence indicator corresponding to the confidence map. can consist of In some embodiments, the display signal may cause the confidence indicator to be displayed on a pixel-by-pixel basis as the transparency of each pixel of the 2D image. In some embodiments, each superframe may consist of a 2D image, a confidence map, and a disparity map corresponding to the 2D image.

The features described above may be used separately or together in any combination in any of the embodiments discussed herein.
Various aspects and embodiments of the technology disclosed herein are described below with reference to the accompanying figures. It should be appreciated that the drawings are not necessarily to scale. Items appearing in multiple figures may be indicated by the same reference numerals. For clarity purposes, not all components may be labeled in all figures. At least one of the accompanying figures, as indicated below, is at least partially drawn in color.

1 is a block diagram of a stereo vision system, in accordance with some embodiments of the present technology; FIG. FIG. 2 illustrates an arrangement of sensors and electronic control units located on a vehicle, in accordance with some embodiments of the present technology; 1 is a block diagram of a stereo vision system coupled to a control system, according to some embodiments of the present technology; FIG. FIG. 4 is a diagram for understanding how cost volume may be determined, in accordance with some embodiments of the present technology; FIG. 13 illustrates an example cost curve, in accordance with some embodiments of the present technology; FIG. 7 illustrates a process performed by a confidence processing procedure, in accordance with some embodiments of the present technology; FIG. 6 shows an example of a video stream and superframes of the video stream (part of FIG. 6 is in color), in accordance with some embodiments of the present technology; FIG. 4 illustrates examples of different types of superframes that may be included within a video stream, in accordance with some embodiments of the present technology; FIG. 4 shows an example of different types of superframes that may be included within a video stream, in accordance with some embodiments of the present technology; FIG. 4 shows an example of different types of superframes that may be included within a video stream, in accordance with some embodiments of the present technology; FIG. 4 shows an example of different types of superframes that may be included within a video stream, in accordance with some embodiments of the present technology; FIG. 4 shows an example of different types of superframes that may be included within a video stream, in accordance with some embodiments of the present technology; FIG. 4 shows an example of different types of superframes that may be included within a video stream, in accordance with some embodiments of the present technology; FIG. 14 schematically depicts a beam region comprising illuminating a scene captured by a sensor, in accordance with some embodiments of the present technology; 4 is a graph depicting the relationship between cross-sectional area and extent for an object within a captured image of a scene, in accordance with some embodiments of the present technology; 9A-9B show examples of disparity maps and confidence maps (some of which are in color in FIG. 9A), in accordance with some embodiments of the present technology; FIG. 4 shows examples of disparity maps and confidence maps, in accordance with some embodiments of the present technology;

To obtain information about the vehicle's surroundings while the vehicle is in motion to enable the vehicle's electronic control system to make decisions about how to operate the vehicle; and/ or different types of sensor systems (e.g., lidar systems, radar systems, monovision camera systems, stereovision cameras, etc.) to provide the driver with useful information to assist him in operating the vehicle. Sensor information from systems, temperature measurement systems (eg, thermocouples), acoustic systems (eg, ultrasonic transducer systems, audio microphone systems, etc.) may be used by the vehicle. For example, sensor information may be used to adjust the speed of the vehicle (e.g., accelerate or decelerate), deploy safety measures (e.g., turn on warning flashing lights, windshield wipers, etc.), or detect objects in the vehicle's path. It can be used by the control system to steer away from, etc. In another example, sensor information may be used by the control system to warn the driver of certain objects in the vehicle's path. In some embodiments of the present technology, the control system may be a centralized computer system consisting of an ECU configured to control various aspects of vehicle operation. Each of the ECUs is configured to receive data from one or more sensors and process the data to output one or more control signals used to operate parts of the vehicle. It may consist of software and/or hardware implemented. As mentioned above, in a moving vehicle there may be over 100 ECUs in operation. Some ECUs may operate independently of other ECUs, and some ECUs may operate interdependently with one or more other ECUs. In some other embodiments of the present technology, the vehicle's electronic control system may be decentralized. For example, the battery management ECU may operate as a separate system, independent from, for example, the speed control ECU. Each ECU may receive sensor information from one type of sensor, or multiple types of sensors. For example, the cabin temperature ECU may receive sensor information from one or more thermometers positioned in different areas of the vehicle cabin and control heaters and/or air conditioners to determine the cabin temperature. can be used to maintain the temperature set by the vehicle occupants. In another example, the steering ECU includes one or more sets of cameras for stereo imaging, one or more radar systems, one or more lidar systems, one or more tire pressure gauges, one Sensor information may be received from various combinations of one or more microphones, one or more cameras for 2D imaging, and one or more navigation systems (e.g., GPS systems) to navigate the vehicle to its destination. Sensor information can be used to determine the best course of action for safe navigation.

The inventors recognize that in order to optimize the use of data from different types of sensors, it is important to know the confidence or certainty level of the measurements from each of the sensors; recognized. According to some embodiments of the present technology, confidence information is used to determine which sensor or combination of sensors should be used for data to determine distance to an object. can be used. Conventional sensors typically report measurements or estimated measurements without reporting error estimates (e.g., error bars) that indicate a level of confidence about the measurements, which making accurate fusion difficult. However, as mentioned above, functionally safe systems, which are of particular importance in automotive applications, rely on sensor measurements to make decisions that can affect the safety of human life. Accordingly, there is an increased interest in providing accurate sensor data to a vehicle's electronic control system, which entails providing the control system with a level of confidence about the sensor data. obtain. Equipped with confidence data, the control system can either trust the sensor data sufficiently to be used to control the vehicle, or the sensor data is not reliable enough and should not be used. You may be able to make better decisions about whether As will be appreciated, sensor fusion may be performed by a central control system to combine data from different types of sensors, but in some embodiments of the present technology sensor fusion is performed by the vehicle's ECU. It may be performed by one or more or by an auxiliary system working in conjunction with the ECU and/or central control system.

We recognize that the confidence level of an estimate can be used in Bayesian inference to update the probability about a hypothesis or estimate as more evidence or information becomes available; recognized. Sensor data from various sensors of the same type and/or different types deployed on the vehicle may be used to corroborate sensor data from a particular sensor deployed on the vehicle. The inventors have found that sensors reporting confidence ranges for depth estimates can lead to safer vehicles by enabling a vehicle's driver assistance systems to make decisions based on reliable data. Recognized and recognized what it could bring. It is especially important for autonomous vehicles, which may not be controlled by a human driver. For example, the vehicle is traveling at cruising speeds typical of highway driving and the sensor (e.g. camera) is subject to the sensor being within close range of the vehicle (e.g. due to debris on the sensor). Based on various factors factored into the analysis of the images captured by the sensors, the stereo vision system according to some embodiments of the present technology may detect an object incorrectly if it is partially obstructed. A confidence factor may be determined and output indicating a relatively low level of certainty for that sensor data corresponding to the object, and thus the sensor data may be ignored. This involves the vehicle being controlled to continue cruising speed on its path, or applying emergency braking to avoid potentially colliding with spurious (i.e., non-existent) objects. For example, instead of controlling the vehicle to increase the likelihood of a rear-end collision, such as suddenly applying the vehicle's brakes, it is controlled to slow down gradually to increase the observation time. can make it possible. In some embodiments, the confidence associated with the sensor data may be used, or sensor data from another sensor with a relatively high confidence should be used instead. can be used to determine whether For example, depth information from sensor data indicates an object in the vehicle's path, but if the depth information is associated with a low confidence value, likely objects are observed over more time. The vehicle may be controlled to slow its travel speed to allow the vehicle to be detected and/or to allow possible objects to be cross-checked by other sensor systems. As discussed below, various embodiments of the present technology employ different types of sensor systems (e.g., radar, lidar, acoustic, etc.) to corroborate or supplement data acquired by stereo vision techniques. based system) can be used.

The inventors have determined that a driver assistance system can determine if a particular sensor is partially or totally malfunctioning (e.g. partially or totally covered by debris). By doing so, the confidence level or degree of certainty of the depth estimate from each sensor can be used advantageously to increase passenger safety. In some embodiments, information derived from confidence levels may be used to warn the driver of anomalies in sensors. In some embodiments, a stereo vision system, or a device operating in conjunction with a stereo vision system, may track occurrences of low confidence (e.g., values below a threshold confidence level) across multiple images. , the low confidence frequency, for example, may be used to determine whether the sensor may be functioning abnormally.

According to some embodiments of the present technology, a stereo vision system may be configured to determine and output a general belief about an image. The terms "belief," "belief range," and "belief level" may be used interchangeably herein. The images can be of scenes captured by sensors on moving vehicles (e.g. cars, trucks, planes, etc.) or fixed structures (e.g. streetlights, control towers, residences, office buildings, etc.). It can be of a scene captured by an on-board sensor. The stereo vision system may be a stand-alone system deployed on the vehicle or may be integrated within the vehicle's control system. In some embodiments, a stereo vision system may determine and output a confidence measure for each region of multiple regions of an image. For example, an image may be divided into quadrants (eg, upper left, upper right, lower left, lower right) and the stereo vision system may determine and output a confidence score for each quadrant. In some embodiments, a stereo vision system may determine and output a confidence score for each pixel of an image.

According to some embodiments of the present technology, a stereo vision system may be configured to output a depth map corresponding to captured images of a scene. The image can be a captured digital image or can be digitized from an analog image. The depth map may be a map of depth values or distances from the stereo vision system's sensors to objects in the scene. A depth map may consist of pixels that correspond to pixels of the image such that each pixel of the depth map (and each pixel of the image) may have an associated depth value. A stereo vision system may also be configured to output confidence data along with the depth map. In some embodiments, the confidence data may be a confidence map that indicates the certainty or confidence of the depth map. In some embodiments, the belief map corresponds to pixels of the depth map (and image) such that each pixel of the belief map (and each pixel of the image) may have an associated belief. can consist of pixels. In some embodiments, the confidence map is displayed as error bars, as standard deviation values, in buckets (e.g., high confidence, medium confidence, low confidence), or as quality levels for each estimated confidence. Any other metric that can indicate the confidence estimate can be represented.

As mentioned above, since the depth of a pixel is inversely proportional to the disparity of the pixel, a depth estimate for each pixel of the image can be calculated from the disparity map. As such, the terms "depth map" and "disparity map" are used because they provide nearly identical information about the captured scene in an image and are related by simple algebraic transformations known in the art. As such, they may be used interchangeably herein.

According to some embodiments of the present technology, autonomous vehicles and/or Advanced Driver Assistance Systems (ADAS) may instruct drivers to avoid accidents and/or when there is unreliable data. Depth maps and confidence values associated with depth maps may be advantageously used to warn. In some embodiments, a confidence value may be provided for each pixel of the depth map so that when there are some pixels with low confidence values, frames of the captured video sequence of the scene Sometimes it is not necessary to discard the whole thing. Alternatively, pixels with low confidence values can be discarded and the remaining pixels with sufficiently high confidence values can be used for depth calculation. This selectivity is attributed to human vision, which can make obstructing objects such as vehicle A-pillars, vehicle windshield wipers, etc. naturally invisible to the driver when processing the scene in the field of view while driving. very similar. That is, the driver automatically ignores obstructing objects while evaluating the scene within the field of view. In some embodiments, the confidence map may increase sensor availability, as frames of the video sequence may be used even if some of the frames are ignored, which may increase the vehicle's Allows the sensor to operate at higher duty cycles in a larger range of environmental conditions. For example, even when part of the sensor's lens is dirty, or when a windshield wiper may partially obstruct the sensor's field of view, or even when a section of the image captured by the sensor is exposed. even when there are too many, or when a section of the image captured by the sensor has low light levels, or when some other pixels of the image may have data that should be ignored. The sensor may operate in any situation where some pixels of the image captured by the sensor may have useful data. By providing a confidence map consisting of per-pixel confidence values, the vehicle's electronic control system is enabled to pay attention to areas of the depth map that are valid and have relatively high confidence values. can be

According to some embodiments of the present technology, autonomous vehicles may fuse information from different sensors, such as cameras, lidar, radar, and/or ultrasonic sensors, to increase reliability and safety. . Such sensors may report or provide information about the distance to an object, but it may be unclear which sensor to trust when different distances are reported for different sensors. In some embodiments, the sensor fusion algorithm may combine data from different sensors, providing more information than is possible when unfused information from one or another sensor is used individually. It can output fused information with a small uncertainty. We enhance the sensor fusion algorithm to increase the certainty of the fused information by providing the algorithm with certainty parameters (e.g. variance) for each of the different sensors. Recognized and recognized that it can be done. In some embodiments, confidence maps may be determined for different sensors on a pixel-by-pixel basis, so sensor fusion may be enabled for two or more sensors with different but overlapping fields of view. In some embodiments, the radar range estimate for the object may be compared to the stereo vision range estimate for the object when there are overlapping fields of view encompassing the object. For example, sensors on a car driving during a sunny day may have very high confidence values for stereovision range determinations based on images captured by cameras on the car; Stereovision range determinations can be trusted by the vehicle's electronic control system for objects at ranges or distances greater than 300 meters where other sensors on the vehicle cannot be expected to return high confidence values. . If the weather deteriorates (e.g., heavy fog, torrential rain, etc.), visibility may be lower, and as a result light attenuation worsens the stereovision range estimate and lowers the associated confidence value. can let The vehicle's control system may then switch to obtaining range estimates from radar data instead of data from the camera. Similarly, when a vehicle is traveling at night or in low ambient light levels without fog or other precipitation, a typical lidar system has its own active lighting source, so the vehicle's control The system may switch to obtain range estimates from lidar data instead of from radar data or data from cameras. In some embodiments, instead of the vehicle's control system determining when to switch between stereo vision range estimates, lidar range estimates, radar range estimates, or acoustic range estimates, The switching may be performed by the car's stereo vision system, which may be the car's primary sensor system. As will be appreciated, the examples above relate to distance estimates for cars, but the technology is not limited to cars, but other vehicles (e.g., trucks and other road vehicles, trains and other railroad vehicles). vehicles, ships and other marine vehicles, as well as airplanes and other air vehicles, etc.).

FIG. 1 illustrates a block diagram of a stereo vision system 1 configured to provide values for stereo matching confidence or certainty, in accordance with some embodiments of the present technology. Stereo vision system 1 may consist of a computer processor 10 coupled to a plurality of sensor systems and configured to receive measurements or sensor data from each of the sensor systems. In some embodiments, the sensor system coupled to processor 10 may comprise an imaging system comprising sensor 100, which may be a camera, lidar sensor, radar sensor, and/or ultrasonic sensor. In some embodiments, the stereo vision system 1 moves autonomously (i.e., without human control) and/or semi-autonomously (i.e., with human control or limited human control). can be mounted on vehicles capable of For example, vehicles can be cars, trucks, robots, marine vessels, flying vehicles, and the like.

According to some embodiments of the present technology, the sensor 100 consists of two stereo cameras 100 configured to capture images of the vehicle's environment at the same time, i.e. at the same or approximately the same instant in time. obtain. To simplify the notation, the cameras 100 may be arranged so that they are either perpendicular to each other (e.g., above and below), or oblique to each other, or (e.g., one camera at the front of the vehicle and the other Although the cameras may be offset and positioned within different range bins (such as at the rear of the vehicle), they may be referred to herein as "left" and "right" cameras. Camera 100 may be, for example, a color CMOS (complementary metal oxide semiconductor) camera, a grayscale CMOS camera, a CCD (charge coupled device) camera, a SWIR (short wavelength infrared) camera, a LWIR (long wavelength infrared) camera, or a focal plane It can be an array sensor.

According to some embodiments of the present technology, sensors S1, S2, S3, S4, S5, S6, S7, S8, S9 are multiple sensors on vehicle 20, as schematically depicted in FIG. can be located at different locations in the In some embodiments, some of the sensors S1, S2, . In some embodiments, some of the sensors S1, S2, . , can be larger than the width of the passenger compartment, which allows objects at relatively greater distances to be detected compared to a baseline limited to the width of the passenger compartment. can be In some embodiments, sensors S1, S2, . . . , S9 receive data from sensors S1, S2, . It may be coupled wirelessly or by wire to one or more ECUs 22, 24 configured to supply power. At least some of the ECUs 22 , 24 may be part of the computer processor 10 in some embodiments. In some other embodiments, at least some of the ECUs 22, 24 may be located external to the computer processor 10 and configured to transmit signals to the computer processor 10, either wirelessly or by wire. can be In some embodiments, a pair of cameras in the headlights (e.g. S1 and S9), or a pair of cameras in the side view mirrors (e.g. S2 and S8), or a pair of cameras in the windshield (e.g. , S3 and S7, or alternatively S3 and S5), or a pair of cameras on the roof (eg S4 and S6) are required.

According to some embodiments of the present technology, the stereo vision system 1 may be coupled to the vehicle's main system controller 30, as shown schematically in FIG. In some embodiments, main system controller 30 may be a vehicle control system that may be configured to control all automated aspects of vehicle operation. In some embodiments, the stereo vision system 1 may be configured to be commanded by a main system controller 30, with signals to and from the main system controller 30 via command and control lines 32. signals can be communicated. As will be appreciated, command and control line 32 may be a wired communication mechanism (eg, data bus, communication line) or may be a wireless communication mechanism using communication techniques known in the art. In some embodiments, main system controller 30 orchestrates high-level functions (e.g., automatic emergency braking, route selection, etc.) and uses various subsystems or ECUs (e.g., It may consist of a computer arranged to communicate with the stereo vision system 1). In some embodiments, a common communication protocol may be used for communication over command and control lines 32 (e.g., Ethernet, CAN (controller area network), I2C (inter-integrated circuit ), LIN (Local Interconnection Network), etc.). Although stereo vision system 1 is shown separate from main system controller 30 in FIG. 3, stereo vision system 1 may be part of main system controller 30 in some embodiments, and may In some embodiments, it may be physically located within the housing of main system controller 30 .

Returning to FIG. 1, the camera 100 may be coupled wirelessly or with a wired connection to the image acquisition module 102, according to some embodiments of the present technology. In some embodiments, the image data of the scene captured by the camera 100 is transmitted through known communication interfaces (e.g., USB (Universal Serial Bus) connector, Ethernet connector, MIPI (Mobile Industry Processor Interface)). CSI (Camera Serial Interface) connector, GMSL (Gigabit Multimedia Serial Link) connector, and Flat Panel Display Link (FPD Link) connector, etc.) to image acquisition module 102 . In some embodiments, the camera 100 transmits image data to the image acquisition module 102 in real-time or near real-time, either directly or through a buffer memory device (e.g., RAM), which may be incorporated into the camera 100. may be configured to transmit via In some embodiments, the camera 100 may be associated with a data storage memory device 140 accessible by the image acquisition module 102 and other portions of the computer processor 10, the camera 100 storing data in the data storage device 140. In return, it may be configured to transmit image data. In some embodiments, camera 100 may be a video camera configured to capture a stream of video data of a scene. A stream of video data may consist of a left stream and a right stream, each stream consisting of a sequence of frames. Accordingly, the term "image data" as used herein may, in some embodiments, refer to a frame of video data.

According to some embodiments of the present technology, image acquisition module 102 may be configured to digitize image data from camera 100 to generate raw digital image data or "raw image data." In some embodiments, image acquisition module 102 may provide raw image data to image preprocessing module 104 . In some embodiments, image acquisition module 102 may provide original image data to memory 140, which may store the original image data for future processing.

According to some embodiments of the present technology, image preprocessing module 104 may be configured to correct the original image data to generate corrected left and right images. For example, the image preprocessing module 104 includes demosaicing, autofocus, autoexposure, and autowhite balance correction, vignetting, noise reduction, bad pixel filtering, HDR (high dynamic range) lookup table color processing, and image compression among: any one or any combination of The corrected left and right images may be forwarded to image rectification module 106 .

According to some embodiments of the present technology, image rectification module 106 distorts corresponding rows of pixels of the corrected left and right images so that they lie on the same epipolar plane. , to parallelize the corrected left and right images. After warping, image rectification module 106 may output left and right rectified 2D images 114, which may be color images or grayscale images. As will be appreciated, image rectification is a known technique used to simplify matching of common objects in corrected left and right images. Image rectification module 106 may provide left and right rectified 2D images 114 to stereo matching module 108 , belief processing module 110 , and encoder module 112 .

According to some embodiments of the present technology, stereo matching module 108 may be configured to calculate disparity between each matching pixel pair in rectified 2D image 114 . The processing performed by stereo matching module 108 may, in some embodiments, consist of four procedures: a cost calculation procedure, a cost aggregation procedure, a disparity calculation procedure, and a disparity refinement procedure, each of which is , discussed below.

According to some embodiments of the present technology, the cost calculation procedure is called a "disparity space image" by calculating the matching cost for each pixel at each disparity value of the set of possible disparity values. It may consist of constructing a three-dimensional (3D) cost volume map 118 . As discussed below, the cost volume (or more correctly, the matching cost volume) can be determined as the product of W×H×D, where W and H are the width and height dimensions of each image. , D is the number of disparity hypotheses or possible disparities. The matching cost for a particular pixel and a particular disparity value represents how unlikely that particular pixel has that particular disparity value. Typically, for a given pixel, the disparity value with the lowest matching cost is chosen to be used in the disparity map, as discussed below. This approach for choosing a disparity value for a given pixel is the so-called winner-take-all (WTA) approach, where the winner is the disparity value with the lowest matching cost, i.e. the best among all disparity hypotheses belongs to. Matching costs are determined using known techniques such as, for example, absolute difference techniques, mutual information (MI) techniques (e.g., hierarchical MI (HMI) techniques), normalized cross-correlation (NCC) techniques, Hamming distance techniques, etc. , can be calculated. In some embodiments, the NCC technique is H.264. H. Hirschmuller et al., "Evaluation of Cost Functions for Stereo Matching" (2007 IEEE Conference on Computer Vision and Pattern Recognition) to consider It can be used to match the costs for two sub-windows (one sub-window in each of the left and right rectified 2D images 114) around the pixels in. In some embodiments, the Hamming distance technique is the S.M. Under consideration, as described in S. Sarika et al., "Census Filtering Based Stereomatching Under Varying Radiometric Conditions" (2015 Procedia Computer Science). Neighboring pixels surrounding a pixel of can be used in a census transform that is mapped to a bitstring depending on whether the intensity values of these pixels are greater or less than that of the pixel under consideration.

FIG. 4A provides a diagram for understanding how the cost volume map 118 may be determined, according to some embodiments of the present technology. In cost volume analysis, cost volume map 118 is constructed by collecting cost curves (e.g., cost curve 400) for every pixel in the left rectified image (or right rectified image) of rectified 2D image 114. can be The cost volume map 118 has H×W×D elements, where H is the height of the image (e.g., H can be the number of pixels in the row direction, or less precision matching is allowed). W is the width of the image (e.g., W can be the number of pixels in the column direction, or the column can be the number of pixel groups in a direction) and D is the number of disparities searched. Simplistically, therefore, the cost volume map 118 can be thought of as comprising a cost curve for each pixel coordinate (row and column). In some embodiments, cost volume map 118 is refined (eg, by filtering) to obtain more reliable matching costs using techniques known in the art. obtain. For example, C.I. C. Rhemann et al., "Fast cost-volume filtering for visual correspondence and beyond," 2011 Conference on Computer Vision and Pattern Recognition on) There is technique.

FIG. 4B shows an example cost curve 400 that may be used to describe cost matching procedures, in accordance with some embodiments of the present technology. As will be appreciated, cost matching typically involves a pixel P having (row, column) coordinates of (H0, W0) in the left rectified image and a coordinate of (H0, W0+D0) in the right rectified image. can be performed between pixels with A cost curve 400, which shows the relationship between matching cost and disparity, may represent the particular pixel under consideration. (The term "matching cost" referred to is sometimes referred to herein as "matching cost" or "cost"). As shown in FIG. _4B , the cost curve 400 has the lowest global minimum cost of c _d1 when the disparity is d1, and the second lowest global minimum cost of c _d2 when the disparity is _d2 . and has the second lowest local minimum cost of c _d2m when the disparity is d _2m . As will be appreciated, the term "global" may be used for values evaluated over all points in cost curve 400, and the term "local" may be used for values evaluated over a portion of cost curve 400. can be used against

According to some embodiments of the present technology, the value for the matching cost spans 5 pixels in the column direction and 5 pixels in the row direction around the pixel under consideration in the left and right rectified 2D images 114. , for subwindows (“5×5” subwindows), can be determined using the NCC technique (see above).

According to some embodiments of the present technology, the cost aggregation procedure may consist of aggregating matching costs over the region of support of each pixel using the results of the cost computation procedure. For "local" stereo matching techniques, the region of support can be understood to be the weighted sum of costs in a group of neighboring pixels around the pixel of interest. For "semi-global" and "global" stereo matching techniques, the area of support can be understood to be a function of cost for all pixels in the image.

According to some embodiments of the present technology, the disparity calculation procedure uses the results of the cost aggregation procedure and uses local or global optimization methods to calculate disparity for each pixel and refinement and generating a non-parallax map. Computational speed versus accuracy may determine the choice between local and global optimization approaches. For example, local methods may be used when speed is desired over accuracy, while global methods may be used when accuracy is desired over speed. In some embodiments, local optimization methods such as block matching may be used. In some embodiments, global optimization methods such as semi-global matching (SGM) may be used.

According to some embodiments of the present technology, the disparity refinement procedure consists of filtering the unrefined disparity map to produce a 2D disparity map 116 for the left and right rectified 2D images 114. obtain. The parallax refinement procedure is an optional procedure for correcting parallax values. Conventional refinement steps include left-right checking, hole filling, smoothing filters, and outlier detection and removal.

According to some embodiments of the present technology, stereo matching module 108 may employ different stereo matching techniques than those described above. For example, the stereo matching module is described in Mohd Saad Hamid et al., "Stereo matching algorithm based on deep learning: A survey" (2020 Journal of King Saud University). - Computer and Information Sciences) and/or one or more of the techniques described in H. H. Hirschmuller, "Stereo Processing by Semi-Global Matching and Mutual Information" (2008 IEEE Transactions on Pattern Analysis and Machine) e Intelligence)1 One or more techniques may be used.

According to some embodiments of the present technology, stereo matching module 108 applies 2D disparity 2D parallax to encoder module 112 and belief processing module 110 of processor 10 of stereo vision system 1, as shown in FIG. Output map 116 . In some embodiments, stereo matching module 108 also outputs 3D cost volume map 118 , discussed above in connection with deriving disparity map 116 , to belief processing module 110 .

According to some embodiments of the present technology, the belief processing module 110 processes the left and right 2D rectified images 114 from the image rectification module 106, the cost volume map 118 from the stereo matching module 108, and the stereo matching From module 108 , it receives as input a 2D disparity map 116 and is configured to determine from these inputs the estimated disparity accuracy for each pixel of the rectified image 114 .

Confidence processing module 110 may be configured to calculate a belief value for each frame of the video stream, according to some embodiments of the present technology. In some embodiments, belief processing module 110 may calculate a belief value for each pixel of each frame of the video stream. Accordingly, it should be understood that the term "image" herein can encompass frames of a video stream.

Returning to FIG. 1, belief processing module 110 may output belief information to encoder module 112, in accordance with some embodiments of the present technology. The belief information, in some embodiments, may be a belief map 120 for each frame of the video stream. Confidence map 120 may have the same dimensions as disparity map 116 output by stereo matching module 108 to encoder module 112 . In some embodiments, confidence map 120 may include confidence information for each pixel of the frame that indicates a value representing the confidence level of the depth estimate for the corresponding pixel in disparity map 116 . In some embodiments, the value representing the confidence level for a particular pixel can be the root mean square error of the disparity or depth determined for that pixel, or the disparity or depth determined for that pixel. Can be 95% confidence intervals, or can be any measure of variability (eg, standard deviation, standard error of the mean, confidence intervals, data ranges, percentiles, etc.). M. M. Poggi et al., "On the confidence of stereo matching in a deep-learning era: a quantitative evaluation" (2021 IEEE Transactions on P Attern Analysis & Machine Intelligence), and X. X. Hu et al., "A Quantitative Evaluation of Confidence Measures for Stereo Vision" (2012 IEEE Transactions on Pattern Analysis and Machine Int) elligence) Other confidence metrics may be used, such as .

FIG. 5 is a diagram illustrating processing performed by belief processing module 110, in accordance with some embodiments of the present technology. In some embodiments, at block 500 the first confidence measure process computes a first confidence map 508 based on the cost volumes 118 received from the stereo matching module 108 . In some embodiments, a cost curve (eg, cost curve 400 ) winner margin for each pixel may be calculated and used as the belief value for that pixel on the first belief map 508 . The winner margin (WMN) is the difference between the second lowest local minimum cost c _d2m and the lowest global minimum cost c _d1 normalized over the entire cost curve, or

where p is the coordinate of the pixel under consideration, c is the match cost, and D is the disparity set to be searched. In some embodiments, the first confidence map 308 is based on the M.O. M. Poggi et al., "On the confidence of stereo matching in a deep-learning era: a quantitative evaluation" (2021 IEEE Transactions on P Attern Analysis & Machine Intelligence) can also be derived from other measures.

According to some embodiments of the present technology, at block 502, the second confidence measure process of the confidence processing module 110, based on the rectified 2D image 114 received from the image rectification module 106, 2 confidence map 510 is computed. In some embodiments, the texture is applied to the rectified 2D image 114, for example, by calculating a derivative (eg, x-derivative) of the image data to determine changes in the rectified 2D image 114. The texture values measured above may be used to derive a second belief map 510 . In some embodiments, image texture may be measured using the x-Sobel operator. As will be appreciated, due to the difficulty in matching untextured features, or features that are the same as multiple other features in the image, on images with little texture, or texture consisting of repeating structures , is known in stereo vision technology to be difficult to perform the 3D reconstruction process. With this difficulty in mind, the second confidence map 310 uses the x-Sobel operator:

can be derived from the grayscale image, convolved with Such Sobel convolution or filtering, which is known to enhance edges in the image, may yield high values when there are edges in the rectified 2D image 114, thus making it easier to perform stereo matching. It can result in sharper, better defined features that can be easier. In some embodiments, in addition to or instead of the Sobel convolution, the second confidence map 510 evaluates each pixel based on the intensity of the pixel, below a minimum threshold or maximum It can be derived by penalizing pixels with intensities above a threshold. For example, a zero or low confidence value may be determined for each low-intensity or under-illuminated pixel having a signal intensity level below the minimum threshold, and similarly for each over-illuminated pixel having a signal intensity level above the maximum value. Or a zero or low confidence value may be determined for saturated pixels.

At block 504, the third confidence measure process of the confidence processing module 110, based on the disparity map 116 received from the stereo matching module 108, determines the third confidence measure 116, according to some embodiments of the present technology. Calculate the degree map 512 . In some embodiments, the confidence value for the third confidence map 312 may be calculated from the disparity variance determined for each pixel of the rectified 2D image 114 . In some embodiments, the variance may be a statistical variance for disparities determined for neighboring pixels (eg, pixels surrounding the pixel under consideration). For example, a pixel with relatively higher variance may indicate that the pixel is part of a region of disparity map 116 that has noisier (eg, more sparse) data; may be assigned lower confidence values. As will be appreciated, noisiness can indicate blurring, where the sensor (e.g., camera 100) used to capture the image, or the area of the sensor, is dirty or partially obscured. It can be shown that there is a possibility that Known techniques (eg, techniques based on Laplacian and/or Sobel filters) may be used to detect variance in disparity map 116 . In some embodiments, a variance below the threshold variance may indicate that the pixel under consideration is blurred or within a blurred region, and thus a low confidence value in the third confidence map 512. can be assigned a degree value.

As will be appreciated, the first belief map 508, the second belief map 510, and the third belief map 512 may, according to some embodiments of the present technology, each pixel of an image, or Although described as consisting of a confidence value for each pixel of each frame of a video stream, in some other embodiments, a first confidence map 508, a second confidence map 510, and a third One or more of the confidence maps 512 of may consist of confidence values that are representative of two or more pixels of the image or frame. For example, each frame of the video stream may consist of pixels categorized into n groups, and the belief processing module 110 provides n belief values (i.e., n groups of n groups) for each frame of the video stream. confidence value for each).

At block 506, the aggregator process of the belief processing module 110 processes the first belief map 508, the second belief map 510, and the third belief map, according to some embodiments of the present technology. 512 is used to combine the estimated confidence values in the first confidence map 508 , the second confidence map 510 , and the third confidence map 512 to generate the confidence map 120 . Thus, confidence map 120 may consist of enhanced confidence values that are the best estimates of certainty based on multiple confidence measures. In some embodiments, the aggregator process may consist of computing a sum of beliefs for each pixel, and the sum may be used as the enhanced belief value for the pixel in the belief map 120. . In some embodiments, the aggregator process weights the beliefs in the first belief map 508, the second belief map 510, and the third belief map 512, and then for each pixel: Calculating a weighted sum of the beliefs may comprise calculating the weighted sum and using the weighted sum as the enhanced belief value for the pixel in the belief map 120 . In some embodiments, the aggregator process converts, for each pixel, the belief for the pixel in the first belief map 508, the second belief map 510, and the third belief map 512 into a belief. It may comprise using three input values to a lookup table that may output a single value to be used as the enhanced confidence value for the pixel in degree map 120 .

Returning to FIG. 1, belief processing module 110 may output belief map 120 to encoder module 112 of computer processor 10, in accordance with some embodiments of the present technology. Encoder module 112 may be configured to receive rectified 2D image 114, disparity map 116, and confidence map 120, encode the received information, and output a video stream 122 consisting of the encoded information. . In some embodiments, the video stream 122 may be provided to the vehicle's main system controller 30 .

According to some embodiments of the present technology, video stream 122 may be encoded such that each 24-bit color value is the range or distance to the scene captured by camera 100 from 24-bit color depth video. It is possible. In some embodiments, distances from 0 to approximately 16800 meters can be represented using 24 bits, with each color representing a different 1 millimeter portion of the 16800 meter range. In some embodiments, the video stream 122 may consist of 2D frames of pixels, each frame pixel corresponding to a pixel in the reconciled 2D image 114 . Each pixel of each frame of video stream 122 may be encoded with an enhanced confidence value. For example, each enhanced confidence value may be an 8-bit unsigned value from 0 to 255, with relatively higher values indicating higher levels of confidence. In some embodiments, using an 8-bit representation for the enhanced belief values of belief map 120 may allow belief map 120 to be displayed as a grayscale image. In some embodiments, video stream 122 may consist of a 24-bit color video stream representing depth or distance, and may also consist of an 8-bit monochrome video stream representing confidence. Outputting the video stream 122 so that the depth data can be separated from the confidence data, in some embodiments, for example, in detecting objects and determining the vehicle's distance to the object In practice, it may facilitate sensor fusion, in which data from different types of sensors (eg, lidar, radar, ultrasound, cameras, etc.) may be combined to provide enhanced reliability. For example, in foggy environments where objects in a scene may not be clearly visible in images captured by a typical camera, confidence values for pixels in an image are generally can be low over In such cases, the vehicle's control system (eg, main system controller 30) may determine that the image is not reliable enough to be used.

According to some embodiments of the present technology, the computer processor 10 of the stereo vision system 1 may consist of a lidar belief processing module 124, the sensor 100 illuminating the scene with laser light and receiving light from the scene. lidar sensor configured to generate lidar image data (e.g., a video stream) and lidar belief data from the reflected light and output a lidar belief map 126 to the encoder module 112. . Optionally, lidar image data may be output along with the lidar belief map 126 . In some embodiments, computer processor 10 may comprise radar confidence processing module 128, sensor 100 illuminates the scene with waves of known wavelength (eg, 76.5 GHz), generated radar image data (e.g., a video stream) and radar confidence data from reflected waves having wavelengths that are reflected from the scene, and outputs a radar confidence map 130 to the encoder module 112. It may consist of a radar sensor configured to: Optionally, radar image data may be output along with radar confidence map 130 . In some embodiments, computer processor 10 may comprise acoustic confidence processing module 132, and sensor 100 scans a scene using acoustic waves (eg, ultrasound) of known wavelength (eg, 20 kHz). Generate acoustic image data (e.g., a video stream) and acoustic confidence data from reflected waves of illuminated and known wavelengths reflected from the scene, and provide the acoustic confidence data to encoder module 112 . It may consist of a transducer configured to output map 134 . Optionally, the acoustic image data may be output together with the acoustic confidence map 134 . Techniques for determining confidence values for lidar image data, radar image data, and acoustic image data are discussed below.

According to some embodiments of the present technology, lidar belief processing module 124 receives disparity map 116 and cost volume map 118 from stereo matching module 108 and belief map 120 from belief processing module 110. can be configured as Although not shown in FIG. 1, the lidar belief processing module 124 also receives one or both of the left and right parallelized 2D images 114 from the image parallelization module 106 in some embodiments. can be configured. The lidar belief processing module 124 performs the comparison and identifies areas where the lidar belief data has values that are higher confidence values than corresponding areas of the belief map 120 . Map 120 and some or all of the other received information (i.e., one or both of disparity map 116, and/or cost volume map 118, and/or rectified 2D image 114) may be used. . In some embodiments, lidar belief processing module 124 may provide comparison information to encoder module 112 along with lidar belief map 126 . Similarly, in some embodiments, disparity map 116 , cost volume map 118 , and confidence map 120 provide comparative information to encoder module 112 along with radar confidence map 130 . It may be provided to a radar confidence processing module 128, which may be configured. Similarly, in some embodiments, disparity map 116, cost volume map 118, and confidence map 120 provide comparative information to encoder module 112 along with acoustic confidence map 134. It may be provided to an acoustic belief processing module 132, which may be configured.

According to some embodiments of the present technology, encoder module 112 encodes video stream 122 to include depth information based solely on sensor data acquired by camera 100 or another sensor system ( For example, a lidar sensor, a radar sensor, an acoustic sensor, another pair of cameras, etc.), or encode the video stream 122 to include depth information based solely on sensors derived from multiple different sensor systems on the vehicle. To combine data, confidence map 120 and one or more of lidar confidence map 126, radar confidence map 130, and acoustic confidence map 134 are used to determine whether to perform sensor fusion. can be used. In some embodiments, encoder module 112 is configured such that the image data, depth data, and/or confidence data in video stream 122 correspond to the highest confidence value determined from different sensor systems. It may be configured to encode the video stream 122 . In some embodiments, the frames of viewstream 122 are followed by one or more frames based on rectified 2D image 114 and one or more frames based on lidar image data. It may consist of a belief map 120 and a lidar belief map 126 followed by one or more frames based on the acoustic image data and the acoustic belief map 134 . In some embodiments, one or more frames of video stream 122 may each consist of a combination of data derived from different sensor systems. For example, each of the frames of the video stream 122 may consist of four quadrants (eg, upper left, upper right, lower left, lower right), and each quadrant has the highest overall confidence (eg, the highest over pixels in the quadrant). consists of sensor data with an average confidence level). Debris on one or both of the cameras 100 therefore causes the upper left quadrant of the confidence map 120 to have the lowest overall confidence compared to the overall confidence of the vehicle's other sensor systems. , the upper left quadrant of the corresponding frame of the video stream 122 has the highest overall confidence value if the other three quadrants of the confidence map 120 have the highest overall confidence value. can be replaced using the data of As will be appreciated, instead of optimizing the frames of the video stream 122 on a frame-by-frame or quadrant-by-quadrant basis, the frames provide highly reliable estimates of scenes captured by sensors of different sensor systems. can be optimized in other ways. In some embodiments, when the vehicle is being operated at night or in a dark environment, the encoder module 112 converts the video stream 122 using, for example, lidar-based data instead of camera-based data. can be encoded.

FIG. 6 depicts an example video stream 122, in accordance with some embodiments of the present technology. Video stream 122 may consist of a sequence of frames, of which only three are shown: frame N 600, frame N+1 602, and frame N+2 604. In some embodiments, each frame includes a left (or right) rectified 2D image 114 (in color or grayscale), a disparity map 116 (in color or grayscale), and a (color or grayscale ), and the belief map 120 . In some embodiments, the superframe 606 includes a velocity magnitude map, a horizontal velocity map (describing motion along the x-axis), a vertical velocity map (describing motion along the y-axis), and a One or more velocity maps may also be included, such as any one or any combination of a velocity map and a radial velocity map (which depicts motion along the z-axis). The velocity map tracks matched points (or pixels) with dense optical flow over two or more frames, and then by determining the difference in disparity map values of the matched points. can be calculated. In this scheme, the superframes 606 may be transmitted like any other stream of video frames, thus advantageously using existing video technologies for encoding, decoding, transmission, etc. . As will be appreciated, the disparity map 116 may instead be a depth map, since both types of maps relate to the distance from the sensor (e.g. camera 100) to the scene captured by the sensor. , since both types of maps are typically inversely proportional to each other. In some embodiments, the "alpha channel" or transparency of each pixel of the rectified 2D image 114 of each superframe may be used to store information. For example, the confidence or certainty value of the depth estimate for a pixel may be stored in the pixel's alpha channel.

7A-7F illustrate six types of superframes that may be included within video stream 122, in accordance with some embodiments of the present technology. As will be appreciated, different types of superframes may consist of different types of information and therefore have different sizes. FIG. 7A shows a vertically concatenated 2D RGB (red-green-blue) color image 700 (e.g., left (or right) rectified 2D image 114), a depth map or disparity map 700 (e.g., disparity map 116), and A superframe 750 is shown, which consists of a belief map 704 (eg, belief map 120). In some embodiments, each pixel of superframe 750 may use 3 bytes to describe the pixel's disparity or range, and 3 bytes are used to describe the pixel's confidence level. obtain. In some embodiments, each pixel of superframe 750 may be specified using 9 bytes. FIG. 7B shows a superframe 752 consisting of the 2D color image 706 and the concatenation of the disparity map and the confidence map. In some embodiments, each pixel of superframe 752 may use 2 bytes for the disparity map and 1 byte for the confidence map, for a total of 3 bytes when concatenated. In some embodiments, each pixel of superframe 752 may be specified using 6 bytes. FIG. 7C shows a superframe 754 consisting of a 2D color image formed from channels including a channel for the red image 710, a channel for the green image 712, and a channel for the blue image 714. . Superframe 754 also consists of a 16-bit disparity map, including upper byte 716 and lower byte 718, and a confidence map. In some embodiments, each pixel of superframe 754 may be specified using 6 bytes. FIG. 7D shows a superframe 756 consisting of a color image formed from red image 722 , green image 724 , and blue image 726 , disparity map 728 and confidence map 730 . In some embodiments, each pixel of disparity map 728 may be specified using 5 bytes. FIG. 7E shows a superframe 758 consisting of a 2D grayscale image 732, a disparity map 734, and a confidence map 736. FIG. In some embodiments, each pixel of superframe 758 may be specified using 3 bytes. FIG. 7F shows a superframe 760 consisting of a 2D color image 738, a disparity map 740 and a confidence map 742. FIG. Each pixel of color image 738 may be specified using one byte. For example, color image 738 may be specified in a color palette by a Bayer filter pattern or lookup table, which may allow superframe 760 to be as compact as in an RGB color image. In some embodiments, each pixel of superframe 760 may be specified using 3 bytes.

In addition to providing stereo vision confidence map 120 based on sensor data from camera 100 to encoder module 112 as described above, lidar confidence map 126, radar confidence map 130, , and acoustic confidence map 134 may be provided to encoder module 112 . The lidar, radar, and acoustic sensors may have the same or similar field of view as camera 100, or may have overlapping fields of view, as described above. In some embodiments, one or more of lidar confidence map 126 , radar confidence map 130 , and acoustic confidence map 134 may be appended to video stream 122 . For example, the lidar belief map 126 (or another belief map) may be placed as separate bits in the belief maps of any of the superframes 750, 752, 754, 756, 758, 760 of FIGS. 7A-7F. It can be encoded or simply appended to the superframe, eg by vertical concatenation.

According to some embodiments of the present technology, if the lidar sensor is configured to provide a lidar return signal for each pixel of the image, the lidar belief map (eg, lidar belief map 126) is a 2D It may consist of a confidence value for each pixel of an image (eg, left (or right) rectified 2D image 114). In some embodiments, the lidar sensor may be configured to provide a lidar return signal, e.g., for every other pixel or every other pixel, or for a predetermined group of pixels, This can result in a relatively sparser, but still useful lidar belief map. If each pixel can be identified by its coordinates (i,j), the confidence value for pixel (i,j) can be denoted as m _ij . In some embodiments, the lidar confidence map is
If R≧R _min then m _ij =a ₀ R ⁻² (P+P ₀ ) ^−1/2
if R<R _min then m _ij =0
where R is the distance or extent to the object at pixel (i,j), as measured by the disparity map 116, and P is , is the background light power (from sunlight or other light source) at pixel (i,j) as measured by the 2D image 114, a ₀ is the normalization constant, and P ₀ is the fitting constant and R _min is the minimum distance of the lidar sensor. In some embodiments, the value of P may be, for example, the grayscale value of 2D image 114 at pixel (i,j) divided by the sensor gain and exposure values for camera 100 . As indicated above, the 2D image and disparity map used in generating the lidar belief map are the 2D image 114 captured by camera 100 and the disparity map determined based on stereo vision processing. 116. In some embodiments, the confidence value for pixel (i,j) may be set to zero when the object at pixel (i,j) is closer than the minimum range of the lidar sensor. The above formula for the confidence value m _ij is inversely proportional to the square of the distance or range, i.e. ^R2 , and also inversely proportional to the square root of the background light power, i.e. P, i.e.
SNR _lidar ∝R ⁻² P ^−1/2
It is derived from an estimate of the lidar signal-to-noise ratio (SNR) such that That is, the farther the object is from the lidar sensor, the lower the return signal, and therefore the accuracy of the lidar estimate decreases in proportion to the number of received photons corresponding to the return signal. Additionally, any background light (eg, sun, moon, man-made) can cause "shot noise" to be perceived by a lidar sensor, where noise energy is equal to the square root of the background light power. Thus, the lidar belief map 126 may be based on two physical quantities: the distance to the object in pixels and the background light level in pixels.

Radar and acoustic sensors, like lidar sensors, can also have an SNR that is proportional to R ⁻² P ^−1/2 . This property is a radar confidence map (e.g., radar confidence map 130), in a manner similar to that described above for computing lidar confidence maps, according to some embodiments of the present technology; and/or to compute an acoustic confidence map (eg, acoustic confidence map 134).

According to some embodiments of the present technology, different sensor systems on the vehicle may be arranged to sense objects at different distance ranges. For example, FIG. 8A schematically depicts a beam area 800 illuminated by a radar or acoustic sensor system. Beam region 800 may encompass a scene having multiple objects A, _B , _C , D at multiple ranges or ranges R _A , R B , R C , R _D . The strength of the return signal reflected from the object is proportional to the cross-sectional area of the object, which can be measured by the stereo vision system 1 (e.g. range is based on the parallax map 116 and the area of the target is left (or Right) given by the rectified 2D image 114). In some embodiments, return signals detected by the sensor system may be categorized into different bins based on signal strength. FIG. 8B is a graph depicting the relationship between cross-sectional area and extent for each of objects A, B, C, and D; The dashed line represents the noise floor 802 of the radar sensor system (or lidar sensor system). The noise floor 802 is a known quantity for radar and lidar technology and may correspond to sensor sensitivity, which is known to degrade as a function of distance squared. As mentioned above, the lidar SNR can be estimated as a value that is inversely proportional to the square of the distance or range. In some embodiments, the lidar (or radar) confidence for a pixel may be determined as a value proportional to the SNR for the pixel. For example, the ratio of the cross-sectional area of an object, which can span multiple pixels, to the range scale value of the noise floor 802 is a value proportional to the SNR and can be used as the confidence value for the pixel that the object appears. .

According to example embodiments of the present technology, the vehicle in which the stereo vision system 1 is installed may be a passenger car. The sensors 100 can be two cameras mounted on the upper left and upper right portions of the windshield of the vehicle (e.g., the locations of sensors S3, S7 in FIG. 2), such as the Superframe 750 discussed above. It may be configured to capture images that are processed and converted into a video stream 122, which may consist of superframes. The RGB image 700 of the superframe 750 can be part of the image preprocessing module 104, or an image captured before (or after) the image is subjected to rectification, before stereo matching is performed. can be analyzed by an object detection module (eg, a monocular object detector), which can be a separate module of computer processor 10, which processes the image data of . For example, the object detection module may be configured to use a trained convolutional neural network to determine bounding boxes of various shapes in a scene of captured images. The shapes were trained to perceive the neural network as typical objects often encountered by cars (e.g. other vehicles, pedestrians, cyclists, traffic signs, traffic lights, etc.). It may correspond to a category of objects. Disparity map 702 of superframe 750 may be averaged over the bounding box to determine the average distance to the detected object. Furthermore, if the ground or road surface is S. S. Kakegawa et al., "Road Surface Segmentation based on Vertically Local Disparity Histogram for Stereo Camera" (Int'l J. ITS Res. .vol. 16, pp. 90-97, 2018), as well as techniques using vertical disparity histograms, as well as H. H. Badino et al., "Free Space Computation Using Stochastic Occupancy Grids and Dynamic Programming," Workshop on Dynamical Vision, ICCV, 2007) can be calculated from the disparity map 702 using known techniques, such as techniques using probabilistic occupancy grids and dynamic programming. From the road surface, unclassified road hazards (eg, motorcycle helmets, bricks, wooden pallets from loose cargo, mufflers, treads separated from tires, etc.) can be found and marked. Small unclassified objects on a typical road surface often have well-defined edge and contour features, which can lead to high confidence values in the confidence map 704, thus reducing false positives and It is possible to prevent unnecessary automatic emergency braking from occurring. In this example embodiment, if there is debris (e.g., mud) on the windshield that blocks part of the lower left portion of the captured image of one or both of the cameras 100, the belief processing module 110 will: The stereo matching module 108 may detect that the lower left portion of the cost volume map 118 output by the stereo matching module 108 repeatedly reports a high cost, thus for the corresponding region of the confidence map 120 a low A confidence value may be provided. The vehicle's main system controller 30 may be configured to determine if the low confidence value is within an acceptable confidence threshold, and if not to invalidate data from the lower left region. can control the computer processor 10 at the same time. That is, in some embodiments, the confidence value for the lower left region may be below the confidence threshold, and thus the calculated depth value for the lower left region may be invalid, so computer processor 10 may: To avoid misreadings and, as a result, erroneous control of the car (e.g. to prevent the car from being controlled to perform evasive actions to avoid objects that are not actually in the scene). Additionally, data from the camera 100 associated with the lower left region can be used to control detection of objects within the scene of the captured image. Alternatively, as discussed above, data from other sensors on the vehicle may be used if the confidence value for the data meets the confidence threshold.

Returning to FIG. 3, the main system controller 30 comprises a plurality of ECUs 34-1, 34-2, 34-2, 34-1, 34-2, configured to control various aspects of the operation of the vehicle, in accordance with some embodiments of the present technology. 34-3, 34-4, . . . , 34-n. For example, the video stream 122 may be provided to the vehicle's display (not shown) and a red tail light ahead of the vehicle's path, which may indicate a traffic jam or other potential hazard. and to main system controller 30, which may use disparity information and confidence information from video stream 122 to determine with high certainty (eg, greater than 90% certainty). Main system controller 30 may then issue control signals to ECU 34-1, which may control the vehicle's powertrain to slow down and stop the vehicle. In some embodiments, processing by computer processor 10 and/or main system controller 30 is performed in real-time with respect to when image data of a scene is captured by camera 100 and transferred to image acquisition module 102. Or it may be fast enough that the display may show video corresponding to video stream 122 in near real-time (eg, less than 2 seconds or less than 1 second). In some embodiments, ECU 34-1 receives control signals in real time or near real time relative to when image data of a scene is captured by camera 100 and transferred to image acquisition module 102. can. As noted above, communication between the main system controller 30 and the ECUs 34-1, 34-2, 34-3, 34-4, . . . , 34-n may be via techniques. For example, the command and control line 32 may be a network or bus internal to the vehicle, and the main system controller 30 and the ECUs 34-1, 34-2, 34-3, 34-4, . . . , 34-n. Known protocols (eg, Ethernet, CAN, I2C, LIN, etc.) may be used to send signals to and from.

9A and 9B show example data resulting from images of a scene captured by two 5-megapixel cameras and processed according to various embodiments of the present technology described above. . The two cameras were mounted on the roof of a standard passenger car such that they were separated by a baseline of 1.2m. FIG. 9A shows a color disparity map blended with a grayscale camera image (not shown) so that both the distance and brightness of the scene can be viewed simultaneously. The disparity map has values from 0 to 1023, encoded using the jet colormap, with red indicating relatively closer objects and blue indicating relatively farther away objects. The transparency of the parallax map is modulated by the grayscale camera image. FIG. 9B shows the corresponding confidence map for the scene. Confidence maps are provided by H.I. H. Hirschmuller, "Stereo Processing by Semi-Global Matching and Mutual Information" (2008 IEEE Transactions on Pattern Analysis and Machine) e Intelligence) , shows the minimum aggregated cost value for each pixel from the semi-global matching algorithm.

According to some embodiments of the present technology, the processing modules and components in FIGS. 1, 3, and 5 are hardware (e.g., programmed to perform the procedures and methods described above). , one or more computer processors) and/or in software (eg, computer executable code). In some embodiments, at least some of the software may be pre-programmed into the computer processor. In some embodiments, at least some of the software may be stored on a non-transitory computer-readable storage medium, or on multiple non-transitory computer-readable storage media, which are accessed by a computer processor, can be carried out. In some embodiments, a process module may be a software module, or a hardware module, or a module combining hardware and software.

A vehicle assistance system according to the techniques described herein may be embodied in different configurations. Exemplary configurations include combinations of configurations (1) through (21) as follows.
(1) An automated vehicle assistance system for supervised or unsupervised vehicle movement, the system comprising a vehicle control system comprising a computer processor and a memory coupled to the computer processor; a first sensor system configured to receive one piece of image data and output a first disparity map and a first confidence map based on the first image data; receives the first disparity map and the first confidence map from the first sensor system and outputs a video stream comprising the first disparity map and the first confidence map. An automated vehicle assistance system, comprising:

(2) The system of configuration (1), wherein in the video stream the first confidence map is encoded to become part of the first disparity map.
(3) The first image data consists of a plurality of pixels, the disparity map consists of disparity data for each of the pixels, and the confidence map consists of confidence data for each of the pixels, configuration (1) or a system of configuration (2).

(4) the first image data comprises data for a left and right two-dimensional (2D) first image, and the first sensor system outputs left and right (2D) collimated data from the first image data; and a first cost volume map, wherein the first sensor system is configured to generate a 2D parallelized first image, a first disparity map, and a first The system of any one of configurations (1)-(3), wherein the system is configured to generate the first confidence map from the cost volume map of the .

(5) the first sensor system based on one or both of a uniqueness value determined from a semi-global matching (SGM) algorithm and an image texture value determined from a Sobel operation on the first image data; The system of any one of configurations (1)-(4), configured to generate a first belief map.

(6) further comprising a second sensor system configured to receive second image data of at least a portion of the scene and output a second confidence map based on the second image data; The vehicle control system receives a second confidence map from the second sensor system and outputs the video stream as a sequence of superframes, each superframe being a first disparity map. and outputting information based on the first confidence map and the second confidence map. system.

(7) Any of configurations (1) to (6), wherein the vehicle control system is configured to output a control signal to an electronic control unit (ECU) of the vehicle based on the information in the video stream. any one system.

(8) the first sensor system is a first sensor module configured to process the first image data to generate a first disparity map and a first confidence map; Yes, the second sensor system is a second sensor module configured to process the second image data to generate a second belief map, the first sensor module and the second any one of configurations (1) through (7), wherein the two sensor modules are stored in the memory and the computer processor is configured to perform the first sensor module and the second sensor module; system.

(9) the video stream comprises at least one superframe consisting of a first disparity map, a first confidence map, a first disparity map and a second confidence map; The system of any one of configurations (1) through (8), consisting of one superframe.

(10) any of configurations (1) through (9), wherein the video stream consists of at least one superframe consisting of a portion of the first confidence map and a portion of the second confidence map; one system.

(11) The system of any one of configurations (1)-(10), wherein the first image data comprises stereo vision data and the second image data comprises lidar data.
(12) further comprising a third sensor system configured to receive third image data of at least a portion of the scene and output a third confidence map based on the third image data; The system of any one of configurations (1) through (11).

(13) The system of any one of configurations (1)-(12), wherein the third image data comprises radar data or acoustic data.
(14) any one of configurations (1) through (13), wherein each superframe of the video stream consists of a two-dimensional (2D) image of the scene, a depth map of the scene, and a certainty map of the scene; system.

(15) The scene certainty map consists of a first confidence map, a second confidence map, or a combination of the first confidence map and the second confidence map, configurations (1) to The system of any one of (14).

(16) the scene depth map consists of a first disparity map modulated with image data corresponding to a 2D image of the scene, and the scene certainty map comprises image data corresponding to the 2D image of the scene; of configurations (1) to (15), consisting of a first confidence map, or a second confidence map, or a combination of the first and second confidence maps, modulated using one of our systems.

(17) any of configurations (1) through (16), wherein the pixels of the 2D image of the scene, the pixels of the depth map of the scene, and the pixels of the certainty map of the scene are temporally and spatially matched; one system.

(18) The vehicle control system encodes the disparity information from the first disparity map and the belief information from the first and second belief maps to reduce the data size of the video stream. The system of any one of configurations (1) through (17), wherein the system is configured to:

(19) further comprising a pair of cameras configured to be mounted on the vehicle, the cameras configured to provide first image data to the first sensor system; A system according to any one of 1) to (18).

(20) The video stream consists of two-dimensional (2D) color images, each 2D color image consisting of a plurality of pixels, the alpha channel transparency of each pixel being proportional to the confidence value for the pixel, configuration (1 ) through (19).

(21) The system of any one of configurations (1)-(20), wherein the color of the 2D color image indicates depth range.
The non-transitory computer readable storage medium, when executed by a computer processor, instructs the computer processor to perform an automated vehicle assistance system method for supervised or unsupervised vehicle movement according to the techniques described herein. It may be configured to store code to be executed. Examples of such computer-readable storage media include combinations of structures (22)-(34) as follows.

(22) A non-transitory computer-readable storage medium storing code that, when executed by a computer processor, causes the computer processor to perform the method of the automated vehicle assistance system for supervised or unsupervised vehicle movement. The method includes, by a computer processor, obtaining a first disparity map and a first belief map, the first disparity map and the first belief map being a first A non-transitory computer-readable storage medium, corresponding to image data, and a computer processor outputting a video stream comprising a first disparity map and a first confidence map.

(23) The computer-readable storage medium of arrangement (22), wherein the step of outputting the video stream comprises the step of the computer processor encoding the first confidence map to become part of the first disparity map. .

(24) The first image data consists of a plurality of pixels, the disparity map consists of disparity data for each of the pixels, and the confidence map consists of confidence data for each of the pixels. (22) or the computer-readable storage medium of configuration (23).

(25) The method comprises the steps of: a computer processor obtaining a second confidence map corresponding to second image data of at least a portion of the scene; and the computer processor outputting the video stream as a sequence of superframes. wherein each superframe consists of information based on the first disparity map, the first confidence map, and the second confidence map; (24) any one computer readable storage medium.

(26) The method of any of (22) to (25) further comprising the step of the computer processor outputting a control signal to an electronic control unit (ECU) of the vehicle based on the information in the video stream. any one computer-readable storage medium of

(27) The method comprises a computer processor processing the first image data to obtain a first disparity map and a first confidence map; and processing the second image data to obtain the map.

(28) Outputting the video stream comprises the computer processor preparing at least one superframe to consist of the first disparity map and the first confidence map; preparing at least one superframe to consist of one disparity map and a second confidence map. .

(29) The step of outputting the video stream comprises the computer processor preparing at least one superframe consisting of a portion of the first confidence map and a portion of the second confidence map. The computer-readable recording medium of any one of (22)-(28).

(30) any one of arrangements (22) through (29), wherein the first image data consists of stereo vision data and the second image data consists of lidar data, or radar data, or acoustic data; one computer-readable storage medium.

(31) The step of outputting the video stream comprises the computer processor preparing each superframe of the video stream to consist of a two-dimensional (2D) image of the scene, a depth map of the scene, and a certainty map of the scene. The computer-readable storage medium of any one of arrangements (22)-(30) comprising the step of:

(32) by a computer processor, preparing each superframe temporally and spatially matches pixels of a 2D image of the scene, pixels of a depth map of the scene, and pixels of a certainty map of the scene; The computer-readable storage medium of any one of arrangements (22)-(31), comprising the steps of:

(33) The step of outputting the video stream comprises: the computer processor disparity information from the first disparity map and from the first and second confidence maps to reduce the data size of the video stream; 33. The computer-readable storage medium of any one of arrangements (22)-(32), comprising the step of encoding the confidence information.

(34) outputting the video stream, wherein each 2D color image is composed of a plurality of pixels, the alpha channel transparency of each pixel is proportional to the confidence value for the pixel, and the color of the 2D color image is the depth range; The computer-readable storage medium of any one of arrangements (22)-(33) comprising the step of providing a two-dimensional (2D) color image, as shown in

A stereo vision system according to the techniques described herein may be embodied in different configurations. Exemplary configurations include combinations of configurations (35)-(46) as follows.

(35) A stereo camera system configured to capture a sequence of image pairs, each pair of images consisting of a first image and a second image captured simultaneously. a camera system and a computer processor, receiving a stream of image data from the stereo camera system, the image data corresponding to a sequence of image pairs, each of the image pairs; rectifying the first image and the second image to generate a two-dimensional (2D) pixmap of matched pixels for , and determining a depth value for each pixel of the pixmap determining, for each pixel of the pixmap, a confidence value for the depth value; and issuing a control signal when at least one of the confidence values indicates an image anomaly. a computer processor programmed to a stereo vision system.

(36) The system of configuration (35), wherein the image anomaly corresponds to one or more pixels of the portion of the confidence map having a confidence value below a predetermined threshold.
(37) the image anomaly corresponds to one or more pixels of the portion of the confidence map that have confidence values below a predetermined threshold for two or more consecutive pairs of images of the sequence; (35) or system of configuration (36).

(38) The system of any one of configurations (35)-(37), wherein the image anomaly consists of a plurality of pixels in contiguous regions of the confidence map.
(39) The system of any one of configurations (35)-(38), wherein the control signal is configured to cause an audible sound.

(40) The system of any one of configurations (35)-(39), wherein the audible tone is a pre-recorded message.
(41) The system of any one of arrangements (35)-(40), wherein the control signal is issued to an engine control module of the vehicle.

(42) For each pixel of the pixelmap, a confidence value is determined based on the presence or absence of an edge at the pixel, the illumination level of the pixel, and the texture values of the first and second images from which the pixelmap was generated. The system of any one of configurations (35)-(41).

(43) the computer processor is programmed to output a sequence of superframes corresponding to the sequence of image pairs, each superframe consisting of a 2D image and a confidence map corresponding to the 2D image; The system of any one of configurations (35)-(42).

(44) The system of any one of configurations (35)-(43), wherein the 2D image is the first image or the second image.
(45) the computer processor is programmed to output the sequence of superframes as a display signal that causes the display to show a 2D image and a visible confidence indicator corresponding to the confidence map; The system of any one of configurations (35)-(44).

(46) The system of any one of arrangements (35)-(45), wherein the display signal causes the confidence indicator to be displayed, pixel by pixel, as transparency for each pixel of the 2D image.

A non-transitory computer-readable storage medium may be configured to store code that, when executed by a computer processor, causes the computer processor to perform methods of vehicle assistance systems in accordance with the techniques described herein. Examples of such computer-readable storage media include combinations of structures (47) through (59) as follows.

(47) A non-transitory computer-readable storage medium storing code that, when executed by a computer processor, causes the computer processor to perform a method of a stereo vision system, the method comprising: receiving a stream of image data from, the image data corresponding to a sequence of pairs of images, each pair of images from the first image and the second image captured simultaneously. and for each pair of images, the computer processor parallelizes the first image and the second image to generate a two-dimensional (2D) pixel map of matched pixels. determining a depth value for each pixel of the pixmap; determining a confidence value for the depth value for each pixel of the pixmap; and the computer processor issuing a control signal when indicating a non-transitory computer-readable storage medium.

(48) The computer-readable storage medium of arrangement (47), wherein the image anomaly corresponds to one or more pixels of the portion of the confidence map having a confidence value below the predetermined threshold.

(49) the image anomaly corresponds to one or more pixels of the portion of the confidence map that have confidence values below a predetermined threshold for two or more consecutive pairs of images of the sequence; (47) or computer readable storage medium of configuration (48).

(50) The computer-readable storage medium of any one of arrangements (47)-(49), wherein the image anomaly consists of a plurality of pixels of contiguous regions of the confidence map.
(51) The computer-readable storage medium of any one of arrangements (47)-(50), wherein the control signal is configured to cause an audible sound.

(52) The computer-readable storage medium of any one of arrangements (47)-(51), wherein the audible tone is a pre-recorded message.
(53) The computer-readable storage medium of any one of arrangements (47)-(52), wherein the control signal is issued to an engine control module of the vehicle.

(54) For each pixel of the pixelmap, a confidence value is determined based on the presence or absence of an edge at the pixel, the illumination level of the pixel, and the texture values of the first and second images from which the pixelmap was generated. The computer-readable storage medium of any one of arrangements (47)-(53).

(55) The method comprises the step of the computer processor outputting a sequence of superframes corresponding to the sequence of image pairs, each superframe being a 2D image and a disparity map corresponding to the 2D image; and a confidence map corresponding to the 2D image.

(56) The computer-readable storage medium of any one of arrangements (47)-(55), wherein the 2D image is the first image or the second image.
(57) Outputting the sequence of superframes comprises outputting a display signal that causes the display to show the 2D image and a visible confidence indicator corresponding to the confidence map, comprising: 47) The computer readable storage medium of any one of (56).

(58) The computer-readable storage medium of any one of arrangements (47)-(57), wherein the display signal causes the confidence indicator to be displayed, pixel by pixel, as transparency for each pixel of the 2D image.

(59) The computer-readable storage medium of any one of arrangements (47)-(58), wherein each superframe consists of a 2D image, a confidence map, and a disparity map corresponding to the 2D image.

Conclusion Various alterations, modifications, and improvements may be made to the structures, compositions, and methods discussed above and are intended to be within the spirit and scope of the invention disclosed herein. It should be understood that Further, while advantages of the invention have been indicated, it should be appreciated that not every embodiment of the invention will include every described advantage. Some embodiments may not implement any features described as advantageous herein. Accordingly, the above description and accompanying drawings are only exemplary.

It will be appreciated that some aspects of the technology may be embodied as one or more methods, and the acts performed as part of the methods of the technology may be ordered in any suitable manner. should. Thus, in various embodiments, the actions shown and/or described may involve performing some acts simultaneously, even though they are shown and/or described as sequential acts. Embodiments may be constructed in which the acts are performed in a different order than they are performed.

Various aspects of the present invention may be used singly, in combination, or in various arrangements not specifically discussed in the above-described embodiments, and therefore, upon application thereof, the above description may be used. It is not limited to the details and arrangements of components defined in or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

The use of ordinal terms such as “first,” “second,” “third,” etc. in the description and claims to modify elements may, by themselves, refer to one element as another. One element or act with a certain name does not imply any precedence, precedence, or order of precedence over the elements, or the temporal order in which the method acts are performed (using ordinal terminology). It is only used as a label to distinguish an element or action (otherwise) from another element or action with the same name.

All definitions, as defined and used herein, supersede dictionary definitions, definitions in documents incorporated herein, and/or the ordinary meaning of the defined terms. should be understood.

As used in this specification and claims, the indefinite articles "a" and "an" shall be understood to mean "at least one," unless clearly indicated to the contrary. .

As used herein and in the claims, the phrase "at least one" in reference to a list of one or more elements means any one or means at least one element selected from a plurality, but not necessarily including at least one of every and every element specifically recited in the list of elements, any of the elements in the list of elements It should be understood that the combination of This definition does not specify whether elements other than the specifically identified elements in the list of elements to which the phrase "at least one" refers are related to or related to those specifically identified elements. It is also allowed that it may optionally be present, whether or not it is present.

As used herein and in the claims, the phrase "equal" or "same" in reference to two values (e.g., distance, width, etc.) means that the two values are within manufacturing tolerances. means they are the same. Thus, two values that are equal or the same can mean that the two values differ from each other by ±5%.

The phrase "and/or" as used herein and in the claims means "either or both" of such conjoined elements, i.e., in some cases conjunctive It should be understood to mean an element that exists implicitly and in other cases disjunctively. Multiple elements listed with "and/or" should be construed in the same fashion, ie, "one or more" of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the "and/or" clause, whether related or unrelated to those elements specifically identified. . Thus, as a non-limiting example, references to "A and/or B," when used in conjunction with open-ended words such as "comprising," in one embodiment refer only to A (and optionally B), in another embodiment, only B (optionally including elements other than A), and in another embodiment, both A and B (optionally including other (including elements) can be mentioned.

As used in the specification and claims, "or" should be understood to have the same meaning as "and/or" as defined above. For example, when separating items in a list, "or" or "and/or" are inclusive, i.e., including at least one of a number of elements or lists of elements. , and, optionally, additional unlisted items. Only words that clearly indicate the opposite, such as "only one" or "exactly one", or "consisting of" when used in a claim, a number of elements or a list of elements refers to including exactly one of Generally, the term "or" as used herein is preceded by terms of exclusivity such as "either," "one," "exactly one," or "exactly one." shall be construed to indicate exclusive alternatives (ie, "one or the other, but not both") only when "Consisting essentially of" when used in the claims shall have its ordinary meaning as used in the field of patent law.

Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of terms such as “including,” “comprising,” “consisting of,” “having,” “including,” and “accompanied by” herein, and variations thereof, herein, shall be used to refer to the items listed thereafter, and equivalents thereof, as well as additional items.

The terms "approximately" and "about" as used herein are in some embodiments within ±20% of a target value, in some embodiments within ±20% of a target value. It can be interpreted to mean within 10%, in some embodiments within ±5% of the target value, and in some embodiments within ±2% of the target value. The terms "approximately" and "about" may equate to a target value.

The term "substantially" as used herein is in some embodiments within 95% of the target value, in some embodiments within 98% of the target value, in some In embodiments, it can be taken to mean within 99% of the target value, and in some embodiments within 99.5% of the target value. In some embodiments, the term "substantially" may equal 100% of the target value.

Claims (60)

An automated vehicle assistance system for supervised or unsupervised vehicle movement, said system comprising:
a vehicle control system comprising a computer processor and a memory coupled to the computer processor;
a first sensor system configured to receive first image data of a scene and to output a first disparity map and a first confidence map based on said first image data; with
The vehicle control system includes:
receiving the first disparity map and the first confidence map from the first sensor system;
configured to output a video stream consisting of the first disparity map and the first confidence map;
Automated vehicle assistance system. 2. The vehicle assistance system of claim 1, wherein in the video stream the first confidence map is encoded to become part of the first disparity map. the first image data consists of a plurality of pixels;
the disparity map comprises disparity data for each of the pixels;
the confidence map consists of confidence data for each of the pixels;
A vehicle assistance system according to claim 1 . the first image data comprises data for left and right two-dimensional (2D) first images;
the first sensor system is configured to generate a first left-right (2D) parallelized image and a first cost volume map from the first image data;
The first sensor system includes:
the 2D parallelized first image; and
the first parallax map;
configured to generate the first confidence map from the first cost volume map;
A vehicle assistance system according to claim 1 . The first sensor system includes:
generating the first confidence map based on one or both of a uniqueness value determined from a semi-global matching (SGM) algorithm and an image texture metric determined from a Sobel operation on the first image data. 2. The vehicle assistance system of claim 1, wherein the vehicle assistance system is configured to: a second sensor system configured to receive second image data of at least a portion of said scene and to output a second confidence map based on said second image data;
The vehicle control system includes:
receiving the second confidence map from the second sensor system;
outputting the video stream as a sequence of superframes, each superframe based on the first disparity map, the first confidence map and the second confidence map; consisting of information configured to output and
A vehicle assistance system according to claim 1 . 7. The vehicle assistance system of claim 6, wherein the vehicle control system is configured to output control signals to a vehicle electronic control unit (ECU) based on the information in the video stream. a first sensor module, wherein the first sensor system is configured to process the first image data to generate the first disparity map and the first confidence map; and
the second sensor system is a second sensor module configured to process the second image data to generate the second belief map;
the first sensor module and the second sensor module are stored in the memory;
the computer processor configured to perform the first sensor module and the second sensor module;
A vehicle assistance system according to claim 6. The video stream is
at least one superframe consisting of the first disparity map and the first confidence map;
7. The vehicle assistance system of claim 6, comprising: at least one superframe comprising said first disparity map and said second confidence map. 7. The vehicle assistance system of claim 6, wherein said video stream comprises at least one superframe comprising a portion of said first belief map and a portion of said second belief map. the first image data comprises stereo vision data;
wherein the second image data comprises lidar data;
A vehicle assistance system according to claim 6. a third sensor system configured to receive third image data of at least a portion of said scene and to output a third confidence map based on said third image data; Item 12. A vehicle assistance system according to Item 11. 13. Vehicle assistance system according to claim 12, wherein said third image data comprises radar data or acoustic data. Each superframe of the video stream comprises:
a two-dimensional (2D) image of the scene;
a depth map of the scene;
7. The vehicle assistance system of claim 6, comprising: a certainty map of the scene; 3. The certainty map of the scene comprises the first confidence map, the second confidence map, or a combination of the first and second confidence maps. 15. Vehicle assistance system according to 14. said depth map of said scene comprising said first disparity map modulated with image data corresponding to said 2D image of said scene;
The certainty map of the scene is the first confidence map, or the second confidence map, or the first confidence map modulated with image data corresponding to the 2D image of the scene. consisting of a combination of a likelihood map and the second confidence map;
15. Vehicle assistance system according to claim 14. 15. The vehicle assistance system of claim 14, wherein pixels of the 2D image of the scene, pixels of the depth map of the scene, and pixels of the certainty map of the scene are temporally and spatially matched. The vehicle control system combines disparity information from the first disparity map and belief information from the first and second belief maps to reduce the data size of the video stream. 18. The vehicle assistance system of claim 17, configured to encode. further comprising a pair of cameras configured to be mounted on the vehicle;
the camera is configured to provide the first image data to the first sensor system;
A vehicle assistance system according to claim 6. 7. The method of claim 6, wherein the video stream consists of two-dimensional (2D) color images, each 2D color image consisting of a plurality of pixels, the alpha channel transparency of each pixel being proportional to the confidence value for the pixel. Vehicle assistance system as described. 21. The vehicle assistance system of claim 20, wherein colors of the 2D color image indicate depth range. A non-transitory computer-readable storage medium storing code that, when executed by a computer processor, causes said computer processor to perform a method of an automated vehicle assistance system for supervised or unsupervised vehicle movement, comprising: The method includes:
the computer processor obtaining a first disparity map and a first confidence map, wherein the first disparity map and the first confidence map are first image data of a scene; a step corresponding to
said computer processor outputting a video stream consisting of said first disparity map and said first confidence map. 23. The computer of claim 22, wherein said step of outputting said video stream comprises said computer processor encoding said first confidence map to become part of said first disparity map. readable storage medium. the first image data consists of a plurality of pixels;
the disparity map comprises disparity data for each of the pixels;
the confidence map consists of confidence data for each of the pixels;
23. A computer-readable storage medium according to claim 22. The method includes:
said computer processor obtaining a second confidence map corresponding to second image data of at least a portion of said scene;
said computer processor outputting said video stream as a sequence of superframes, each superframe comprising said first disparity map, said first confidence map and said second confidence 23. The computer-readable storage medium of claim 22, further comprising the step of: information based on the map. The method includes:
26. The computer readable storage medium of claim 25, further comprising: said computer processor outputting a control signal to a vehicle electronic control unit (ECU) based on said information in said video stream. The method includes:
said computer processor processing said first image data to obtain said first disparity map and said first confidence map;
26. The computer-readable storage medium of claim 25, further comprising: said computer processor processing said second image data to obtain said second confidence map. The step of outputting the video stream comprises:
the computer processor preparing at least one superframe to consist of the first disparity map and the first confidence map;
26. The computer-readable storage medium of claim 25, comprising: said computer processor preparing at least one superframe to consist of said first disparity map and said second confidence map. Said step of outputting said video stream comprises said computer processor preparing at least one superframe consisting of a portion of said first belief map and a portion of said second belief map. 26. The computer readable storage medium of claim 25. the first image data comprises stereo vision data;
wherein the second image data consists of lidar data, radar data, or acoustic data;
26. A computer-readable storage medium according to claim 25. The step of outputting the video stream comprises: the computer processor;
a two-dimensional (2D) image of the scene;
a depth map of the scene;
26. The computer-readable storage medium of claim 25, comprising preparing each superframe of the video stream to consist of: a certainty map of the scene; By the computer processor, the step of preparing each superframe includes pixels of the 2D image of the scene, pixels of the depth map of the scene, pixels of the certainty map of the scene, and 32. The computer-readable storage medium of Claim 31, comprising the step of spatially matching. In the step of outputting the video stream, the computer processor performs disparity information from the first disparity map and disparity information from the first confidence map and the second confidence map to reduce the data size of the video stream. 32. The computer-readable storage medium of claim 31, comprising encoding belief information from the belief map. The step of outputting the video stream comprises:
each 2D color image consists of a plurality of pixels,
the alpha channel transparency of each pixel is proportional to the confidence value for that pixel;
34. The computer-readable storage medium of claim 33, comprising preparing a two-dimensional (2D) color image such that colors of the 2D color image indicate depth range. A stereo camera system configured to capture a sequence of image pairs, each pair of images consisting of a first image and a second image captured simultaneously; ,
a computer processor,
receiving a stream of image data from the stereo camera system, the image data corresponding to the sequence of image pairs;
For each of said pairs of images,
rectifying the first image and the second image to generate a two-dimensional (2D) pixel map of matched pixels;
determining a depth value for each pixel of the pixel map;
determining a confidence value for the depth value for each pixel of the pixel map;
and a computer processor programmed to issue a control signal when at least one of said confidence values indicates an image anomaly. 36. The system of claim 35, wherein the image anomaly corresponds to one or more pixels of a portion of a confidence map having a confidence value below a predetermined threshold. 3. The image anomalies correspond to one or more pixels of a portion of a confidence map having confidence values below a predetermined threshold for two or more consecutive pairs of images of the sequence. 35. The system according to 35. 36. The system of claim 35, wherein the image anomaly consists of a plurality of pixels in contiguous regions of a confidence map. 36. The system of Claim 35, wherein the control signal is configured to cause an audible sound. 40. The system of Claim 39, wherein the audible tone is a pre-recorded message. 36. The system of claim 35, wherein the control signal is issued to an engine control module of the vehicle. For each pixel of the pixelmap, the confidence value is the presence or absence of an edge at the pixel, the illumination level of the pixel, and the texture values of the first and second images from which the pixelmap was generated. 36. The system of claim 35, determined based on: The computer processor is programmed to output a sequence of superframes corresponding to the sequence of image pairs, each superframe consisting of a 2D image and a confidence map corresponding to the 2D image. 36. The system of claim 35. 44. The system of claim 43, wherein said 2D image is said first image or said second image. The computer processor is programmed to output the sequence of superframes as a display signal that causes a display to show the 2D image and a visual confidence indicator corresponding to the confidence map. 44. The system of claim 43. 46. The system of claim 45, wherein the display signal causes the confidence indicator to be displayed on a pixel-by-pixel basis as the transparency of each pixel of the 2D image. 44. The system of claim 43, wherein each of said superframes consists of said 2D image, said confidence map, and a disparity map corresponding to said 2D image. A non-transitory computer-readable storage medium storing code that, when executed by a computer processor, causes the computer processor to perform a method of a stereo vision system, the method comprising:
the computer processor receiving a stream of image data from a stereo camera system, the image data corresponding to a sequence of image pairs, each pair of images captured simultaneously; a step consisting of an image and a second image;
for each of said pairs of images, said computer processor:
rectifying the first image and the second image to generate a two-dimensional (2D) pixel map of matched pixels;
determining a depth value for each pixel of the pixel map;
determining a confidence value for the depth value for each pixel of the pixel map;
and said computer processor issuing a control signal when at least one of the confidence maps indicates an image anomaly. 49. The computer-readable storage medium of claim 48, wherein the image anomaly corresponds to one or more pixels of the portion of the confidence map having a confidence value below a predetermined threshold. wherein the image anomaly corresponds to one or more pixels of the portion of the confidence map that have confidence values below a predetermined threshold for two or more consecutive pairs of images of the sequence. 49. The computer-readable storage medium of Clause 48. 49. The computer-readable storage medium of claim 48, wherein the image anomaly consists of a plurality of pixels in contiguous regions of the confidence map. 49. The computer-readable storage medium of Claim 48, wherein the control signal is configured to cause an audible sound. 53. The computer-readable storage medium of Claim 52, wherein the audible tone is a pre-recorded message. 49. The computer readable storage medium of Claim 48, wherein the control signal is issued to an engine control module of a vehicle. For each pixel of the pixelmap, the confidence value is the presence or absence of an edge at the pixel, the illumination level of the pixel, and the texture values of the first and second images from which the pixelmap was generated. 49. The computer-readable storage medium of claim 48, determined based on. The method includes:
said computer processor outputting a sequence of superframes corresponding to said sequence of image pairs, each of said superframes comprising a 2D image, a disparity map corresponding to said 2D image, and said 2D 49. The computer-readable storage medium of claim 48, further comprising: a confidence map corresponding to the image. 57. The computer-readable storage medium of claim 56, wherein said 2D image is said first image or said second image. 3. The step of outputting the sequence of superframes comprises outputting a display signal to cause a display to show the 2D image and a visual confidence indicator corresponding to the confidence map. 57. Computer readable storage medium according to Clause 56. 59. The computer-readable storage medium of Claim 58, wherein the display signal causes the confidence indicator to be displayed on a pixel-by-pixel basis as transparency for each pixel of the 2D image. 57. The computer-readable storage medium of Claim 56, wherein each of said superframes consists of said 2D image, said confidence map, and a disparity map corresponding to said 2D image.

Images