KR20230142640A - Methods and systems for providing depth maps with confidence estimates - Google Patents

Abstract

An automated vehicle assistance system is provided for supervised or unsupervised vehicle movement. The system includes a control system and a first sensor system. The first sensor system may receive first image data of the scene and output a first difference map and a first confidence map based on the first image data. The control system may output a video stream based on the first difference map and the first confidence map. The vehicle assistance system may also include a second sensor system that receives second image data of at least a portion of the scene and outputs a second confidence map based on the second image data. A video stream may include super frames, and each super frame includes a 2D image of the scene, a depth map corresponding to the 2D image, and a certainty map corresponding to the depth map.

Description

Method and system for providing confidence estimates to depth maps {METHODS AND SYSTEMS FOR PROVIDING DEPTH MAPS WITH CONFIDENCE ESTIMATES}

Cross-reference to related applications

This application claims the benefit of priority from U.S. Provisional Application No. 63/229,102, filed on August 4, 2021, and entitled “Depth Sensing System with Confidence Map,” filed on January 22, 2020, and International Application No. 62/964,148, entitled “Untethered Stereo Vision Camera System,” filed January 6, 2021, and entitled “Non-Rigid Stereo Vision Camera System.” This is a partial continuation application of PCT/US2021/12294. The entire contents of these applications are incorporated herein by reference.

field of invention

The present technology relates to stereo vision systems. In particular, the technology captures images of a scene and outputs two-dimensional (“2D”) image data, depth-estimate data regarding distances to objects in the scene, and confidence data regarding the level of certainty of the depth-estimate data. It relates to a stereo vision system (e.g., a stereo camera system).

Stereo vision systems typically use two cameras (e.g., left and right cameras) to measure the disparity, or difference, between matching pixels of the image captured by the left camera and the image captured by the right camera. Distance is estimated by measuring parallax. For example, US8208716B2 discloses a stereo vision system and method that outputs a difference map, which is a 2D matrix containing pixel-shift data corresponding to a calibrated image captured by one of the cameras (often the left camera). An estimate of the depth for each pixel in the image, corresponding to the distance from the camera to the imaged object in the pixel, can be easily calculated from the difference map because the depth of a pixel is inversely proportional to the difference between pixels. As such, the terms “depth map” and “difference map” may be used interchangeably herein because they provide much the same information about the scene in the image.

A problem with existing stereo vision systems is that depth estimates are typically provided without any indication of how reliable (or unreliable) the estimates are on a per-pixel basis or even on a frame-by-frame basis. In conventional driver assistance systems, automated decisions to stop or steer the vehicle are made using multiple sensors (e.g., radar, lidar, cameras) without knowledge of the reliability of information from any of the sensors. etc.) can be done based on information from. This causes driver assistance systems to overcompensate for the uncertainty of conflicting information from sensors by performing unnecessary procedures (e.g. traveling at too low speeds, braking too frequently, etc.), reducing passenger comfort. Alternatively, driver assistance systems may arbitrarily select one sensor over another, undercompensating for the uncertainty of conflicting information and increasing passenger risk. As can be appreciated, decisions made by driver assistance systems can greatly affect passenger safety and, at times, can be critical life-staking decisions. For example, when a vehicle is traveling at a cruising speed typical of highway driving, accurate knowledge of the distance to an object determines how the driver assistance system will control the vehicle to avoid hitting the object while maintaining or nearly maintaining cruising speed. It may be important to decide whether to

Autonomous vehicles and advanced driver assistance systems use the vehicle's control system to obtain information about the vehicle's surroundings to determine how to steer the vehicle and how to adjust the vehicle's speed (e.g., whether to accelerate or decelerate). ), various types of sensors (e.g. lidar, radar, cameras, ultrasonic, stereo cameras, etc.) to be able to make decisions about whether to deploy safety measures (e.g. to turn on warning flashers), etc. ) can be used. As will be appreciated, a vehicle's control system may include dozens of electronic control modules or units (“ECUs”), and in some cases, more than 100 ECUs, each ECU responsible for controlling the operation of the vehicle. Control aspects (e.g., speed control ECU, brake control ECU, transmission control ECU, engine control ECU, battery management ECU, etc.).

The present inventors have recognized and understood that in order to best fuse or combine data from different types of sensors, it is important to know the level of reliability or certainty of measurements from each of the sensors. In some aspects of the present technology, high-resolution depth information indicating the distance to a feature appearing in a pair of images may be determined through stereo matching of the pair of images. Stereo matching may be performed to provide distance certainty information based on any one or any combination of difference features, previous images, cost curves, and local attributes. Range certainty information can be used to minimize or eliminate problems in sensor fusion by allowing the reliability of depth information to be evaluated, thereby increasing the safety of decisions made using depth information.

According to one aspect of the present technology, an automated vehicle assistance system for supervised or unsupervised vehicle movement is provided. The system includes a vehicle control system consisting of a computer processor and a memory coupled to the computer processor; and a first sensor system configured to receive first image data of the scene and output a first difference map and a first confidence map based on the first image data. The vehicle control system may be configured to receive a first difference map and a first confidence map from the first sensor system and output a video stream consisting of the first difference map and the first confidence map.

In some embodiments of this aspect, in a video stream, a first confidence map may be encoded to be part of a first difference map. In some embodiments, the difference map may consist of difference data for each of the pixels, and the confidence map may consist of reliability data for each of the pixels. In some embodiments, the first image data may consist of data for left and right two-dimensional (2D) first images, and the first sensor system may provide left and right (2D) calibrated first images from the first image data. 1 images and a first cost volume map, wherein the first sensor system is configured to generate a first confidence map from the 2D calibrated first images, a first difference map, and a first cost volume map. It can be. In some embodiments, the first sensor system generates a first confidence map based on one or both of a uniqueness value determined from a semi-global matching (SGM) algorithm and an image texture metric determined from a Sobel operation on the first image data. Can be configured to generate.

In some embodiments of this aspect, the vehicle assistance system may be further configured with a second sensor system configured to receive second image data of at least a portion of the scene and output a second confidence map based on the second image data. there is. The vehicle control system receives the second confidence map from the second sensor system and outputs the video stream as a sequence of super frames, each super frame containing information based on the first difference map, the first confidence map and the second confidence map. consists of - consists of. In some embodiments, a vehicle control system may be configured to output a control signal to the vehicle's electronic control unit (ECU) based on information in the video stream. In some embodiments, the first sensor system may be a first sensor module configured to process the first image data to generate a first difference map and a first confidence map, and the second sensor system may be configured to process the second image data. may be a second sensor module configured to generate a second reliability map, the first and second sensor modules may be stored in a memory, and the computer processor may be configured to execute the first sensor module and the second sensor module. there is. In some embodiments, a video stream may be comprised of at least one super frame comprised of a first difference map and a first reliability map, and at least one super frame comprised of a first difference map and a second reliability map. In some embodiments, a video stream may consist of at least one super frame comprised of a portion of a first reliability map and a portion of a second reliability map. In some embodiments, the first image data may consist of stereo-vision data and the second image data may consist of LIDAR data.

In some embodiments of this aspect, the vehicle assistance system may be further configured with a third sensor system configured to receive third image data of at least a portion of the scene and output a third confidence map based on the third image data. there is. The third image data consists of radar data or acoustic data.

In some embodiments of this aspect, each super frame of the video stream may be comprised of a two-dimensional (2D) image of the scene, a depth map of the scene, and a certainty map of the scene. In some embodiments, the certainty map of the scene may consist of a first confidence map or a second confidence map or a combination of the first and second confidence maps. In some embodiments, the depth map of the scene may consist of a first difference map modulated with image data corresponding to a 2D image of the scene, and the certainty map of the scene may be a first confidence map or a second confidence map, or a first confidence map of the scene. It may be composed of a combination of first and second reliability maps modulated with image data corresponding to the 2D image. In some embodiments, pixels of a 2D image of a scene, pixels of a depth map of a scene, and pixels of a certainty map of a scene may be matched temporally and spatially. In some embodiments, the vehicle control system can be configured to encode difference information from the first difference map and reliability information from the first and second reliability maps to reduce the data size of the video stream.

In some embodiments of this aspect, the vehicle assistance system may further consist of a pair of cameras configured to be mounted on the vehicle. The cameras may be configured to provide first image data to the first sensor system.

In some embodiments of this aspect, the video stream may consist of two-dimensional (2D) color images, each 2D color image consisting of a plurality of pixels, and the alpha-channel transparency of each pixel may be It is proportional to the reliability value. In some embodiments, the colors of a 2D color image may indicate depth range.

According to another aspect of the present technology, there is provided a non-transitory computer-readable storage medium storing code that, when executed by a computer processor, causes the computer processor to perform a method of an automated vehicle assistance system for supervised or unsupervised vehicle movement. do. The method includes obtaining, by a computer processor, a first difference map and a first confidence map, wherein the first difference map and the first confidence map correspond to first image data of the scene, and obtaining, by the computer processor, a first difference map and a first confidence map. 1 It may consist of outputting a video stream composed of a reliability map.

In some embodiments of this aspect, outputting the video stream may consist of the computer processor encoding the first confidence map to be part of the first difference map. In some embodiments, the first image data may consist of a plurality of pixels, the difference map may consist of difference data for each of the pixels, and the reliability map may consist of reliability data for each of the pixels. there is. In some embodiments, the method includes the computer processor obtaining a second confidence map corresponding to second image data of at least a portion of the scene, the computer processor outputting the video stream as a sequence of super frames, each super frame may further include - consisting of information based on the first difference map, the first reliability map, and the second reliability map. In some embodiments, the method may further consist of the computer processor outputting a control signal to the vehicle's electronic control unit (ECU) based on information in the video stream. In some embodiments, the method includes the computer processor processing the first image data to obtain a first difference map and the first confidence map, and the computer processor processing the second image data to obtain the second confidence map. It may be configured additionally.

In some embodiments of this aspect, the step of outputting the video stream may include: the computer processor preparing at least one super frame including the first difference map and the first confidence map, and the computer processor preparing the first difference map and the second confidence map. It may consist of preparing at least one super frame including a map. In some embodiments, outputting the video stream may further consist of the computer processor preparing at least one super frame comprised of a portion of the first reliability map and a portion of the second reliability map. In some embodiments, the first image data may consist of stereo-vision data, and the second image data may consist of LIDAR data, radar data, or acoustic data.

In some embodiments of this aspect, outputting the video stream may consist of the computer processor preparing each super frame of the video stream, wherein each super frame of the video stream is a two-dimensional (2D) image of the scene. , a depth map of the scene, and a certainty map of the scene. In some embodiments, the preparation of each super frame by a computer processor consists of temporally and spatially matching pixels of a 2D image of the scene, pixels of a depth map of the scene, and pixels of a certainty map of the scene. It can be. In some embodiments, outputting the video stream comprises the computer processor encoding the encode difference information from the first difference map and the reliability information from the first and second reliability maps to reduce the data size of the video stream. It can be configured. In some embodiments, outputting a video stream may include preparing a two-dimensional (2D) color image as follows: each 2D color image consisting of a plurality of pixels, and the alpha-channel of each pixel. Transparency is proportional to the confidence value for the pixel, and the color of a 2D color image represents the depth range.

According to another aspect of the present technology, a stereo vision system is provided. The system includes a stereo camera system configured to capture a sequence of pairs of images, each pair of images including a first image and a second image captured simultaneously; It may consist of a computer processor programmed to receive a stream of image data from a stereo camera system, where the image data corresponds to a sequence of pairs of images. The computer processor may also be configured to, for each pair of images, calibrate the first and second images to generate a two-dimensional (2D) pixel map of the matched pixels, determine a depth value for each pixel of the pixel map, and configure the pixel map. Can be programmed to determine a confidence value for the depth value for each pixel. The computer processor may also be programmed to issue a control signal when at least one of the confidence values indicates an image anomaly.

In some embodiments of this aspect, an image anomaly may correspond to one or more pixels in a portion of the confidence map that have a confidence value below a predetermined threshold. In some embodiments, an image anomaly may correspond to one or more pixels that are part of a confidence map that have confidence values below a predetermined threshold for two or more consecutive image pairs in the sequence. In some embodiments, an image anomaly may consist of a plurality of pixels in a contiguous region of the confidence map. In some embodiments, the control signal may be configured to cause an audible sound, which may be a pre-recorded message. In some embodiments, a control signal may be issued to the vehicle's engine control module. In some embodiments, for each pixel in the pixel map, a confidence value may be determined based on the presence or absence of an edge within the pixel, the illumination level of the pixel, and the texture values of the first and second images from which the pixel map is generated. You can.

In some embodiments of this aspect, a computer processor can be programmed to output a sequence of super frames corresponding to a sequence of pairs of images, each of the super frames consisting of a 2D image and a confidence map corresponding to the 2D image. . In some embodiments, the 2D image may be a first image or a second image. In some embodiments, a computer processor can be programmed to output a sequence of super frames as a display signal that causes the display to show a 2D image and a visible confidence indicator corresponding to the confidence map. The display signal may cause the confidence indicator to be displayed on a pixel-by-pixel basis as the transparency of each pixel of the 2D image. In some embodiments, each super frame may consist of a 2D image, a confidence map, and a difference map corresponding to the 2D image.

According to another aspect of the present technology, a non-transitory computer-readable storage medium is provided having code stored thereon that, when executed by a computer processor, causes the computer processor to perform a method of a stereo vision system. The method includes the steps of a computer processor receiving a stream of image data from a stereo camera system, wherein the image data corresponds to a sequence of pairs of images, each image pair comprising a first image and a second image captured simultaneously. ; For each pair of images, a computer processor calibrates the first and second images to generate a two-dimensional (2D) pixel map of the matched pixels, determines a depth value for each pixel of the pixel map, and generates a pixel map. determining a confidence value for a depth value for each pixel in; and issuing, by the computer processor, a control signal when at least one of the reliability maps indicates an image abnormality.

In some embodiments of this aspect, an image anomaly may correspond to one or more pixels in a portion of the confidence map that have a confidence value below a predetermined threshold. In some embodiments, an image anomaly may correspond to one or more pixels that are part of a confidence map that have confidence values below a predetermined threshold for two or more consecutive image pairs in the sequence. In some embodiments, an image anomaly may consist of a plurality of pixels in a contiguous region of the confidence map.

In some embodiments of this aspect, the control signal can be configured to generate an audible sound. For example, an audible sound may be a pre-recorded message. In some embodiments, a control signal may be issued to the vehicle's engine control module. In some embodiments, for each pixel in the pixel map, a confidence value may be determined based on the presence or absence of an edge within the pixel, the illumination level of the pixel, and the texture values of the first and second images from which the pixel map is generated. You can.

In some embodiments of this aspect, the method may further comprise the computer processor outputting a sequence of super frames corresponding to the sequence of pairs of images, each of the super frames being a 2D image, the 2D image It consists of a corresponding difference map and a confidence map corresponding to the 2D image. In some embodiments, the 2D image may be a first image or a second image. In some embodiments, outputting the sequence of super frames may consist of outputting a display signal that causes the display to show a 2D image and a visible confidence indicator corresponding to the confidence map. In some embodiments, the display signal may cause the confidence indicator to be displayed on a per-pixel basis as the transparency of each pixel of the 2D image. In some embodiments, each super frame may consist of a 2D image, a confidence map, and a difference map corresponding to the 2D image.

The above-described features may be used individually or together in any combination in any of the embodiments discussed herein.

Various aspects and embodiments of the technology disclosed herein are described below with reference to the accompanying drawings. It should be understood that the drawings are not necessarily drawn to scale. Items appearing in multiple drawings may be indicated by the same reference number. For clarity, not all components may be shown in all drawings. At least one of the accompanying drawings is at least partially in color, as indicated below.
1 shows a block diagram of a stereo vision system, according to some embodiments of the present technology.
2 shows an arrangement of sensors and electronic control units arranged in a vehicle, according to some embodiments of the present technology.
3 shows a block diagram of a stereo vision system coupled to a control system, according to some embodiments of the present technology.
4A shows a diagram to understand how cost volume may be determined, according to some embodiments of the present technology.
4B shows an example cost curve, according to some embodiments of the present technology.
5 is a diagram illustrating processes performed by a reliability processing procedure, according to some embodiments of the present technology.
6 shows an example of a video stream and a super frame of a video stream according to some embodiments of the present technology. Portions of Figure 6 are in color.
7A-7F illustrate examples of different types of super frames that may be included in a video stream, according to some embodiments of the present technology.
Figure 8A schematically shows a beam region comprising illuminating a scene captured by a sensor, according to some embodiments of the present technology.
FIG. 8B is a graph showing the relationship between cross-sectional area and extent for objects in a captured image of a scene, according to some embodiments of the present technology.
9A and 9B show examples of a difference map and a confidence map, respectively, according to some embodiments of the present technology. Portions of Figure 9A are in color.

Different types of sensor systems (e.g. lidar systems, radar systems, mono-vision camera systems, stereo-vision camera systems, temperature measurement systems (e.g. thermocouples), acoustic systems (e.g. ultrasonic transducers) Sensor information from sensor systems, audible-sound microphone systems, etc.) obtains information about the vehicle's surroundings while the vehicle is in motion, enables the vehicle's electronic control system to make decisions about how to operate the vehicle, and/or allows the driver to make decisions about how to operate the vehicle. It can be used in vehicles to provide useful information for the driver to help operate the vehicle, for example, to adjust the vehicle's speed (e.g. accelerate or decelerate) and to provide safety information. It may be used by the control system to deploy actions (e.g., turn on warning flashers, windshield wipers, etc.), steer away from objects in the vehicle's path, etc. In another example, sensor information may be used by the vehicle. In some embodiments of the present technology, the control system may be used by the control system to alert the driver to a specific object in the path of the vehicle. It may be a centralized computer system, where each of the ECUs may be comprised of software and/or hardware configured to receive data from one or more sensors and process the data to output one or more control signals used to operate parts of the vehicle. As mentioned above, there may be more than 100 ECUs operating in a moving vehicle. Some ECUs may operate independently of other ECUs, and some ECUs may operate interdependently with one or more other ECUs. In some other embodiments of the present technology, the electronic control system of the vehicle may be distributed.For example, the battery management ECU may operate as a separate system independent of, for example, the speed control ECU.Each ECU Can receive sensor information from one type of sensor or multiple types of sensors. For example, the cabin-temperature ECU may receive sensor information from one or more thermometers located in different areas of the vehicle's cabin and may operate heaters and/or heaters to maintain the temperature of the cabin at a temperature set by the vehicle's occupants. Alternatively, sensor information can be used to control the air conditioner. In another example, the steering ECU may include one or more sets of cameras for stereo imaging, one or more radar systems, one or more LiDAR systems, one or more tire-pressure gauges, one or more microphones, one or more cameras for 2D imaging, and one or more It may receive sensor information from various combinations of navigation systems (e.g., GPS systems) and use the sensor information to determine the best course of action to safely navigate the vehicle to its destination.

The present inventors have recognized and understood that in order to optimize the use of data from different types of sensors, it is important to know the level of reliability or certainty of measurements from each of the sensors. According to some embodiments of the present technology, reliability information may be used to determine which sensor or combination of sensors has been or will be used for data to determine distances to objects. Conventional sensors typically report measurements or estimated measurements without reporting error estimates (e.g., error bars) that indicate the level of confidence in the measurements, making accurate fusion difficult. However, as discussed above, functional safety systems, which are particularly important for automotive applications, rely on sensor measurements to make decisions that can affect the safety of human life. Accordingly, there is a growing interest in providing accurate sensor data to a vehicle's electronic control system, which may involve providing the control system with a level of confidence for the sensor data. Armed with reliability data, the control system may be better able to make decisions regarding whether the sensor data is reliable enough to be used to control the vehicle or whether the sensor data is not reliable enough and should not be used. As will be appreciated, sensor fusion may be performed by a central control system to combine data from different types of sensors, but in some embodiments of the present technology, sensor fusion may be performed by one or more of the vehicle's ECUs. It may be performed by or by an auxiliary system operating in conjunction with ECUs and/or a central control system.

The inventors recognize and understand that the confidence level of an estimate can be used in Bayesian inference to update probabilities for a hypothesis or estimate as more evidence or information becomes available. Sensor data from various sensors of the same and/or different types deployed on the vehicle may be used to corroborate sensor data from a specific sensor deployed on the vehicle. The present inventors recognized and understood that sensors that report confidence ranges for depth estimates could make vehicles safer by enabling their driver assistance systems to make decisions based on reliable data, which would allow humans to This is especially important for autonomous vehicles that may not be controlled by drivers. For example, if a vehicle is traveling at a cruising speed typical of highway driving and the sensor (e.g., camera) is partially obscured, the image captured by the sensor (e.g., due to a foreign object on the sensor) Based on various factors that factor into the analysis, if an object is inaccurately detected as being in a close range to the vehicle, the stereo vision system according to some embodiments of the present technology may detect the sensor data corresponding to the object. A confidence level can be determined and output indicating a relatively low level of certainty about the sensor data, and therefore the sensor data can be ignored. This could potentially control emergency braking to avoid hitting the wrong (i.e. non-existent) object, e.g. to apply the vehicle's brakes suddenly, increasing the likelihood of the vehicle being rear-ended. Alternatively, the vehicle could continue on its path at cruising speed or allow the vehicle to gradually reduce speed to increase observation time. In some embodiments, the reliability associated with the sensor data may be used to determine whether the sensor data can be used or whether sensor data from another sensor with a relatively higher reliability should be used instead. For example, if depth information from sensor data indicates objects in the vehicle's path, but the depth information is associated with low confidence values, the vehicle may want to ensure that known objects are observed over more time and/ Or a known object may be controlled to reduce its movement speed so that it can be cross-checked by other sensor systems. As described below, different types of sensor systems (e.g., systems based on radar, lidar, acoustics, etc.) can be used to corroborate data acquired by stereo-vision techniques, according to various embodiments of the present technology. It can be used to complement or supplement.

The present inventors have determined that driver assistance systems can determine whether a particular sensor is partially or fully malfunctioning (e.g., partially or completely obscured by a foreign object), thereby reducing It is recognized and understood that the level of confidence or degree of certainty in depth estimates can be used to advantage. In some embodiments, information derived from confidence levels can be used to warn the driver of a sensor anomaly. In some embodiments, a stereo vision system or a device operating in conjunction with a stereo vision system can track occurrences of low confidence values (e.g., values below a threshold confidence level) across a plurality of images, e.g. For example, the frequency of low confidences can be used to determine whether a sensor may be malfunctioning.

According to some embodiments of the present technology, a stereo vision system may be configured to determine and output an overall confidence level for an image. The terms “reliability”, “confidence range”, and “confidence level” may be used interchangeably herein. The image may be a scene captured by sensors on a moving vehicle (e.g., a car, truck, airplane, etc.), or on a stationary structure (e.g., a street lamp, an airport tower, a house, an office building, etc.). It may be a scene captured by mounted sensors. The stereo vision system may be a standalone system deployed in the vehicle or may be integrated into the vehicle's control system. In some embodiments, a stereo vision system may determine and output a confidence level for each of a plurality of regions of an image. For example, an image can be divided into quadrants (e.g., top left, top right, bottom left, bottom right) and the stereo vision system can determine and output a confidence level for each quadrant. In some embodiments, the stereo vision system can determine and output a confidence level for each pixel in the image.

According to some embodiments of the present technology, a stereo vision system may be configured to output a depth map corresponding to a captured image of a scene. The image may be a captured digital image or may be digitized from an analog image. A depth map can be a map of depth values or distances from sensors in a stereo vision system to objects in a scene. A depth map can be made up of pixels that correspond to pixels in the image, such that each pixel in the depth map (and each pixel in the image) has an associated depth value. The stereo vision system may also be configured to output reliability data along with a depth map. In some embodiments, the confidence data may be a confidence map that indicates the certainty or reliability of the depth map. In some embodiments, the confidence map may be composed of pixels that correspond to pixels in the depth map (and the image), such that each pixel in the confidence map (and each pixel in the image) will have an associated confidence level. You can. In some embodiments, the confidence map organizes estimates of confidence as error bars, standard deviation values, buckets (e.g., high confidence, medium confidence, low confidence), or a quality level of each estimated confidence. It can be expressed as any other displayable metric.

As described above, an estimate of the depth for each pixel in an image can be calculated from the difference map because the depth of a pixel is inversely proportional to the difference between the pixels. As such, the terms “depth map” and “difference map” may be used interchangeably herein because they provide substantially the same information about the captured scene in the image and are well known in the art. This is because they are related by simple algebraic transformations.

According to some embodiments of the present technology, autonomous vehicles and/or advanced driver assistance systems (ADAS) may advantageously use depth maps and depth maps to avoid accidents and/or to warn drivers in the presence of unreliable data. Reliability values associated with can be used. In some embodiments, because a confidence value may be provided for each pixel of the depth map, there may be no need to discard entire frames of a captured video sequence of a scene when there are some pixels with low confidence values. . Instead, pixels with low confidence values can be discarded, and the remaining pixels with sufficiently high confidence values can be used for depth calculations. This selectivity is very similar to human vision, which can naturally ignore distracting objects such as a vehicle's A-pillar, a vehicle's windshield wipers, etc., when processing a scene within the driver's field of view while driving. That is, the driver will automatically ignore distracting objects while evaluating the scene within the field of view. In some embodiments, confidence maps can increase sensor availability because frames of a video sequence can be used even if some parts of the frame can be ignored, allowing the vehicle's sensors to perform higher duty across a larger range of environmental situations. Allows operation in cycles. For example, the sensor may detect contaminants on part of the sensor's lens, or windshield wipers may partially obstruct the sensor's field of view, or a section of the image captured by the sensor may be overexposed. or when a section of the image captured by the sensor has low light levels, or in any case where some pixels of the image captured by the sensor may have useful data even though some other pixels of the image may have data that should be ignored. It can work in any situation. By providing a confidence map comprised of confidence values on a pixel-by-pixel basis, the vehicle's electronic control system may be able to direct attention to regions of the depth map that have valid and relatively high confidence values.

According to some embodiments of the present technology, autonomous vehicles can fuse information from different sensors, such as camera, lidar, radar, and/or ultrasonic sensors, to increase reliability and safety. These sensors can report or provide information about distances to objects, but when different distances are reported for different sensors, it can be unclear which sensor to trust. In some embodiments, sensor fusion algorithms can combine data from different sensors and produce fused information with less uncertainty than is possible when unfused information from one or the other of the sensors is used individually. can be output. The inventors have recognized and understood that sensor fusion algorithms can be improved to increase the certainty of the fused information by providing the algorithms with certainty parameters (e.g., variance) for each of the different sensors. . In some embodiments, sensor fusion may be enabled for two or more sensors with different but overlapping fields of view because confidence maps may be determined for different sensors on a pixel-by-pixel basis. In some embodiments, radar range estimates for an object may be compared to stereo-vision distance estimates for an object when there are overlapping fields of view surrounding the object. For example, sensors on a car moving in clear weather during the day may have very high confidence values for stereo-vision distance determinations based on images captured by cameras on the car, and thus stereo-vision distance determination. They may be trusted by the car's electronic control system, especially for objects at ranges or distances greater than 300 meters, and other sensors on the car may not be expected to return high confidence values. If the weather turns unfavorable (e.g., dense fog, pouring rain, etc.), visibility may be lower and, as a result, stereo-vision distance estimates may deteriorate due to light attenuation and associated confidence values may be lowered. At this time, the car's control system can be switched to obtain range estimates from radar data instead of data from cameras. Similarly, if the car is traveling at night or in low ambient light levels without fog or other precipitation, the car's control system can switch to obtain range estimates from lidar data instead of radar data or data from cameras. This is because a typical LiDAR system has its own active lighting source. In some embodiments, instead of the vehicle's control system determining when to switch between stereo-vision range estimates, LiDAR range estimates, radar range estimates, or acoustic range estimates, switching is performed on the vehicle's main range estimates. This may be performed by the car's stereo-vision system, which may be a sensor system. As can be appreciated, although the foregoing examples relate to distance estimates for automobiles, the present technique is not limited to automobiles and may be applied to other vehicles (e.g., trucks and other road vehicles, trains and other rail vehicles, boats and other sailing vehicles). It may be applicable to vehicles, airplanes and other aerial vehicles, etc.).

1 shows a block diagram of a stereo vision system 1 configured to provide values for stereo-matching confidence or certainty, according to some embodiments of the present technology. Stereo vision system 1 may be comprised of a computer processor 10 coupled to a plurality of sensor systems and configured to receive measurements or sensor data from each of the sensor systems. In some embodiments, sensor systems coupled to processor 10 may be configured as an imaging system comprised of sensors 100, which may be cameras, lidar sensors, radar sensors, and/or ultrasonic sensors. . In some embodiments, the stereo vision system 1 may be mounted on a vehicle capable of autonomous movement (i.e., without human control) and/or semi-autonomous movement (i.e., with human control or limited human control). . For example, a vehicle may be a car, truck, robot, sailing vessel, air vehicle, etc.

According to some embodiments of the present technology, the sensors 100 may consist of two stereo cameras 100 configured to capture images of the vehicle's environment simultaneously, that is, at the same or approximately the same moment in time. To simplify notation, cameras 100 may be referred to herein as “left” and “right” cameras, although they are positioned perpendicularly to each other (e.g., top and bottom), or to each other. They may be positioned diagonally relative to each other, or offset in different range bins (eg, one camera in the front portion of the vehicle and the other camera in the rear portion of the vehicle). Cameras 100 may include, for example, color complementary metal-oxide-semiconductor (CMOS) cameras, grayscale CMOS cameras, charge-coupled device (CCD) cameras, short-wavelength infrared (SWIR) cameras, These may be long-wavelength infrared (LWIR) cameras, or focal plane array sensors.

According to some embodiments of the present technology, sensors S1, S2, S3, S4, S5, S6, S7, S8, S9 are configured to detect a plurality of different sensors on vehicle 20, as schematically shown in FIG. 2. Can be located in locations. In some embodiments, some of the sensors S1, S2, ..., S9 may be located inside the cabin of the vehicle 20 and thus protected from dust and rain. In some embodiments, some of the sensors S1, S2, ..., S9 may be located outside the cabin of vehicle 20 so that the baseline (i.e. the distance between the two sensors) is located outside the cabin of the vehicle 20. It can be larger than the width, which can allow objects at relatively greater distances to be detected compared to baselines limited to the width of the room. In some embodiments, sensors S1, S2, ..., S9 receive data from sensors S1, S2, ..., S9 and/or sensors S1, S2, ... , S9) may be wirelessly or wiredly coupled to one or more ECUs 22, 24 configured to supply power. In some embodiments, at least some of ECUs 22, 24 may be part of computer processor 10. In some other embodiments, at least some of the ECUs 22, 24 may be located external to the computer processor 10 and may be configured to transmit signals wirelessly or wired to the computer processor 10. In some embodiments, a pair of cameras in the headlights (eg, S1 and S9); or a pair of cameras in the side view mirrors (eg S2 and S8); or a pair of cameras in the windshield (eg S3 and S7 or alternatively S3 and S5); Or only two cameras are needed, such as a pair of cameras on the loop (eg S4 and S6).

According to some embodiments of the present technology, the stereo vision system 1 may be coupled to the main system controller 30 of the vehicle, as schematically shown in FIG. 3 . In some embodiments, main system controller 30 may be the vehicle's control system, which may be configured to control all automated aspects of vehicle operation. In some embodiments, stereo vision system 1 may be configured to receive commands from main system controller 30 and communicate signals to and from main system controller 30 via command and control lines 32. can do. As will be appreciated, command and control lines 32 may be a wired communication mechanism (e.g., data bus, communication line) or may be a wireless communication mechanism utilizing communication techniques known in the art. In some embodiments, main system controller 30 coordinates high-level functions (e.g., automatic emergency braking, route selection, etc.) and various sub-systems or ECUs (e.g., stereo vision system It may consist of a computer configured to perform high-level functions by communicating with (1)). In some embodiments, common communication protocols include communication over command and control lines 32 (e.g., Ethernet, Controller Area Network (CAN), Inter-Integrated Circuit (I2C), Local Interconnect Network (LIN), etc. can be used for Although the stereo vision system 1 is shown in FIG. 3 as separate from the main system controller 30, the stereo vision system 1 may, in some embodiments, be part of the main system controller 30 and may be part of the main system controller 30. In embodiments, it may be physically located in the housing of main system controller 30.

Referring to FIG. 1 , cameras 100 may be coupled to image acquisition module 102 wirelessly or by a wired connection, according to some embodiments of the present technology. In some embodiments, image data of a scene captured by cameras 100 may be transmitted through a known communication interface (e.g., a Universal Serial Bus (USB) connector, an Ethernet connector, a Mobile Industry Processor Interface (MIPI), a CSI (Camera Serial Interface) connector, Gigabit Multimedia Serial Link (GMSL) connector, Flat Panel Display Link (FPD-Link) connector, etc.) may be transmitted to the image acquisition module 102. In some embodiments, cameras 100 may be connected to image acquisition module 102 in real time or near real time, or directly, through a buffer memory device (e.g., RAM) that may be integrated into cameras 100. Can be configured to transmit image data. In some embodiments, cameras 100 may be associated with a data storage memory device 140 accessible by image acquisition module 102 as well as other portions of computer processor 10, and the cameras ( 100 may be configured to transmit image data to data storage device 140 . In some embodiments, cameras 100 may be video cameras configured to capture streams of video data of a scene. Streams of video data may consist of a left stream and a right stream, with each stream consisting of a sequence of frames. Accordingly, the term “image data” as used herein may refer to frames of video data in some embodiments.

According to some embodiments of the present technology, image acquisition module 102 may be configured to digitize image data from cameras 100 to generate raw digital image data or “raw image data.” In some embodiments, image acquisition module 102 may provide raw image data to image pre-processing module 104. In some embodiments, image acquisition module 102 may provide raw image data to memory 140, and memory 140 may store the raw image data for future processing.

According to some embodiments of the present technology, the image pre-processing module 104 may be configured to correct raw image data to generate corrected left and right images. For example, image pre-processing module 104 may perform demosaicing; Autofocus, autoexposure and automatic white balance correction; Vignetting; noise reduction; Bad pixel filtering; high-dynamic-range (HDR) lookup table color processing; and image compression. The corrected left and right images may be passed to the image correction module 106.

According to some embodiments of the present technology, the image correction module 106 is configured to correct the corrected left and right images by warping them so that the corresponding rows of pixels of the corrected left and right images are on the same epipolar plane. It can be. After warping, image correction module 106 may output left and right corrected 2D images 114, which may be color images or grayscale images. As will be appreciated, image correction is a known technique used to simplify the matching of common objects in the corrected left and right images. Image correction module 106 may provide left and right corrected 2D images 114 to stereo matching module 108, reliability processing module 110, and encoder module 112.

According to some embodiments of the present technology, stereo matching module 108 may be configured to calculate the difference between each pair of matching pixels in the calibrated 2D images 114. The processing performed by the stereo matching module 108 may, in some embodiments, consist of the following four procedures: a cost calculation procedure, a cost aggregation procedure, a difference calculation procedure, and a difference refinement procedure—each of these. is described later -.

According to some embodiments of the present technology, a cost calculation procedure constructs a three-dimensional (3D) cost volume map 118 by calculating matching costs for each pixel at each difference value in a set of possible difference values. It may be composed of, and may also be referred to as a “difference space image.” As described below, the cost volume (or more accurately the matching cost volume) can be determined as the product of W x H x D, where W and H are the width and height dimensions of each image, and where D is the difference hypothesis. Or is the number of possible differences. The cost of matching a particular pixel and a particular difference value indicates how unlikely it is that a particular pixel has that particular difference value. Typically, for a given pixel, the difference value with the lowest matching cost is selected to be used in the difference map, as described below. This approach to selecting a difference value for a given pixel is the so-called winner-takes-all (WTA) approach, where the winner is the difference value with the lowest matching cost, i.e. the best difference value among all difference hypotheses. Matching costs are calculated using known techniques, such as absolute deviation techniques, mutual information (MI) techniques (e.g., hierarchical MI (HMI) techniques), normalized cross-correlation (NCC) techniques, Hamming distance techniques, etc. It can be. In some embodiments, the NCC technique includes two sub-circuits around the pixel under consideration, as described in “Evaluation of Cost Functions for Stereo Matching” (2007 IEEE Conference on Computer Vision and Pattern Recognition) by H. Hirschmuller et al. It can be used to match costs for windows (one sub-window in each of the left and right calibrated 2D images 114). In some embodiments, the Hamming distance technique may be used in census transformation, where the pixel under consideration is Surrounding neighboring pixels are mapped to a bit string depending on whether their intensity values are greater or less than the intensity value of the pixel under consideration.

FIG. 4A shows a diagram to understand how cost volume map 118 may be determined, in accordance with some embodiments of the present technology. In cost-volume analysis, the cost is calculated by collecting a cost curve (e.g., cost curve 400) for every pixel in the left calibrated image (or right calibrated image) of calibrated 2D image 114. A volume map 118 may be constructed. Cost volume map 118 has H may be the number of pixel groups in the column direction), and W is the width of the image (e.g., W may be the number of pixels in the column direction, or, if less accurate matching is allowed, the number of pixel groups in the column direction. ), D is the number of retrieved differences. Thus, briefly, cost volume map 118 can be thought of as containing a cost curve for each pixel coordinate (row and column). In some embodiments, cost volume map 118 may be refined (e.g., by filtering) to obtain more reliable matching costs, using techniques known in the art. For example, there are techniques described in "Fast cost-volume filtering for visual correspondence and beyond" (2011 Conference on Computer Vision and Pattern Recognition) by C. Rhemann et al.

FIG. 4B shows an example cost curve 400 that can be used to describe a cost matching procedure, according to some embodiments of the present technology. As can be appreciated, typically the cost between the pixel P (row, column) with coordinates of (H0, W0) in the left-corrected image and the pixel with coordinates of (H0, W0+D0) in the right-corrected image. Matching may be performed. Cost curve 400, which represents the relationship between matching cost and difference, may represent the specific pixel under consideration. (The term “matching cost” referred to may also be referred to herein as “match cost” or “cost.”) As shown in Figure 4B, cost curve 400 has c when the difference is d ₁ has the lowest global minimum cost of _d1 , when the difference is _d2 , c has the second lowest global minimum cost of _d2 , and when the difference is _d2m , c has the second lowest local minimum cost of _d2m . As will be understood, the term “global” may be used for values evaluated across all points within cost curve 400 and the term “local” may be used for values evaluated across a portion of cost curve 400. can be used

According to some embodiments of the present technology, the value for matching cost spans 5 pixels in the column direction and 5 pixels in the row around the pixel under consideration in the left and right corrected 2D image 114. It can be determined using the NCC technique (see above) for sub-windows (“5x5” sub-windows).

According to some embodiments of the present technology, the cost aggregation procedure may consist of aggregating matching costs over the support area of each pixel, using the results of the cost calculation procedure. For “local” stereo matching techniques, the area of support can be understood to be a weighted sum of costs in a group of neighboring pixels around the pixel of interest. For “semi-global” and “global” stereo matching techniques, the area of support can be understood to be a function of the costs for all pixels of the image.

According to some embodiments of the present technology, the difference calculation procedure consists of calculating the difference for each pixel using local or global optimization methods, using the results of the cost aggregation procedure, and generating an unrefined difference map. It can be configured. Computational speed versus accuracy may determine the choice between local and global optimization methods. For example, a local method may be used when speed is required over accuracy, whereas a global method may be used when accuracy is required over speed. In some embodiments, local optimization methods such as block matching may be used. In some embodiments, a global optimization method such as Semi-Global Matching (SGM) may be used.

According to some embodiments of the present technology, the difference refinement procedure may consist of filtering the unrefined difference map to generate a 2D difference map 116 for the left and right corrected 2D images 114. . The difference refinement procedure is an optional procedure to correct difference values. Traditional refinement steps include left-right checking, hole filling, smoothing filters, and outlier detection and removal.

According to some embodiments of the present technology, stereo matching module 108 may utilize different stereo matching techniques than those described above. For example, the stereo matching module may use one or more techniques described in: “Stereo matching algorithm based on deep learning: A survey” by Mohd Saad Hamid et al. (2020 Journal of King Saud University - Computer and Information Sciences) and/or one or more of the techniques described in “Stereo Processing by Semi-Global Matching and Mutual Information” by H. Hirschmuller (2008 IEEE Transactions on Pattern Analysis and Machine Intelligence).

According to some embodiments of the present technology, the stereo matching module 108 matches the 2D difference map 116 to the encoder module 112 of the processor 10 of the stereo vision system 1 and the 2D difference map 116, as shown in FIG. It is output to the reliability processing module 110. In some embodiments, the stereo matching module 108 also outputs the 3D cost volume map 118 described above in connection with deriving the difference map 116 to the reliability processing module 110.

According to some embodiments of the present technology, the reliability processing module 110 may receive left and right 2D corrected images 114 from the image correction module 106, a cost volume map 118 from the stereo matching module 108, and receive the 2D difference map 116 from the stereo matching module 108 as inputs, and determine the accuracy of the estimated difference for each pixel of the calibrated images 114 from these inputs.

The reliability processing module 110 may be configured to calculate a reliability value for each frame of the video stream, according to some embodiments of the present technology. In some embodiments, reliability processing module 110 may calculate a reliability value for each pixel of each frame of the video stream. Accordingly, it should be understood that the term “image” herein may include frames of a video stream.

Referring back to FIG. 1 , according to some embodiments of the present technology, reliability processing module 110 may output reliability information to encoder module 112. In some embodiments, the reliability information may be a reliability map 120 for each frame of the video stream. Confidence map 120 may have the same dimensions as difference map 116 output by stereo matching module 108 to encoder module 112. In some embodiments, confidence map 120 may include confidence information for each pixel in the frame, representing a value indicative of a confidence level of the depth estimate for the corresponding pixel in difference map 116. In some embodiments, a value indicative of the level of confidence for a particular pixel may be the root mean square error of the difference or depth determined for that pixel, or a 95% confidence interval of the difference or depth determined for that pixel, or or it can be any measure of variability (e.g., standard deviation, standard error of the mean, confidence interval, data range, percentile, etc.). “On the confidence of stereo matching in a deep-learning era: a quantitative evaluation” (2021 IEEE Transactions on Pattern Analysis & Machine Intelligence) by M. Poggi et al. and “A Quantitative Evaluation of Confidence Measures for Stereo Vision” by X. Hu et al. Other reliability metrics may be used, such as those described in "(2012 IEEE Transactions on Pattern Analysis and Machine Intelligence).

FIG. 5 is a diagram illustrating processes performed by reliability processing module 110, according to some embodiments of the present technology. In some embodiments, at block 500, the first confidence measure process calculates a first confidence map 508 based on the cost volume 118 received from the stereo matching module 108. In some embodiments, the winning margin of the cost curve (e.g., cost curve 400) for each pixel may be calculated and used as the confidence value for that pixel on first confidence map 508. Winner Margin (WMN) is defined as the difference between the second lowest local minimum cost c _d2m and the lowest global minimum cost c _d1 , normalized over the entire cost curve, or

ego,

Here, p is the coordinate of the pixel under consideration, c is the matching cost, and D is the set of differences retrieved. In some embodiments, the first confidence map 308 may also be described in M. Poggi et al., “On the confidence of stereo matching in a deep-learning era: a quantitative evaluation” (2021 IEEE Transactions on Pattern Analysis & Machine Intelligence). It can be derived from other measures such as those summarized.

According to some embodiments of the present technology, at block 502, the second reliability measurement process of reliability processing module 110 performs a first reliability measurement process based on the corrected 2D images 114 received from image correction module 106. 2 Calculate the reliability map 510. In some embodiments, a texture is applied to the calibrated 2D image 114, for example, by calculating a derivative (e.g., x derivative) of the image data to determine the change in the calibrated 2D image 114. As measured, the texture values can be used to derive a second confidence map 510. In some embodiments, image texture may be measured using the x-Sobel operator. As can be appreciated, in stereo vision techniques, 3D reconstruction is performed on images with little or no texture consisting of repeating structures, due to the difficulty in matching textureless features or features that are identical to a plurality of other features in the image. It is known that the processing is difficult to carry out. Given these difficulties, the second confidence map 310 can be derived from the grayscale image convolved with the x-Sobel operator:

.

This Sobel convolution or filtering, which is known to emphasize edges in an image, can yield higher values when there are edges in the calibrated 2D image 114, and thus a sharper image that can be easier to stereo-match. and can result in more well-defined characteristics. In some embodiments, in addition to or instead of the Sobel convolution, the second confidence map 510 evaluates each pixel based on the intensity of the pixel, with the intensity below the minimum threshold or above the maximum threshold. It can be derived by penalizing pixels with . For example, a zero or low confidence value may be determined for each low-intensity or under-illuminated pixel with a signal-intensity level below a minimum threshold, and similarly, a zero or low confidence value may be determined for each low-intensity or under-illuminated pixel with a signal-intensity level below a maximum value. A signal-intensity level of can be determined for each over-illuminated or saturated pixel.

According to some embodiments of the present technology, at block 504, the third reliability measurement process of reliability processing module 110 generates a third reliability map based on the difference map 116 received from stereo matching module 108. Calculate (512). In some embodiments, confidence values for third confidence map 312 may be calculated from the variance of the difference determined for each pixel of the calibrated 2D images 114. In some embodiments, the variance may be the statistical variance of the differences determined for neighboring pixels (e.g., pixels surrounding the pixel under consideration). For example, a pixel with relatively higher variance may indicate that the pixel is part of a region of the difference map 116 that has more noisy (e.g., more scattered) data, and thus has relatively higher variance. It may be assigned a low reliability value. As can be appreciated, noise can indicate blurriness and may indicate that the sensors used to capture images (e.g., cameras 100) or areas of the sensors may be dirty or partially obscured. You can. Known techniques (e.g., techniques based on Laplace filters and/or Sobel filters) may be used to detect variance in difference map 116. In some embodiments, a variance below the threshold variance may indicate that the pixel under consideration is blurry or in a hazy region and may therefore be assigned a low confidence value in the third confidence map 512.

As will be appreciated, although the first, second and third confidence maps 508, 510, 512 have been described as consisting of confidence values for each pixel of an image or each pixel of each frame of a video stream, According to some embodiments of the technology, in some other embodiments, one or more of the first, second and third confidence maps 508, 510, 512 are configured with confidence values representing two or more pixels of an image or frame. It can be configured. For example, each frame of the video stream may be comprised of pixels categorized into n groups, and reliability processing module 110 may generate n reliability values for each frame of the video stream (i.e., n It may be configured to calculate a reliability value for each group.

According to some embodiments of the present technology, at block 506, the aggregator process of reliability processing module 110 uses the first, second, and third reliability maps 508, 510, and 512, and , the reliability map 120 is generated by combining the estimated reliability values in the second and third reliability maps 508, 510, and 512. Accordingly, confidence map 120 may be comprised of enhanced confidence values that are best certainty estimates based on a plurality of confidence measures. In some embodiments, the aggregator process may consist of calculating a sum of the confidence values for each pixel and using that sum as an improved confidence value for the pixel in confidence map 120. In some embodiments, the aggregator process may include weighting the confidences in the first, second and third confidence maps 508, 510, 512 and then calculating a weighted sum of the confidence values for each pixel. and the weighted sum can be used as an improved confidence value for the pixels in confidence map 120. In some embodiments, the aggregator process may, for each pixel, store the reliability values for the pixel in the first, second, and third reliability maps 508, 510, and 512 as three input values to the lookup table. The lookup table can output a single value that is used as an enhanced confidence value for a pixel in confidence map 120.

Referring to FIG. 1 , according to some embodiments of the present technology, the reliability processing module 110 may output a reliability map 120 to the encoder module 112 of the computer processor 10. The encoder module 112 is configured to receive the calibrated 2D images 114, the difference map 116 and the confidence map 120, encode the received information, and output a video stream 122 consisting of the encoded information. It can be configured. In some embodiments, video stream 122 may be provided to the vehicle's main system controller 30.

According to some embodiments of the present technology, video stream 122 may consist of 24-bit color depth video, with each 24-bit color value representing a range or distance relative to the scene captured by cameras 100. It can be encoded as much as possible. In some embodiments, a distance from 0 to approximately 16,800 meters may be represented in 24 bits, with each color representing a different 1 millimeter portion of the 16,800 meter range. In some embodiments, video stream 122 may be comprised of 2D frames made up of pixels, with each frame's pixel corresponding to a pixel in calibrated 2D image 114. Each pixel of each frame of video stream 122 may be encoded with an improved confidence value. For example, each enhanced confidence value could be an 8-bit unsigned value from 0 to 255, with relatively higher values indicating higher confidence levels. In some embodiments, using an 8-bit representation for the enhanced confidence values of confidence map 120 may enable confidence map 120 to be displayed as a grayscale image. In some embodiments, video stream 122 may consist of a 24-bit color video stream representing depth or distance, and an 8-bit monochrome video stream also representing reliability. Outputting a video stream 122 so that depth data can be separated from reliability data may, in some embodiments, facilitate sensor fusion, where different types of sensors (e.g., lidar, , ultrasound, cameras, etc.) can be combined to provide improved reliability, for example, in detecting objects and determining the vehicle's distances to the objects. For example, in foggy environments where objects within the scene may not be clearly visible in images captured by typical cameras, confidence values for pixels in the image may generally be low throughout the entire image. In such cases, the vehicle's control system (e.g., main system controller 30) may determine that the image is not reliable enough to be used.

According to some embodiments of the present technology, the computer processor 10 of the stereo vision system 1 may be configured with a LIDAR reliability processing module 124, and the sensors 100 illuminate the scene with laser light. and generate LiDAR image data (e.g., a video stream) and LiDAR reliability data from reflected light received from the scene, and output the LiDAR reliability map 126 to the encoder module 112. It can be composed of: Optionally, LiDAR image data may be output along with a LiDAR confidence map 126. In some embodiments, computer processor 10 may be configured with radar reliability processing module 128 and sensors 100 to illuminate the scene with waves of a known wavelength (e.g., 76.5 GHz) and , a radar configured to generate radar image data (e.g., a video stream) and radar reliability data from reflected waves reflected from the scene with known wavelengths, and output the radar reliability map 130 to the encoder module 112. It may consist of a sensor. Optionally, radar image data may be output along with a radar reliability map 130. In some embodiments, computer processor 10 may be configured with an acoustic reliability processing module 132 and sensors 100 may detect acoustic waves of a known wavelength (e.g., 20 kHz) (e.g., illuminate the scene with ultrasound waves), generate acoustic image data (e.g., a video stream) and acoustic reliability data from reflected waves reflected from the scene with known wavelengths, and generate an acoustic reliability map 134 into the encoder module 112. ) may be composed of a transducer configured to output. Optionally, acoustic image data may be output along with an acoustic reliability map 134. Techniques for determining reliability values for lidar image data, radar image data, and acoustic image data are described below.

According to some embodiments of the present technology, the lidar reliability processing module 124 receives the difference map 116 and the cost volume map 118 from the stereo matching module 108 and receives the reliability processing module 110 from the reliability processing module 110. Can be configured to receive map 120. Although not shown in FIG. 1, lidar confidence processing module 124 is also configured, in some embodiments, to receive one or both left and right corrected 2D images 114 from image correction module 106. It may be. The LiDAR reliability processing module 124 may use the reliability map 120 and other received information to perform comparisons and identify areas where the LiDAR reliability data has higher reliability values than corresponding areas of the reliability map 120. Some or all (i.e., one or both of difference map 116 and/or cost volume map 118 and/or calibrated 2D images 114) may be used. In some embodiments, LiDAR reliability processing module 124 may provide comparison information along with LiDAR reliability map 126 to encoder module 112. Similarly, in some embodiments, difference map 116, cost volume map 118, and confidence map 120 may be provided to radar reliability processing module 128, which may Together, they may be configured to provide comparison information to the encoder module 112. Likewise, in some embodiments, difference map 116, cost volume map 118, and confidence map 120 may be provided to acoustic reliability processing module 132, which compares them with acoustic confidence map 134. It may be configured to provide information to the encoder module 112.

According to some embodiments of the present technology, encoder module 112 uses confidence map 120 and one or more of lidar confidence map 126, radar confidence map 130, and acoustic confidence map 134. , to include depth information based only on sensor data acquired by cameras 100 or to include depth information based only on other sensor systems (e.g., lidar sensors, radar sensors, acoustic sensors, other camera pairs, etc.) A decision may be made whether to encode the video stream 122 or perform sensor fusion to combine sensor data derived from a plurality of different sensor systems on the vehicle. In some embodiments, encoder module 112 is configured to encode video stream 122 such that the image data, depth data, and/or reliability data within video stream 122 correspond to the highest reliability values determined from different sensor systems. It can be. In some embodiments, the frames of view stream 122 are one or more frames based on calibrated 2D images 114 and confidence map 120, followed by one based on LIDAR image data and LIDAR confidence map 126. It may consist of one or more frames, followed by one or more frames based on acoustic image data and acoustic reliability map 134. In some embodiments, each of one or more frames of video stream 122 may consist of a combination of data derived from different sensor systems. For example, each of the frames of video stream 122 may be comprised of four quadrants (e.g., top left, top right, bottom left, bottom right), with each quadrant having the highest overall confidence (e.g., It consists of sensor data with the highest average confidence level for the pixels in the quadrant. Therefore, foreign matter on one or both cameras 100 will cause the upper left quadrant of confidence map 120 to have the lowest overall confidence compared to the overall confidence of the vehicle's other sensor systems, but the other three quadrants of confidence map 120 If a quadrant has the highest overall confidence values, then the upper left quadrant of the corresponding frame of video stream 122 may be replaced with data from the sensor system with the highest overall confidence values. As can be appreciated, instead of optimizing the frames of video stream 122 on a frame-by-frame basis or on a quadrant basis, the frames are different ways to provide high-reliability estimates of the scene captured by sensors of different sensor systems. can be optimized. In some embodiments, when the vehicle is operating at night or in a dark environment, encoder module 112 may encode video stream 122 with, for example, lidar-based data instead of camera-based data. .

Figure 6 shows an example video stream 122 according to some embodiments of the present technology. Video stream 122 may consist of a sequence of frames, only three of which are shown: Frame N (600), Frame N+1 (602), and Frame N+2 (604). In some embodiments, each frame includes a left (or right) corrected 2D image 114 (color or grayscale), a difference map 116 (color or grayscale), and a confidence map 120 (color or grayscale). It may be a super frame 606 composed of grayscale). In some embodiments, super frame 606 may also include a velocity magnitude map, a horizontal velocity map (representing movement along the x-axis), a vertical velocity map (representing movement along the y-axis), and a line-of-sight velocity map (representing movement along the z-axis). may include one or more velocity maps, such as any one or any combination of (representing). Velocity maps can be computed by tracking matched points (or pixels) in dense optical flow over two or more frames and then determining the difference in difference map values of the matched points. In this way, the super frame 606 can be transmitted like any other video frame stream and thus advantageously utilize existing video technologies for encoding, decoding, transmission, etc. As will be appreciated, difference map 116 may instead be a depth map, since both types of maps relate to the distance from a sensor (e.g., cameras 100) to the scene captured by the sensor. This is because the maps of both types are typically inversely proportional to each other. In some embodiments, the “alpha channel” or transparency of each pixel of the calibrated 2D image 114 of each super frame may be used to store information. For example, a value for the confidence or certainty of the depth estimate for a pixel may be stored in the pixel's alpha channel.

7A-7F illustrate six types of super frames that may be included in video stream 122, according to some embodiments of the present technology. As will be appreciated, different types of super frames consist of different types of information and therefore may have different sizes. 7A shows a 2D Red Green Blue (RGB) color image 700 (e.g., left (or right) corrected 2D image 114), a depth map or difference map 700 (e.g., a difference map ( 116)), and a vertically connected reliability map 704 (e.g., reliability map 120). In some embodiments, each pixel in super frame 750 may use three bytes to describe the difference or range of the pixel, and three bytes may be used to describe the confidence level of the pixel. In some embodiments, each pixel of super frame 750 may be designated by 9 bytes. Figure 7b shows a 2D color image 706 and a super frame 752 consisting of a chain of difference maps and confidence maps. In some embodiments, each pixel in super frame 752 may use 2 bytes for the difference map and 1 byte for the confidence map, for a total of 3 bytes for the concatenation. In some embodiments, each pixel of super frame 752 may be designated by 6 bytes. 7C shows a super frame 754 consisting of a 2D color image formed of channels including a channel for the red image 710, a channel for the green image 712, and a channel for the blue image 714. . Super frame 754 also includes a 16-bit difference map consisting of high-order byte 716 and low-order byte 718, and a confidence map. In some embodiments, each pixel of super frame 754 may be designated by 6 bytes. FIG. 7D shows a super frame 756 consisting of a color image formed by a red image 722, a green image 724, and a blue image 726, a difference map 728, and a confidence map 730. In some embodiments, each pixel in difference map 728 may be designated by 5 bytes. FIG. 7E shows a super frame 758 consisting of a 2D grayscale image 732, a difference map 734, and a confidence map 736. In some embodiments, each pixel of super frame 758 may be designated by 3 bytes. 7F shows a super frame 760 consisting of a 2D color image 738, a difference map 740, and a confidence map 742. Each pixel of color image 738 may be designated by 1 byte. For example, color image 738 can be specified in a color palette by a Bayer filter pattern or lookup table, allowing super frame 760 to be as compact as an RGB color image. In some embodiments, each pixel of super frame 760 may be designated by 3 bytes.

As described above, in addition to providing a stereo-vision reliability map 120 to the encoder module 112 based on sensor data from cameras 100, a lidar reliability map 126, a radar reliability map ( 130), and an acoustic reliability map 134 may be provided to the encoder module 112. As described above, the lidar sensor, radar sensor, and acoustic sensor may have the same or similar fields of view as the cameras 100, or may have fields of view that overlap. In some embodiments, one or more of lidar reliability map 126, radar reliability map 130, and acoustic reliability map 134 may be attached to video stream 122. For example, LIDAR reliability map 126 (or other reliability map) is an individual bit within the reliability map of any of the super frames 750, 752, 754, 756, 758, and 760 of FIGS. 7A-7F. , or simply attached to a super frame, for example by vertical concatenation.

According to some embodiments of the present technology, when a LiDAR sensor is configured to provide a LiDAR return signal for each pixel of an image, a LiDAR reliability map (e.g., LiDAR reliability map 126) is It may consist of reliability values for each pixel of a 2D image (e.g., left (or right) corrected 2D image 114). In some embodiments, the LiDAR sensor may be configured to provide a LiDAR return signal for, for example, every other pixel or every other pixel or a predetermined group of pixels, which may be relatively sparse. But it can also still result in a useful lidar confidence map. If each pixel can be identified by its coordinates (i,j), the confidence value of pixel (i,j) can be denoted as m _ij . In some embodiments, the lidar confidence map may be a function of the 2D image 114 and difference map 116 according to:

For R≥R _min , m _ij = a ₀ R ^-2 (P + P ₀ ) ^-1/2

For R<R _min , m _ij = 0;

where R is the distance or range to the object at pixel (i,j) as measured by the difference map 116, and P is the pixel (from sunlight or other light sources) as measured by the 2D image 114. is the background light power in (i,j), a ₀ is the normalization constant, P ₀ is the fitting constant, and R _min is the minimum range of the lidar sensor. In some embodiments, the value of P may be, for example, the grayscale value of 2D image 114 at pixel (i,j) divided by the sensor gain and exposure values for cameras 100. . As described above, the 2D image and difference map used to generate the LIDAR confidence map may be the 2D image 114 captured by the cameras 100 and the difference map 116 determined based on stereo-vision processing. You can. In some embodiments, when an object within pixel (i,j) is closer than the minimum range of the LIDAR sensor, the confidence value for pixel (i,j) may be set to 0. The above expression for the confidence value m _ij is derived from an estimate of the lidar signal-to-noise ratio (SNR), which is inversely proportional to the square of the distance or range, i.e. R ² , and is also inversely proportional to the square root of the background light power P, or

SNR _lidar ∝ R ^-2 P ^-1/2 .

That is, the further an object is from the LIDAR sensor, the return signal decreases, and thus the accuracy of the LIDAR estimate decreases proportionally to the number of received photons corresponding to the return signal. Additionally, any background light (e.g., solar, rural, artificial) may cause “shot noise” to be detected by the LiDAR sensor, with noise energy equal to the square root of the background light power. Accordingly, the lidar confidence map 126 may be based on two physical quantities: the distance to the object within the pixel and the level of background light within the pixel.

Acoustic sensors, such as radar sensors and LiDAR sensors, can have SNRs proportional to R ^-2 P ^-1/2 . This characteristic may be used in a radar reliability map (e.g., radar reliability map 130) and/or an acoustic reliability map (e.g., For example, it can be used to calculate an acoustic reliability map 134).

According to some embodiments of the present technology, different sensor systems on a vehicle may be arranged to detect objects at different distance ranges. For example, Figure 8A schematically shows a beam area 800 illuminated by a radar sensor system or an acoustic sensor system. Beam area 800 may include a scene with multiple objects A, B, C, D at multiple distances or ranges R _A , R _B , R _C and R _D . The intensity of the return signal reflected from an object is proportional to the cross-sectional area of the object, which can be measured by the stereo vision system 1 (e.g., the range is based on the difference map 116, the area of the target is on the left (or right) given by the corrected 2D image 114). In some embodiments, return signals detected by the sensor system may be categorized into different bins based on signal strength. Figure 8b is a graph showing the relationship between the cross-sectional area and range for each of the objects (A, B, C, and D). The dashed line represents the noise floor 802 of the radar sensor system (or LiDAR sensor system). Noise floor 802 is a known quantity for radar and lidar technologies and may correspond to sensor sensitivity, which is known to degrade as a function of the square of distance. As described above, LIDAR SNR can be estimated as a value inversely proportional to the square of the distance or range. In some embodiments, LIDAR (or radar) reliability for a pixel may be determined as a value proportional to the SNR for the pixel. For example, the ratio of the cross-sectional area of an object that may span a plurality of pixels to the range-scaled value of the noise floor 802 is a value proportional to the SNR and a confidence value for the pixels in which the object appears. can be used

According to an exemplary embodiment of the present technology, the vehicle in which the stereo vision system 1 is installed may be a passenger car. Sensors 100 may be two cameras mounted on the upper left and upper right portions of the windshield of the vehicle (e.g., at the positions of sensors S3 and S7 in Figure 2) and are processed to produce a video stream. 122 may be configured to capture images that are converted into super frames, such as super frame 750 described above. RGB image 700 of super frame 750 may be parsed by an object detection module (e.g., a monocular object detector), which may be part of image preprocessing module 104, or stereo matching may be performed. It may be a separate module of the computer processor 10 that processes the image data of the captured images before (or after) the images are calibrated. For example, an object detection module may be configured to use a trained convolutional neural network to determine bounding boxes of various shapes in a scene of captured images. The shapes may correspond to categories of objects that the neural network is trained to recognize as typical objects that cars often encounter (e.g., other vehicles, pedestrians, cyclists, traffic signs, traffic lights, etc.). The difference map 702 of the super frame 750 may be averaged over the bounding boxes to determine the average distance to the detected objects. In addition, the ground plane or road surface is described in "Road Surface Segmentation based on Vertically Local Disparity Histogram for Stereo Camera" by S. Kakegawa et al. (Int'l J. ITS Res. vol. 16, pp. 90-97, 2018) Techniques using vertical difference histograms, and techniques using stochastic occupancy grids and dynamic programming described in "Free Space Computation Using Stochastic Occupancy Grids and Dynamic Programming" (Workshop on Dynamical Vision, ICCV, 2007) by H. Badino et al. It can be calculated from the difference map 702 using well-known techniques such as . From the road surface, unclassified road hazards (eg, motorcycle helmets, bricks, wooden pallets, loose cargo, mufflers, trees separated from tires, etc.) can be located and marked. Small unclassified objects on typical road surfaces often have well-defined edges and contour features, which can result in high confidence values in the confidence map 704, thus resulting in false positives and unnecessary automated emergency braking. can be prevented. In this example embodiment, when there is a foreign body (e.g., mud) on the windshield that blocks the lower left portion of the captured image of one or both cameras 100, the reliability processing module 110 The stereo matching module 108 repeatedly reports high costs in the lower left portion of the cost volume map 118 output by the stereo matching module 108 and therefore low confidence for the corresponding region of the confidence map 120. It can be detected that values can be provided. The vehicle's main system controller 30 may be configured to determine whether low confidence values are within an acceptable confidence threshold and, if not, control the computer processor 10 to invalidate data from the lower left region. can do. That is, in some embodiments, computer processor 10 may cause the computer processor 10 to To detect objects, to avoid false readings, and consequently to avoid erroneous control of the car (e.g. to prevent the car from being controlled to perform evasive maneuvers to avoid objects that are not actually in the scene). (To do this) can be controlled not to use data from the cameras 100 regarding the lower left area. Alternatively, as described above, data from other sensors on the vehicle may be used if the confidence values for the data meet a confidence threshold.

Referring back to Figure 3, according to some embodiments of the present technology, main system controller 30 includes a plurality of ECUs 34-1, 34-2, 34-3, configured to control various aspects of the operation of the vehicle. 34-4, ..., 34-n). For example, video stream 122 may be provided to a vehicle's display (not shown) and also to main system controller 30, which may use difference information and reliability information from video stream 122 to: It can be determined with high certainty (e.g., greater than 90% certainty) that red tail lights are ahead of the vehicle's path, which may indicate traffic congestion or other potential hazards. Thereafter, the main system controller 30 issues a control signal capable of controlling the vehicle's power train to the ECU 34-1, causing the vehicle to decelerate and stop. In some embodiments, processing by computer processor 10 and/or main system controller 30 continues until image data of the scene is captured by cameras 100 and delivered to image acquisition module 102. The display may be fast enough to show video corresponding to video stream 122 in real time or near real time (e.g., less than 2 seconds or less than 1 second). In some embodiments, ECU 34-1 may receive control signals in real time or near real time as image data of a scene is captured by cameras 100 and delivered to image acquisition module 102. As described above, communication between the main system controller 30 and the ECUs 34-1, 34-2, 34-3, 34-4, ..., 34-n may be via techniques. For example, the command and control line 32 may be a network or bus mounted on the vehicle and is connected to the main system controller 30 using known protocols (e.g., Ethernet, CAN, I2C, LIN, etc.). Signals can be transmitted between ECUs (34-1, 34-2, 34-3, 34-4, ..., 34-n).

9A and 9B show examples of data resulting from images of a scene captured by two 5-megapixel cameras and processed according to various embodiments of the technology described above. The two cameras are mounted on the roof of a standard passenger vehicle, separated by a 1.2-m baseline. Figure 9A shows a color difference map mixed with a grayscale camera image (not shown) so that both distance and luminance of the scene can be viewed simultaneously. The difference map has values from 0 to 1023 encoded as a jet colormap, where red colors represent relatively close objects and blue colors represent relatively more distant objects. The transparency of the difference map is modulated by the grayscale camera image. Figure 9b shows the corresponding confidence map of the scene. The confidence map is the minimum aggregation for each pixel from a semi-global matching algorithm as described in "Stereo Processing by Semi-Global Matching and Mutual Information" by H. Hirschmuller (2008 IEEE Transactions on Pattern Analysis and Machine Intelligence). Shows cost values.

According to some embodiments of the present technology, the processing modules and components of FIGS. 1, 3, and 5 may include hardware (e.g., one or more computer processor(s) programmed to perform the procedures and methods described above) and/ or may be implemented in software (e.g., computer-executable code). In some embodiments, at least a portion of the software may be pre-programmed in the computer processor(s). In some embodiments, at least a portion of the software may be stored on a non-transitory computer-readable storage medium or a plurality of non-transitory computer-readable storage media, which may be accessed and executed by a computer processor(s). In some embodiments, a process module may be a software module, a hardware module, or a module combining hardware and software.

Vehicle assistance systems according to the technology described herein may be implemented in different configurations. Exemplary configurations include combinations of configurations (1) through (21) as follows:

(1) An automated vehicle assistance system for supervised or unsupervised vehicle movement, comprising: a vehicle control system comprising a computer processor and a memory coupled to the computer processor; a first sensor system configured to receive first image data of a scene, and output a first difference map and a first confidence map based on the first image data, wherein the vehicle control system receives first image data from the first sensor system; configured to receive a difference map and a first confidence map, and output a video stream comprised of the first difference map and the first confidence map.

(2) In the system of configuration (1), in the video stream, the first confidence map is encoded to be part of the first difference map.

(3) In the system of configuration (1) or configuration (2), the first image data is composed of a plurality of pixels, the difference map is composed of difference data for each of the pixels, and the confidence map is composed of difference data for each of the pixels. It consists of reliability data for

(4) In the system of any one of configurations (1) to (3), the first image data consists of data for left and right two-dimensional (2D) first images, and the first sensor system includes the first image data. configured to generate left and right (2D) corrected first images and a first cost volume map from the 1 image data, wherein the first sensor system is configured to generate the 2D corrected first images, the first difference map, and the first cost volume map. and configured to generate a first reliability map from the volume map.

(5) The system of any of configurations (1) to (4), wherein the first sensor system is configured to generate the first reliability map based on one or both of the following: Semi-global matching A uniqueness value determined from the (SGM) algorithm and an image texture value determined from a Sobel operation on the first image data.

(6) The system of any one of configurations (1) to (5), configured to receive second image data of at least a portion of the scene and output a second reliability map based on the second image data. further comprising a second sensor system, wherein the vehicle control system receives a second reliability map from the second sensor system, and outputs the video stream as a sequence of super frames, each super frame comprising a first difference map, a first reliability map, -consists of information based on the map and the second confidence map.

(7) In the system of any one of configurations (1) to (6), the vehicle control system is configured to output a control signal to the electronic control unit (ECU) of the vehicle based on information in the video stream.

(8) The system of any one of configurations (1) to (7), wherein the first sensor system is configured to process the first image data to generate a first difference map and a first confidence map. A sensor module, the second sensor system is a second sensor module configured to process the second image data to generate a second reliability map, the first and second sensor modules are stored in a memory, and the computer processor is configured to process the second image data to generate a second reliability map, and a second sensor module.

(9) In the system of any one of configurations (1) to (8), the video stream includes at least one super frame composed of a first difference map and a first reliability map, and a first difference map and a second reliability map. It consists of at least one super frame consisting of a map.

(10) In the system of any one of configurations (1) to (9), the video stream is composed of at least one super frame composed of a portion of the first reliability map and a portion of the second reliability map.

(11) In the system of any one of configurations (1) to (10), where the first image data is composed of stereo-vision data, and the second image data is composed of lidar data.

(12) The system of any one of configurations (1) to (11), configured to receive third image data of at least a portion of a scene and output a third reliability map based on the third image data. It further includes a third sensor system.

(13) In the system of any one of configurations (1) to (12), the third image data consists of radar data or acoustic data.

(14) The system of any one of configurations (1) to (13), wherein each super frame of the video stream is composed of a two-dimensional (2D) image of the scene, a depth map of the scene, and a certainty map of the scene. .

(15) In the system of any one of configurations (1) to (14), the certainty map of the scene is composed of a first confidence map or a second confidence map or a combination of the first and second confidence maps.

(16) The system of any one of configurations (1) to (15), wherein the depth map of the scene is comprised of a first difference map modulated with image data corresponding to a 2D image of the scene, and a certainty map of the scene. consists of a first confidence map or a second confidence map, or a combination of first and second confidence maps modulated with image data corresponding to a 2D image of the scene.

(17) The system of any one of configurations (1) to (16), wherein pixels of the 2D image of the scene, pixels of the depth map of the scene, and pixels of the certainty map of the scene are temporally and spatially It matches.

(18) In the system of any one of configurations (1) to (17), the vehicle control system includes difference information from the first difference map and the first and second confidence maps to reduce the data size of the video stream. configured to encode reliability information from.

(19) The system of any one of configurations (1) to (18), further comprising a pair of cameras configured to be mounted on a vehicle, wherein the cameras are configured to provide first image data to the first sensor system. do.

(20) In the system of any one of configurations (1) to (19), the video stream is composed of two-dimensional (2D) color images, each 2D color image is composed of a plurality of pixels, and each A pixel's alpha-channel transparency is proportional to the confidence value for that pixel.

(21) In the system of any one of configurations (1) to (20), the colors of the 2D color images represent depth ranges.

A non-transitory computer-readable storage medium stores code that, when executed by a computer processor, causes the computer processor to perform methods of an automated vehicle assistance system for supervised or unsupervised vehicle movement according to the techniques described herein. It can be configured to do so. Examples of such computer-readable storage media include combinations of configurations (22) through (34) as follows:

(22) A non-transitory computer-readable storage medium storing code, wherein the code, when executed by a computer processor, causes the computer processor to perform a method of an automated vehicle assistance system for supervised or unsupervised vehicle movement, The method includes obtaining, by a computer processor, a first difference map and a first confidence map, wherein the first difference map and the first confidence map correspond to first image data of the scene, and obtaining, by the computer processor, a first difference map and a first confidence map. It consists of outputting a video stream composed of a reliability map.

(23) The computer-readable storage medium of configuration (22), wherein outputting the video stream comprises, by the computer processor, encoding the first confidence map to be part of the first difference map.

(24) The computer-readable storage medium of configuration (22) or (23), wherein the first image data is comprised of a plurality of pixels, the difference map is comprised of difference data for each of the pixels, and the reliability The map consists of reliability data for each pixel.

(25) The computer-readable storage medium of any one of configurations (22) through (24), wherein the method comprises: the computer processor obtaining a second confidence map corresponding to second image data of at least a portion of the scene; The computer processor outputs the video stream as a sequence of super frames, each super frame consisting of information based on a first difference map, a first confidence map, and a second confidence map.

(26) In the computer-readable storage medium of any one of configurations (22) to (25), the method includes the computer processor outputting a control signal to the electronic control unit (ECU) of the vehicle based on information in the video stream. It consists of additional steps:

(27) The computer-readable storage medium of any one of configurations (22) to (26), wherein the method includes: the computer processor processing the first image data to obtain a first difference map and a first confidence map; The computer processor further comprises processing the second image data to obtain a second confidence map.

(28) In the computer-readable storage medium of any one of configurations (22) to (27), outputting the video stream causes the computer processor to display at least one super file including a first difference map and a first reliability map. Preparing a frame, and having a computer processor prepare at least one super frame including a first difference map and a second confidence map.

(29) In the computer-readable storage medium of any one of configurations (22) to (28), the step of outputting the video stream comprises at least one of a computer processor configured to include a portion of a first reliability map and a portion of a second reliability map. It consists of the steps of preparing a super frame.

(30) The computer-readable storage medium of any one of configurations (22) to (29), wherein the first image data consists of stereo-vision data and the second image data is lidar data or radar. It consists of data or sound data.

(31) In the computer-readable storage medium of any one of configurations (22) to (30), the step of outputting the video stream consists of preparing each super frame of the video stream by the computer processor, A frame consists of a two-dimensional (2D) image of the scene, a depth map of the scene, and a certainty map of the scene.

(32) In the computer-readable storage medium of any one of configurations (22) to (31), preparing each super frame by a computer processor includes pixels of a 2D image of the scene, a depth map of the scene, It consists of temporally and spatially matching pixels, and pixels of a certainty map of a scene.

(33) The computer-readable storage medium of any one of configurations (22) to (32), wherein the step of outputting the video stream includes the computer processor generating difference information from the first difference map and the first and second confidence maps. It consists of reducing the data size of the video stream by encoding reliability information from the video stream.

(34) The computer-readable storage medium of any one of configurations (22) to (33), wherein the step of outputting the video stream includes preparing a two-dimensional (2D) color image, and each 2D color image consists of a plurality of pixels, the alpha-channel transparency of each pixel is proportional to the confidence value for the pixel, and the color of the 2D color image represents the depth range.

Stereo vision systems according to the technology described herein may be implemented in different configurations. Exemplary configurations include combinations of configurations 35-46 as follows:

(35) A stereo vision system, comprising: a stereo camera system configured to capture a sequence of pairs of images, each pair of images including a first image and a second image captured simultaneously; comprising a computer processor, the computer processor receiving a stream of image data from the stereo camera system, wherein the image data corresponds to a sequence of pairs of images, and for each pair of images: a two-dimensional (2D) pixel map of matched pixels. Calibrate the first and second images to generate, determine a depth value for each pixel of the pixel map, determine a confidence value for the depth value for each pixel of the pixel map, and determine at least one of the confidence values. It is programmed to issue a control signal when one exhibits an image abnormality.

(36) The system of configuration (35), wherein the image anomaly corresponds to one or more pixels of a portion of the confidence map having a confidence value less than a predetermined threshold.

(37) The system of configuration (35) or configuration (36), wherein the image anomaly is one or more pixels that are part of a confidence map having a confidence value less than a predetermined threshold for two or more consecutive image pairs in the sequence. corresponds to

(38) In the system of any one of configurations (35) to (37), the image anomaly is composed of a plurality of pixels in a continuous area of the confidence map.

(39) In the system of any one of configurations (35) to (38), the control signal is configured to cause an audible sound.

(40) In the system of any one of configurations (35) through (39), the audible sound is a pre-recorded message.

(41) In the system of any one of configurations (35) to (40), a control signal is issued to the engine control module of the vehicle.

(42) The system of any of configurations (35) to (41), wherein, for each pixel in the pixel map, the confidence value is determined by the presence or absence of an edge in the pixel, the illuminance level of the pixel, and the pixel map. It is determined based on the texture values of the generated first and second images.

(43) The system of any of configurations (35) to (42), wherein the computer processor is programmed to output a sequence of super frames corresponding to the sequence of pairs of images, each of the super frames being a 2D image and It consists of a confidence map corresponding to a 2D image.

(44) In the system of any one of configurations (35) to (43), the 2D image is the first image or the second image.

(45) The system of any of configurations (35) through (44), wherein the computer processor outputs a sequence of super frames as a display signal to cause the display to show a 2D image and a visible confidence indicator corresponding to the confidence map. It is programmed to output.

(46) The system of any of configurations (35) to (45), wherein the display signal causes the confidence indicator to be displayed on a pixel-by-pixel basis as the transparency of each pixel of the 2D image.

A non-transitory computer-readable storage medium may be configured to store code that, when executed by a computer processor, causes the computer processor to perform methods of a vehicle assistance system according to the technology described herein. Examples of such computer-readable storage media include combinations of configurations (47) through (59) as follows:

(47) A non-transitory computer-readable storage medium storing code, wherein the code, when executed by a computer processor, causes the computer processor to perform a method of a stereo vision system, the method comprising: receiving a stream of, wherein the image data corresponds to a sequence of pairs of images, each image pair consisting of a first image and a second image captured simultaneously; For each pair of images, a computer processor calibrates the first and second images to generate a two-dimensional (2D) pixel map of the matched pixels, determines a depth value for each pixel of the pixel map, and generates a pixel map. determining a confidence value for a depth value for each pixel in; and the computer processor issuing a control signal when at least one of the confidence maps indicates an image abnormality.

(48) The computer-readable storage medium of configuration (47), wherein the image anomaly corresponds to one or more pixels of a portion of the confidence map having a confidence value less than a predetermined threshold.

(49) The computer-readable storage medium of configuration (47) or configuration (48), wherein the image anomaly is a portion of the confidence map having a confidence value less than a predetermined threshold for two or more consecutive image pairs of the sequence. Corresponds to one or more pixels.

(50) The computer-readable storage medium of any one of configurations (47) to (49), wherein the image anomaly is comprised of a plurality of pixels in a continuous region of a confidence map.

(51) The computer-readable storage medium of any one of configurations (47) to (50), wherein the control signal is configured to cause an audible sound.

(52) The computer-readable storage medium of any one of configurations (47) to (51), wherein the audible sound is a pre-recorded message.

(53) In the computer-readable storage medium of any one of configurations (47) to (52), a control signal is issued to an engine control module of a vehicle.

(54) The computer-readable storage medium of any of configurations (47) to (53), wherein, for each pixel of the pixel map, the reliability value includes the presence or absence of an edge within the pixel, the illuminance level of the pixel, and a pixel map is determined based on the texture values of the first and second images from which the pixel map is generated.

(55) The computer-readable storage medium of any one of configurations (47) to (54), wherein the method includes outputting, by a computer processor, a sequence of super frames corresponding to the sequence of image pairs—each of the super frames. is further composed of - containing a 2D image, a difference map corresponding to the 2D image, and a confidence map corresponding to the 2D image.

(56) In the computer-readable storage medium of any one of configurations (47) to (55), the 2D image is the first image or the second image.

(57) The computer-readable storage medium of any one of configurations (47) through (56), wherein outputting the sequence of super frames comprises displaying a visible confidence indicator corresponding to the 2D image and a confidence map. It consists of the step of outputting a display signal to display.

(58) The computer-readable storage medium of any one of configurations (47) to (57), wherein the display signal causes the reliability indicator to be displayed on a pixel-by-pixel basis as the transparency of each pixel of the 2D image.

(59) In the computer-readable storage medium of any one of configurations (47) to (58), each of the super frames is composed of a 2D image, a confidence map, and a difference map corresponding to the 2D image.

References

Mohd Saad Hamid, Nurul Fajar Abd Manap, Rostam Affendi Hamzah, and Ahmad Fauzan Kadmin, "Stereo matching algorithm based on deep learning: A survey," Journal of King Saud University - Computer and Information Sciences, 2020, (available at: https: //doi.org/10.1016/j.jksuci.2020.08.011).

H. Hirschmuller, “Stereo Processing by Semiglobal Matching and Mutual Information,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 328-341, Feb. 2008 (available at https://core.ac.uk/download/pdf/11134866.pdf).

Matteo Poggi, Seungryong Kim, Fabio Tosi, Sunok Kim, Filippo Aleotti, Dongbo Min, Kwanghoon Sohn, Stefano Mattoccia, "On the confidence of stereo matching in a deep-learning era: a quantitative evaluation," IEEE Trans Pattern Analysis & Machine Intelligence , April 2021 (available at https://arxiv.org/pdf/2101.00431.pdf).

H. Hirschmuller and D. Scharstein, “Evaluation of Cost Functions for Stereo Matching,” IEEE Conference on Computer Vision and Pattern Recognition, 2007, pp. 1-8.

S. Sarika et al., “Census Filtering Based Stereomatching Under Varying Radiometric Conditions,” Procedia Computer Science, vol. 58, pp. 315-320, 2015.

C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, "Fast cost-volume filtering for visual correspondence and beyond," Conference on Computer Vision and Pattern Recognition 2011, pp. 3017-3024.

X. Hu and P. Mordohai, “A Quantitative Evaluation of Confidence Measures for Stereo Vision,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 11, pp. 2121-2133, 2012.

S. Kakegawa, H. Matono, H. Kido, and T. Shima, "Road Surface Segmentation based on Vertically Local Disparity Histogram for Stereo Camera," International Journal of Intelligent Transportation Systems Research, vol. 16, pp. 90-97, 2018 (available at https://doi.org/10.1007/s13177-017-0140-8 and https://link.springer.com/content/pdf/10.1007/s13177-017-0140-8. pdf).

H. Badino, U. Franke, R. Mester, "Free Space Computation Using Stochastic Occupancy Grids and Dynamic Programming," Workshop on Dynamical Vision, ^11th IEEE International Conference on Computer Vision, ICCV, 2007, (available at http:// vision.jhu.edu/iccv2007-wdv/WDV07-badino.pdf).

conclusion

It should be understood that various changes, modifications, and improvements may be made to the structures, configurations, and methods discussed above and are intended to remain within the spirit and scope of the invention disclosed herein. Additionally, while advantages of the invention have been indicated, it should be understood that not all embodiments of the invention include all of the advantages described. Some embodiments may not implement any features described as advantageous herein. Accordingly, the foregoing description and accompanying drawings are examples only.

It should be understood that some aspects of the present technology may be implemented in more than one method, and that the operations performed as part of the method of the present technology may be arranged in any suitable manner. Accordingly, embodiments may be configured in which the operations are performed in a different order than that shown and/or described, including performing some operations simultaneously although they are shown and/or described as sequential operations in various embodiments. can do.

The various aspects of the invention can be used alone, in combination, or in various arrangements not specifically discussed in the embodiments described in the foregoing, and thus may be used in their applications as set forth in the foregoing description or illustrated in the drawings. It is not limited to the details and arrangement of components. For example, aspects described in one embodiment may be combined in any way with aspects described in other embodiments.

The use of ordinal terms such as “first,” “second,” “third,” etc. in the description and claims to modify elements does not by itself imply any priority, precedence, or priority of one element over another. It does not imply the order, or temporal order in which the operations of the method are performed. However, they are only used as marks to distinguish elements or actions, distinguishing one element or action with a particular name from another element or action with the same name (but using an ordinal term).

All definitions, as defined and used herein, are to be construed to take precedence over dictionary definitions, definitions in documents incorporated by reference, and/or ordinary meanings of defined terms.

As used in this specification and claims, the indefinite articles “a” and “an” are to be understood to mean “at least one,” unless clearly indicated otherwise.

As used in this specification and the claims, with respect to a list of one or more elements, the phrase “at least one” means at least one element selected from any one or more of the elements in the list of elements, but It should be understood that it does not necessarily include at least one of each and every element specifically listed within the list, nor does it exclude any combination of elements within the list of elements. This definition also allows that elements other than those specifically identified within the list of elements referred to by the phrase "at least one" may optionally be present, whether or not related to the specifically identified elements. do.

As used in the specification and claims, the phrase “equal” or “equal” with respect to two values (e.g., distances, widths, etc.) indicates that the two values are identical within manufacturing tolerances. it means. Accordingly, two values being the same or equal may mean that the two values differ from each other by ±5%.

As used in the specification and claims, the phrase “and/or” refers to “either or both” of the elements so combined, that is, elements that exist in combination in some cases and separately in other cases. It must be understood as meaning. Multiple elements listed as “and/or” should be interpreted in the same way, i.e., as “one or more” of the elements so combined. Other elements, other than those specifically identified by the “and/or” clause, may optionally be present, whether related or unrelated to those specifically identified elements. Thus, by way of non-limiting example, reference to “A and/or B”, when used with open language such as “comprising,” means that, in one embodiment, only A (optionally including elements other than B) box); In another embodiment, B only (optionally including elements other than A); In another embodiment, it may refer to both A and B (optionally including other elements), etc.

As used in this specification and the claims, “or” should be understood to have the same meaning as “and/or” as defined above. For example, when separating items in a list, "or" or "and/or" is inclusive, that is, includes at least one of a number of elements or a list of elements, but also includes two or more. , optionally, should be construed to include additional unlisted items. Only clearly alternative terms such as “only one of” or “exactly one of” or “consisting of” when used in the claims indicate the inclusion of exactly one element of a plurality of elements or a list of elements. will refer to Generally, the term "or" as used refers to exclusive alternatives ( That is, it should be interpreted as indicating “one or the other, but not both.” When used in the claims, “consisting essentially of” will have its ordinary meaning when used in the field of patent law.

Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. As used herein, “including,” “comprising,” “comprised of,” “having,” “containing,” and “involving.” The use of terms such as and variations thereof are intended to include additional items as well as the items listed thereafter and their equivalents.

The terms “approximately” and “about” when used herein mean, in some embodiments, within ±20% of the target value, in some embodiments, within ±10% of the target value, and in some embodiments, within ±20% of the target value. It can be interpreted to mean within ±5%, and in some embodiments, within ±2% of the target value. The terms “approximately” and “about” may be equivalent to a target value.

The term “substantially” when used herein means within 95% of the target value in some embodiments, within 98% of the target value in some embodiments, within 99% of the target value in some embodiments, and in some embodiments. In embodiments, it may be interpreted to mean within 99.5% of the target value. In some embodiments, the term “substantially” can be equal to 100% of the target value.

Claims (21)

A vision system for supervised or unsupervised movement of a vehicle, comprising:
a pair of camera sensors configured to capture first image data of the scene;
an auxiliary sensor configured to capture second image data of the scene; and
sensor system
It includes, and the sensor system includes,
Receive the first image data from the pair of camera sensors,
Receive the second image data from the auxiliary sensor,
Calculate first distance values and first confidence values from the first image data,
Generating a first distance map including the first distance values, generating a first confidence map including the first confidence values,
Generating a second distance map including second distance values obtained from the second image data,
Using the first confidence map, fuse the first distance map with the second distance map to obtain a fused distance map, and
The vision system is configured to output a first control signal to an electronic control unit (ECU) of the vehicle to control the vehicle based on the fused distance map. According to paragraph 1,
The sensor system, at least in part, fuses the first distance map with the second distance map,
determine whether a low confidence portion is included in the first confidence map, wherein the low confidence portion includes at least one confidence value below a predetermined threshold, and
If it is determined that the first confidence map includes the low confidence portion, the fused map is generated by replacing the portion of the first distance map corresponding to the low confidence portion with a replacement portion obtained from the second distance map. By creating,
A vision system configured to obtain a fused distance map. According to paragraph 1,
and the first control signal is output in real time or near real time until the scene is captured by the pair of camera sensors and the auxiliary sensor. According to paragraph 1,
The auxiliary sensor is,
radar sensor,
lidar sensor,
a secondary camera sensor, and
acoustic sensor
A vision system, comprising any one or any combination thereof. The vision system of claim 1, wherein the sensor system is configured to determine a confidence value for each pixel of the first image data. The vision system of claim 1, wherein the sensor system is configured to determine a confidence value for each region of the plurality of regions of the first image data. The vision system of claim 1, wherein the fused distance map includes the entire second distance map. 1. A method of a vision system for supervised or unsupervised movement of a vehicle, comprising:
Receiving captured first image data of a scene from a pair of camera sensors on the vehicle;
receiving captured second image data of the scene from an auxiliary sensor on the vehicle;
calculating first distance values and first confidence values from the first image data;
generating a first distance map including the first distance values, and generating a first confidence map including the first confidence values;
generating a second distance map including second distance values obtained from the second image data;
using the first reliability map to fuse the first distance map with the second distance map to obtain a fused distance map; and
outputting a first control signal to an electronic control unit (ECU) of the vehicle to control the vehicle based on the fused distance map.
How to include . The method of claim 8, wherein fusing the first distance map with the second distance map to obtain a fused distance map comprises:
determining whether a low confidence portion is included in the first confidence map, wherein the low confidence portion includes at least one confidence value below a predetermined threshold, and
If it is determined that the first confidence map includes the low confidence portion, the fused distance map is generated by replacing the portion of the first distance map corresponding to the low confidence portion with a replacement portion obtained from the second distance map. Steps to create
Method, including. According to clause 8,
wherein the first control signal is output to a vehicle control system in real time or near real time until the scene is captured by the pair of camera sensors and the auxiliary sensor. The method of claim 8, wherein the auxiliary sensor is:
radar sensor,
lidar sensor,
a secondary camera sensor, and
acoustic sensor
A method comprising any one or any combination of the following. 9. The method of claim 8, wherein the calculating step includes determining a confidence value for each pixel of the first image data. The method of claim 8, wherein the calculating step includes determining a confidence value for each region of the plurality of regions of the first image data. 9. The method of claim 8, wherein the fused distance map includes the entire second distance map. A non-transitory computer-readable storage medium storing code, the code, when executed by a computer processor, causing the computer processor to perform a method of a vision system for supervised or unsupervised movement of a vehicle, the method comprising: ,
calculating first distance values and first confidence values from first image data of a scene captured by a pair of cameras on the vehicle;
generating a first distance map including the first distance values, and generating a first confidence map including the first confidence values;
generating a second distance map including second distance values obtained from second image data captured of the scene by an auxiliary sensor on the vehicle;
using the first reliability map to fuse the first distance map with the second distance map to obtain a fused distance map; and
outputting a first control signal to an electronic control unit (ECU) of the vehicle to control the vehicle based on the fused distance map.
A non-transitory computer-readable storage medium comprising: According to clause 15,
The step of fusing the first distance map with the second distance map to obtain a fused distance map includes:
determining whether a low confidence portion is included in the first confidence map, wherein the low confidence portion includes at least one confidence value below a predetermined threshold, and
If it is determined that the first confidence map includes the low confidence portion, the fused distance map is generated by replacing the portion of the first distance map corresponding to the low confidence portion with a replacement portion obtained from the second distance map. Steps to create
A non-transitory computer-readable storage medium comprising: According to clause 15,
wherein the first control signal is output in real time or near real time until the scene is captured by the pair of camera sensors and the auxiliary sensor. According to clause 15,
The auxiliary sensor is,
radar sensor,
lidar sensor,
a secondary camera sensor, and
acoustic sensor
A non-transitory computer-readable storage medium comprising any one or any combination of the following. According to clause 15,
The calculating step includes determining a confidence value for each pixel of the first image data. According to clause 15,
The calculating step includes determining a reliability value for each region of the plurality of regions of the first image data. A vision system for supervised or unsupervised movement of a vehicle, comprising:
a pair of camera sensors configured to capture first image data of the scene;
an auxiliary sensor configured to capture second image data of the scene; and
sensor system
It includes, and the sensor system includes,
Receive the first image data from the pair of camera sensors,
Receive the second image data from the auxiliary sensor,
Calculate first distance values and first confidence values from the first image data,
Generating a first distance map including the first distance values, generating a first confidence map including the first confidence values,
generate a second distance map including second distance values obtained from the second image data, and
The vision system is configured to output a first control signal to an electronic control unit (ECU) of the vehicle to control the vehicle based on the first confidence values, the first distance values, and the second distance values.

Images