EP2005361A1 - Multi-sensorial hypothesis based object detector and object pursuer - Google Patents

Abstract

The invention relates to a method for multi-sensorial object detection, wherein sensor information is evaluated together from several different sensor signal flows having different sensor signal properties. For said evaluation, the at least two sensor signal flows are not adapted to each other and/or projected onto each other, but object hypotheses are generated in each of the at least two sensor signal flows and characteristics for at least one classifier are generated based of said object hypotheses. Said object hypotheses are subsequently evaluated by means of a classifier and are associated with one or more categories. At least two categories are identified and the object is associated with one of the two categories.

Description

 Multi-sensory hypothesis-based object detector and object tracker

The invention relates to a method for multi-sensor object detection.

The computer-based evaluation of sensor signals for object detection and object tracking is already known from the prior art. For example, driver assistance systems for road vehicles are available, which detect and track vehicles ahead by means of radar, for example, to automatically regulate the speed and the distance of the own vehicle to the preceding traffic. In addition, various types of sensors, such as radar, laser and camera sensors are already known for use in the vehicle environment. These sensors are very different in their properties and have different advantages and disadvantages. For example, such sensors differ in their resolution or in the spectral sensitivity. It would therefore be particularly advantageous if several different sensors would be used simultaneously in a driver assistance system. A multi-sensorial use is currently hardly possible, however, since variables detected by means of different types of sensors can be directly compared or combined in a suitable manner only with considerable effort in the evaluation of the signal. In the systems known from the prior art, therefore, the individual sensor currents are first matched to each other before they are fused together. For example, the images of two cameras with different resolving power are first of all imaged in pixel-precise manner in a complex manner and only then fused together.

The invention is therefore based on the object to provide a method for multi-sensor object detection, which objects can be detected and tracked in a simple and reliable manner.

The object is achieved according to the invention by a method having the features of patent claim 1. Advantages Embodiments and developments are shown in the subclaims.

According to the invention, a method for multi-sensor object detection is provided, wherein sensor information from at least two different sensor signal currents with different

Sensor signal properties are used for joint evaluation. The sensor signal currents are not adapted to each other for evaluation and / or imaged each other. On the basis of the at least two sensor signal streams, object hypotheses are first of all generated and, on the basis of these object hypotheses, features for at least one classifier are then generated. The object hypotheses are then evaluated by means of the at least one classifier and assigned to one or more classes. At least two classes are defined, one of the two classes being assigned objects. With the method according to the invention is thus a simple and Reliable object recognition only possible. An elaborate adaptation of different sensor signal currents to one another or an image on each other is thereby completely eliminated in a particularly profitable manner. In the context of the method according to the invention, the sensor information from the at least two sensor signal streams is combined directly with one another or fused together. This considerably simplifies the evaluation and enables shorter calculation times. The fact that no additional steps for the adaptation of the individual sensor signal currents are needed, the number of possible sources of error in the evaluation is minimized.

The object hypotheses can either be clearly assigned to a class, or they are assigned to several classes, the respective assignment is occupied with a probability.

In a profitable manner, the object hypotheses are independently generated individually in each sensor signal stream, the object hypotheses of different sensor signal currents are then assigned to each other via assignment rules. First, the object hypotheses are generated in each sensor signal stream by means of search windows in a previously defined 3D state space, which is defined by physical variables. Due to the defined 3D state space, the object hypotheses generated in the individual sensor signal streams can later be assigned to one another. For example, the object hypotheses from two different sensor signal streams are paired later in the subsequent classification, forming an object-hypothesis-out-of-a-search-window pair if there are more than two sensor signal streams Accordingly, from each sensor signal stream in each case a search window used and formed from an object hypothesis, which is then passed to the classifier for joint evaluation. The physical quantities for spanning the 3D state space can be, for example, one or more component (s) of the object extent, a speed and / or acceleration parameter, a time, etc. The state space can also be made higher dimensional.

In another profitable way of the invention, object hypotheses are generated in a sensor signal stream (primary stream) and the object hypotheses of the primary stream are then projected into other image streams (secondary streams), wherein an object hypothesis of the primary stream generates one or more object hypotheses in the secondary stream. When using a camera sensor, the object hypotheses in the primary stream are generated, for example, by means of a search window within the image recordings recorded by means of the camera sensor. The object hypotheses generated in the primary stream are then computationally projected into one or more other sensor streams. In a further advantageous manner, the projection of object hypotheses of the primary current into a secondary current is based on the sensor models used and / or the positions of search windows within the primary current or on the epipolar geometry of the sensors used. Projection can also create ambiguity in this context. An object hypothesis / search window of the primary stream generates, for example, due to different object distances of the individual sensors, several -Ob ^~ j ^~ e ^~ kthypot-hesen - / - Suchfenst_e_r in the secondary stream. The object hypotheses generated with it are then preferably pass in pairs to the classifier. In each case, pairs are formed from the object hypothesis of the primary stream and in each case one object hypothesis of the secondary stream and then transferred to the classifier. However, there is also the possibility that in addition to the object hypothesis of the primary flow, all object hypotheses or parts thereof generated in the secondary flows are also passed to the classifier.

In the context of the invention, object hypotheses are profitably determined by their object type, object position, object extent, object orientation, object motion parameters such as direction of motion and velocity,

Object hazard potential or any combination thereof. It can also be any other parameter that describes the object properties. For example, an object associated speed and / or acceleration values. This is particularly advantageous if the inventive method is used in addition to the object recognition in addition to the object tracking and the evaluation includes tracking.

In a further advantageous manner of the invention object hypotheses are randomly scattered in a physical search space or generated in a grid. For example, search windows are varied with a predetermined step size within the search space using a grid. However, there is also the possibility that search windows are used only within predetermined regions of the state space at which objects occur with high probability and thus object hypotheses are generated. In addition, the object hypotheses in a physical search space can also be determined by a physical search space Model originated. The search space may be adaptively constrained by external constraints such as aperture angles, range ranges, statistical characteristics obtained locally in the image, and / or measurements from other sensors.

In the context of the invention, the different sensor signal properties in the sensor signal currents are based essentially on different positions and / or orientations and / or sensor variables of the sensors used. In addition to position and / or orientation deviations or individual components thereof, deviations in the sensor variables used mainly cause different sensor signal properties in the individual sensor signal currents. For example, camera sensors having a different resolving power cause differences in sizes in image capturing. Also, due to different camera optics, different sized image areas are often detected. Furthermore, e.g. the physical properties of the camera chips can be completely different, so that, for example, one camera captures environmental information in the visible wavelength spectrum and another camera acquires environmental information in the infrared spectrum, wherein the image recordings can have a completely different resolution.

In the context of the evaluation is advantageously the possibility that each object hypothesis is individually classified for themselves and the results of the individual classifications are combined, at least one classifier is provided. If several classifiers are used, e.g. for each different

Art-of-jbj-ekt - j-ewe-i-1-s-a-K-lassif-ikat.or viox.ge.sjehen.

If only one classifier is provided first classify each object hypothesis by means of the classifier and then combine the results of several individual classifications into one overall result. For this purpose, the expert in the field of pattern recognition and classification different evaluation strategies are known. In a further advantageous manner of the invention, however, it is also possible for features of object hypotheses of different sensor signal currents to be jointly evaluated in the at least one classifier and combined to form a classification result. For the reliable recognition of a specific object, a predetermined number of object hypotheses, for example, must achieve a minimum probability in class membership of this particular object class. Also, the expert in the field of pattern recognition and classification in this context a wide variety of evaluation strategies are known.

Furthermore, it is of great advantage if the grid in which the object hypotheses are generated, is adaptively adjusted depending on the classification result. For example, the grid width is adapted adaptively as a function of the classification result, object hypotheses being generated only at the grid points or search windows being positioned only at grid points. If object hypotheses are increasingly not assigned to any object class or no object hypotheses are generated, the screen ruling is preferably chosen to be smaller. In contrast, the grid size is larger if object hypotheses are increasingly assigned to an object class or if the probability of object class membership increases. Also in this context is a use of a hierarehi.s.chen. Structure-for-the-hypothesis grid possible. In addition, the grid in Depending on the classification result of a previous time step adaptively adapted, possibly taking into account a dynamic system model.

In a further advantageous manner, the evaluation method, by means of which the object hypotheses are evaluated, is adjusted automatically as a function of at least one previous evaluation. In this case, for example, only the last preceding classification result or several previous classification results are taken into account. By way of example, only individual parameters of an evaluation method and / or a plurality of evaluation methods are selected here for a suitable evaluation method. In principle, the most varied evaluation methods are possible in this connection, which can be based, for example, on statistical and / or model-based approaches. The type of evaluation method provided for the selection also depends on the type of sensors used.

Furthermore, there is also the possibility that, as a function of the classification result, both the grid is adapted adaptively, and the evaluation method used for the evaluation is adapted. The raster is refined in a profitable manner only at the positions in the search space where the probability or score for the presence of objects is high enough, with the score being derived from the last raster levels.

The different sensor signal currents can also be used zi-tgie-i-eh- ₇ -a-be-r-auch_zei_tv_ers_e £, zt. Similarly, in an advantageous manner, a single Sensor signal stream together with at least one time-shifted version of the same can be used.

The inventive method can be used except for object detection and tracking of detected objects.

The inventive method can be used in particular for environmental detection and / or object tracking in a road vehicle. For example, a combination of a color camera sensitive in the visible wavelength spectrum and a camera sensitive in the infrared wavelength spectrum is suitable for use in a road vehicle. Thus, on the one hand, persons and, on the other hand, the colored signal lights of traffic lights in the vicinity of the road vehicle can be reliably detected at night. The information provided by the two sensors is evaluated by the method according to the invention for multisensorial object recognition in order to recognize and track, for example, persons contained therein. The sensor information is thereby preferably presented to the driver on a display unit arranged in the vehicle cockpit in the form of image data, persons and signal lights of traffic light installations being highlighted in the displayed image information. For use in a road vehicle are in the context of the inventive method as sensors in addition to cameras especially radar and Lidarsensoren. The present invention is method further for use unterschiedlichster types of image sensors, and any other prior art known sensors g ^ ^~ eϊgϊτet Further features and advantages of the invention will become apparent from the following description of preferred embodiments with reference to the figures. Showing:

1 shows on the left a surrounding scene detected by means of an NIR camera and on the right a scene detected by means of an FIR camera FIG. 2 shows a suboptimal assignment of two sensor signal streams FIG. 3 the feature formation in connection with FIG

Multi current detector

4 shows the geometric determination of the search space. FIG. 5 shows a resulting set of hypotheses for a

Single-stream hypothesis generator

6 shows the epipolar geometry of a two-camera system. FIG. 7 shows the epipolar geometry using the example of FIG

Pedestrian Detection Fig. 8 shows the cause of scaling differences in

Correspondence Search Window Fig. 9 resulting correspondences in the NIR image for a

Search window in the FIR image

Fig. 10 shows the relaxation of the correspondence condition Fig. 11 correspondence error between label and

Correspondence search window

FIG. 12 shows how multi-stream hypotheses arise FIG. 13 Comparison of detection rates with different ones

Grid width Fig. 14 shows the detector response as a function of the achieved

detection stage

15 shows a coarse-to-fine search in the one-dimensional case. FIG. 16 shows by way of example the neighborhood definition FIG. 17 a hypothesis tree

In the figure 1 is on the left one by means of an NIR camera

_and right-ei-ferrous -mi-trfee-1 s ^ a FTR- camera erf ^{'a ~} s ^~ ste ^"~

Environmental scene shown. The two camera sensors and the recorded intensity images differ greatly. The NIR image shown on the left has a high variance depending on the lighting conditions and surface properties. In contrast, the heat rays detected by the FIR camera, which are shown in the right-hand part of the picture, are almost exclusively direct emissions of the objects. Due to their intrinsic heat, pedestrians in particular generate a pronounced signature in thermal images and stand out greatly from the background in country road scenarios. However, this obvious advantage of the FIR sensor is contrasted with its resolution: it is four times smaller in the X and Y direction than that of the NIR camera. Due to this rough sampling, important high-frequency signal components are lost. For example, a pedestrian 50 meters away in the FIR image only has a height of 10 pixels. The quantization also differs, although both cameras deliver 12-bit gray value images, however, the dynamic range relevant for the detection task extends to 9 bits for the NIR camera and to only 6 bits for the FIR camera. This results in an 8 times larger quantization error. In the NIR camera image object structures are clearly visible, the image is dependent on lighting and surface structure and it has a high intensity variance. In contrast, in the FIR camera image object structures are difficult to detect, the image is dependent on emissions, with the pedestrian stands out clearly from the cold background. Due to the fact that both sensors have various advantages, namely that the strengths of one are the weaknesses of the other, the use of these sensors in the context of the inventive method is particularly advantageous. Here are the advantages of both sensors in one. Combine classifier, which significantly exceeds the detection performance of single-stream classifiers.

The term sensor fusion refers to the use of multiple sensors and the generation of a common representation. The goal is to increase the accuracy of the information obtained. Characteristic here is the union of measured data in a perceptual system. The sensor integration, however, refers to the use of different sensors for several subtasks, such as image recognition for localization and haptic sensors for subsequent manipulation with actuators.

Fusion approaches can be categorized based on their resulting representations. For example, the following four fusion levels are distinguished:

• Fusion at the signal level: The raw signals are viewed directly here. An example is the localization of acoustic sources due to phase shifts.

• Pixel-level fusion: In contrast to the signal plane, the spatial reference of pixels to objects in space is considered. Examples are extraction of depth information with stereo cameras or the calculation of optical flow in image sequences.

• Feature-level fusion: Feature-level fusion independently extracts features from both sensors. These are combined, for example, in a classifier or a localization method. • Symbol-level fusion: For example, symbolic representations are words or phrases used in speech recognition. Grammars create logical relationships between words. These in turn can control the interpretation of acoustic and visual signals.

Another form of fusion is the classifier fusion. Here the results of several classifiers are united. The data sources or the sensors are not necessarily different. The goal here is to reduce the classification error by redundancy. The decisive factor is that the individual classifiers have errors that are as uncorrelated as possible. Some methods for merging classifiers include:

• Weighted majority decision: A simple principle is the majority decision, ie the choice of class issued by most classifiers. Each classifier can be weighted according to its reliability. Using learning data ideal weights can be determined.

• Bayes combination: For each classifier a confusion matrix can be calculated. This is a confusion matrix indicating the frequency of all classifier results for each actual class. It can be used to approximate conditional probabilities for resulting classes. Now all classifications are mapped to probabilities for class membership using the Bayes theorem. As the final result, the maximum is then selected.

• Stacked Generalizatioju_j3le__I_dee_j3_eji. di.esem_Ansatz_ist-die_

Use of the classifier results as inputs or Characteristics of another classifier. The further classifier can be trained with the vector of the results and the label of the first classifier.

Possible fusion concepts in the detection of pedestrians are the detector fusion and a merger on feature level. There are already acceptable solutions to the detection problem with only one sensor, so combining by classifier fusion is possible. In the case of two classifiers and a two class problem considered here, a merger by weighted majority vote or Bayesian combination results in either a simple and operation or an or operation of the single detectors. The AND operation has the consequence that (with the same parameterization) the number of detections and thus the detection rate can only be reduced. With an OR operation, the false alarm rate can not get better. How meaningful the respective links are can be determined by determining the confusion matrices and analyzing the correlations. However, it is possible to make a statement about the resulting effort: In the case of the OR operation, the images of both streams must be scanned; the effort is at least the sum of the expenditure of both single-stream detectors. As an alternative to the AND or OR operation, the detector result of the cascade classifier can be interpreted as a probability of inference by mapping the achieved level and the last activation to a detection probability. This allows a decision function to be defined on nonbinary values. Another possibility is to use the one classifier for attention control and the other classifier for detection. The former should be parameterized so that the detection rate (ate Fa-1-sch-aia-rmr be borne de ^~ r ^~ -) - is high. This may reduce the amount of data of the detecting classifier, making it easier to classify. Feature-level fusion is mainly due to the availability of boosting techniques. The concrete combination of features from both streams can thus be done automatically with the already used method based on the training data. The result represents approximately an optimal choice and weighting of the features from both streams. An advantage here is the extended feature space. If certain subsets of the data can only be easily separated in one of the individual flow feature spaces, then the combination can simplify the separation of all data. For example, the pedestrian silhouette is clearly visible in the NIR image, whereas the FIR image shows an illumination-independent contrast between the pedestrian and the background. In practice, it has been shown that feature-level fusion can drastically reduce the number of features required.

The architecture of the multi-stream classifier used will now be described. Extending the single-stream classifier to the multi-stream classifier requires reworking many parts of the classifier architecture. An exception is the core algorithm eg AdaBoost, which does not necessarily have to be modified. Nevertheless, some implementation-technical optimizations can be made which reduce the duration of an NIR training run by a predetermined parameterization by a multiple. The entire table of characteristic values for all examples is kept in memory. Another point is the optimizations in the example generation. This allowed training exercises to be carried out in practice πdrt 1 ^" & Sequen-z-en in to be finished about 24 hours. Before this optimization, training with only three sequences took two weeks. The integration of further streams into the application takes place in the course of a redesign of the implementations. The expansion of the hypothesis generator requires the most modifications and innovations.

In the following, the essential extensions with regard to the data preprocessing will be described. The resulting detector will be used in the form of a real-time system and with live data from the two cameras. The training uses labeled data. An extensive database of sequences and labels is available for this purpose, which includes country road scenes with pedestrians running by the roadside, cars and cyclists. Although the two sensors used record approx. 25 pictures per second, the temporal sampling is done asynchronously due to the hardware, the times of both pictures are independent. Because of fluctuations. At the recording times even a significant difference in the number of images of the two cameras for a sequence is common. An application of the detector is not possible as soon as only one feature is not available. For example, if one were to replace the respective terms in the Stronglearner equation by zeros for missing features, the behavior is undefined. This makes sequential execution of the individual images of the multi-stream data impossible and requires synchronization of the sensor data streams both for training and for the application of a multi-stream detector. In this case, image pairs must be formed. Since the recording time of the images of a pair are not exactly the same, each is a different state of the environment abgejoildet_ __p _i _._ h DIE Bos-i-fe-i-.beta.n ^¬ the vehicle and the pedestrian is in each other. Around To minimize any influence of environmental dynamics, the image pairs must be formed so that the differences of the two time stamps become minimal. Because of the mentioned different number of measurements per unit of time, either images from a stream must be used several times, or images are omitted. Two reasons are in favor of the latter: First, it minimizes the average time stamp difference, and secondly, multiple use online would lead to occasional spikes in computational overhead. The following algorithm describes the data synchronization:

I Given: 2

3 image sequences I _s (i) for each stream s ^e {l, 2}

4

5 timestamp t _s (i) for all images for each stream s

6

7 Expected time stamp difference E (t _s (i + l) -t _s (i)) for each stream s 8 9 Largest expected time stamp difference deviation ^ε _s for each stream s 10

II initialization: 12

13 Begin with the first images of the streams: 14

15 i = 1

16 j = 1 17 P = O 18

19 Algorithm: -20 21 As long as the images I l (i) and 12 (j) exist: 22

23 If It ₁ (D - t ₂ (j) |> min _s ( ^~ (E (t _s (i + 1) -t _s (i)) + ^ ₈ ))

24

25 If t ₁ (±) <t2 (j)

26 i = i + 1

27 otherwise

28 j = j + 1

29 otherwise 30

31 Make a pair (i, j): 32

33 P = P ^ (i, j)

34 i = i + 1

35 j = j + 1 36

37 result:

38

39 image pairs P

In this case, ^ε ₃ should be chosen as a function of the distribution of t _s (i + 1) -t _s (i) and should be about 3 ^σ . With small ^ _{3 there} is the possibility that some image pairs are not found, for large ^ε _s the expected time stamp difference increases. The assignment rule corresponds to a greedy strategy and is therefore generally less than optimal with respect to minimizing the mean time stamp difference. However, it can be used both in training and in online operation of the application. In the case of V ar (t _a (i + 1) -t _a (i)) = 0 and ^ε _s = 0 V ₃ , it is advantageously optimal.

[-0Θ2-8 -] - I-n-the E-igur__2__will__become a suboptimal example

Assignment of two sensor signal currents shown. Here is in particular the result of the mapping algorithm shown above. In this example, the mapping is sub-optimal with respect to minimizing the mean timestamp difference. The assignment algorithm can be used in this form for the application, there are advantageously no delays due to waiting for potential assignment candidates.

The concept for the search window plays a central role in the feature formation, especially in the extension of the detector for multi-sensorial use, with multiple sensor signal currents are present. In a single-current detector, the localization of all objects in a picture consists of examining a set of hypotheses. A hypothesis stands for a position and scaling of the object in the image. This results in the search window, ie the image section, which is used for the feature calculation. In the multi-stream case, one hypothesis consists of a search window pair, that is, one search window in each stream. It should be noted that for a single search window in one stream due to the parallax problem, different combinations of search windows in the other stream may occur. Thus, a very large number of multi-stream hypotheses can arise. A hypothesis generation for any camera arrangements will be shown later. The classification is based on features of two search windows, as illustrated by FIG. FIG. 3 shows the feature formation in connection with a multi-stream detector. A multi-stream feature set corresponds to the union of the two feature sets that result for the single-stream detectors. A multi-stream feature is defined by filter type, position, scaling and sensor flow. Smaller filters can be used in the NIR search window due to the higher image resolution-ais-im- PIR search window. The number of NIR features is thus higher than the number of FIR features. In this embodiment, approximately 7000 NIR features and approximately 3000 FIR features were used.

In an advantageous manner, new training examples are continuously selected during the training process. Before training by means of each classifier stage, a new example set is generated using all stages already trained. In multistrom training, the training examples, like the hypotheses, consist of a search window in each stream. Positive examples result from labels that are present in each stream. In connection with automatically generated negative examples, a mapping problem occurs: The randomly selected search windows must be consistent with respect to the projection geometry of the camera system so that the training examples match the multistrom hypotheses of the later application. To achieve this, a specific hypothesis generator, which will be described in detail below, is used in the determination of the negative examples. Instead of selecting the position and size of the search window from negative examples independently and randomly, a random set of hypotheses is now used. In addition to consistent search window pairs, the set of hypotheses has a smarter, world model-based distribution of the hypotheses in the image. This hypothesis generator can also be used for single-stream training. Here, the negative examples are determined with the same search strategy, which later serves in the application of the detector for generating hypotheses. Thus, the example set for the multi-stream training is made up of positive and negative examples, which include-again-by-the-next-search windows in both streams. For the training For example, AdaBoost is used, and all features of each example are calculated. In the feature selection, only the number of features changes compared to the single-stream training, as it is abstracted due to its definition and the associated multistrom data source.

The architecture of a multi-stream detector application is very similar to that of a single-stream detector. The required modifications to the system are on the one hand adaptations for the general handling of multiple sensor signal streams, which changes are required at almost all points of the implementation. On the other hand, the hypothesis generator is extended. For the generation of multi-stream hypotheses, a correspondence condition for search windows of both streams is necessary, which is based on world and camera models. Thus, a multi-stream camera calibration must be integrated into the hypothesis generation. Although the brood force search in the hypothesis space used for single-stream detectors can be transferred to multi-stream detectors, it often turns out to be too inefficient. The search space increases significantly and the number of hypotheses multiplies. However, to remain real-time capable, the set of hypotheses must be reduced again and more intelligent search strategies are required. The fusion approach pursued in connection with this exemplary embodiment corresponds to a merger at feature level. By means of Ada-Boost a combination of features of both streams is chosen. Other methods could be used here for feature selection and fusion. The required changes to the detector are an extended feature set, a synchronization of the data as well as the generation of a hypothesis set, which takes into account the geometric relationships of the camera models. The derivation of a correspondence rule, the search space sampling and further profitable optimizations are presented below. The trained single-stream cascade classifier evaluates individual search windows one after the other. As a result, the classifier provides a statement as to whether an object was detected in exactly this position and scaling. In each image, pedestrians can appear in different positions with different scales. Therefore, when using the classifier as a detector, a large number of positions or hypotheses must be checked in each image. This set of hypotheses can be reduced by subsampling and search range constraints. Thus, the calculation effort can be reduced without affecting the detection performance. For this purpose, hypothesis generators for single-stream applications are already known from the prior art. In the multistrom detector presented in connection with this exemplary embodiment, hypotheses are defined via a search window pair, that is to say via a search window in each stream. Although the search windows can be generated in two streams with two single-stream hypothesis generators, the link to the multistrom hypothesis set is not trivial due to the parallax. The assignment of two search windows from different streams to a multi-stream hypothesis must fulfill certain geometric conditions. In order to achieve robustness against calibration errors and dynamic influences, furthermore, relaxations of these geometric correspondence conditions are introduced. Finally, a concrete sampling and allocation strategy is chosen. There are many more hypotheses than with ___ single.els-trom = -detek-feo-ren- um ± e ^~ to ensure real-time capability of the multistrom detector, Further optimization strategies are shown below, including a very effective method for hypothesis reduction via a dynamic local control of the hypothesis density, which can also be used in connection with single-stream detectors. The simplest search strategy for finding objects at all positions in the image is the pixel-by-pixel scanning of the entire image in all possible search window sizes. This results in a picture with 640 x 480 pixels a hypothesis with about 64 million elements. This set of hypotheses is referred to below as the complete search space of the single-stream detector. With the aid of a range restriction described below on the basis of a simple world model and a scaling-dependent subsampling of the search space, the number of hypotheses to be investigated can be reduced in a particularly advantageous manner to approximately 320,000. The basis for the range restriction is on the one hand the so-called "ground plane assumption", the assumption that the world is flat, with the objects to be detected and the vehicle being on the same level. On the other hand, due to the object size in the image and an assumption regarding the real object size, a unique position in space can be derived. All the hypotheses of a scaling in the image lie on a horizontal line. Both assumptions, ie the "Ground Plane Assumption" as well as the. A fixed real object size usually does not apply. The restrictions are therefore relaxed, so that a certain tolerance range is permitted for the object position as well as for their size in space, this situation is illustrated in FIG. The relaxation of the "ground plane assumption" is indicated by an angle ^ε , which in this embodiment is for example 1 °. So be Or-i-ENFE-i erungsfehl ^~ e ^~ f iπT

Camera model compensated, which, for example, by Nick movements of the vehicle may arise. In addition to the described range limitation, the number of hypotheses to be examined is further reduced by a scaling-dependent subsampling. The step size of the scanning in the u and v directions in FIG. 4 are chosen to be proportional to the height of the hypothesis, that is to say the scaling, and in this example amounts to approximately 5% of the hypothesis height. The search window heights themselves are the result of a series of scaling, each increasing by 5% starting with 25 pixels in the NIR image (8 pixels in FIR image). This type of quantization can be motivated with a property of the detector, namely the fact that the size scaling of the features also increases the blurring of their localization in the image, as is the case, for example, with a Haarwavelet or similar filters. The features are defined here in a fixed grid and are scaled according to the size of the hypothesis. With the hypothesis generation described, a reduction of the 64 million hypotheses of the complete search space to 320,000 results in this case in the NIR image. Due to the low image resolution in the FIR image, there are 50,000 hypotheses. Reference is also made to FIG. 5. For the consideration of the restrictions defined in three-dimensional space, a transformation between image coordinates and world coordinates is necessary. The basis for this are the intrinsic and extrinsic camera parameters determined by the calibration. The geometrical relationships for the projection of a 3D point onto the image plane are known to those skilled in the field of image analysis. Due to the small distortion in both cameras, a hole camera model can be used in this embodiment. FIG. 4 illustrates the geometric determination of the search space. Here, the search area is displayed, which results in a fixed scaling. An upper and lower limit is calculated for the upper search window edge in the image. The limits (v _m i _n and v _max ) arise when the object is projected onto the image plane once with the smallest and once with the largest expected object size (obj _m i _n or obj _ma χ). Here, the distance (z _m i _n and Zma _x ) is chosen so that the correct scaling arises in the image. Due to the relaxed restriction of the ground plane assumption, the spatial position lies between the dashed planes. The smallest and the largest object are moved up and down to calculate the limits.

FIG. 5 shows the resulting hypothesis set of the single-stream hypothesis generator. In this case, search windows with a grid-like arrangement are generated. For different scalings different square lattices are created with adapted lattice spacings and own area restrictions. In the sense of a clear representation, only one search window for each scaling as well as the centers of all other hypotheses is visualized in FIG. The illustration is exemplary, with large scaling and position increments selected.

From the single-stream hypotheses thus arise by suitable pairing Multistrom hypotheses. The epipolar geometry is the basis for pair formation, which describes the geometric relationships. FIG. 6 shows the epipolar geometry of a two-camera system. The epipolar marginal number specifies the set of possible correspondence points for a point in an image plane. Epipolar lines and an epipolar plane can be constructed for every point p in the image. The possible correspondence points for points of an epipolar line in an image are exactly the same on the corresponding epipolar line of the other image plane. FIG. 6 shows in particular the geometry of a multi-camera system with two arbitrarily arranged cameras with the centers Oi ^e R ³ and O ₂ ^e R ³ and an arbitrary point P ^e R ³ . Oi, O ₂ and P span the so-called epipolar plane. It cuts the image planes in the epipolar lines. The epipoles are the intersections of the image planes with the line OiO ₂ . 0i0 ₂ is contained in all epipolar planes of all possible points P. All occurring epipolar lines thus intersect in the respective epipole. The Epipolarlinien have the following meaning in the correspondence finding: Epipolar lines and an epipolar plane can be constructed for each point p in the picture. The possible correspondence points for points of an epipolar line in one image are exactly those on the corresponding epipolar line of the other image plane.

Let point P ^e R ^{3 be} a point in space. Pl, P2 ^e R ³ let P represent the camera coordinate systems originating O ₁ and O _2, respectively. Then there is a rotation matrix R ^e R ^3x3 and a translation vector T ^e R ³ for which the following applies:

P ₂ = R [P _x -T). (5.1)

R and T are clearly defined by the relative extrinsic parameters of the camera system. Pi, T and Pi- T are coplanar, i.

(P ₁ -T) ^τ ■ (TxP,) = 0. (5.2) Equation (5.1) and the orthonormality of the rotation matrix give:

, (5.1)

O = (P ₁ -T) ^{7 "} (TxP ₁ ) = (A- ¹ Pj (T-XP ₁ J = (^ ^p J (TxP ₁ ). (5.3)

The cross product can now be rewritten into a scalar product:

This results from equation (5.3)

o = (RV ₂ J (SP ₁ ) = (PfRXsP ₁ ) = Pl (RS) P ₁ = P ₂ ^T EP _X , '.5.5)

with E: = RS of the Essential Matrix. Now a relationship between Pi and P2 is established. Projected by means of

this results in:

0 = plEpl (5.6)

Here, fi, 2 is the focal length and Zi, 2 is the Z component of P ₁ ^. Thus, the set of all possible pixels p2 in the second image, which may correspond to a point pi of the first image, is exactly the one for which equation (5.6) is erf-ü-1-lt. Mrt of this correspondence condition for individual pixels can now consistent search window pairs from the Single-stream hypotheses are formed as follows: The aspect ratio of the search window is preferably fixed by definition, ie a search window can be uniquely described by the midpoints of the upper and lower edge. With the correspondence condition for pixels, two epipolar lines thus result in the image of the second camera for the possible midpoints of the upper and lower edges of all corresponding search windows, as shown for example in FIG. FIG. 7 shows the epipolar geometry using the example of pedestrian detection. Here, an ambiguous projection of a search window from the image of the right camera into that of the left camera takes place. The correspondence search windows result from the epipolar lines of the centers of the search window bottom and top edges. For reasons of clarity, the illustration is only illustrative here. The set of possible search window pairs should include all those search window pairs that describe objects of realistic size. If one calculates the backprojection of the objects into the space, the position and size of the object can be determined by means of triangulation. The range of epipolar lines is then reduced to correspondences with valid object size, as shown by the dotted line in Figure 7.

It will now be described the optimization of the correspondence space, resulting in the projection of a search window from a sensor current in the other sensor current multiple correspondence search window with different scales. However, this difference in scale disappears if the camera positions and orientations are the same except for a lateral offset. For scaling, therefore, only ___ an offset d between the centers Oi and O ₂ in the longitudinal direction of the camera system is relevant, as in this case. Figure 8 is shown. The difference in orientation of both cameras is negligible in this example. FIG. 8 shows, in particular, the cause of the scaling differences arising in the correspondence search windows, and in the projection of a search window from the first to the second sensor stream, a plurality of correspondence search windows with different scaling results. Here, the geometric relationship between camera arrangement, object sizes and scaling differences is shown in detail.

There is a fixed search window size hi in the first image. The following is the relationship

T min H ₂

where h ₂ ^min or h ₂ ^{max is} the minimum or maximum occurring scaling of the correspondence search window in the second sensor current to the search window h _x in the first sensor current. Let H ^min = Im the height of a nearby pedestrian and H ^max = 2m the height of a distant pedestrian, with only pedestrians being considered to have a minimum size of Im and a maximum size of 2m. Both pedestrians are so far away that they have the height h _x in the picture of the first camera. Let Z _λ ^mxn , Z ₁ ¹ "**, Z ₂ ^min and Z ₂ ^{max be} the object distances of both objects to both cameras, then follows

Z min.max • - _/ min.max j

'2 ₂ = ^Z i ^~ d (5, 7) and

The scaling ratio then results

Λ xrr min H ma

(5 8) ^ mm Z y

A ₂ ^max z> j, ₂ min y max rr min (57) 2 ' ^{mω dd} mm rr min _ ^Λ 2

Λ y min

A ₇ ^mm TT man y min TT man. y mm £ ^■ max (5.9)

Tj max ^■ ^ 1 zy, ₂ max

For long distances, the scaling ratio goes to 1. For an application of the classifier as an early warning system in country road scenarios, you can limit the choice of Zi ^min to values greater than 20m. The offset of the cameras is about 2m in the test carrier. Together with the above suggested values for pedestrian sizes, it follows that

.Max

applies. Thus, the correspondence space for a search window in the first stream, that is to say the set of corresponding search windows in the second stream, can be simplified as follows: The scaling of all corresponding search windows is standardized. The scaling h _{2 used} for all correspondences is the mean of the minimum and maximum scaling:

Max

K = ₂ • (5.10)

The resulting scaling error is a maximum of 2.75%. FIG. 9 shows resultant correspondences in the NIR image for a search window in the FIR image. A unified scaling is used for all corresponding search windows. For modeling the correspondence error, the pairing described above for generating multistrom hypotheses is often inadequate in real applications. In a profitable way, the following factors are taken into account in addition:

• Errors in extrinsic and intrinsic camera parameters caused by measurement errors during camera calibration.

• Influences of the environmental dynamics.

Thus, there is an unknown error in the camera model. This creates a fuzziness for both the position and the scaling of the correlated search window, it is referred to below as a correspondence error. The scaling error is neglected for the following reasons: First, the influence of the dynamics on the scaling is very small if the object is at least 20m away. Secondly, a significant insensitivity of the detector response can be seen in terms of the accuracy of the hypothesis scaling. This can be seen by multiple detections, whose centers hardly vary, but the scales vary greatly. To compensate for the translational error, a relaxation of the correspondence condition is introduced. For this purpose, a tolerance range for the position of the correlated search window is defined. For each of these correspondences, an ellipse-shaped tolerance range is defined in the image with the radii e _x and e _y , in which further correspondences arise, as shown with reference to FIG. The correspondence error is identical for each search window scaling. The resulting tolerance range is therefore glejLc_h__selected-t- for each scaling. FIG. 10 shows the relaxation of the correspondence condition. The positions of the correlating search windows are not limited to one route only. You can now lie within an elliptical area around this distance. In the NIR image, only the center points of the search windows are drawn. In relation to this correspondence error, data labeled with the radii are used. The radii of the elliptical tolerance range are determined as follows:

• For each multistrom label, the search windows in both streams are determined.

• All possible correspondence search windows in the second stream are calculated for the respective search window in the first stream. In this case, a non-relaxed correspondence condition is used.

• The correspondence search window that comes closest to the label search window in the second stream is used for error determination. The proximity of two search windows can be defined here either by the overlap, in particular by the ratio of the intersection of two rectangles to their union surface (also called coverage) or by the spacing of the search window center points. The latter definition was chosen in this embodiment, since this neglects the scaling error that is uncritical for the detector response.

• For all labels, the distance in the X and Y directions is determined between the label search window and the closest correspondence search window. This results in a frequency distribution for the X and Y distances. A histogram over the distance in the X and Y directions is shown in FIG. • Now the radii e _x and e _y are derived from the distribution of the distances. In this work e _x = 2 ^σ x and e _y = 2 ^er y were chosen. The next step after defining the correspondence space for a search window is the search space scan. As with single-stream subsampling, the number of hypotheses should also be minimized with as little loss as possible in the detection performance.

FIG. 11 shows the correspondence error between label and correspondence search window. The illustrated correspondence error is the smallest pixel distance of a label search window to the correspondent search windows of the corresponding label, so the projected label of the other sensor signal stream. In the illustrated measurement, FIR labels are projected into the NIR image and a histogram is formed over the distances of the search window centers.

The method for the search space sampling proceeds as follows: In both streams, single-stream hypotheses, ie search windows, are scattered with the single-stream hypothesis generator. In this case, the resulting scaling stages must be matched to one another, wherein in the first stream the scalings are determined by the hypothesis generator. For each of these scaling levels, the correspondence space of a prototypical search window is then determined. The scaling of the second stream results from the scaling of the correspondence spaces of all prototypical search windows. This creates the same number of scaling levels in both streams. Now, search window pairs are formed, resulting in the multi-stream hypotheses. It is then possible to select one of the two streams in order to select the respective one for each search window

Correspondence to be determined in the other stream. All

Search window of the second stream, which the ricntige Scales that are within this range are used together with the fixed search window of the first stream for pairing, this will be illustrated with reference to FIG 12. FIG. 12 shows the resulting multistrom hypotheses. Here, three search windows in the FIR image and their correspondence regions in the NIR image are drawn. Couples are formed with the search windows scattered by single-stream hypothesis generators. A multi-stream hypothesis corresponds to a search window pair.

If one selects the position and scaling step widths of 5% of the search window height for the internally used single-current hypothesis generators, approximately 400,000 single-stream hypotheses result in the NIR image, and approximately 50,000 in the FIR image. However, there are about 1.2 million multistrom hypotheses. In practical use, a processing speed of 2 images per second could be achieved. To ensure the real-time capability of the application, further optimizations are presented below. On the one hand, a so-called Weaklearner cache is described, which reduces the number of necessary feature calculations. In addition, a method for the dynamic reduction of the hypothesis set is presented, hereafter referred to as a multiraster hypothesis tree. The third optimization, which is called backtracking, reduces unnecessary effort in connection with multiple detections in case of detection.

Evaluating multiple multi-stream hypotheses that share a search window results in multiple runners being computed multiple times on the same data. To avoid all redundant calculations is now a Cachingverfahren_e, ing.esetzt-It-is-f-u-r-search every ^{~ 'feTϊs ~} ter ^~ in both partial streams and, for each Strong Learner Totals of the Stronglearner calculation stored in tables. A stronglearner H ^{k of} the cascade stage k is defined by:

"! 'WH | - ^1: s ^S o ^t n ^{{s x t) ≥ @t"} "s'Mi" XtfM (5.1d

k with the weave learners h, ^e {-l, l} and hypothesis x.

S ^k (x) can be split into two sums that only

Weaklearners with features of a stream include:

with W _s = \ t I Af Weaklearner is in the stream s \.

If several hypotheses xi in a stream s have the same search window, then in each stage k for the stream s the sum S ₅ (xi) is the same for all xi. The result is preferably buffered and used several times. If it is possible to resort to already calculated values for a Stronglearner calculation, the effort is reduced in a profitable manner to a sum operation and a threshold operation. As far as the size of the tables is concerned, in this exemplary embodiment 12.5 million entries result for a total of 500,000 search windows and 25 cascade stages. 64-bit floating-point numbers require 100 MB of memory. For an effort estimate, the number of feature calculations can be considered both with and without a Weaklearner cache. In the former case, the number of hypotheses per image and the number of all features are decisive. The number of hypotheses can be reduced by the number of search windows R ₃ in the streams, s cc-protected by O (R1-R2). However, the factor hidden in the O notation is very small here, since the correspondence area is small is opposite to the entire image area. The number of calculated features is then in the worst case O (R1-R2- (M1 + M2)) where Ms is the number of features in each stream s. In the second case, each feature in each search window is calculated at most once per image. Thus, the number of calculated features is at most O (Rl -M1 + R2 -M2). The effort is reduced in the worst case by the factor min (Rl, R2). A complexity analysis for the average case, however, is more complex because the relationship between the average number of calculated features per hypothesis or search window in the first case and in the second case is not linear.

The following is a description of the multiraster hypothesis tree. The search space of the multistrom detector was detected in this example with two single-stream hypothesis generators and a relaxed correspondence relationship. In this case, however, it is difficult to find an optimal parameterization, especially the finding of the appropriate sampling step sizes. On the one hand, they have a major influence on the detection performance and, on the other hand, on the resulting computational effort. For the single-stream detectors, acceptable compromises could be found in a practical experiment, which could ensure a real-time capability in the FIR case because of the lower image resolution, but in the NIR case this was not possible with the hardware used. The performance of the experimental computer used was also insufficient when using a fusion detector with Weaklearner cache and resulted in longer response times in complex scenes. Of course, these problems can be solved with more powerful hardware.

F0047_1 At. Practical 3-i-nsa-fe-z-ve-rschi-edene parameterizations of the hypothesis generator and the detector tested. Several search grid densities and different step restrictions were evaluated. It has been found that, even with very coarse scanning, every pedestrian to be detected is already recognized with the first stages of the detector. In this case, the rear cascade stages were switched off successively, resulting in a high false alarm rate. The measured values recorded during practical use are shown in Figure 13. The number of hypotheses were starting with the finest grid density: approximately 1,200,000, 200,000, 7,000 and 2,000.

FIG. 13 shows the comparison of the detection rates of different screen rulings, wherein four different hypothesis grid densities are compared. For each screen ruling, the detection rate of a fusion detector is plotted against the number of stages used. The detection rate is defined by the number of pedestrians found divided by the number of pedestrians. The reason for the phenomenon that has occurred is the following property of the detector: The detector response, ie the cascade stage reached, is at most a hypothesis which is positioned exactly on the pedestrian. If one pushes the hypothesis step by step away from the pedestrian, the detector result does not drop abruptly to zero, but there is an area in which the detector result varies greatly and tends to decrease. This behavior of the cascade detector is referred to below as a characteristic detector response. An experiment in which an image is scanned in pixel steps is visualized in FIG. It uses a multistrom detector and fixed scale hypotheses. You can do that

Range for which d-ie-Defeekfeoranfewort-delayed-wood waste prod ^~ l ^{~ ~} ϊt; ^"clearly visible. Furthermore, it has been shown that the detector has similar characteristics in a fixed-position experiment with varying scaling. Thus, the detection performance of the shortened detector applied to a rough hypothesis grid to explain, because the "target area" for a pedestrian increases for lower levels.

FIG. 14 shows the detector response as a function of the achieved detection stage. In this case, a multistrom detector is applied to a set of hypotheses in a scaling with pixel-precise grid. The last cascade level reached is plotted for each hypothesis at its midpoint. During training, no training examples slightly offset to a label are used. Only exact positive examples are used as well as negative examples, which have a large distance to each positive example. Thus, the behavior of the detector is undefined in hypotheses that are slightly offset from an object. Therefore, the characteristic detector response is experimentally investigated for each detector. The central idea for reducing the number of hypotheses is a coarse-to-fine search, whereby each image is searched in the first step with a roughly resolved set of hypotheses. Depending on the detector result, further hypotheses with higher density are scattered in the image. In addition, the local neighborhood is searched for hypotheses that suggest an object in its vicinity. By the above described behavior of the detector, the achieved number of stages can be taken as criteria for the refinement of the search. Following the same principle, the local neighborhood of the new hypotheses can then be searched again until the finest hypothesis grid is reached. For the refinement step, a threshold is used with which the achieved cascade level of each hypothesis is compared.

FIG. 15 shows a coarse-to-fine search in the one-dimensional case. For this purpose, an image line from the image acquisition shown in FIG. 14 was used, which is shown in the form of a function in FIG. From left to right you can see the steps of the search process. The hypothesis results are horizontal and the thresholds for local refinement are shown horizontally. For the threshold determination, the experiment described above can be used. The detection rate of each screen density is almost identical for the first stages of the detector. The threshold value is the maximum level for which the affected screen density still has almost the same detection rate as the maximum achievable. For the threshold level k of a grid density L becomes a

L

Detection rate D t required, so

≥a-D ?.

D "

Ok here denotes the detection rate of the finest screen density H in step k. If n is the number of refinements, then a detection rate results for the last step K of the detector

D _κ = a "-D _K ^H

For α, in this example mainly values between 0.98 and 0.999 are suitable. In the definition of the neighborhood, the Hypothesian is considered. The hypothesis space is now not one-dimensional but in the case of the single-stream detector three-dimensional or six-dimensional in the fusion detector. The problem of gradual refinement in all dimensions is solved with the hypothesis generator. There are two possibilities for defining the neighborhood, of which the second is used in this embodiment. On the one hand, a minimum value for the coverage of two adjacent search windows can be defined. In this case, however, it is not clear how to choose the minimum value, since gaps can arise in the refined sets of hypotheses, that is, areas that are not close enough to any hypothesis of the coarse set of hypotheses. Therefore, different thresholds must be set for each grid density. On the other hand, the neighborhood can be defined with a modified checkerboard distance. Thus, the mentioned gaps are avoided and it can be defined a uniform threshold for all screen densities. The chessboard distance is defined by

dύt (p _ι , p ₂ ) = max \ p _lιX -p _2ιX \, \ p _Uy -p _2ty \) with p _λ , p ₂ e <R ² . (5.13)

The array density of a current is defined by r _x, r _y, r _h ^e R. The grid spacings are for a search window height h then in the _x direction r _x • h and in the _y direction r _y ^• h. For a search window height hi, the next largest search window height h is ₂ hi- (1 + rh). The neighborhood criterion for a search window with position S ₁ ^e R ² and search window height Ia ₁ to a search window S ₂ ^e R ^{2 of} a finer hypothesis set with height h? is with _a_scalar <? DEFINE-t ^ Max <δ Λ h ₂ e [h _x {\ ₊ r _h ) - ^δ , hfi ₊ r _h y ^δ }. r _x 'K (5.14)

A,

The resulting interval boundaries are visualized in FIG. 16. In the multi-stream case, there is a three-dimensional neighborhood criterion in each stream. For adjacent multi-stream hypotheses, the neighborhood condition in both streams must be met. If you choose r _x = r _y and δ = 0.5, then all neighborhoods are disjoint except for the edges. If the step sizes r _* and for the

Refinement hypothesis quantities are successively halved and the added hypotheses to the limits of

This value makes sense for δ because the finer hypotheses are linked to all of the coarser adjacent hypotheses. However, this does not apply if the refined sets of hypotheses are arbitrary

Grid intervals have. Then by choosing ^> 0.5 it must be achieved that the neighborhoods of adjacent coarse set hypotheses overlap and the fine grid hypotheses overlap several coarse hypotheses

Assigned to rasters. The required value for δ must be determined by experiment, i. it must be adapted to the characteristic detector response.

The neighborhood definition is shown in FIG. 16: the neighborhood is drawn for three of the hypotheses of the same scaling level, and on the right side there are three different scalings and their resultant

Scaling neighborhood depicted. For δ, 0.75 was chosen here.

The generation of the refined hypothesis wanrenα αer application would be too time-consuming and may as well Preprocessing step done. The generation of all refined hypothesis sets is done by means of the hypothesis generator. First, the set of hypotheses for each refinement level is generated. Then the hypotheses are linked to the neighborhood criterion, each hypothesis being compared to each hypothesis of the next finer set of hypotheses. If these are close, they are linked. This results in a tree-like structure whose roots correspond to the hypotheses of the coarsest stage. In FIG. 17, the edges represent the calculated neighborhood relationships. Since a certain search effort is associated with the generation of the hypothesis tree, the calculations required for this purpose are preferably realized via a separate tool and stored in the form of a file.

FIG. 17 shows the resulting hypothesis tree. The hypothesis tree / search tree has several roots and is searched from the roots to the leaf level, if the detection result of a node is greater than the threshold value. When processing an image (or image pair in the case of the multi-stream detector), the hypothesis tree is traversed. Beginning with the first tree root, the tree is searched with a depth or breadth first search. The hypothesis of the root is evaluated. As long as the corresponding threshold value is exceeded, the tree is descended and the respective child node hypotheses are examined. Then the search continues at the next tree root. Along with the backtracking method described below, the depth search is most effective. Since node may have multiple parent node, care must be taken that each node wird._ only once examined by the use of a multi-grid tree hypothesis here results in advantageous manner a reduction in d ^~ he ^~ Hypothesis number, which affects the detection performance.

The number of multiple detections is very high in the multi-stream detector and in the FIR detector. Multiple detections therefore have a major impact on computation time as they traverse the entire cascade. Therefore, a so-called backtracking method is used. With a change in the search strategy, a large part of the multiple detections can be avoided, and in the case of detection, the search in the hypothesis tree is aborted and continued at the next tree root. This will locally reduce the density of hypotheses as soon as an object is found. In order not to create a systematic error, all child nodes are randomly permuted so that their order does not correlate with their order in the image. For example, if the first child hypotheses are always in the upper left corner of the neighborhood, detection tends to shift in that direction.

On the basis of this embodiment, a method was thus developed starting from the single-stream Hyothesengenerator by modeling a relaxed correspondence area and finally by various optimizations, which requires very little computing time despite the complex search space of the multi-stream data. An important contribution is made by the multiraster hypothesis tree.

The use of the multiraster hypothesis tree is not only in the context of multi-sensor fusion of great advantage, but is particularly suitable for interaction with cascade classifiers in general and this leads to significantly better classification results.

Claims

DaimlerChrysler AG BöpplePatentansprüche

1. A method for multi-sensor object detection, wherein sensor information from at least two different sensor signal currents are used with different sensor signal properties for common evaluation, wherein the at least two sensor signal streams are not matched to each other for evaluation and / or mapped to each other, in which case generated in each of the at least two sensor signal streams object hypotheses in which features for at least one classifier are generated on the basis of these object hypotheses and wherein the object hypotheses are evaluated by means of the at least one classifier and assigned to one or more classes, at least two classes being defined and objects being assigned to one of the two classes.

2. The method according to claim 1, characterized in that the object hypotheses are uniquely assigned to a class.

3. The method according to claim 1, characterized in that the object hypotheses are assigned to several classes, wherein the respective assignment is assigned a probability.

4. The method according to any one of the preceding claims, characterized in that the object hypotheses are generated independently in each sensor signal individually, wherein the object hypotheses of different sensor signal currents are then assigned to each other via assignment rules.

5. The method according to any one of claims 1 to 3, characterized in that object hypotheses are generated in a sensor signal current (primary current) and object hypotheses of

Primary current can be projected into other sensor signal currents (secondary currents), with a

Object hypothesis of the primary flow one or more

Object hypotheses generated in the secondary stream.

6. The method according to claim 5, characterized in that the projection of object hypotheses of the primary current into a secondary current based on the sensor models used and / or the positions of search windows within the primary current or on the Epipolargeometrie.

7. Method according to one of the preceding claims, characterized in that object hypotheses are distinguished by their object type _, _ object position, object extent, object orientation, Object motion parameters such as direction of motion and velocity, object danger potential, or any combination thereof are described.

8. The method according to any one of the preceding claims, characterized in that object hypotheses are randomly scattered in a physical search space or generated in a grid or generated by a physical model.

9. The method according to claim 8, characterized in that the search space is adaptively limited by external specifications such as opening angle, range ranges, statistical characteristics that are obtained locally in the image, and / or measurements of other sensors.

10. The method according to any one of the preceding claims, characterized in that the different sensor signal properties in the sensor signal currents based on different positions and / or orientations and / or sensor sizes of the sensors used.

11. The method according to claim 10, characterized in that each object hypothesis is individually classified for itself and the results of the individual classifications are combined, wherein at least one classifier is provided.

12. The method according to claim 10, characterized in that in the at least one classifier features of Object hypotheses of different sensor signal currents are evaluated together and combined to form a classification result.

13. The method according to any one of the preceding claims, characterized in that the grid in which the object hypotheses are generated, is adaptively adjusted as a function of the classification result.

14. The method according to any one of claims 1 to 12, characterized in that the grid in which the object hypotheses are generated, is adaptively adjusted in dependence on the classification result of a previous time step.

15. The method according to any one of the preceding claims, characterized in that the evaluation method, by means of which the object hypotheses are evaluated, is automatically adjusted depending on at least one previous evaluation.

16. The method according to any one of the preceding claims, characterized in that at least two different sensor signal currents are used with a time delay or that a single sensor signal current is used together with at least one time-shifted version of the same.

17. Use of the method after any one of the following: ors-teh.enden claims for tracking detected objects.

8. Use of the method according to one or more of claims 1 to 16 for environmental detection and / or object tracking in a road vehicle.