DE102007013664A1 - Tool e.g. blade, measuring and/or adjusting device, has rolling nut designed as roller ring transmission comprising set of roller-supported roller rings with variable upward gradient - Google Patents

Abstract

The device has a positioning device (16) comprising a rolling nut surrounding a shaft, where the nut converts a rotation of the shaft into a linear movement of a carriage (14). The rolling nut is designed as a roller ring transmission (18) with a set of roller-supported roller rings (20, 22, 24) with a variable upward gradient. The positioning device comprises an adjusting organ (60) for adjusting the inclination of roller. The positioning device comprises a motor for driving the adjusting organ.

Description

The The invention relates to a method for multisensor object recognition.

The Computer-based evaluation of sensor signals for object recognition and object tracking is already known in the art. For example, driver assistance systems for road vehicles are available, which vehicles ahead detect by means of radar and after track, e.g. the speed and distance of one's own Automatically control the vehicle to the preceding traffic. For use in the vehicle environment are above In addition, various types of sensors, such as. Radar, laser and camera sensors already known. These sensors are in theirs Properties very different and own different Advantages and disadvantages. For example, such differ Sensors in their resolution or in the spectral sensitivity. It would therefore be particularly advantageous if several different sensors simultaneously in a driver assistance system would be used. A multi-sensorial use is currently hardly possible because Sizes detected by different types of sensors only compare with considerable effort in the signal evaluation directly or combine as appropriate.

at the known from the prior art systems are therefore the individual sensor currents first together adjusted before they are fused together. For example The images of two cameras with different resolution are first in consuming manner pixel-accurately mapped and only then fused together.

Of the The invention is therefore based on the object of a method for multi-sensorial To create object recognition, bringing objects to a simple and reliable Can be detected and tracked.

The Task is according to the invention solved by a method having the features of claim 1. advantages Embodiments and developments are shown in the subclaims.

According to the invention a method for multi-sensor object recognition is provided, wherein sensor information from at least two different Sensor signal streams with different sensor signal characteristics to the common Evaluation are used. The sensor signal currents are not adapted to each other for evaluation and / or each other displayed. Based on the at least two sensor signal currents first Object hypotheses are generated and based on these object hypotheses then become features for at least one classifier generated. The object hypotheses are subsequently evaluated by the at least one classifier and one or assigned to several classes. There are at least two classes defined, wherein one of the two classes are assigned objects. With the method according to the invention Thus, a simple and reliable object recognition only possible. A complex adaptation of different sensor signal currents to each other or an illustration of each other is omitted here in a particularly profitable Way complete. In the context of the method according to the invention, the sensor information from the at least two sensor signal streams directly with each other combined or fused together. This will make the evaluation much easier and shorter Calculation times are possible. Because there is no additional Steps for the adaptation of the individual sensor signal currents are needed, the number potential Sources of error in the evaluation minimized.

The Object hypotheses can either be uniquely assigned to one class or they will be multiple Classes associated with each assignment with a probability is occupied.

In a profitable manner, the object hypotheses are independently generated individually in each sensor signal stream, wherein the object hypotheses of different sensor signal currents can then be assigned to each other via assignment rules. First, the object hypotheses are generated in each sensor signal stream by means of search windows in a previously defined 3D state space, which is defined by physical variables. Due to the defined 3D state space, the object hypotheses generated in the individual sensor signal streams can later be assigned to one another. For example, the object hypotheses from two different sensor signal streams are paired later in the subsequent classification, where an object hypothesis is formed from a search window pair. If more than two sensor signal currents are present, a search window is accordingly used from each sensor signal stream and an object hypothesis is formed from this, which is then transferred to the classifier for joint evaluation. The physical quantities for spanning the 3D state space can be, for example, one or more component (s) of the Object extent, a speed and / or acceleration parameter, a time, etc. act. The state space can also be made higher dimensional.

In Another beneficial mode of the invention will be object hypotheses in a sensor signal current (primary current) generated and the object hypotheses of the primary stream then projected into other image streams (secondary streams), where an object hypothesis of the primary stream is one or more object hypotheses in the secondary current generated. When using a camera sensor, the object hypotheses become in the primary stream thereby for example by means of a search window within the means generated by the camera sensor recorded images. The im primary current generated object hypotheses are then computationally in or several other sensor currents projected. In a further advantageous manner, the projection is based of object hypotheses of the primary stream into a secondary stream thereby on the used sensor models and / or the positions search windows within the primary stream or on the epipolar geometry the sensors used. In the projection can in this context Ambiguities also arise. An object hypothesis / search window of the primary current generated, e.g. due to different object distances of the individual sensors, several object hypotheses / search windows in the secondary current. The object hypotheses generated with it are then preferably pairwise passed to the classifier. Where pairs from the object hypothesis of the primary stream and in each case an object hypothesis of the secondary current are formed and then handed over to the classifier become. But there is also the possibility that in addition to the object hypothesis of the primary current also all generated in the secondary currents Object hypotheses or parts thereof passed to the classifier become.

in the In connection with the invention, object hypotheses become more profitable Way by their object type, object position, object extent, object orientation, Object movement parameters such as direction of movement and speed, Object hazard potential or any combination thereof to be discribed. It can also be any act further parameters that describe the object properties. For example, associated with an object speed and / or Acceleration. This is particularly advantageous if the inventive method in addition to pure object recognition in addition to object tracking is used and the evaluation includes a tracking with.

In Another advantageous embodiment of the invention becomes object hypotheses randomly scattered in a physical search space or in one Raster generated. For example, search windows are given a predetermined Increment within the search space varies based on a grid. But there is also the possibility that search window only within predetermined areas of the state space where objects are likely to occur and thus object hypotheses are generated. In addition, the Object hypotheses in a physical search space also by a physical model emerged. The search space can be determined by external specifications like opening angle, Distance ranges, statistical characteristics that are obtained locally in the picture , and / or measurements of other sensors are adaptively restricted.

in the In connection with the invention, the different sensor signal properties are based in the sensor signal currents essentially on different positions and / or orientations and / or Sensor sizes of used sensors. In addition to position and / or orientation deviations or individual components thereof cause mainly deviations in the used sensor sizes different Sensor signal properties in the individual sensor signal currents. For example, cause Camera sensors with a different resolution differ in the sizes the pictures. Also, become common due to different camera optics different sized image areas detected. Furthermore you can e.g. the physical properties of the camera chips completely different so that, for example, a camera has environmental information in it visible wavelength spectrum and another camera environment information in the infrared spectrum recorded, the images have a completely different resolution can.

In the context of the evaluation, there is advantageously the possibility that each object hypothesis is individually classified for itself and the results of the individual classifications are combined, at least one classifier being provided. If several classifiers are used, a classifier can be provided for each different type of object, for example. If only one classifier is provided, each object hypothesis is first classified by means of the classifier, and then the results of several individual classifications are combined into a total result. For this purpose, the expert in the field of pattern recognition and classification different evaluation strategies are known. In a further advantageous manner of the invention, however, it is also possible that in the at least a classifier characteristics of object hypotheses of different sensor signal currents are evaluated together and combined to form a classification result. For the reliable recognition of a specific object, a predetermined number of object hypotheses, for example, must achieve a minimum probability in class membership of this particular object class. Also, the expert in the field of pattern recognition and classification in this context a wide variety of evaluation strategies are known.

Farther is it great Advantage, if the grid in which generates the object hypotheses be, depending the classification result is adapted adaptively. For example the screen ruling becomes dependent the classification result adapted adaptively, where object hypotheses only generated at the grid points or search window only on Grid points are positioned. If object hypotheses increasingly No object class can be assigned or no object hypotheses are generated, the grid width is preferably selected smaller. in the In contrast, the grid size is larger if object hypotheses are increasingly assigned to an object class or the probability of an object class affiliation increases. Also in this context is a use of a hierarchical Structure for the hypothesis grid possible. Furthermore can the grid depending on the classification result of a previous time step be adapted adaptively, possibly taking into account a dynamic System model.

In In a further advantageous manner, the evaluation method, by means of which the object hypotheses are evaluated, depending on automatically adjusted by at least one previous rating becomes. Here, for example, only the last preceding Classification result or several previous classification results considered. For example, only individual parameters of a Evaluation method and / or from several evaluation a suitable Evaluation method selected. in principle are in this context the most diverse evaluation methods possible, which, for example, based on statistical and / or model-based approaches can be based. The type of for the choice available depends on the evaluation procedure This also depends on the type of sensors used.

Farther there is also the possibility that in dependence the classification result adaptively adapted both the grid will, as well as for the evaluation method used is adjusted. The Raster will be profitable only at positions Refined in the search space, where the probability or rating for the Presence of objects is high enough, with the rating off derived from the last raster steps.

The different sensor signal currents can be simultaneously, but also be used with a time delay. The same can be done in an advantageous way also a single sensor signal current together with at least one delayed version of the same.

The inventive method can except used for object recognition also for tracking detected objects become.

The inventive method can in particular for environment detection and / or object tracking in a road vehicle be used. For example, it is suitable for use in a road vehicle a combination of a sensitive in the visible wavelength spectrum Color camera and sensitive in the infrared wavelength spectrum Camera. With that you can at night on the one hand people and on the other hand the colored signal lights of traffic lights around the road vehicle in a reliable manner be recorded. The information provided by the two sensors be doing with the inventive method for multi-sensorial Object detection evaluated, for example, contained therein Recognize and track people. The sensor information The driver is preferably on a in the cockpit arranged display unit presented in the form of image data, where people and signal lights of traffic lights in the displayed Image information is highlighted. For use with a road vehicle are suitable in connection with the method according to the invention as sensors in addition to cameras, especially radar and lidar sensors. The inventive method is about it out for the use of different types of image sensors and any other known from the prior art sensors suitable.

Further Features and advantages of the invention will become apparent from the following Description of preferred embodiments with reference to Characters. Showing:

1 on the left, an environmental scene captured by means of an FIR camera, on the left, and on the right, an environment scene

2 a suboptimal assignment of two sensor signal currents

3 the feature building in connection with a multi-stream detector

4 the geometric determination of the search space

5 a resulting set of hypotheses in a single-stream hypothesis generator

6 the epipolar geometry of a two-camera system

7 the epipolar geometry using the example of pedestrian detection

8th the cause of scaling differences in correspondence search windows

9 resulting correspondences in the NIR image for a search window in the FIR image

10 the relaxation of the correspondence condition

11 Correspondence error between label and correspondence search window

12 how multistrom hypotheses arise

13 Comparison of detection rates with different screen rulings

14 the detector response as a function of the detection level achieved

15 a coarse-to-fine search in the one-dimensional case

16 for example, the neighborhood definition

17 a hypothesis tree

In the 1 on the left is an environmental scene captured by means of an NIR camera and on the right a scene captured by means of an FIR camera. The two camera sensors and the intensity images recorded with them differ greatly. The NIR image shown on the left has a high variance depending on lighting conditions and surface properties. In contrast, the heat rays detected by the FIR camera, which are shown in the right-hand part of the picture, are almost exclusively direct emissions of the objects. Due to their intrinsic heat, pedestrians in particular generate a pronounced signature in thermal images and stand out greatly from the background in country road scenarios. However, this obvious advantage of the FIR sensor is contrasted with its resolution: it is four times smaller in the X and Y direction than that of the NIR camera. Due to this rough sampling, important high-frequency signal components are lost. For example, a pedestrian 50 meters away in the FIR image only has a height of 10 pixels. The quantization also differs, with both cameras providing 12-bit gray scale images, however, the dynamic range relevant for the detection task extends to 9 bits for the NIR camera and to only 6 bits for the FIR camera. This results in an 8 times larger quantization error. In the NIR camera image object structures are clearly visible, the image is dependent on lighting and surface structure and it has a high intensity variance. In contrast, in the FIR camera image object structures are difficult to detect, the image is dependent on emissions, with the pedestrian stands out clearly from the cold background. Due to the fact that both sensors have various advantages, namely that the strengths of one are the weaknesses of the other, the use of these sensors in the context of the inventive method is particularly advantageous. The advantages of both sensors can be combined in one classifier, which clearly outperforms the detection performance of single-stream classifiers.

The term sensor fusion refers to the use of multiple sensors and the generation of a common representation. The goal is to increase the accuracy of the information obtained. Characteristic here is the union of measured data in a perceptual system. The sensor integration By contrast, the use of different sensors for several subtasks, such as image recognition for localization and haptic sensors for subsequent manipulation with actuators.

• • Fusion auf Signalebene: Hierbei werden direkt die Rohsignale betrachtet. Ein Beispiel ist die Lokalisierung von akustischen Quellen aufgrund von Phasenverschiebungen.
• • Fusion auf Pixelebene: Im Gegensatz zur Signalebene wird der räumliche Bezug von Pixeln zu Objekten im Raum betrachtet. Beispiele sind Extraktion von Tiefeninformation mit Stereokameras oder auch die Berechnung des optischen Flusses in Bildfolgen.
• • Fusion auf Merkmalsebene: Bei der Fusion auf Merkmalsebene werden unabhängig Merkmale beider Sensoren extrahiert. Diese werden z.B. in einem Klassifikator oder einem Lokalisierungsverfahren kombiniert.
• • Fusion auf Symbolebene: Symbolische Repräsentationen sind beispielsweise Wörter oder Sätze bei der Spracherkennung. Durch Grammatiken entstehen logische Beziehungen zwischen Wörtern. Diese wiederum können die Interpretation von akustischen und visuellen Signalen steuern.

Fusion approaches can be categorized based on their resulting representations. For example, the following four fusion levels are distinguished:

• • Fusion at the signal level: The raw signals are viewed directly here. An example is the localization of acoustic sources due to phase shifts.
• • Pixel-level fusion: In contrast to the signal plane, the spatial reference of pixels to objects in space is considered. Examples are extraction of depth information with stereo cameras or the calculation of optical flow in image sequences.
• • Feature-level fusion: Feature-level fusion independently extracts features from both sensors. These are combined, for example, in a classifier or a localization method.
• • Symbol-level fusion: For example, symbolic representations are words or phrases used in speech recognition. Grammars create logical relationships between words. These in turn can control the interpretation of acoustic and visual signals.

• • Gewichtete Mehrheitsentscheidung: Ein einfaches Prinzip ist die Mehrheitsentscheidung, also die Wahl der Klasse die von den meisten Klassifikatoren ausgegeben wurde. Jeder Klassifikator kann entsprechend seiner Zuverlässigkeit gewichtet werden. Mittels Lerndaten können ideale Gewichte ermittelt werden.
• • Bayes-Kombination: Für jeden Klassifikator kann eine Konfusionsmatrix berechnet werden. Das ist eine Verwechslungsmatrix, die die Häufigkeit von allen Klassifikatorergebnissen für jede tatsächliche Klasse angibt. Mit ihr können bedingte Wahrscheinlichkeiten für resultierende Klassen approximiert werden. Nun werden alle Klassifikationen mit Hilfe des Bayes-Theorems auf Wahrscheinlichkeiten für Klassenzugehörigkeiten abgebildet. Als Endresultat wird sodann das Maximum gewählt.
• • Stacked Generalization: Die Idee bei diesem Ansatz ist die Verwendung der Klassifikatorergebnisse als Eingänge bzw. Merkmale eines weiteren Klassifikators. Der weitere Klassifikator kann dabei mit dem Vektor der Ergebnisse und dem Label des ersten Klassifikators trainiert werden.

Another form of fusion is the classifier fusion. Here the results of several classifiers are united. The data sources or the sensors are not necessarily different. The goal here is to reduce the classification error by redundancy. The decisive factor is that the individual classifiers have errors that are as uncorrelated as possible. Some methods for merging classifiers include:

• • Weighted majority decision: A simple principle is the majority decision, ie the choice of class issued by most classifiers. Each classifier can be weighted according to its reliability. Using learning data ideal weights can be determined.
• • Bayes combination: For each classifier a confusion matrix can be calculated. This is a confusion matrix indicating the frequency of all classifier results for each actual class. It can be used to approximate conditional probabilities for resulting classes. Now all classifications are mapped to probabilities for class membership using the Bayes theorem. As the final result, the maximum is then selected.
• • Stacked Generalization: The idea behind this approach is to use the classifier results as inputs or characteristics of another classifier. The further classifier can be trained with the vector of the results and the label of the first classifier.

Possible fusion concepts in the detection of pedestrians are the detector fusion and a merger on feature level. There are already acceptable solutions to the detection problem with only one sensor, so combining by classifier fusion is possible. In the case of two classifiers and a two class problem considered here, a merger by weighted majority vote or Bayesian combination results in either a simple and operation or an or operation of the single detectors. The AND operation has the consequence that (with the same parameterization) the number of detections and thus the detection rate can only be reduced. With an OR operation, the false alarm rate can not get better. How meaningful the respective links are can be determined by determining the confusion matrices and analyzing the correlations. However, it is possible to make a statement about the resulting effort: In the case of the OR operation, the images of both streams must be scanned; the effort is at least the sum of the expenditure of both single-stream detectors. As an alternative to the AND or OR combination, the detector result of the cascade classifier can be interpreted as a probability of inference by mapping the achieved level and the last activation to a detection probability. This allows a decision function to be defined on non-binary values. Another possibility is to use the one classifier for attention control and the other classifier for detection. The former should be parameterized so that the detection rate (at the expense of the false alarm rate) is high. This may reduce the amount of data of the detecting classifier, making it easier to classify. Feature-level fusion is mainly due to the availability of boosting techniques. The concrete combination of features from both streams can thus be done automatically with the already used method based on the training data. The result represents approximately an optimal choice and weighting of the features from both streams. An advantage here is the extended feature space. If certain subsets of the data can only be separated easily in one of the individual flow feature spaces, then the combination can simplify the separation of all data. For example, the pedestrian silhouette is clearly visible in the NIR image, whereas the FIR image shows an illumination-independent contrast between pedestrian and background. In practice it has been shown that with the merger at characteristic level the number of necessary Features can be drastically reduced.

following the architecture of the used multistrom classifier is described. For the Extension of the single-stream classifier to the multi-stream classifier It is necessary that many parts of the classifier architecture be revised become. One exception is the kernel algorithm, e.g. AdaBoost, which does not necessarily have to be modified. Still, some can implementation-technical optimizations are made, which the duration of an NIR training run with a predetermined parameterization many times lower. It will be the complete table the characteristic values for all examples kept in memory. Another point is the Optimizations in the example generation. This could be practical Use training runs ended with 16 sequences in about 24 hours. Before this optimization Training with only three sequences took two weeks. The integration from further streams The application is implemented as part of a redesign of the implementations. In the expansion of the hypothesis generator are the most Modifications and innovations needed.

The following describes the major enhancements in terms of data preprocessing. The resulting detector will be used in the form of a real-time system and with live data from the two cameras. The training uses labeled data. An extensive database of sequences and labels is available for this purpose, which includes country road scenes with pedestrians running by the roadside, cars and cyclists. Although the two sensors used record approx. 25 pictures per second, the temporal sampling is done asynchronously due to the hardware, the times of both pictures are independent. Because of variations in the recording times even a significant difference in the number of images of the two cameras for a sequence is common. An application of the detector is not possible as soon as only one feature is not available. If, for example, you replace the respective terms in the Stronglearner equation with zeros for missing features, the behavior is undefined. This makes sequential execution of the individual images of the multi-stream data impossible and requires synchronization of the sensor data streams both for training and for the application of a multi-stream detector. In this case, image pairs must be formed. Since the recording times of the images of a pair are not exactly the same, a different state of the environment is shown in each case. That is, the position of the vehicle and that of the pedestrian is a different one. In order to minimize any influence of environmental dynamics, the image pairs must be formed so that the differences of the two time stamps become minimal. Because of the mentioned different number of measurements per unit of time, either images from a stream must be used several times, or images are omitted. Two reasons are in favor of the latter: First, it minimizes the average time stamp difference, and secondly, multiple use online would lead to occasional spikes in computational overhead. The following algorithm describes the data synchronization:

In this case, ε _s should be chosen as a function of the distribution of t _s (i + 1) -t _s (i) and should be about 3 ^σ . With small ε _{s, there} is the possibility that some image pairs are not found, for large ε _s the expected time stamp difference increases. The assignment rule corresponds to a greedy strategy and is therefore generally less than optimal with respect to minimizing the mean time stamp difference. However, it can be used both in training and in online operation of the application. In the case of V ar (t _s (i + 1) -t _s (i)) = 0 and ε _s = 0 ∀ _s , it is advantageously optimal.

In the 2 For example, a suboptimal assignment of two sensor signal currents is shown. Here, in particular, the result of the allocation algorithm shown above is shown. In this example, the mapping is sub-optimal with respect to minimizing the mean timestamp difference. The assignment algorithm can be used in this form for the application, there are advantageously no delays due to waiting for potential assignment candidates.

The search window concept plays a central role in feature formation, especially in the extension of the multi-sensor deployment detector, where multiple sensor signal streams are present. In a single-current detector, the localization of all objects in a picture consists of examining a set of hypotheses. A hypothesis stands for a position and scaling of the object in the image. This results in the search window, ie the image section, which is used for the feature calculation. In the multi-stream case, one hypothesis consists of a search window pair, that is, one search window in each stream. It should be noted that for a single search window in one stream due to the parallax problem, different combinations of search windows in the other stream may occur. Thus, a very large number of multi-stream hypotheses can arise. A hypothesis generation for any camera arrangements will be shown later. The classification is based on features of two search windows, as shown by the 3 is clarified. The 3 shows the feature formation in connection with a multi-stream detector. A multi-stream feature set corresponds to the union of the two feature sets that result for the single-stream detectors. A multi-stream feature is defined by filter type, position, scale, and sensor current. In the NIR search window smaller filters can be used due to the higher image resolution than in the FIR search window. The number of NIR features is thus higher than the number of FIR features. In this embodiment, approximately 7000 NIR features and approximately 3000 FIR features were used.

In An advantageous way will be during the training process continuously selected new training examples. Before the training by means of each classifier level is calculated using all already trained levels created a new example set. Compete in multi-stream training the training examples like the hypotheses from a search window in every stream. Positive examples result from labels, which are present in every stream. In connection with automatically generated Negative examples come here a mapping problem: The randomly selected search window have to consistent regarding the projection geometry of the camera system, so the training examples agree with the multistrom hypotheses of later application. To achieve this, a special hypothesis generator is created will be described in detail below, in the determination used the negative examples. Instead of the position as before and size of the search window independent of negative examples and by chance to choose, will now be random into a set of hypotheses. In this case, the set of hypotheses in addition to consistent search window pairs a smarter, on world models based distribution of hypotheses in the picture. Also for single stream training This hypothesis generator can be used. Here are the negative examples with the same search strategy determines which later in the application of the detector for the generation of hypotheses. The example set for the multistrom training consists of positive and negative examples, which in turn each contain a search window in both streams. For the Training is used, for example, AdaBoost, with all the features all examples are calculated. The feature selection changes across from the single-stream training only the number of features, because due their definition and the associated multistrom data source is abstracted.

The Architecture of a multi-stream detector application is that of a single-current detector very similar. The necessary modifications to the system are on the one hand adjustments for the general handling of multiple sensor signal currents, resulting in almost all Make the implementation changes required are. On the other hand, the hypothesis generator is extended. For the Generation of multi-stream hypotheses is a correspondence condition for search window both streams necessary, which is based on world and camera models. Thus, must a multi-stream camera calibration be integrated into the hypothesis generation. The for single-stream detectors used brute force search in the hypothesis space while transmitting to multi-stream detectors, but then it turns out to be common as too inefficient. The search space increases significantly and the number of hypotheses multiplies. To still have real-time capability, must the set of hypotheses are scaled down again and they are smarter Search strategies required. The in connection with this embodiment pursued fusion approach corresponds to a merger at feature level. Ada-Boost will be a combination of features of both streams selected. Other methods could used here for feature selection and merger. The required changes at the detector is an extended feature set, a synchronization the data as well as the generation of a hypothesis set, which geometrical relationships the camera models are taken into account.

The derivation of a correspondence rule, search space sampling and other profitable optimizations are presented below. The trained single-stream cascade classifier evaluates individual search windows one after the other. As a result, the classifier provides a statement as to whether an object was detected in exactly this position and scaling. In each image, pedestrians can appear in different positions with different scales. Therefore, when using the classifier as a detector, a large number of positions or hypotheses must be checked in each image. This set of hypotheses can be reduced by subsampling and search range constraints. Thus, the calculation effort can be reduced without affecting the detection performance. For this purpose, hypothesis generators for single-stream applications are already known from the prior art. In the multistrom detector presented in connection with this exemplary embodiment, hypotheses are defined via a search window pair, that is to say via a search window in each stream. Although the search windows can be generated in two streams with two single-stream hypothesis generators, the link to the multistrom hypothesis set is not trivial due to the parallax. The assignment of two search windows from different streams to a multi-stream hypothesis must fulfill certain geometric conditions. In order to achieve robustness against calibration errors and dynamic influences, furthermore, relaxations of these geometric correspondence conditions are introduced. Finally, a concrete sampling and allocation strategy is chosen. There are many more hypotheses than single-stream detectors. In order to ensure the real-time capability of the multistrom detector, further optimization strategies are shown below, including a very effective method for hypothesis reduction via a dynamic local control of the hypothesis density, which can also be used in connection with single-stream detectors. The simplest search strategy for finding objects at all positions in the image is the pixel-by-pixel scanning of the entire image in all possible search window sizes. This results in a picture with 640 × 480 pixels a hypothesis amount of about 64 million Ele instruments. This set of hypotheses is referred to below as the complete search space of the single-stream detector. With the aid of a range restriction described below on the basis of a simple world model and a scaling-dependent subsampling of the search space, the number of hypotheses to be examined can be reduced in a particularly advantageous manner to approximately 320,000. The basis for the range restriction is on the one hand the so-called "ground plane assumption", the assumption that the world is flat, with the objects to be detected and the vehicle being on the same level. On the other hand, due to the object size in the image and an assumption regarding the real object size, a unique position in space can be derived. All the hypotheses of a scaling in the image lie on a horizontal line. Both assumptions, ie the "ground plane assumption" as well as with respect to a fixed real object size are generally not true. The restrictions are therefore relaxed, so that a certain tolerance range is allowed for the object position as well as for their size in space, this situation is in the 4 illustrated. The relaxation of the "ground plane assumption" is indicated by an angle ε, which in this embodiment is for example 1 °. Thus, orientation errors in the camera model are compensated, which can arise, for example, by pitching movements of the vehicle. In addition to the described range limitation, the number of hypotheses to be examined is further reduced by a scaling-dependent subsampling. The step size of the scanning in u- and v-direction in the 4 are chosen proportional to the height of the hypothesis, ie the scaling, and in this example is about 5% of the hypothesis height. The search window heights themselves are the result of a series of scaling, each increasing by 5% starting with 25 pixels in the NIR image (8 pixels in FIR image). This type of quantization can be motivated with a property of the detector, namely the fact that the size scaling of the features also increases the blurring of their localization in the image, as is the case, for example, with a Haarwavelet or similar filters. The features are defined here in a fixed grid and are scaled according to the size of the hypothesis. With the hypothesis generation described in this case, a reduction of the 64 million hypotheses of the complete search space to 320,000 results in the NIR image. In the FIR image, there are due to the low resolution 50,000 hypotheses, this is also on 5 directed. For the consideration of the restrictions defined in three-dimensional space, a transformation between image coordinates and world coordinates is necessary. The basis for this are the intrinsic and extrinsic camera parameters determined by the calibration. The geometrical relationships for the projection of a 3D point onto the image plane are known to those skilled in the field of image analysis. Due to the small distortion in both cameras, a hole camera model can be used in this embodiment.

The 4 illustrates the geometric determination of the search space. Here, the search area is displayed, which results in a fixed scaling. An upper and lower limit is calculated for the upper search window edge in the image. The limits (v _min and v _max ) result when the object is projected onto the image plane once with the smallest and once with the largest expected object size (obj _min or obj _max ). Here, the distance (z _min and z _max ) is chosen so that the correct scaling is created in the image. Due to the relaxed restriction of the ground plane assumption, the spatial position lies between the dashed planes. The smallest and the largest object are moved up and down to calculate the limits.

In the 5 the resulting set of hypotheses of the single-stream hypothesis generator is shown. In this case, search windows with a grid-like arrangement are generated. For different scalings different square lattices are created with adapted lattice spacings and own area restrictions. In the sense of a clear representation is in the 5 visualized only one search window for each scaling as well as the midpoints of all other hypotheses. The illustration is exemplary, with large scaling and position increments selected.

From the single-stream hypotheses thus arise by suitable pairing multistrom hypotheses. The epipolar geometry is the basis for pair formation, which describes the geometric relationships. In the 6 the epipolar geometry of a dual-camera system is shown. The epipolar geometry describes the set of possible correspondence points for a point in an image plane. Epipolar lines and an epipolar plane can be constructed for every point p in the image. The possible correspondence points for points of an epipolar line in an image are exactly the same on the corresponding epipolar line of the other image plane. In the 6 In particular, the geometry of a multi-camera system with two arbitrarily arranged cameras with the centers O ₁ ∈ R ³ and O ₂ ∈ R ³ and an arbitrary point P ∈ R ^{3 is} shown. O ₁ , O ₂ and P thereby span the so-called epipolar plane. It cuts the image planes in the epipolar lines. The epipoles are the intersections of the image planes with the line O ₁ O ₂ . O ₁ O ₂ is contained in all epipolar planes of all possible points P. All occurring epipolar lines thus intersect in the respective epipole. The Epipolarlinien have the following meaning in the correspondence: For each Point p in the image can be used to construct epipolar lines and an epipolar plane. The possible correspondence points for points of an epipolar line in one image are exactly those on the corresponding epipolar line of the other image plane.

Let point P ∈ R ^{3 be} a point in space. Let P1, P2 ∈ R ³ be the representation of P in the camera coordinate systems with origin O ₁ or O ₂ . Then there is a rotation matrix R ∈ R ^{3 × 3} and a translation vector T ∈ R ³ for which holds P 2 = R (p 1 - T). (5.1)

R and T are clearly defined by the relative extrinsic parameters of the camera system. P ₁ , T and P ₁ - T are coplanar, ie (P 1 - T) T · (T × P 1 ) = 0. (5.2)

Equation (5.1) and the orthonormality of the rotation matrix give:

The cross product can now be rewritten into a scalar product:

mit E := RS der Essentiellen Matrix. Nun ist eine Beziehung zwischen P₁ und P₂, hergestellt. Projiziert man sie mittels

so ergibt sich:

This results from equation (5.3)

with E: = RS of the Essential Matrix. Now, a relation between P ₁ and P _{2 is} established. Projected by means of

this results in:

Here f _{1,2 is} the focal length and Z _{1,2 is} the Z component of P _1,2 . Thus, the set of all possible pixels p ₂ in the second image that can correspond to a point p _{1 of} the first image are exactly those for which the equation (5.6) is satisfied. With this correspondence condition for individual pixels, consistent search window pairs from the single-stream hypotheses can now be formed as follows: The aspect ratio of the search windows is preferably fixed by definition, ie a search window can be unambiguously described by the midpoints of the upper and lower edge. With the correspondence condition for pixels, two epipolar lines thus result in the image of the second camera for the possible midpoints of the upper and lower edges of all corresponding search windows, as described, for example, in US Pat 7 is pictured. The 7 shows the epipolar geometry using the example of pedestrian detection. Here, an ambiguous projection of a search window from the image of the right camera into that of the left camera takes place. The correspondence search windows result from the epipolar lines of the centers of the search window lower and upper edges. For reasons of clarity, the illustration is only illustrative here. The set of possible search window pairs should include all those search window pairs that describe objects of realistic size. If one calculates the backprojection of the objects into the space, the position and size of the object can be determined by means of triangulation. The range of epipolar lines is then reduced to correspondences with valid object size, as indicated by the dotted line in 7 is shown.

It will now be described the optimization of the correspondence space, wherein in the projection result in a search window from one sensor current in the other sensor current several correspondence search window with different scaling. However, this difference in scale disappears if the camera positions and orientations are the same except for a lateral offset. For the scaling, therefore, only one offset d between the centers O ₁ and O ₂ in the longitudinal direction of the camera system is relevant, as described in US Pat

8th will be shown. The difference in orientation of both cameras is negligible in this example. In the 8th In this case, the cause of the scaling differences arising in the correspondence search windows is shown in particular, and in the projection of a search window from the first into the second sensor stream, a plurality of correspondence search windows with different scaling results. Here, the geometric relationship between camera arrangement, object sizes and scaling differences is shown in detail.

untersucht werden, wobei h₂ ^min bzw. h₂ ^max die minimale bzw. maximale auftretende Skalierung der Korrespondenzsuchfenster im zweiten Sensorstrom zum Suchfenster h₁ im ersten Sensorstrom ist. Es seien H^min = 1m die Höhe eines nahen Fußgängers und H^max = 2m die Höhe eines weit entfernten Fußgängers, wobei hier nur Fußgänger betrachtet werden, welche eine minimale Größe von 1m und eine maximale Größe von 2m aufweisen. Beide Fußgänger seien so weit entfernt, dass sie im Bild der ersten Kamera die Höhe h1 aufweisen.There is a fixed search window size h ₁ in the first image. The following is the relationship

where h ₂ ^min or h ₂ ^{max is} the minimum or maximum occurring scaling of the correspondence search window in the second sensor current to the search window h ₁ in the first sensor current. Let H ^min = 1m the height of a nearby pedestrian and H ^max = 2m the height of a faraway pedestrian, only pedestrians being considered here having a minimum size of 1m and a maximum size of 2m. Both pedestrians are so far away that they have the height h1 in the picture of the first camera.

Continue to be Z ₁ ^min , Z ₁ ^max , Z ₂ ^min and Z ₂ ^max the object distances of both objects to both cameras, then follows Z 2 min, max = Z 1 min, max - d (5.7) and

The scaling ratio then results

gilt. Es lässt sich somit der Korrespondenzraum für ein Suchfenster im ersten Strom, also die Menge der korrespondierenden Suchfenster im zweiten Strom folgendermaßen vereinfachen: Die Skalierung aller korrespondierenden Suchfenster wird vereinheitlicht. Die verwendete Skalierung h₂ für alle Korrespondenzen ist der Mittelwert der minimal und maximal auftretenden Skalierung:

For long distances, the scaling ratio goes to 1. For an application of the classifier as an early warning system in country road scenarios, one can limit itself to values greater than 20m when choosing Z ₁ ^min . The offset of the cameras is about 2m in the test carrier. Together with the above suggested values for pedestrian sizes, it follows that

applies. Thus, the correspondence space for a search window in the first stream, that is to say the set of corresponding search windows in the second stream, can be simplified as follows: The scaling of all corresponding search windows is standardized. The scaling h _{2 used} for all correspondences is the mean of the minimum and maximum scaling:

The resulting scaling error is a maximum of 2.75%. In the 9 Resulting correspondences in the NIR image are shown for a search window in the FIR image. It becomes a unified Ska used for all corresponding search windows.

• • Fehler in den extrinsischen und intrinsischen Kameraparametern, verursacht durch Messfehler während der Kamerakalibrierung.
• • Einflüsse der Umgebungsdynamik.

For modeling the correspondence error, in practice the pairing described above for generating multi-stream hypotheses is often inadequate. In a profitable way, the following factors are taken into account in addition:

• • Errors in extrinsic and intrinsic camera parameters caused by measurement errors during camera calibration.
• • Influences of the environmental dynamics.

So there is an unknown error in the camera model. This creates a blur for both the position and the scaling of the correlated search windows, it is referred to below as a correspondence error. The scaling error is neglected for the following reasons: First, the influence of the dynamics on the scaling is very small if the object is at least 20m away. Secondly, a significant insensitivity of the detector response can be seen in terms of the accuracy of the hypothesis scaling. This can be seen by multiple detections, whose centers hardly vary, but the scales vary greatly. To compensate for the translational error, a relaxation of the correspondence condition is introduced. For this purpose, a tolerance range for the position of the correlated search window is defined. For each of these correspondences an ellipse-shaped tolerance range is defined in the image with the radii e _x and e _y , in which further correspondences arise, as shown by the 10 will be shown. The correspondence error is identical for each search window scaling. The resulting tolerance range is therefore chosen to be the same for every scaling.

• • Für jedes Multistrom-Label werden die Suchfenster in beiden Strömen bestimmt.
• • Zu dem jeweiligen Suchfenster im ersten Strom werden alle möglichen Korrespondenzsuchfenster im zweiten Strom berechnet. Dabei wird eine nicht-relaxierte Korrespondenzbedingung verwendet.
• • Das Korrespondenzsuchfenster, das dem Labelsuchfenster im zweiten Strom am nächsten kommt, wird für die Fehlerbestimmung herangezogen. Die Nähe zweier Suchfenster kann hier entweder durch die Überdeckung definiert sein, insbesondere durch das Verhältnis der Schnittfläche zweier Rechtecke zu deren Vereinigungsfläche (auch Coverage genannt) oder durch den Abstand der Suchfenstermittelpunkte. Letztere Definition wurde bei diesem Ausführungsbeispiel gewählt, da so der für die Detektorantwort unkritische Skalierungsfehler vernachlässigt wird.
• • Für alle Labels wird der Abstand in X- und Y-Richtung zwischen dem Labelsuchfenster und dem am nächsten gelegenen Korrespondenzsuchfenster bestimmt. Es entsteht dabei eine Häufigkeitsverteilung für die X- und Y-Abstände. Ein Histogramm über den Abstand in X- und Y-Richtung ist in der 11 dargestellt.
• • Nun werden die Radien e_x und e_y von der Verteilung der Abstände abgeleitet. In dieser Arbeit wurde e_x = 2^σ x und e_y = 2^σ y gewählt. Der nächste Schritt nach der Definition des Korrespondenzraumes für ein Suchfenster ist die Suchraumabtastung. Wie bei der Einzelstrom-Unterabtastung soll auch hier die Hypothesenzahl bei möglichst geringen Einbußen bei der Detektionsleistung minimiert werden.

In the 10 the relaxation of the correspondence condition is shown. The positions of the correlating search windows are not limited to one route only. You can now lie within an elliptical area around this distance. The NIR image shows only the center points of the search windows. In relation to this correspondence error, data labeled with the radii are used. The radii of the elliptical tolerance range are determined as follows:

• • For each multistrom label, the search windows in both streams are determined.
• • All possible correspondence search windows in the second stream are calculated for the respective search window in the first stream. In this case, a non-relaxed correspondence condition is used.
• • The correspondence search window that comes closest to the label search window in the second stream is used for error determination. The proximity of two search windows can be defined here either by the overlap, in particular by the ratio of the intersection of two rectangles to their union surface (also called coverage) or by the spacing of the search window center points. The latter definition was chosen in this embodiment, since this neglects the scaling error that is uncritical for the detector response.
• • For all labels, the distance in the X and Y directions is determined between the label search window and the closest correspondence search window. This results in a frequency distribution for the X and Y distances. A histogram of the distance in the X and Y directions is in the 11 shown.
• • Now the radii e _x and e _y are derived from the distribution of the distances. In this work e _x = 2 ^σ x and e _y = 2 ^σ y were chosen. The next step after defining the correspondence space for a search window is the search space scan. As with single-stream subsampling, the number of hypotheses should also be minimized with as little loss as possible in the detection performance.

11 shows the correspondence error between label and correspondence search window. The illustrated correspondence error is the smallest pixel distance of a label search window to the correspondent search windows of the corresponding label, so the projected label of the other sensor signal stream. In the illustrated measurement, FIR labels are projected into the NIR image and a histogram is formed over the distances of the search window centers.

The procedure for the search space sampling proceeds as follows: In both streams, single-stream hypotheses, ie search windows, are scattered with the single-stream hypothesis generator. In this case, the resulting scaling stages must be matched to one another, wherein in the first stream the scalings are determined by the hypothesis generator. For each of these scaling levels, the correspondence space of a prototypical search window is then determined. The scaling of the second stream results from the scaling of the correspondence spaces of all prototypical search windows. This creates the same number of scaling levels in both streams. Now, search window pairs are formed, resulting in the multi-stream hypotheses. One of the two streams can then be selected in order to determine the respective correspondence area in the other stream for each search window. All search windows of the second stream, which have the correct scaling, which are within this range, become together with the fixed search window of the first stream used for pairing, this is based on the 12 clarified. It shows the 12 the resulting multistrom hypotheses. There are three search windows in the FIR image and their correspondence areas in the NIR image. Couples are formed with the search windows scattered by single-stream hypothesis generators. A multi-stream hypothesis corresponds to a search window pair.

You choose for the Internally used single-current prosthetic generators Position and scale increments of 5% of the search window height, as shown In the NIR picture about 400,000 single-stream hypotheses, in the FIR image about 50,000. However, there are about 1.2 million multistrom hypotheses. In practical use could process at 2 frames per second be achieved. To the real-time capability to ensure the application Further optimizations will be presented below. For one thing a so-called Weaklearner cache described the number of necessary feature calculations reduced. In addition, a procedure for the dynamic reduction of the hypothesis set, below denoted by multiraster hypothesis tree. The third optimization, which is called backtracking, reduced in the case of a Detection unnecessary Effort in connection with multiple detections.

mit den Weaklearnern h k / t ∊ {–1, 1} und Hypothese x.Evaluating multiple multi-stream hypotheses that share a search window will cause Weaklearner to be computed multiple times on the same data. To avoid all redundant calculations, a caching method is now used. In this case, partial sums of the Stronglearner calculation are stored in tables for each search window in both streams and for each Stronglearner. A stronglearner H ^{k of} the cascade stage k is defined by:

with the Weaklearnern hk / t ε {-1, 1} and Hypothesis x.

S ^k (x) can be separated into two sums, which contain only weapon learners with characteristics of a stream:

If several hypotheses x _i in a stream s have the same search window, then in each stage k for the stream s the sum S _s (x _i ) is the same for all x _i . The result is preferably buffered and used several times. If it is possible to resort to already calculated values for a Stronglearner calculation, the effort is reduced in a profitable manner to a sum operation and a threshold operation. As far as the size of the tables is concerned, in this exemplary embodiment 12.5 million entries result for a total of 500,000 search windows and 25 cascade stages. 64-bit floating-point numbers require 100 MB of memory. For an effort estimate, the number of feature calculations can be considered both with and without a Weaklearner cache. In the former case, the number of hypotheses per image and the number of all features are decisive. The number of hypotheses can be estimated by the number of search windows R _s in the streams s with O (R1 * R2). However, the factor hidden in the O-notation is very small here, since the correspondence area is small compared to the entire image area. The number of calculated features is then in the worst case O (R1 * R2 * (M1 + M2)) where Ms is the number of features in each stream s. In the second case, each feature in each search window is calculated at most once per image. Thus, the number of calculated features is at most O (R1 * M1 + R2 * M2). In the worst case, the effort is reduced by the factor min (R1, R2). A complexity analysis for the average case, however, is more complex because the relationship between the average number of calculated features per hypothesis or search window in the first case and in the second case is not linear.

Below are comments on the multiraster hypothesis tree. The search space of the multistrom detector was detected in this example with two single-stream hypothesis generators and a relaxed correspondence relationship. In this case, however, it is difficult to find an optimal parameterization, especially the finding of the appropriate sampling step sizes. On the one hand, they have a major influence on the detection performance and, on the other hand, on the resulting computational effort. For the single-stream detectors could at a prak In the FIR case acceptable compromises could be found in the FIR case because of the lower image resolution, but in the NIR case this was not possible with the hardware used. The performance of the experimental computer used was also insufficient when using a fusion detector with Weaklearner cache and resulted in longer response times in complex scenes. Of course, these problems can be solved with more powerful hardware.

In practical use, various parameterizations of the hypothesis generator and the detector were tested. Several search grid densities and different step restrictions were evaluated. It has been found that, even with very coarse scanning, every pedestrian to be detected is already recognized with the first stages of the detector. In this case, the rear cascade stages were switched off successively, resulting in a high false alarm rate. The measured values recorded during practical use are in the 13 shown. The number of hypotheses were starting with the finest grid density: approximately 1,200,000, 200,000, 7,000 and 2,000.

The 13 shows the comparison of the detection rates of different screen widths, whereby four different hypothesis grid densities are compared. For each screen ruling, the detection rate of a fusion detector is plotted against the number of stages used. The detection rate is defined by the number of pedestrians found divided by the number of pedestrians. The reason for the phenomenon that has occurred is the following property of the detector: The detector response, ie the cascade stage reached, is at most a hypothesis which is positioned exactly on the pedestrian. If one pushes the hypothesis step by step away from the pedestrian, the detector result does not drop abruptly to zero, but there is an area in which the detector result varies greatly and tends to decrease. This behavior of the cascade detector is referred to below as a characteristic detector response. An experiment in which an image is scanned in pixel steps is in the 14 visualized. It uses a multistrom detector and fixed scale hypotheses. One can easily see the area for which the detector response decays. Furthermore, it has been found that the detector has similar characteristics in a fixed position, varying scale experiment. Thus, the detection performance of the shortened detector applied to a rough hypothesis grid to explain, because the "target area" for a pedestrian increases for lower levels.

In the 14 the detector response is shown as a function of the detection level achieved. In this case, a multistrom detector is applied to a set of hypotheses in a scaling with pixel-precise grid. The last cascade level reached is plotted for each hypothesis at its midpoint. During training, no training examples slightly offset to a label are used. Only exact positive examples are used as well as negative examples, which have a large distance to each positive example. Thus, the behavior of the detector is undefined in hypotheses that are slightly offset from an object. Therefore, the characteristic detector response is experimentally investigated for each detector. The central idea for reducing the number of hypotheses is a coarse-to-fine search, whereby each image is searched in the first step with a roughly resolved set of hypotheses. Depending on the detector result, further hypotheses with higher density are scattered in the image. In addition, the local neighborhood is searched for hypotheses that suggest an object in its vicinity. By the above described behavior of the detector, the achieved number of stages can be taken as criteria for the refinement of the search. Following the same principle, the local neighborhood of the new hypotheses can then be searched again until the finest hypothesis grid is reached. For each refinement step, a threshold is used to compare the achieved cascade level of each hypothesis.

15 shows a coarse-to-fine search in the one-dimensional case. For this purpose, a picture line from the in 14 shown image recording, which in the form of a function in the 15 is shown. From left to right you can see the steps of the search process. The hypothesis results are horizontal and the thresholds for local refinement are shown horizontally. For the threshold determination, the experiment described above can be used. The detection rate of each screen density is almost identical for the first stages of the detector. The threshold value is the maximum level for which the affected screen density still has almost the same detection rate as the maximum achievable. For the threshold level k of a grid density L, a detection rate DL / k is required, so

DH / k denotes the detection rate of the finest screen density H in step k. If n is the number of Refinements, then results for the last stage K of the detector, a detection rate D K = α n · D H K

For α are in this example mainly Values between 0.98 and 0.999 are suitable.

In defining the neighborhood, the hypothesis space is considered. The hypothesis space is now not one-dimensional but in the case of the single-stream detector three-dimensional or six-dimensional in the fusion detector. The problem of gradual refinement in all dimensions is solved with the hypothesis generator. There are two possibilities for defining the neighborhood, of which the second is used in this embodiment. On the one hand, a minimum value for the coverage of two adjacent search windows can be defined. In this case, however, it is not clear how to choose the minimum value, since gaps can arise in the refined sets of hypotheses, ie areas that are not close enough to any hypothesis of the coarse hypothesis set. Therefore, different thresholds must be set for each grid density. On the other hand, the neighborhood can be defined with a modified checkerboard distance. Thus, the mentioned gaps are avoided and it can be defined a uniform threshold for all screen densities. The chessboard distance is defined by

The array density of a current is defined by r _x, r _y, r _h ^ε R. The grid spacings are for a search window height h then in the _x direction r _x · h and in the _y direction r _y · h. For a search window height h ₁ , the next largest search window height is h ₂ = h ₁ * (1 + rh). The neighborhood criterion for a search window with position s ₁ ε R ² and search window height h ₁ to a search window s ₂ ε R ^{2 of} a finer hypothesis set with height h ₂ is defined with a scalar δ:

The resulting interval limits are in the 16 visualized. In the multi-stream case, there is a three-dimensional neighborhood criterion in each stream. For adjacent multi-stream hypotheses, the neighborhood condition in both streams must be met. If you choose r _x = r _y and δ = 0.5, then all neighborhoods are disjoint except for the edges. If the step sizes r _x and r _y for the refinement hypothesis sets are successively halved and the additional hypotheses fall precisely on the boundaries of the neighborhood areas, this value makes sense for δ, since the finer hypotheses are linked to all adjacent coarser hypotheses. However, this does not apply if the refined sets of hypotheses have any grid spacing. Then, by choosing δ> 0.5, it must be achieved that the neighborhoods of adjacent coarse set hypotheses overlap and the fine-grid hypotheses are assigned to multiple coarse-grid hypotheses. The required value for δ must be determined by experiments, ie it must be adapted to the characteristic detector response.

In the 16 the neighborhood definition is shown: The neighborhood is plotted for three of the hypotheses of the same scaling level, and on the right three different scales and their resulting scaling neighborhood are shown. For δ, 0.75 was chosen here.

The generation of the refined hypotheses during the application would be too time consuming and may as well be done as a pre-processing step. The generation of all refined hypothesis sets is done by means of the hypothesis generator. First, the set of hypotheses for each refinement level is generated. Then the hypotheses are linked to the neighborhood criterion, each hypothesis being compared to each hypothesis of the next finer set of hypotheses. If these are close, they are linked. This results in a tree-like structure whose roots correspond to the hypotheses of the coarsest stage. In the 17 Since the edges represent the calculated neighborhood relationships. Since the generation of the hypothesis tree involves a certain search effort, the calculations required for this purpose are preferably realized via a separate tool and stored in the form of a file.

17 shows the resulting hypothesis tree. The hypothesis tree / search tree has several roots and is searched from the roots to the leaf level, if the detection result of a node is greater than the threshold. When processing an image (or image pair in the case of the multi-stream detector), the hypothesis tree is traversed. Beginning with the first tree root, the tree is searched with a depth or breadth first search. The hypothesis of the root is evaluated. As long as the corresponding threshold value is exceeded, the tree is descended and the respective child node hypotheses are examined. Then the search continues at the next tree root. Along with the backtracking method described below, the depth search is most effective. Since nodes can have multiple parent nodes, care must be taken to examine each node only once. The use of a multiraster hypothesis tree results in a profitable way a reduction in the number of hypotheses, which affects the detection performance.

The The number of multiple detections is very high for the multi-stream detector and the FIR detector high. Multiple detections therefore have a major influence on the computing time, as they go through the entire cascade. It will therefore be a so-called Backtracking method used. With a change of search strategy can a large part the multiple detections are avoided, wherein in the case of detection the search in the hypothesis tree is aborted and at the next tree root will continue. This will reduce the density of hypotheses locally, as soon as an object is found. Not a systematic mistake All child nodes are randomly permuted so that their Order is not correlated with their arrangement in the picture. If the first child hypotheses, for example, always in the upper left corner of the neighborhood lie, the detection tends to shift in that direction become.

Based this embodiment was thus starting from the single-stream Hyothesengenerator, by Modeling a relaxed correspondence area and finally by Various optimizations developed a procedure which, despite the complex search space of the multistrom data has very little computation time requires. An important contribution is made by the multiraster hypothesis tree.

The Use of the Multiraster Hypothesis tree is not just in the frame the multisensor fusion of great Advantage, but is particularly suitable for interaction with cascade classifiers in general and leads to significant better classification results.

Claims (18)

Method for multisensor object recognition, wherein sensor information from at least two different Sensor signal streams with different sensor signal properties for common evaluation be used, wherein the at least two sensor signal streams for evaluation not adapted to each other and / or mapped to each other, in which case in each of the at least two sensor signal streams object hypotheses generated, based on these object hypotheses characteristics for at least a classifier are generated and where the object hypotheses evaluated by the at least one classifier and one or be assigned to several classes, with at least two classes are defined and objects are assigned to one of the two classes. Method according to claim 1, characterized in that that the object hypotheses are uniquely assigned to a class. Method according to claim 1, characterized in that that the object hypotheses are assigned to several classes, where the respective assignment is assigned a probability. Method according to one of the preceding claims, characterized characterized in that the object hypotheses independently in each sensor signal current be generated individually, the object hypotheses of different Sensor signal streams then over Assignment rules are assignable to each other. Method according to one of claims 1 to 3, characterized that object hypotheses in a sensor signal current (primary current) are generated and object hypotheses of the primary current in other sensor signal currents (secondary currents) projected where an object hypothesis of the primary stream is one or more object hypotheses in the secondary current generated. Method according to claim 5, characterized in that that the projection of object hypotheses of the primary stream into a secondary stream on the sensor models used and / or the positions of search windows within the primary stream or based on the Epipolargeometrie. Method according to one of the preceding claims, characterized in that object hypotheses are described by their object type, object position, object extent, object orientation, object motion parameters such as direction of motion and velocity, object danger potential, or any combination thereof. Method according to one of the preceding claims, characterized characterized in that object hypotheses in a physical search space fortuitously scattered or generated in a grid or by a physical Model generated. Method according to claim 8, characterized in that that the search space is determined by external specifications such as opening angle, distance ranges, statistical parameters that obtained locally in the image, and / or measurements of other sensors adaptively restricted becomes. Method according to one of the preceding claims, characterized characterized in that the different sensor signal characteristics in the sensor signal currents in different positions and / or orientations and / or Sensor sizes of used sensors are based. Method according to claim 10, characterized in that that each object hypothesis is classified individually for itself and the results of each classifications are combined, wherein at least one classifier is provided. Method according to claim 10, characterized in that in that in the at least one classifier features of object hypotheses evaluated together and to different sensor signal currents summarized a classification result. Method according to one of the preceding claims, characterized characterized in that the grid in which the object hypotheses be generated, depending the classification result is adapted adaptively. Method according to one of claims 1 to 12, characterized that the grid in which the object hypotheses are generated dependent on the classification result of a previous time step adaptively adapted. Method according to one of the preceding claims, characterized in that the evaluation method by means of which the Object hypotheses are evaluated, depending on at least one automatically adjusted to the previous rating. Method according to one of the preceding claims, characterized characterized in that at least two different sensor signal currents offset in time be used or that a single sensor signal current in common used with at least one time-shifted version of the same becomes. Use of the method according to one of the preceding claims for tracking detected objects. Use of the method according to one or more the claims 1 to 16 for environmental detection and / or object tracking at a Road vehicle.