DE102015200437A1 - Method and device for determining the confidence of an object recognition - Google Patents

Abstract

A method (400) for determining the reliability of an object recognition system for detecting an object in an image (200) is described. The image (200) comprises a plurality of pixels (203). The method (400) includes determining (401) a global lighting situation of an image (200) based on the plurality of pixels (203). In addition, the method (400) comprises determining an image section (202) for the detection of the object, wherein the image section (202) comprises a subset of the plurality of image points (203). Furthermore, the method (400) comprises determining (402) a local illumination situation of the image detail (202) on the basis of the subset of the pixels (203). In addition, the method (400) includes determining (403) the reliability of the object recognition system for detecting an object in the image pane (202) based on the global lighting situation and based on the local lighting situation.

Description

The invention relates to a method and a corresponding device for detecting objects, in particular for detecting persons in the environment of a vehicle.

To capture the surrounding situation, modern vehicles are equipped with camera systems that record image data relating to an environment of the vehicles. Object classifiers can be applied to the image data to detect objects (such as people) in front of the vehicle. However, the appearance of persons in an image is not constant, but varies significantly with the lighting conditions of the environment in which a person resides. Due to the different appearances, the detection quality of the object classifiers typically decreases.

The present document deals with the technical task of improving the detection quality of object classifiers with changing lighting conditions. Furthermore, the present document deals with the technical task to determine in a precise manner the confidence of an object recognition. In addition, the present document addresses the technical problem of efficiently providing object classifiers with high detection quality.

One or more of the o.g. Tasks are solved by the independent claims. Advantageous embodiments are described i.a. in the dependent claims.

According to one aspect, a method for determining the reliability of an object recognition system or an object recognition method for detecting an object in an image is described. The image includes a plurality of pixels (also referred to as pixels). The pixels may have different gray levels and / or brightnesses. The image may have been captured by a camera (e.g., a stereo camera). The camera may be mounted on a vehicle and may be configured to detect an environment of the vehicle (e.g., an environment in front of the vehicle). The object to be detected may be e.g. to be a person (especially a pedestrian).

The reliability can be indicated by values for one or more confidence measures. The one or more confidence measures may include one or more of, a correct classification rate of the object recognition system, an error rate of the object recognition system, a precision of the object recognition system, a recall of the object recognition system, a user's accuracy of the object recognition system, and / or a producer's accurary of the object recognition system.

The method includes determining a global lighting situation of an image based on the plurality of pixels. In other words, a situation of illuminating the entire image can be determined (e.g., by analyzing all the pixels of the image). In particular, an illumination class can be determined for the overall image, which describes the global illumination situation of the image.

Furthermore, the method comprises determining an image section for the detection of the object, the image section comprising a subset of the plurality of pixels. Typically, an image comprises a plurality of image sections. The plurality of image sections can be determined by moving a search window along a horizontal axis and / or along a vertical axis of the image with a predefined step size.

In particular, such a scanning of the image can be used to determine a sequence of image sections, wherein for each image section of the sequence or the plurality of image sections, an expected reliability of the object recognition system or object recognition method can be determined.

An image section may have a specific size and / or shape. The size and / or the shape of the image section may depend on the object to be detected. This ensures that a frame of the image section can surround the object to be detected. When choosing the size of the image sections, a considered distance of the object to be detected can also be taken into account. In particular, image sections of different sizes can be analyzed to detect objects at different distances from the camera with which the image was acquired.

The method further comprises determining a local illumination situation of an image detail on the basis of the subset of the pixels of the image detail. In other words, a situation of illuminating the image section can be determined (for example, by analyzing all pixels of the image section). In particular, an illumination class which describes the local illumination situation of the image section can be determined for the image detail. In an analogous manner, the local lighting situations for each image section from the sequence or the plurality of image sections of the image can be determined.

In addition, the method includes determining, based on the global lighting situation and based on the local lighting situation, the reliability of the object recognition system for the detection of an object in a frame having the specific combination of global lighting situation and local lighting situation.

By determining the combination of global illumination situation and local illumination situation of an image section, an expected reliability of the object recognition can thus be determined even before an object recognition method or before the application of an object recognition system. Thus, the computational effort for the object recognition can be reduced in one image. In particular, such an order in which the plurality of image sections of an image are searched for objects can be optimized (for example, to speed up the detection of an object).

The reliability of the object recognition system (or the object recognition method) for the detection of an object in the image detail can be determined on the basis of predefined reliability data. In doing so, the reliability data may assign at least one reliability value (e.g., for one or more confidence measures) to different combinations of a global lighting situation and a local lighting situation. Thus, the reliability of the object recognition for a frame can be determined efficiently.

The reliability data may have been determined based on a test data set having a plurality of test images. Alternatively or additionally, the reliability data may be arranged in a look-up table that indicates expected reliability (e.g., values of one or more confidence measures) for each possible combination of a global lighting situation and a local lighting situation.

As already stated above, the image typically comprises a plurality of image sections. The method may include determining a plurality of local lighting situations for the plurality of frames.

Furthermore, the method may include determining, based on the plurality of local lighting situations and based on the global lighting situation, a plurality of reliability values of the object recognition system and the object recognition method for detecting an object in the plurality of image sections, respectively. Furthermore, depending on the plurality of reliability values, an order of the plurality of image sections for the detection of an object can be determined. Thus, the computational effort for the object recognition can be reduced or at least prioritized.

The object recognition system / method may e.g. have a first reliability value for a first image section of the plurality of image sections, and the object recognition system / method can have a second reliability value for a second image section of the plurality of image sections. The first reliability value may be higher than the second reliability value. The order of the plurality of image sections can then be selected such that it is first determined in the first image section and only then in the second image section whether the respective image section has the object to be detected. Thus, a prioritization of the object recognition can be carried out depending on the expected reliability. Thus, (on average) the detection of an object within an image can be accelerated.

Determining a lighting situation on the basis of pixels (eg determining a global lighting situation or determining a local lighting situation) may include determining, based on the pixels (eg the pixels of the image or the pixels of the image), a plurality of quality measures. Values for a corresponding variety of quality measures. Exemplary quality measures are a statistical quantity from a histogram of brightness values of the pixels and / or a central moment which describes a center of gravity of the brightness values of the pixels. The plurality of quality measure values may form a feature vector for the pixels for which a lighting situation is to be determined.

Furthermore, determining a lighting situation may include determining, based on the plurality of goodness-of-values, one lighting class from a plurality of predefined lighting classes, the determined lighting class indicating the lighting situation of the pixels. The plurality of illumination classes may include one or more of, a lighting class indicative of a lighting situation with daylight, and / or a lighting class indicating a lighting situation with artificial light, in particular in a rural region, in a residential area or in a city center.

The illumination class for the pixels can be determined by means of an illumination classifier that is set up, feature vectors of one (and possibly several) Assign lighting classes from the wide range of lighting classes. Furthermore, based on the illumination classifier, a probability can be determined with which the feature vector (and therefore the image or the image detail) falls into an illumination class. Possibly. For example, an image section or an image can be assigned by the illumination classifier to a plurality of illumination classes, and it can be determined with which probability the image section or the image has the respective illumination class. The illumination classifier may include a Bayes classifier, an Expectation Maximization classifier, a K-Means classifier, and / or a Support Vector Machine classifier. The illumination classifier may have been trained based on a training record having a plurality of training images.

Thus, by using lighting classes and possibly by using a lighting classifier in an efficient and robust manner, the lighting situation of pixels (e.g., pixels of an image or pixels of a frame) can be determined.

Determining whether an image portion of the image comprises an object to be detected may include determining, based on the subset of pixels of the image detail, a descriptor for the image detail. The descriptor may e.g. according to a HOG (Histogram of Oriented Gradients) model, a LBP (Local Binary Patterns) model, and / or a wavelet model.

Furthermore, determining whether an image section of the image comprises an object to be detected comprises classifying the image section using an object classifier. In this case, the object classifier is set up to associate the descriptor of the image section with a positive object class or a negative object class. The positive object class may indicate that the frame has the object to be detected, and the negative object class may indicate that the frame does not have the object to be detected. The object classifier may be e.g. a Bayes classifier, an Expectation Maximization classifier, a K-Means classifier, and / or a Support Vector Machine classifier. The object classifier may have been trained based on a training record having a plurality of training images.

The detection method which is used for the detection in an image section may depend on the illumination situation of the image section. In particular, a type of the descriptor, a resolution of the descriptor, a type of the object classifier and / or a training data set which was used to teach the object classifier may depend on the global lighting situation and on the local lighting situation of the image section. Thus, the detection method used can be adapted in a precise manner to the illumination situation of the image section, and thus the reliability of the object recognition method can be increased.

The image may correspond to a current image from a sequence of temporally successive images. The global illumination situation of the current image can then be determined based on the pixels of a plurality of images from the sequence of images. In particular, the plurality of images may include the current image and one or more images captured (possibly directly) temporally prior to the current image. In an analogous manner, the local illumination situation of an image section of the current image can be determined on the basis of corresponding subsets of the pixels of the plurality of images. By considering a plurality of images, the quality of the determined lighting situations can be increased.

As already explained, the image may have been detected by a camera of a vehicle. The camera may be configured to detect an environment in front of the vehicle. The global and / or the local lighting situation can also be determined on the basis of a-priori knowledge regarding the lighting situation in the environment in front of the vehicle. The a-priori knowledge may include one or more of: a time of day the image was acquired; a date on which the image was captured; the rising and / or sunset time of the sun on the day the image was captured; and / or a geographic location of the environment represented in the image (e.g., GPS coordinates in conjunction with digital map information). By considering a-priori knowledge, the quality of the determined lighting situations can be increased.

In addition to the camera, the vehicle may comprise one or more environment sensors, in particular a radar sensor and / or a lidar sensor, which are set up to capture surroundings data relating to the surroundings of the vehicle. The global and / or the local lighting situation can then also be determined on the basis of the environmental data in order to increase the quality of the determined lighting situation. In particular, the method may further include detecting, based on the environmental data, a static or dynamic object in the environment of the vehicle that may affect the lighting situation of the image. The static object can For example, an (illuminated) street lamp and the dynamic object may include, for example, an oncoming vehicle. The light sources of these objects can lead to a change in the localized lighting conditions, which can be taken into account when determining the lighting situation. In particular, the global and / or local lighting situation can also be determined on the basis of the detected static or dynamic object in order to increase the quality of the determined lighting situation.

The method may further comprise determining an expected distance of the object to be detected. The subset of the plurality of pixels for determining the local illumination situation of the image section can then be determined on the basis of the expected distance of the object to be detected. In particular, a resolution of pixels considered can be reduced with decreasing expected distance of the object to be detected (and thus an increasing number of pixels in which the object to be detected is shown).

The resolution of an image section for determining the illumination situation or the illumination class can be reduced adaptively, in particular as a function of the distance in which the object to be detected is to be expected. For example, a person in the vicinity includes a high number of pixels, which typically are not all needed for the determination of the lighting situation. Here, by reducing the resolution, computing time can be saved for determining the lighting situation. On the other hand, a person in the far field typically includes a few pixels which, if possible, should all be used to determine the lighting situation.

In one aspect, a controller configured to perform one or more of the methods described in this document is described.

In another aspect, a vehicle (e.g., a passenger car, a truck, or a motorcycle) is described that includes a control unit described in this document.

In another aspect, a software (SW) program is described. The SW program may be set up to run on a processor (e.g., on a controller) and thereby perform the method described in this document.

In another aspect, a storage medium is described. The storage medium may include a SW program that is set up to run on a processor and thereby perform the method described in this document.

It should be understood that the methods, devices and systems described herein may be used alone as well as in combination with other methods, devices and systems described in this document. Furthermore, any aspects of the methods, apparatus, and systems described herein may be combined in a variety of ways. In particular, the features of the claims can be combined in a variety of ways.

Furthermore, the invention will be described in more detail with reference to exemplary embodiments. Show

1a an exemplary vehicle with an environment camera;

1b exemplary components of a vehicle;

2 exemplary image excerpts from an image captured by the surrounding camera;

3a a flowchart of an exemplary method for detecting an object taking into account the lighting situation;

3b an exemplary method for classifying a lighting situation;

3c a flowchart of an exemplary method for detecting an object;

4 a flowchart of an exemplary method for determining the reliability of an object detection method;

5a exemplary stereo images and a disparity image determined therefrom;

5b a flowchart of an exemplary method for detecting an object, taking into account disparity;

5c an exemplary histogram of disparity values;

6a an exemplary distribution of the probability of objects to be detected in image sections with different lighting situations; and

6b a flowchart of an exemplary method for determining training data for an object classifier.

As set forth above, the present document is concerned with the reliable recognition of an object (especially a person) based on image data. In this context shows 1a an exemplary vehicle 100 , The vehicle 100 has a camera 101 which is set up, an environment in front of the vehicle 100 capture. In the example shown, the camera is 101 at a rearview mirror in the interior of the vehicle 100 arranged. As in 1a shown, the camera can 101 a stereo camera, which is arranged, a right image and a left image of the environment of the vehicle 100 capture. On the basis of the right image and the left image depth information can be determined (eg a disparity). In particular, based on the right image and the left image, information regarding a distance of a detected object to the vehicle may be obtained 100 be determined.

1b shows a block diagram with exemplary components of a vehicle 100 , The from the camera 101 captured image data 111 (eg a sequence of images) can be sent to a control unit 102 be passed on. The control unit 102 is set up based on the image data 111 to determine if in the front of the vehicle 100 an object (especially a person) is or is not. Furthermore, the control unit 102 be arranged to initiate one or more actions when an object has been detected. In particular, one or more control devices 103 of the vehicle 100 be prompted to carry out one or more measures. Exemplary measures are the issuing of an (audible and / or visual) warning to a driver of the vehicle 100 , or performing an automatic braking of the vehicle 100 in order to avoid a collision with the detected object or to reduce the consequences of a collision.

2 shows an exemplary picture 200 from the image data 111 , The picture 200 has a plurality of pixels 203 on, which is an environment of the vehicle 100 play. The picture 200 can be analyzed to get one in the picture 200 to recognize the represented object (especially a person). The analysis of the picture 200 may possibly be on a partial area of the image 200 be limited because, for example, only objects are to be detected, which has a predefined distance to the vehicle 100 do not exceed. For example, the analysis of the image 200 on a lower screen area below a horizontal line 201 be limited because objects are above this line 201 too far away from the vehicle 100 are located. Thus, the computational effort to recognize an object can be reduced.

To recognize an object, individual image sections 202 (also referred to as samples) of the part of the image to be analyzed 200 to be viewed as. A picture detail 202 can have a rectangular shape, with a height 211 and a width 212 , The height 211 and / or the width 212 a picture detail 202 can be changed to different sized objects (eg an adult or a child) and / or objects at different distances to the vehicle 100 to detect. In this case, depending on typical sizes and / or distances from an object to be detected, a minimum height / width or a maximum height / width of image sections may be necessary 202 be determined. So can the computational effort for the search for a picture detail 202 that includes an object can be reduced.

It can thus be a picture detail 202 from the picture 200 to be viewed as. The picture section 202 typically includes a smaller number of pixels than the image 200 , Based on the procedure described in this document, it can be determined whether the image section 202 includes an object to be recognized, eg whether on the image section 202 a person is recognizable. Furthermore, by the size (eg the height 211 and the width 212 ) of the image section 202 Information regarding a distance of a detected object to the vehicle 100 be determined.

It can thereby different image details 202 from the picture 200 to be viewed as. In particular, different image sections 202 along a horizontal image axis 213 and / or along a vertical image axis 214 to be viewed as. This can also be overlapping image sections 202 to be viewed as. For example, a plurality of image sections 202 are considered, each by a certain increment along the horizontal and / or vertical axis 213 . 214 differ from each other. Alternatively or additionally, image sections 202 with different heights / widths 211 . 212 to be viewed as.

The picture section 202 may also be referred to as a search window, wherein the positions of the search window by a raster-shaped scanning of the image 200 result. The form of the search window as well as the raster width of the scan result from the respective application. For the detection of pedestrians, for example, a high-edged rectangle (as in 2 shown).

As stated above, different lighting conditions typically lead to significant differences in the reliability of detecting objects in an image 200 , 3a shows a flowchart of an exemplary method 300 to recognize an object in an image 200 , The procedure 300 includes determining 301 a global lighting class for the image 200 , In other words, the pixels 203 of the picture 200 can be analyzed to the entire picture 200 (or the entire part of the picture 200 to search for an object) to a specific lighting class. This will be the pixels 203 of the entire picture 200 or the entire part of the image to be analyzed 200 considered. Exemplary lighting classes are, for example, a tag class indicating a lighting situation in daylight and a night class indicating a lighting situation in artificial light. Due to very different lighting situations at night, the night class can be divided into other sub-classes. For example, for a night situation, a highway class, a residential class, and a city center class may be defined which indicate different levels of artificial lighting situations.

The procedure 300 includes, determining 302 a local lighting situation 302 for a picture section 202 of the picture 200 , In particular, the image section 202 a lighting class (eg one of the above lighting classes) are assigned. Thereby only the pixels become 203 of the respective image section 202 considered. Thus, for each image section 202 a picture 200 a global lighting class (GK), which illustrates the lighting situation of the image 200 reproduces from which the image section 202 comes, and a local lighting class (LK) are determined.

The picture 200 for which a GK and / or a LK is to be determined may correspond to a current image from a sequence of temporally successive images. The GK of the current image can then be based on the pixels 203 be determined from a variety of images from the sequence of images. In particular, the plurality of images may include the current image and one or more images captured (possibly directly) temporally prior to the current image. In a similar way, the LK of a picture detail 202 of the current image based on corresponding subsets of the pixels 203 the plurality of images are determined. By considering a plurality of images, the quality of the determined BK and / or LK can be increased.

The picture 200 can, as already stated, by a camera 101 of a vehicle 100 have been recorded. The camera 101 can be set up, an environment in front of the vehicle 100 capture. The GK and / or the LK can also on the basis of a-priori knowledge regarding the lighting situation in the environment in front of the vehicle 100 be determined. The a-priori knowledge may include one or more of: a time of day on which the image 200 was recorded; a date on which the picture 200 was recorded; the rising and / or sunset time of the sun on the day on which the picture 200 was recorded; and / or a geographic location of the environment contained in the image 200 is shown (eg the GPS coordinates in conjunction with digital map information). By considering a-priori knowledge, the quality of the determined GK and / or LK can be increased.

The vehicle 100 can over the camera 101 In addition, one or more environment sensors, in particular a radar sensor and / or a lidar sensor, which are set up, environmental data relating to the environment of the vehicle 100 capture. The GK and / or the LK can then also be determined on the basis of the environmental data in order to increase the quality of the determined GK and / or LK. In particular, the method can 300 further comprising detecting, based on the environmental data, a static or a dynamic object in the environment of the vehicle 100 , the influence on the lighting situation of the picture 200 may have. For example, the static object may be an (illuminated) street lamp, and the dynamic object may include an oncoming vehicle, for example. The light sources of these objects can lead to a change in the localized illumination conditions, which can be taken into account when determining the GK and / or LK. In particular, the GK and / or LK can also be determined on the basis of the detected static or dynamic object in order to increase the quality of the determined GK and / or LK.

The procedure 300 may further comprise determining an expected distance of the object to be detected. The number of pixels 203 for determining the LK of a picture detail 202 can then be determined based on the expected distance of the object to be detected. In particular, a resolution of pixels considered can be used 203 with decreasing expected distance of the object to be detected (and a thus increasing number of pixels 203 , in which the object to be detected is shown) can be reduced.

The resolution of a picture detail 202 to determine the LK can thus be reduced adaptively, in particular depending on the distance in which the object to be detected is to be expected. For example, a close-up person includes a large number of pixels 203 which are typically not all needed for the determination of LC. Here, by reducing the resolution, computation time for the determination of the LK be saved. On the other hand, a person in the far field typically includes a few pixels 203 , which should be used, if possible, all for the determination of LK. In addition, the process includes 300 the performing 303 an object recognition within the image section 202 , In this case, the object recognition takes place taking into account the GK and / or the LK of the image section 202 , In particular, an algorithm for detecting an object within the image section 202 from the GK and / or the LK of the image section 202 depend. By taking into account the GK and / or the LK of a picture excerpt 202 For example, the reliability (ie, confidence) of the object recognition can be substantially improved. This is especially true in nocturnal lighting situations.

3b shows an exemplary method 310 for determining a local illumination class for a picture detail 202 , The procedure 310 may be analogous to determining a global illumination class for an image 200 be used. The procedure 310 includes determining 311 of one or more quality measures based on the pixels 203 of the image section 202 , The quality measures can also be referred to as image features. Typically, for a picture detail 202 determined several quality measures, which together a feature vector of the image section 202 represent. By means of the feature vector, the illumination situation of the image section becomes 202 described.

The quality measures are used to the content and / or the quality of a picture 202 to describe. A single quality measure usually provides no direct information about the lighting conditions. However, by interpretation or by combining several quality measures, a description of the illumination can typically be derived.

An exemplary measure of quality is a quality measure without a reference, based solely on the pixels 203 of the image section 202 is determined. Another exemplary quality measure is a quality measure with or with limited reference, which is determined by comparison with a reference. For example, (eg by Gaussian smoothing), a background image of the image section 202 which can be used as a reference. The original image section 202 can then be compared to the background image to determine one or more quality measures.

• • ein Unschärfemaß in Bezug auf Kanten in einem Bildausschnitt 202;
• • ein zentrales Moment, das einen Schwerpunkt von einer Helligkeitsverteilung von Bildpunkten 203 eines Bildausschnittes 202 beschreibt;
• • ein Kompressionsmaß, welches die Unschärfe und Blockbildung in einem Bildausschnitt 202 anzeigt;
• • ein Maß, welches eine statische Auswertung von DCT (diskrete Cosinustransformation) Koeffizienten des Bildausschnittes 202 umfasst.

The quality measures may include, for example, information-theoretical quantities (such as eg the entropy and / or the anisotropy) and / or statistical variables which are based, for example, on the pixels 203 and in particular based on a histogram of the brightness and / or colors of the pixels 203 a picture section 202 can be determined. Other examples of quality measures without reference are

• • a blur on edges in a frame 202 ;
• • a central moment that focuses on a brightness distribution of pixels 203 a picture detail 202 describes;
• • a compression measure, which blurs and blocks in a frame 202 displays;
• A measure, which is a static evaluation of DCT (discrete cosine transformation) coefficients of the image section 202 includes.

• • ein Distanzmaß, welches z.B. einen Unterschied zwischen dem originalen Bildausschnitt 202 und dem Hintergrundbild des Bildausschnitts 202 anzeigt;
• • statistische Größen, welche aus dem originalen Bildausschnitt 202 und dem Hintergrundbild ermittelt werden;
• • ein Kontrastmaß, welches Helligkeitsunterschiede des originalen Bildausschnitts 202 und des Hintergrundbilds anzeigt;
• • Referenzbasierte Unschärfemaße; und/oder
• • Gradient-basierte Maße, welche Gradienten im originalen Bildausschnitt 202 und im Hintergrundbild vergleichen.

Examples of quality measures with reference reference are

• • a distance measure, which eg a difference between the original picture detail 202 and the background image of the image section 202 displays;
• • statistical quantities, which are taken from the original image section 202 and the background image;
• • a contrast measure, which brightness differences of the original picture detail 202 and the background image;
• • reference-based blur measures; and or
• • Gradient-based measures, which gradients in the original image section 202 and compare in the background image.

The procedure 310 further includes mapping 312 of the image section 202 to a lighting class from a variety of predefined lighting classes. The assignment 312 takes place on the basis of one or more determined quality measures for the image section 202 , For this purpose, a lighting classifier can be used which is set up, the feature vector of an image section 202 assign a lighting class. The lighting classifier may include, for example, a Bayes classifier, an Expectation Maximization classifier, a K-Means classifier, and / or a Support Vector Machine classifier.

The illumination classifier is typically learned based on training data (ie, a training dataset). Thus, the parameters of the classifier model of the classifier (eg one reference feature vector per illumination class) can be determined. The learning methods can be divided into supervised and unsupervised algorithms. Monitored algorithms use a training record in which for each sample (ie, for each frame 202 or for each picture 200 of the data record) it is known to which class the respective random sample is assigned. The samples are "labeled". Unmonitored algorithms use a record from which that is not known. A sample is a picture 200 (to determine the global illumination class) or image detail 202 (to determine the local lighting class) on which all quality measures used were calculated. As already explained above, the values of the quality measures determined for a sample can be combined into a feature vector for this sample.

It can thus be determined from the training record a plurality of feature vectors for the plurality of samples of the training record.

If the illumination classes of the plurality of samples are known (in the case of a monitored algorithm), a plurality of feature vectors can be determined for each illumination class, which represents the corresponding illumination class. For example, a reference feature vector may thus be determined for each illumination class (e.g., as the average of the plurality of feature vectors of that illumination class). The reference feature vectors may then be used by the illumination classifier for classification. For example, to classify a first sample for which the illumination class is not known, the illumination classifier may compare a first feature vector of that first sample with the reference feature vectors, and determine the reference feature vector closest to the first feature vector (according to FIG predefined distance measure). The first sample can then be assigned to the illumination class that corresponds to the determined reference feature vector.

It can thus by means of the method 310 efficiently and reliably the lighting class of an image 200 or a picture detail 202 be determined. The GK and LK of a picture excerpt 202 can then be used to one to the lighting situation of the image 202 perform customized object detection. Furthermore, based on the GK and the LK of a picture detail 202 the expected quality, ie the reliability, of the object recognition are determined (see the procedure 400 ).

The object recognition in a picture detail 202 can be based on a descriptor-classifier combination. This will be a descriptor of the image section 202 used to display the displayed content of the image section 202 to describe. Based on the description provided by the descriptor of a picture detail 202 accepts or discards the classifier for a particular frame 202 the hypothesis of whether an object (eg a person) in the image detail 202 is present. This classifier is also referred to as an object classifier in this document.

3c shows an exemplary method 320 for detecting an object in a picture detail 202 , The procedure 320 includes determining 321 from a descriptor for the image section 202 , A specific descriptor model can be used. Exemplary descriptor models are the HOG (Histogram of Oriented Gradients) model and the LBP model (Local Binary Patterns).

The HOG model is a gradient-based approach, that is, the HOG model captures the shape of the objects to be detected. Based on the gray values or the color values of the pixels 203 a picture detail 202 be for each pixel 203 Brightness gradient or color gradient determined. Furthermore, for every pixel 203 the directions of the maximum and / or minimum brightness gradient or color gradient are determined.

The method determines as feature vector a cell-wise histogram of the gradient directions. To compute the feature vector, an image section 202 divided into overlapping blocks. Each block in turn consists of a certain number of cells, which in turn consist of several pixels 203 consist. The cells do not overlap but are flush with each other. You can choose square blocks as well as cells, even if the image section 202 not square. For every pixel 203 the gradient is determined in the vertical and horizontal image direction. From these quantities, the strength and orientation of the gradient can be calculated using known formulas. From this, a histogram of the gradient directions is created for each cell. The directions are discretized into equidistant values. Usually, the discretized values are related only to the semicircle of 0 ° to 180 °, which means that it is not discriminated whether the gradient describes a brightness transition from dark to light or vice versa. The histogram entries are weighted with the associated gradient strength. All histograms of a block are then normalized. For normalization, a specific normalization method, such as the L2 standard, is applied to all histogram entries in the block. Since the block normalization depends on the gradient strengths, this results in a balance between blocks with higher and lower contrast. If individual block values are higher than a predefined threshold, they are previously limited to the threshold value. The concatenation of the histograms of all cells of an image section 202 forms the descriptor of the image section 202 ,

Alternatively or additionally, the LBP model can be used as a descriptor. Like the HOG model, it is based on the evaluation of gray value differences of the pixels 203 a picture section 202 , The descriptor consists of a block-wise assembled histogram with an indication of the sign of the gray value differences of the pixels 203 in a block.

The procedure 320 further includes mapping 322 of the image section 202 to an object class. The assignment is based on the descriptor of the image section 202 , An object classifier can be used for the assignment. Exemplary classifiers include a Bayes classifier, an expectation maximization classifier, a k-means classifier, and / or a support vector machine classifier. The classifier can be learned based on a training record (analogous to the learning method described for the illumination classifier, wherein the descriptors of the image sections 202 have the analogous function to the above-mentioned feature vectors of a lighting classifier).

A security value can be defined for the object classifier. This is especially true for a binary object classifier that indicates in binary form whether a frame is 202 contains an object or not. The security value can be used to set the reliability of the object classifier. In particular, the security value can be used to balance false classifications (where an image detail 202 , which in fact does not represent an object, is falsely classified as an image detail with a detected object, also referred to as a so-called false positive) and misclassifications (in which a picture detail 202 , which actually represents an object that is falsely classified as an image section without an object, also referred to as a so-called false negative).

For example, the object classifier may include a reference descriptor determined based on training data. For the classification of a first image section 202 can be a first descriptor of the first image section 202 be compared with the reference descriptor. In particular, a distance between the first descriptor and the reference descriptor (according to a predefined distance measure) can be determined. It may then be determined whether the determined distance is equal to or less than a predefined distance threshold. If this is the case, then the first image section 202 as a picture section 202 be classified with detected object. On the other hand, the first image section 202 as a picture section 202 to be classified without an object. The distance threshold may depend on the security value. The safety value can thus be used to shift the relationship between false classifications and misclassifications.

Thus, with the method 320 In an effective and accurate way, it can be determined whether in a particular frame 202 a picture 200 an object (eg a person) is or is not. The reliability of the object recognition can be increased by the fact that the object recognition method, which is suitable for a specific image section 202 is applied, of the lighting situation of this image detail 202 (ie in particular from the GK and / or the LK of this image section 202 ) depends.

• • das verwendete Deskriptor-Modell (z.B. HOG-Modell, LDP-Modell, Haar/Edgelet-Wavelets-Modell, etc.);
• • die verwendete Auflösung eines Deskriptors;
• • der Typ des Objekt-Klassifikators (z.B. Bayes-Klassifikator, Expectation Maximization Klassifikator, K-Means Klassifikator, Support Vector Machine Klassifikator, etc.); und/oder
• • die Trainingsdaten, welche zum Anlernen des Objekt-Klassifikators verwendet werden.

For example, one or more of the following parameters of the object recognition method may be dependent on the illumination situation of the image detail 202 depend:

• • the descriptor model used (eg HOG model, LDP model, Haar / Edgelet wavelets model, etc.);
• • the used resolution of a descriptor;
• • the type of the object classifier (eg Bayes classifier, expectation maximization classifier, K-means classifier, support vector machine classifier, etc.); and or
• • the training data used to teach the object classifier.

In other words, this document becomes a procedure 300 to recognize an object based on an image 202 (eg a gray value image) described. For different lighting situations, different discrete lighting categories (ie lighting classes), such as day sun, day shadow, night with street lamps, night without lighting, etc., are defined. Based on the pixels 203 a picture detail 202 For example, quality measures, such as blur measurements, anisotropy, central moments, etc., can be determined in order to determine the lighting situation of the image section 202 to describe. Furthermore, based on the pixels 203 a picture detail 202 a descriptor, such as a HOG, LBP, and / or Haar / Edgelet wavelet descriptor, is determined to obtain the image detail 202 to describe in relation to an object to be recognized.

By means of the camera 101 becomes a picture 200 detected. There are typically no possibilities for influencing which aperture or exposure time the camera 101 used. Accordingly, different exposures result from image to image. These exposures are also affected by the lighting in the area. The lighting situations can thus change from picture to picture.

It can be the lighting category of the imaged environment in the picture 200 be determined. This is for the entire picture 200 a feature vector is calculated with quality measures and classified according to the lighting categories (using a lighting classifier). It can thus provide a global illumination class (GK) for the image 200 be determined. The feature vector contains as an entry one or more quality measures for the description of the lighting situation of the image 200 ,

Furthermore, by a sliding window method in a similar manner for a plurality of image sections 202 a local lighting classification determined. In other words, it can be used for the large number of image sections 202 the local lighting classes (LK) are determined. The LKs of the large number of image sections 202 can be displayed on the screen, resulting in a "heat map" regarding the illumination of the image 200 ,

As stated above, a variety of different quality measures may be used for a frame 202 or for the picture 200 be determined. It can be determined based on principal component analysis and / or Support Vector Machine (SVM) classification, which of the different quality measures for the assessment of the global or the local lighting situation are relevant. It is thus possible to determine the smallest possible number of quality measures for the feature vector, by means of which, nevertheless, a possible feature space can be described as completely as possible. In other words, by judiciously choosing the quality measures for a feature vector, the dimension of the feature vector can be reduced with minimal impact on the accuracy of the illumination classification. Thus, the computational effort of the illumination classification can be reduced.

Depending on the detected global and local lighting category, different object classifiers may be applied to determine if and which object is in a frame 202 located. For this purpose, an object classifier can be trained for a training record with uniform lighting conditions. This is preferably done for data sets from all lighting categories. For relatively low lighting conditions (eg, for a night class), edge-based features such as HOG or LBP can be used with a relatively low resolution of the detector window. For relatively high illumination conditions (eg for a tag class), finer resolutions of the detector window are to be preferred due to the greater detail depth. In addition, texture-based features are also suitable as descriptors in such cases.

The procedure 300 Thus, through the determination and the consideration of the lighting situation, an increased quality of the person detection in comparison to other classifiers is possible. It can thus be a higher proportion of the one picture 200 recognized persons are recognized. Furthermore, the number of assignments of other objects to the group of detected persons (ie the number of so-called "false positives") can be reduced. In addition, the choice of a (for the lighting situation) suitable object classifier, the complexity of the object detection can be reduced. In particular, multiple object classifiers do not have to be used in order to achieve the best possible recognition rate. Thus, the computing time to achieve the highest possible recognition rate can be reduced.

As stated above, a plurality of quality measures can be determined to determine a local or a global lighting situation. In order to reduce the number of quality measures required, methods such as the principal component analysis or support vector machine may be used to determine a relevant subset of quality measures representing (almost) the total information content of the total quantity of quality measures. Thus, the computational effort to determine a lighting class can be substantially reduced.

The principal component analysis (PCA) method is a transformation method in which a multivariate size is represented by so-called major components. A main component is formed by linear combination of the elements of the original representation. These contribute to each major component different degrees. The weights are determined so that as many properties of the original representation remain.

The transformation parameters from the original representation to the main components are determined from the eigenvectors of the covariance matrix of the original representation. It tries to maximize the covariance between the data. This leads to a low connection and thus high information content of the individual elements. Thus, all information of the original representation can be represented by as few elements as possible in the new representation. The associated eigenvalues contain a statement about the relevance of each main component.

For data reduction, the principal component analysis can be calculated with all the quality-of-measure vectors of a picture sequence (ie a training record). To avoid loss of information in Having to take purchase, as many major components are chosen as a measure of quality vector has elements. From this, those quality measures are determined which make the largest contribution to the main components. With this information, less relevant quality measures can be discarded and the illumination classification can be restricted exclusively to the relevant quality measures. In order to determine the relevant quality measures, a histogram can be created for all main components, which records the frequency of individual quality measures. For this purpose, the absolute value of the weighting factor of each quality measure is added up over all main components. The highest values in the histogram correspond to the quality measures which provide the most important contribution to the main components. It is assumed that the contribution to the main components provides a statement about the relevance of a quality measure. Thus, those quality measures that are the most relevant are considered most relevant

SVM preservation determines the hyperplane that has the greatest possible distance between the feature vectors of different classes. This level serves as a separation level for classifying feature vectors into one or the other class. It is determined by support vectors. The support vectors from both classes are those feature vectors that each have the smallest distance to the separation plane. For describing the parting plane, the other feature vectors are thus not necessary.

In addition to the standard case of two-class SVM, there is a one-class SVM. The separation level of the one-class SVM indicates whether an element is to be assigned to the trained class or not. This approach can be used, for example, if a class can be well described, but not the possible deviations, which should then no longer be assigned to the class.

For data reduction, such a one-class SVM can be used. In doing so, for each quality measure, a vector is created over all images of a sequence (i.e., a training dataset). This sequence contains images of all lighting classes. The quality measure vectors finally form the feature vectors of the SVM, from which the separation plane is then determined. The support vectors are then considered to be the relevant quality measures to describe the lighting classes. All quality measures that do not serve as support vectors can be considered irrelevant.

It is thus possible on the basis of a principal component analysis and / or by means of a support vector machine method to determine the smallest possible number of quality measures by which the space of possible lighting situations can be described as comprehensively as possible.

As already stated above, it is often necessary to make a statement about the reliability of an object recognition. The reliability can be indicated by one or more confidence measures. The reliability measures can be divided into two groups, the first group of reliability measures providing a statement of reliability for each determined class, and the second group of reliability measures providing a statement of reliability for the overall system of object detection. The reliability baseline is typically three sizes determined by comparing samples with a priori class assignment and the classification results for the same samples. From this, the number or the proportion of correct assignments can be determined for each class (TP, "true positive"). In addition, for each class, you can determine the number or proportion of samples assigned to a wrong class, even though they actually belong to the reference data of another class. This information is referred to as FN (false negative) because the classification result of such samples is erroneously negative from the perspective of a particular class. This is also referred to as a second type error or misclassification. A first type error (or misclassification), on the other hand, occurs when the classifier assigns a sample to a particular class, but actually belongs to a different class. This information is also known as FP (false positive). From this information, different reliability measures can be derived. The confidence measures typically indicate a probability and thus have values from 0% to 100%, the reliability decreasing as the percentage decreases.

An exemplary measure of reliability is the correct classification rate (OA, Overall Accuracy). This measure indicates the proportion of correct classification results for all samples. For a two-class classifier, it can be calculated as follows: OA = TP + TN / TP + TN + FP + FN

Another exemplary measure of reliability is the error rate (E, Error Rate). This size indicates the proportion of errors in the classification results. It does not matter if it's a first or second type error. For a two-class classifier, it can be calculated as follows: E = FP + FN / TP + TN + FP + FN.

Other exemplary reliability measures are Precision (P) and Recall (R). Precision describes the proportion of correctly classified positive samples on all positive detected samples: P = TP / TP + FP.

The recall, on the other hand, sets the correct positive samples of a class relative to the actual samples of the same class: R = TP / TP + FN.

Further exemplary reliability measures are user's accuracy (UA) and producer's accuracy (PA). User's Accuracy gives the ratio between the number of correctly classified samples of a class and the total number of samples correctly or incorrectly assigned to that class. This measure indicates the probability that a sample actually belongs to the class that the classifier chose. The Producer's Accuracy is the ratio of the number of correctly classified samples in a class and the total number of actual samples in that class. It thus indicates the probability that a reference sample was correctly classified. This measure is named because it tells the developer of a classifier how well a particular class can be recognized, while the user's accuracy is more important to the user. User's and Producer's Accuracy can be interpreted as a classwise interpretation of Precision and Recall.

To determine quality statements via a classifier, a so-called validation can be carried out. The classifier is applied to a test record, which should be different from the training record with which the classifier was trained. From the classification results for the test data set one or more of the above-mentioned Reliability measures determined. The size of the test record may be e.g. about 20% of the size of the training record.

As stated above, depending on the lighting situation of an image section 202 different descriptor-classifier combinations are used. For each of the descriptor-classifier combinations used, one or more confidence measures can be determined. The one or more reliability measures can be determined by means of a test data set, which test image sections 202 with certain lighting situations. That is, one or more reliability measures can be determined for each possible lighting situation, which indicate with which quality or with which reliability or with which confidence an object can be detected in the presence of a corresponding lighting situation. In particular, a table or a look-up table (LUT) can be determined, which specify one or more reliability measures for each illumination situation, eg for each combination of GK and LK.

4 shows an exemplary method 400 for determining a reliability measure for the object recognition in a picture detail 202 , As already related to the 3a . 3b , and 3c set out for a picture 200 and for the different parts of the picture 202 a picture 200 be determined by means of a lighting classifier lighting situations. In particular, for each image section 202 a combination of a global lighting class (GK) and a local lighting class (LK) can be determined. This can be for a variety of pictures 200 of a test record. The combination of a particular global and a particular local lighting class may be referred to as a "lighting combination".

Different object classifiers can be used for different lighting situations. In particular, the features used (i.e., descriptors) may be adapted for object detection (e.g., HOG, LBP, Haar / Edgelet wavelets, etc.). Alternatively or additionally, the classifiers used may be adapted (e.g., SVM, Bayes classifier, neural network, etc.). Furthermore, the training data sets used can be adapted to the lighting situations.

The confidence represents a measure of the reliability of a classifier and may be e.g. through the o.g. Dimensions UA, PA and / or OA are expressed. The confidence can be determined by means of a test dataset in two ways. First, a common statement for all object classifiers used, i. not separately for each classifier. Secondly, a confidence can be determined separately for each illumination combination.

To determine the confidence, a look-up table can be determined. For this purpose, a relatively high number (eg several hundred) of local image sections 202 from each lighting class used as a test record. The image sections used correspond to this 202 Areas with correct, false or false detections of the object classifier. For any global and local lighting combination, for the entire test Dataset the parameters TP, FP and FN are determined. From this, for example, the reliability measures UA, PA and OA can be derived.

• • Für einen gesamten Test-Datensatz eine zusammenfassende Konfidenzaussage über die Eignung des Objekterkennungssystems eine bestimmte Objektart zu erkennen, unabhängig davon, welcher Objekt-Klassifikator in einem bestimmten Bild 200 oder Bildausschnitt 202 verwendet wurde. Das Objekterkennungssystem umfasst dabei den in diesem Dokument beschriebenen Beleuchtungs-Klassifikator und Objekt-Klassifikator.
• • Eine Konfidenzaussage für eine bestimmte globale und lokale Beleuchtungskombination. Diese Aussage kann gegebenenfalls auf mehrere Test-Datensätze bezogen sein, wenn mehrere Test-Datensätze verarbeitet wurden.

The result is as follows, depending on the type of confidence:

• For a whole test record, a summary confidence statement about the object recognition system's ability to recognize a particular object type, regardless of which object classifier in a given image 200 or picture detail 202 has been used. The object recognition system comprises the illumination classifier and the object classifier described in this document.
• • A confidence statement for a specific global and local lighting combination. This statement may be related to multiple test records if multiple test records have been processed.

Thus, a look-up table can be provided which includes values for UA, PA and OA for each illumination combination.

Is an object classifier on an image 200 applied, the object classifier scans all image sections during the sliding window process 202 from. For every picture section 202 a combination of illumination (ie a GK and a LK) can be determined (method steps 401 . 402 of the procedure 400 ). Furthermore, the look-up table can be evaluated as a function of the determined illumination combination (method step 403 ). In particular, from the look-up table for each picture detail 202 a statement about the confidence determined 403 with the object classifier used for this illumination combination, the detection in this image section 202 can perform. It is therefore always based on the suitable for the lighting combination object classifier. This is already before calculation of the object classifier for a picture detail 202 a statement about the expected confidence possible. Furthermore, if the above-mentioned method is applied to an entire test data set, it is possible to derive a judgment of the respectively suitable object classifier for this test data set without executing the object classification.

The o.g. The method thus makes it possible to determine which lighting combinations allow object detections with higher security and which do not. Thus, the existing resources for object detection (e.g., computation time) can be better classified. In particular, it is possible to restrict the search space of the object recognition system. In areas of very low expected confidence, it can be assumed that no detection will be possible. Thus, other areas where higher confidence is expected can be prioritized higher.

The camera 101 of the vehicle 100 may include a stereo camera that is set up at a particular time, ie, at the same time, a left image 501 and a right picture 502 to capture (see 5a ). The left picture 501 and the right picture 502 put the same environment of the vehicle 100 from slightly different angles. This makes it possible to obtain information regarding the distance of objects that appear on the images 501 . 502 are shown. In particular, based on the left image 501 and the right picture 502 a disparity picture 503 be determined.

Disparity is a quantity that is indirectly proportional to the distance of the objects from the camera 101 , In a disparity picture 503 is thus recognizable which objects closer to the camera 101 or further from the camera 101 are removed. The coordinate system of the disparity image 503 typically agrees with the coordinate system of rectified images 501 . 502 (eg grayscale images) of the stereo camera system 101 match.

As stated above, there are two types of errors in object detection. On the one hand the misclassification (ie the FN, False Negatives) and on the other the misclassification (ie the FP, False Positives). These are dependent on each other via a security value of the object classifier. To evaluate an object classifier, one determines a threshold for the safety value at which detections are considered valid. This means that the higher the threshold for the security value, the fewer false classifications (FP) occur, but the more misclassifications (FN) occur. It is therefore not possible to reduce both errors, a balance must be made, for which purpose, for example, a precision recall curve can be used. The following describes a method independent of this, which describes the disparity picture 503 used to reduce the number of false classifications (FP) without increasing the number of misclassifications (FN). The method starts after applying the object classifier and checks every detection of an object. In other words, any "positive" detection can be checked.

The object classifier recognizes that a first image detail 202 includes an object. Depending on the reliability of the object classifier, a certain probability can be a false classification (FP). Based on the disparity picture 503 can such misclassifications (at least in part) be detected. For this purpose, the size of the first image section 202 (eg the height 211 and / or the width 212 ) to be viewed as. Furthermore, based on the disparity picture 503 for this first image section 202 Information concerning the distance of the detected object in this first image section 202 be determined. A plausibility check can then be carried out in which it is judged whether a detected object at the determined distance is one of the first image detail 202 may have the appropriate size. For example, it is checked whether a detected person at the determined distance a the first image section 202 appropriate size (height 211 and / or width 212 ). If this is not the case, it can be determined that it is the first image section 202 is a misclassification. Thus, inappropriate detections can be discarded. If at the same time the above-mentioned safety threshold is reduced, then both the misclassifications and the false classifications can be reduced.

To determine the metric size of a detection, first the enclosing rectangle of the first image section 202 into the disparity picture 503 be projected. In the case of the correct detection of a person, the rectangle contains its image as well as a certain amount of background. Since the images are typically in the same coordinate system, no transformations are necessary. Subsequently, a disparity value of the person's image is determined. For this, the background portion in the enclosing rectangle (ie in the first image section 202 ) determined and discarded. With a correct detection, it can be assumed that the person is clearly in front of objects in the image background and thus has a higher disparity value than the background.

To determine the background part, you can use a histogram 520 for the disparity values of the first image section 202 be created (see 5c ). It can be expected that the background has smaller disparity values than the detected person. Furthermore, it can be expected that both the background and the detected person each have similar disparity values. The distribution of the disparity values of the background and the person each have local maxima in the histogram 520 on. By a so-called peakiness test, the minimum 521 be determined between these maxima. This test determines relevant minima between two maxima based on the ratio of histogram entries. All disparity values less than the disparity of the minimum 521 are assigned to the background. For example, for all remaining disparity values in the enclosing rectangle, the median (or possibly an average) is determined. It is thus determined from the disparity values that are not assigned to the background, a disparity value D for the detected object. This disparity value D can be determined with the aid of orientation parameters of the camera 101 in a distance indication d for the detected object (in particular for the detected person) from the camera 101 being transformed. It can be used as orientation parameter of the camera 101 the distance, ie the baseline, between the left and right camera and a focal length of the camera 101 be taken into account.

Furthermore, the height 211 and the width 212 of the image section 202 by means of the distance d for the detected object and by means of one or more orientation parameters of the camera 101 (eg the focal length of the camera 101 ) are converted into metric sizes (eg, a metric height and a metric width) of the detected object. These metric magnitudes of the detected object can then be compared with typical magnitudes of the object to be detected. If the metric variables of the detected object are outside of a permissible value range, then the detected image section 202 be considered as a misclassification.

5b shows an exemplary method 510 for the classification of a picture detail 202 , The procedure 510 includes determining 511 a disparity value D for a picture detail 202 , The disparity value can be determined as set out above. Furthermore, the method includes 510 the determining 512 the size 211 . 213 of the image section 202 , The picture section 202 can then be based on the disparity value and on the size of the image detail 202 classified 513 become. In particular, it can be determined whether a picture detail 202 actually has a previously detected object or not.

It is thus proposed that the time-synchronized recording of a left image 501 and a right picture 502 through a camera 101 to reduce misclassifications. These can be the captured images 501 . 502 based on the orientation parameters of the camera system 101 be rectified. Furthermore, from the left and right rectified image 501 . 502 a disparity picture 503 be determined.

Through a sliding window process, at least one of the two images 501 . 502 object detection may be performed using a descriptor classifier technique (eg, as described in this document). This can be the right or the left stereo image 501 . 502 is used. For each detection of an object, a safety value can be determined or used as a basis. Furthermore can the size 211 . 212 of the image section 202 be determined in which an object was detected.

For all detections whose safety value is above a predefined safety threshold, the metric size of the corresponding image section 202 be determined. It can then be checked whether this metric size agrees with real size information of the object to be detected (eg of a human). If so, the detection may be considered valid, otherwise the detection may be discarded.

For determining the metric size of the image section 202 a detection becomes from the grayscale / color images 501 . 502 a picture section 202 corresponding section of the disparity image 503 determined. These are typically the orientation parameters of the camera 101 used. The picture section 202 (Also called Boundingbox) is thus in Disparitätsbild 503 at a corresponding location to the gray scale / color images 501 . 502 ,

If an object has been detected correctly, they are in the image section 202 in the disparity picture 503 the image of the object as well as a certain background area around the object. When background can be assumed that its distance to the camera 101 is greater than the distance of the detected object. Thus, the background has lower disparity values. For the picture section 202 can be a histogram 520 be created from the disparity values. The aim is to determine the correct disparity value of the detected object. For this, the median of the pixels 203 the disparity picture 503 calculated, which can be attributed to the detected object. For this purpose, in the histogram 520 the most striking minimum 521 certainly. All disparity values less than disparity at minimum 521 are assigned to the background and are excluded from further processing.

From the median of the disparity values attributed to the person, a value for the distance d of the object from the camera is calculated from the orientation parameters 101 determined. With the help of the distance indication d the size becomes 211 . 212 of the image section 202 converted into metric units. Again, typically orientation parameters of the camera 101 used. Thus, a comparison can be made with metric size specifications for real objects.

The pixel associated with the object 203 from the disparity picture 503 not only can be used to determine the distance d, but in addition the shape, stature and / or structure of the pixel associated with the object 203 to be examined. In particular, it can be checked whether the structure of the detected object coincides with a typical target structure of the object. Thus, if necessary, false classifications can be further reduced.

The striking minimum 521 in the histogram 520 can be determined by the so-called peakiness test. In addition the two highest maxima in the histogram become 520 determined and in between the "global" minimum 521 determined. The ratio of the values between the higher maximum and the minimum describes the degree of relevance of a minimum. Subsequently, the highest maximum is excluded from the further calculation. This algorithm is applied iteratively until the newly detected minima are considered no longer relevant compared to the most relevant minima already determined.

By using the disparity image 503 For example, the security threshold for the detections may be set lower than without using the disparity image 503 , Insecure detections, which previously had to be discarded to keep the number of false classifications low, can now be subjected to the size and / or shape check described above and then accepted or discarded. Thus, both misclassifications and misclassifications can be reduced.

Furthermore, restriction of the search space for the object classifier is possible since detector windows (ie image sections 202 ) can be discarded with too small or too large a metric size for an object type to be detected even before the classifier is applied. That is, before applying the object classifier to a frame 202 can based on the disparity picture 503 be determined, whether in the image section 202 due to the size and / or the shape can be located at all an object to be detected. The object classifier can then only on image sections 202 applied, for which such a preselection has come to a positive conclusion. Thus, the sequence of the disparity analysis described in this document and the object classification can be reversed. Thus, the computational effort for the object classification can be reduced.

Furthermore, the disparity analysis described in this document uses information of the disparity image 503 without direct an object classifier on the disparity image 503 apply. This is advantageous because a direct application due to the lower resolution of the Disparitätsbildes 503 (compared to the gray / color images 501 . 502 ) would be disadvantageous.

As already stated above, a test data set is typically used for the evaluation of an object classifier, wherein the test data record differs from the training data record with which the object classifier was trained. In the context of the evaluation, e.g. One or more reliability measures for the object classifier can be determined on the basis of the test data record.

In the context of the development of suitable object classifiers, it would be advantageous to be able to reduce the evaluation effort for an object classifier, without reducing the validity of the determined one or more reliability measures. Typically, the detection quality of an object classifier increases with increasing resolution of test images 200 at. As the resolution increases, however, the computational effort for the evaluation of the object classifier also increases.

One way to reduce the evaluation effort would be to increase the number of test images 200 to reduce in a test record. However, the detection quality of an object classifier typically increases with the number of test images tested 200 , Thus, the minimum number of required test images 200 typically relatively high for reliable evaluation. The consequence is that if the number of test images is too small 200 no reliable statement about the confidence of a classifier is possible.

In this document, it is proposed to provide insights regarding the lighting situation of image sections 202 in a variety of test pictures 200 of a test dataset to reduce the scope of the classifications to be performed as part of the evaluation of an object classifier, without reducing the significance of the object classifier confidence.

The test data set comprises a plurality of gray value / color images 200 through a camera 101 of a vehicle 100 were recorded. In this case, there is typically no possibility of influencing the duration of illumination, depth of field, aperture, number of pixels etc. with respect to image acquisition. For all images 200 the test record can be a lighting situation in the overall picture 200 as well as in all local image sections 202 determined by the lighting classifier described in this document in the form of discrete lighting categories or lighting classes. In particular, for each image section 202 and for every picture 200 of the test data record a combination of global illumination class (GK) and local illumination class (LK) are determined. The association of a particular global and particular local lighting category may be referred to as a particular "lighting combination".

When determining the illumination category of the local image sections 202 become the image sections 202 (in particular the shape of the image sections 202 ) is typically selected depending on the type of object to be detected. For example, a rectangular cutout with an aspect ratio of 2: 1 is suitable for person detections.

Furthermore, the image sections 202 of the test record "labeled". In particular, the one or more image sections 202 a picture 200 in which actual imaged objects of the desired object type are present in the image 200 manually recorded. It thus results in a test record, which for each image section 202 indicating the GK and LK, and that for each frame 202 indicates whether the image section 202 an object to be detected comprises or not.

The object classifier to be tested may include one of the classifiers described in this document. For example, the object classifier may use one or more of the following features / descriptors: HOG, LBP, Haar / Edgelet wavelets. Furthermore, the object classifier may include one of the following classifiers: SVM, Bayes classifier, neural network, etc.

Based on the test data set, a desired distribution of the illumination combinations can be determined. In other words, a probability within the test data set can be determined for each possible illumination combination. Alternatively or additionally, based on the test data set, a desired distribution of the illumination combinations in which an object to be detected is located can be determined. In other words, it can only be based on the image details 202 of the test data set comprising an object to be detected (so-called "ground truth" image sections), a probability for this combination of illumination within the test data set is determined for each possible illumination combination.

For example, at the same time as determining the local illumination class for each image section 202 be determined, whether in the respective image section 202 an object to be detected is or is not. If in a picture section 202 an object to be detected is located, it can be a function of the illumination combination of the image section 202 a corresponding counter can be increased by one. After checking the entire test data set, the value of each counter can be compared to the total number of image sections for all lighting combinations 202 be set with this lighting combination. It can thus be determined for each illumination combination the probability that the test record for this illumination combination a picture detail 202 comprising an object to be detected. In other words, the determined ratio for a particular combination of illumination can be considered as a probability that an image section 202 contains a searched object with this lighting situation.

6a shows an exemplary distribution 600 of probabilities 602 for different lighting combinations 601 , The probability 602 a lighting combination 601 indicates with what probability the test record for this illumination combination 601 picture details 202 comprising an object to be detected. An association of global and local lighting class (ie the lighting combination 601 ) to the probability 602 can for example be stored in a look-up table.

During the validation of an object classifier by means of the test data set, it is possible to use the illumination combination during the sliding window method 601 for a picture section 202 the right probability 602 be determined for the occurrence of an object. For this purpose, the above look-up table can be used.

Based on a threshold can then be decided whether the object classifier on a particular image detail 202 should be applied or not. In particular, the probability 602 for the picture section 202 be compared with a predefined probability threshold. Is the probability 620 below the probability threshold, the object classifier will not be on that frame 202 applied. Otherwise, an application of the object classifier to this image section 202 respectively. With the following image sections 202 can be proceeded in an analogous manner.

6b shows an exemplary method 610 to evaluate an object classifier. The procedure 610 includes providing 611 a test dataset with a large number of images 200 , Furthermore, the method includes 610 the determining 612 a distribution 600 of probabilities 602 for the presence of image sections 202 with an object to be detected within the test data set. The probabilities 602 be for different lighting combinations 601 determined. Furthermore, the method includes 610 the application 613 of the object classifier to a subset of the image sections 202 of the test record, depending on the distribution 600 ,

Thus, the object classification of the test data set can be accelerated. In particular, not all image sections have to be changed 202 a picture 200 to be evaluated for the evaluation of an object classifier. Improbable image excerpts 202 can be excluded from the exam. This reduces the required computing time to derive a quality statement of the classifier. On the other hand, with the same calculation time, a higher number of data records can be tested on the basis of the described method. Thus, with the same amount of time, a classifier can be tested for a wider range of conditions.

The present invention is not limited to the embodiments shown. In particular, it should be noted that the description and figures are intended to illustrate only the principle of the proposed methods, apparatus and systems.

Claims (14)

Procedure ( 400 ) for determining the reliability of an object recognition system for detecting an object in an image ( 200 ), where the image ( 200 ) a plurality of pixels ( 203 ), the method ( 400 ), - determining ( 401 ) of a global lighting situation of an image ( 200 ) based on the plurality of pixels ( 203 ); - determining a picture section ( 202 ) for the detection of the object, wherein the image section ( 202 ) a subset of the plurality of pixels ( 203 ); - Determine ( 402 ) of a local lighting situation of the image section ( 202 ) based on the subset of pixels ( 203 ); - certain ( 403 ), based on the global lighting situation and on the basis of the local lighting situation, the reliability of the object recognition system for the detection of an object in the image section ( 202 ). Procedure ( 400 ) according to claim 1, wherein - the reliability of the object recognition system for the detection of an object in the image section ( 202 ) is determined on the basis of predefined reliability data; and - each of the reliability data assigns a reliability value to various combinations of a global lighting situation and a local lighting situation. Procedure ( 400 ) according to claim 2, wherein the reliability data - based on a test data set with a plurality of test images ( 200 ) has been determined; and / or - are arranged in a look-up table indicating an expected reliability for each possible combination of a global lighting situation and a local lighting situation. Procedure ( 400 ) according to one of the preceding claims, wherein the reliability comprises one or more of - a correct classification rate of the object recognition system; An error rate of the object recognition system; A precision of the object recognition system; A recall of the object recognition system; A User's Accurary of the object recognition system; and / or - a producer's accurary of the object recognition system. Procedure ( 400 ) according to one of the preceding claims, wherein - the image ( 200 ) a plurality of image sections ( 202 ); - the procedure ( 400 ), determining a plurality of local illumination situations for the plurality of image sections ( 202 ); - the procedure ( 400 ), certain ( 403 ) based on the plurality of local lighting situations and on the basis of the global lighting situation, of a plurality of reliability values of the object recognition system for the detection of an object in the plurality of image sections (US Pat. 202 ); and - the method ( 400 ), determining, in dependence on the plurality of reliability values, an order of the plurality of image sections ( 202 ) for the detection of an object. Procedure ( 400 ) according to claim 5, wherein - the object recognition system for a first image section ( 202 ) from the plurality of image sections ( 202 ) has a first reliability value; The object recognition system for a second image section ( 202 ) from the plurality of image sections ( 202 ) has a second reliability value; The first reliability value is higher than the second reliability value; and the order of the plurality of image sections ( 202 ) is such that first in the first image section ( 202 ) and then in the second image section ( 202 ) determines whether the respective image section ( 202 ) has the object to be detected. Procedure ( 400 ) according to one of the preceding claims, wherein the determination of a lighting situation on the basis of pixels ( 203 ), - determining ( 311 ), based on the pixels ( 203 ), a plurality of quality measure values for a corresponding plurality of quality measures, wherein plurality of quality measure values comprises a feature vector for the pixels ( 203 ) forms; and - determining ( 312 ), based on the plurality of Gütemaß values, a lighting class from a plurality of predefined lighting classes, wherein the determined illumination class the lighting situation of the pixels ( 203 ), where the illumination class for the pixels ( 203 ) is determined by means of a lighting classifier which is set up to assign feature vectors to each of a lighting class from the plurality of lighting classes. Procedure ( 400 ) according to one of the preceding claims, wherein - the object recognition system is set up on the basis of the subset of pixels ( 203 ) a descriptor for the image section ( 202 ) to investigate; - the object recognition system is set up, by means of an object classifier, the image section ( 202 ), wherein the object classifier is arranged, the descriptor of the image section ( 202 ) to a positive object class or a negative object class, wherein the positive object class indicates that the image section ( 202 ) has the object to be detected, and wherein the negative object class indicates that the image section ( 202 ) does not have the object to be detected; and - the descriptor and / or the object classifier depend on the global lighting situation and the local lighting situation. Procedure ( 400 ) according to one of the preceding claims, wherein the size and / or the shape of the image section ( 202 ) depend on the object to be detected, so that a frame of the image section ( 202 ) can enclose the object to be detected. Procedure ( 400 ) according to one of the preceding claims, further comprising, - moving a search window along a horizontal axis ( 213 ) and / or along a vertical axis ( 214 ) of the picture ( 200 ) with a predefined increment in order to produce a sequence of image sections ( 202 ) to investigate; Determining a local lighting situation for each image section ( 202 ) from the sequence of image sections ( 202 ); and determining, for each image section ( 202 ) from the sequence of image sections ( 202 ) and depending on the particular local Lighting situation and depending on the global lighting situation of the image ( 200 ), the reliability of the object recognition system. Procedure ( 400 ) according to one of the preceding claims, wherein - the image ( 200 ) corresponds to a current picture from a sequence of temporally successive pictures; The global illumination situation of the current image based on the pixels ( 203 ) is determined from a plurality of images from the sequence of images; and - the local lighting situation of an image section ( 202 ) of the current image on the basis of corresponding subsets of the pixels ( 203 ) of the plurality of images is determined. Procedure ( 400 ) according to one of the preceding claims, wherein - the image ( 200 ) through a camera ( 101 ) of a vehicle ( 100 ) was recorded; - the camera ( 101 ), an environment in front of the vehicle ( 100 ) capture; The global and / or the local lighting situation also on the basis of a-priori knowledge regarding the lighting situation in the environment in front of the vehicle ( 100 ) is determined; and - the a-priori knowledge, in particular one or more of: • a time of day at which the image ( 200 ) was recorded; • a date on which the picture ( 200 ) was recorded; • the rising and / or sunset time of the sun on the day the image 200 ) was recorded; and / or • a geographic position of the environment contained in the image ( 200 ) is shown. Procedure ( 400 ) according to one of the preceding claims, wherein - the image ( 200 ) through a camera ( 101 ) of a vehicle ( 100 ) was recorded; - the camera ( 101 ), an environment of the vehicle ( 100 ) capture; - the vehicle ( 100 ) comprises one or more environment sensors, in particular a radar sensor and / or a lidar sensor, which are set up, environment data relating to the environment of the vehicle ( 100 ) capture; - the procedure ( 300 ) further comprising detecting, based on the environmental data, a static or a dynamic object in the environment of the vehicle ( 100 ), the influence on the lighting situation of the image ( 200 ) may have; and - the global and / or local lighting situation is also determined on the basis of the detected static or dynamic object. Procedure ( 400 ) according to one of the preceding claims, further comprising, - determining an expected distance of the object to be detected; and determining the subset of the plurality of pixels ( 203 ) for the determination of the local illumination situation of the image section ( 202 ) based on the expected distance of the object to be detected.

Images