DE102019126074A1 - Analysis of an environment of an active optical sensor system - Google Patents

Abstract

According to a method for analyzing an environment (3) of an active optical sensor system (2), a point cloud of scanning points of the environment (3) is generated by means of the sensor system (2), the point cloud having a coordinate matrix for a spatial dimension. The coordinate matrix is modified by means of a computing unit (6) by replacing error entries with a first reference number value, generating an integral image of the modified coordinate matrix, defining a square sub-area for each entry in the modified coordinate matrix, and defining a code number for each sub-area depending on the number of replaced error entries For each entry of the modified coordinate matrix, a mean value of entries within the respective sub-area is calculated based on the integral image and the characteristic number in order to generate an averaged coordinate matrix, and a surface normal (8) is determined for at least one of the sampling points based on the averaged coordinate matrix.

Description

The present invention relates to a method for analyzing the surroundings of an active optical sensor system, an ordered point cloud of scanning points of the surroundings being generated by means of an active optical sensor system, the point cloud having a coordinate matrix for each spatial dimension. The invention also relates to a sensor device with an active optical sensor system and a computing unit, a motor vehicle, a computer program, and a computer-readable storage medium.

Active optical sensor systems, for example lidar systems, which are sometimes also referred to as laser scanners, can be mounted on motor vehicles in order to implement various functions of electronic vehicle guidance systems or driver assistance systems. These functions include, for example, distance measurements, distance control algorithms, lane keeping assistants, object tracking functions and so on.

These functions are usually based on the analysis of the spatial surroundings of the motor vehicle and thus of the active optical sensor system by means of image processing methods. For example, surface normals, that is to say normal vectors that are perpendicular to the surface of objects in the environment, can be used to analyze the environment or the objects. Different objects are characterized, for example, by characteristic directions, directional distributions or courses of their surface normals. For example, surface normals of a road surface can be approximately parallel to one another over a large area, while surface normals of a motor vehicle point in different directions.

The publication "Adaptive Neighborhood Selection for Real-Time Surface Normal Estimation from Organized Point Cloud Data Using Integral Images", 2012 IEEE / RSJ International Conference on Intelligent Robots and Systems, 2012, Portugal by S. Holzer et al. describes a method for determining surface normals for robotic applications on the basis of camera images .

Laser scanners, for example, are also known from the prior art, which can generate distance images of the environment by scanning the environment for discrete scanning directions and a corresponding distance measurement, for example by means of time-of-flight measurement.

In various application areas, especially in the field of automotive technology, the computing capacity, especially of CPUs, is a limited resource.

In addition, when the environment is scanned with the aid of the laser scanner, it may be the case that no measurement can be made for individual scanning directions or that a measurement is faulty. This can lead to the point cloud, which has a matrix structure, for example, having incorrect or missing entries. Such entries can negatively influence the accuracy of the calculation of the surface normals and make the effects of signal noise appear more pronounced.

Against this background, it is an object of the present invention to provide an improved concept for analyzing the surroundings of an active optical sensor system which enables a more precise estimation of surface normals with little or less computational effort.

According to the invention, this object is achieved by the respective subject matter of the independent claims. Advantageous developments and preferred embodiments are the subject of the dependent claims.

The improved concept is based on the idea of averaging a coordinate matrix of the point cloud in order to reduce the influence of noise and to explicitly take into account missing entries or missing sampling points or error entries in the point cloud when averaging.

According to an independent aspect of the improved concept, a method for analyzing an environment of an active optical sensor system is specified, with the active optical sensor system, in particular with the sensor system and a computing unit, generating an ordered point cloud of sampling points in the environment, the point cloud being a coordinate matrix for each spatial dimension, in particular for Cartesian spatial dimensions. The coordinate matrix is modified by means of the computing unit by adding error entries, in particular all error entries, to the coordinate matrix are each replaced by a first reference numerical value. An integral image of the modified coordinate matrix is generated by means of the computing unit and an, in particular square, partial area of the modified coordinate matrix is defined for each entry in the modified coordinate matrix. For each of the sub-areas an associated code number is determined by means of the arithmetic unit as a function of the number of error entries replaced in the respective sub-area. For each entry of the modified coordinate matrix, the computing unit calculates an average of entries, in particular of all entries, within the respective sub-area based on the integral image and based on the code number of the respective sub-area in order to generate an averaged coordinate matrix. By means of the computing unit, a surface normal is determined for at least one of the scanning points based on the averaged coordinate matrix.

For each of the scanning directions, the active optical sensor system transmits, for example, associated sensor data to the computing unit. The computing unit can then generate the distance image based on the sensor data for the various scanning directions. Alternatively, a further processing unit can generate the distance image based on the sensor data for the various scanning directions and transmit this to the processing unit.

The distance that is determined by means of the active optical sensor system for each of the scanning directions corresponds in particular to a distance of the active optical sensor system, in particular an optical detector of the active optical sensor system, from an object in the vicinity that is in the direction of the active optical sensor system the respective scanning direction is located.

By definition, an active optical sensor system has a light source for emitting light or light pulses. The light source can in particular be designed as a laser. Furthermore, by definition, an active optical sensor system has at least one optical detector in order to detect reflected components of the emitted light. The active optical sensor system is set up in particular to generate and process or output one or more sensor signals based on the detected components of the light, in particular to generate the sensor data.

Here and in the following, the term “light” can be understood to include electromagnetic waves in the visible range, in the infrared range and / or in the ultraviolet range. Accordingly, the term “optical” can also be understood in such a way that it relates to light according to this understanding.

The corresponding distance of the optical detector from the reflecting object can then be determined based on a transit time difference between the emission of the light pulses and the detection of the light pulses.

The active optical sensor system can contain, for example, a deflection device for deflecting the emitted and the reflected light pulses. In particular, light pulses can be emitted in different directions and light pulses from different directions of incidence can be detected by being directed onto the one or more optical detectors by means of the deflection device.

The active optical sensor system can contain, for example, a deflection device for deflecting the emitted and the reflected light pulses. In particular, light pulses can be emitted in different directions and light pulses from different directions of incidence can be detected by being directed onto the one or more optical detectors by means of the deflection device.

The deflection device can, for example, contain a rotatable mirror on which the light from the light source strikes and is accordingly emitted at different times in different emission directions. As a result, an emission plane of the active optical sensor system is defined perpendicular to the axis of rotation of the mirror.

Depending on the direction of incidence of the reflected light and depending on the current mirror position of the rotating mirror, reflected light can be directed by means of the mirror from different directions of incidence onto the optical detector or detectors. For example, the different horizontal scanning angles can be defined by the different mirror positions.

The active optical sensor system can for example have several optical detectors, for example three or more optical detectors. The optical detectors can be arranged, for example, along a direction parallel to the axis of rotation of the mirror or perpendicular to the emission plane. Ever according to the direction of the incident reflected light, the light can therefore be directed to different of the optical detectors by means of the mirror. In this way, for example, the vertical scanning angle can be defined; in particular, each discrete value for the vertical scanning angle can correspond to a corresponding one of the optical detectors.

The sensor data therefore include the horizontal scanning angle for each detected light pulse, for example determined based on the current mirror position, the associated vertical scanning angle, identified via the corresponding excited optical detector, as well as the associated distance of the reflecting object from the optical detector or the corresponding light transit time.

The coordinate matrix can also be viewed as an ordered set of coordinates of the scanning points in the corresponding spatial dimension, the set being ordered as described in accordance with the vertical and horizontal scanning angles. For this reason, the point cloud, which also has this structure, is also referred to as an ordered point cloud.

In particular, the point cloud can have a corresponding coordinate matrix for each of the three Cartesian spatial dimensions. The steps described with regard to a coordinate matrix can also be carried out analogously for the other coordinate matrices of the other spatial dimensions.

wobei M die Anzahl der Zeilen und N die Anzahl der Spalten der Matrix ist.A matrix can be understood in particular as an image F, which is defined as follows: $$\text{Q:}\left\{\mathrm{1..}\text{M.}\right\}\times \left\{\mathrm{1..}\text{N}\right\}\to ℝ,\left(\text{i},\text{j}\right)\to {\text{a}}_{\text{ij}},$$

where M is the number of rows and N is the number of columns in the matrix.

A scanning point of the point cloud can in particular be understood as a three-dimensional coordinate tuple, the x, y and z coordinates of the scanning point being given by the entries of the various coordinate matrices belonging to one another, i.e. belonging to a specific scanning direction.

An error entry corresponds to an entry in the coordinate matrix for which the scanning point, or the light transit time or the corresponding distance, could not be determined or could not be determined correctly. The error entry can correspond, for example, to a “Not a number” entry (NaN). In other words, an error entry can correspond to a predefined placeholder value.

In order to modify the coordinate matrix, the computing unit checks, for example, all entries in the coordinate matrix and, if there is an incorrect entry, replaces it with the first reference numerical value. The first reference numerical value can in particular be zero. If a numerical value other than zero is used for the first reference numerical value, the equations listed below may have to be modified accordingly.

The square sub-area is defined by means of the computing unit, for example, in such a way that a size r of the sub-area is defined for each entry in the coordinate matrix, and the sub-area ^{corresponds to a square with r 2} entries of the coordinate matrix that contains the entry under consideration for which the sub-area is defined, includes. In this case, the value for r can, for example, be fixed or determined individually for different entries in the coordinate matrix by means of the computing unit. The individual determination can be made, for example, based on the sensor data, a distance image or a corresponding depth change map. For example, the value for r can be defined in such a way that there is no edge in each of the subregions.

The mean is not necessarily an arithmetic mean. Rather, other definitions of a mean can be used. In particular, a trimmed mean can be used.

The mean value corresponds in particular to a mean value over all entries in the sub-area that have not been replaced by the reference numerical value. This ensures that the mean value only takes correctly determined distances or coordinates of scanning points into account.

The code number for a specific sub-area can, for example, be equal to the number of error entries replaced in this sub-area. The code number can also be equal to the number of entries in the sub-area that were not replaced by the reference numerical value, i.e. 4r ² minus the number of replaced error entries.

In order to generate the averaged coordinate matrix, the computing unit calculates the mean value in particular for each entry in the coordinate matrix, as described, and replaces the entry with the mean value calculated in this way.

According to the improved concept, the surface normal determined as described can be used to analyze the surroundings, wherein the analysis of the surroundings can include or consist of the determination of the surface normal.

According to at least one embodiment, the environment is analyzed by means of the computing unit based on the surface normal.

In particular, the surface normal can be determined for all sampling points as described.

To determine the surface normal, the computing unit can use the averaged coordinate matrix as well as the other two coordinate matrices or corresponding further averaged coordinate matrices, the averaging being carried out analogously to that described above.

By calculating the mean value and using it to determine the surface normals, the influence of signal noise or other noise effects is reduced, which leads to a particularly precise determination of the surface normals. By determining the surface normal on the basis of the matrix modifications required for the steps described, the method can be implemented according to the improved concept with a particularly low computational effort.

Because the number of error entries in the respective sub-area used for calculating the mean value is explicitly taken into account, the accuracy in determining the surface normals can be increased and the influence of noise can be further reduced. In particular, larger sub-areas can be used for averaging, since the error entries in the sub-area are explicitly taken into account and processed. In particular, the sub-area does not have to be selected to be particularly small due to any error entries in order to completely avoid averaging over sub-areas with error entries. As a result, a more effective averaging, that is to say in particular a greater reduction in noise, can be achieved.

For a matrix according to the definition above ( 1 ) the entry (m, n) of an integral image is defined as follows: $$\text{I.}\left(\text{F.},\text{m},\text{n}\right):={\displaystyle {\sum }_{\text{i}=0}^{\text{n}}{\displaystyle {\sum }_{\text{j}=0}^{\text{n}}{\text{a}}_{\text{ij}}.}}$$

wobei k die Anzahl der Einträge in dem entsprechenden Teilbereich bezeichnet, die nicht durch den ersten Referenzwert ersetzt wurde.The following then applies to the mean value O ' _mod (m, n) of an entry O _mod (m, n) of the modified coordinate matrix: $$\begin{array}{l}{\text{O}}_{\text{mod}}^{\text{'}}\left(\text{m},\text{n}\right)=\\ \left[\text{I.}\left({\text{O}}_{\text{mod}},\text{m}+\text{r},\text{n}-\text{r}\right)-\text{I.}\left({\text{O}}_{\text{mod}},\text{m}+\text{r},\text{n}-\text{r}\right)+\text{I.}\left({\text{O}}_{\text{mod}},\text{m}-\text{r},\text{n}-\text{r}\right)\right]/\text{k},\end{array}$$

where k denotes the number of entries in the corresponding sub-area that have not been replaced by the first reference value.

By using the integral image for calculating the mean value, the number of computation steps required can be reduced, which further reduces the computational effort of the method. In particular, the integral image only needs to be determined once and can then be used for all averaging.

In particular, this enables the implementation of the improved concept or the method according to the improved concept on field-programmable gate arrays, so that the computing steps do not have to be carried out on a CPU. A field programmable gate array is also referred to as a field programmable gate array, FPGA.

According to at least one embodiment of the method for analyzing the environment according to the improved concept, the point cloud has an associated coordinate matrix for each of the three Cartesian spatial dimensions. Each of the coordinate matrices is modified by means of the arithmetic unit by replacing error entries in the respective coordinate matrices with the first reference numerical value. For each of the modified coordinate matrices, an associated integral image is generated by means of the computing unit. The sub-area is defined for each entry of each modified coordinate matrix, the same sub-area, that is to say in particular a sub-area of the same size and shape, being assigned to entries of different coordinate matrices that correspond to one another. An associated code number is determined for each of the modified coordinate matrices for each sub-area, the code number depending on the number of error entries replaced in the corresponding coordinate matrix in the respective sub-area. For each entry of the modified coordinate matrices, an average value of entries within the respective sub-area is calculated based on the associated integral image of the modified coordinate matrix and the respective characteristic number of the corresponding modified coordinate matrix in order to generate corresponding averaged coordinate matrices. The surface normal is determined for the at least one scanning point based on all three determined coordinate matrices.

According to at least one embodiment, a first vector is determined by means of the computing unit for a sampling point for which the surface normal is to be determined, which connects two horizontally adjacent sampling points of the sampling point under consideration, the coordinates of the averaged coordinate matrices being used, if available. Correspondingly, a second vector is determined which connects two vertically adjacent sampling points with one another. The surface normal is determined based on the first and the second vector, in particular based on a vector product or cross product of the first and the second vector.

The vertically adjacent scanning points are in particular the closest neighbors of the scanning point under consideration, for which the surface normal is to be determined. According to the ordered structure of the coordinate matrix, the two horizontally adjacent scanning points belong to the same vertical scanning angle and the two vertical adjacent scanning points belong to the same horizontal scanning angle.

The first and the second vector thus span a plane which approximately corresponds to an object surface in the surroundings, in the center of which the averaged scanning point under consideration or the corresponding scanning direction lies. A normal vector that is perpendicular to this plane spanned by the two vectors can therefore be viewed approximately as a surface normal for the scanning direction under consideration or at the corresponding scanning point.

According to at least one embodiment, the computing unit generates an error matrix which has a second reference numerical value at positions which correspond to the error entries in the coordinate matrix or one of the further coordinate matrices. The key figure is determined by means of the arithmetic unit based on the error matrix.

In particular, the error matrix has the same number of rows and columns as the coordinate matrix.

In particular, all entries in the error matrix that do not correspond to the positions of the error entries, that is to say that do not have the second reference numerical value, are equal to zero.

The second reference numerical value is in particular equal to 1. If a second reference numerical value that deviates therefrom is used, the equations explained below may need to be corrected.

According to at least one embodiment, an integral image of the error matrix is generated by means of the computing unit and the characteristic number is determined based on the integral image of the error matrix and based on the integral image of the modified coordinate matrix.

This enables a particularly simple and less computationally expensive calculation of the mean value or the key figure. In particular, the key figure can be calculated as the number k as follows, where C (m, n) denotes a corresponding entry in the error matrix: $$\text{k}=\text{k}\left(\text{m},\text{n}\right)=\mathrm{4th}{\text{r}}^2-\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}-\text{r}\right),-\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}-\text{r}\right),-\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}-\text{r}\right),+\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}-\text{r}\right),$$

Together with equation (3) this results in the mean value: $$\text{O'mod}\left(\text{m},\text{n}\right):=\frac{\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}+\text{r}\right)-\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}-\text{r}\right)-\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}+\text{r}\right)+\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}-\text{r}\right)}{\mathrm{4th}{r}^2-\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}+\text{r}\right)-\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}-\text{r}\right)-\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}+\text{r}\right)+\text{I.}\left(\text{C.},\text{m}+\text{r},\text{n}-\text{r}\right)}.$$

According to at least one embodiment, a distance image of the surroundings is generated by using the active optical sensor system to determine the associated distance for the plurality of scanning directions, each of the scanning directions being defined by the associated horizontal scanning angle and the associated vertical scanning angle. The point cloud is generated based on the distance image by means of the computing unit.

The distance image corresponds, for example, to a matrix in which different rows correspond to different vertical scanning angles and different columns correspond to different horizontal scanning angles or vice versa. The respective entries in the matrix correspond to the associated distances.

“Horizontal” and “vertical” are defined in particular with regard to the sensor system, for example with regard to the emission plane. If, for example, a first scanning direction with a first vertical and a first vertical scanning angle lies within the transmission plane, a second scanning direction with the first vertical scanning angle and a second horizontal scanning angle deviating from the first horizontal scanning angle can also lie in the transmission plane. A third scanning direction with the first horizontal scanning angle and a second vertical scanning angle different from the first vertical scanning angle does not lie, for example, in the transmission plane, but in a plane perpendicular thereto.

In particular, the computing unit carries out a coordinate transformation based on the distance image in order to generate the ordered point cloud. The coordinate transformation corresponds to a transformation of polar coordinates, in which the distance image, as described above, is given, into Cartesian coordinates. However, the aforementioned matrix structure is retained.

Accordingly, the ordered point cloud contains, for example, a corresponding coordinate matrix for each of the three Cartesian spatial dimensions, with rows and columns corresponding to the various vertical and horizontal scanning angles, as described for the distance image, and the entries in the coordinate matrices to the corresponding Cartesian coordinates, i.e. x-, y- and z coordinates, respectively.

In particular, this ensures that neighboring points in the distance image, that is to say neighboring matrix entries in the distance image, are also mapped onto correspondingly neighboring entries in the respective coordinate matrices.

In accordance with at least one embodiment, the computing unit is used to perform image segmentation, in particular of the surroundings, that is to say of the objects in the surroundings, depending on the surface normal, in particular depending on all determined surface normal. The analysis of the environment includes, in particular, image segmentation.

During the segmentation, objects or object parts in the environment that are connected or correlated in some other way are identified as such. A further analysis of the surroundings can be carried out based on the image segmentation.

Alternatively or additionally, the segmented image areas can be used for further processing, which can be accompanied by a data reduction.

According to at least one embodiment, an object classification and / or object tracking is carried out by means of the computing unit based on a result of the image segmentation. The analysis of the environment includes, in particular, object classification.

In the context of object classification and / or object tracking, the processing unit assigns individual objects in the environment, for example, to a specific predefined class. In particular, the classification can correspond to the assignment of semantics to the corresponding objects or object parts.

For example, the object classification can be used to differentiate between different types of objects in the environment. For example, a road surface can be assigned to a different class than, for example, a building wall, another motor vehicle or a pedestrian.

According to at least one embodiment, a bounding box for an object in the environment is determined by means of the computing unit based on a result of the image segmentation, that is to say an envelope body or an envelope figure that encloses the object. The analysis of the environment includes in particular the creation or determination of the bounding box.

According to at least one embodiment, the surroundings of the active optical sensor system are surroundings of a motor vehicle on or in which the active optical sensor system is mounted.

In other words, it is a method for analyzing the surroundings of the motor vehicle, the active optical sensor system being mounted on or in the motor vehicle.

Results of the analysis can be used, for example, by a driver assistance system of the motor vehicle and / or for partially autonomous or fully autonomous control of the motor vehicle by means of an electronic vehicle guidance system.

According to a further independent aspect of the improved concept, a sensor device, in particular for a motor vehicle, is specified. The sensor device has an active optical sensor system which is set up to determine an associated distance for a plurality of scanning directions and to generate associated sensor data based on the distance. The sensor device has a computing unit which is set up to generate an ordered point cloud of sampling points in the surroundings based on the sensor data of the plurality of scanning directions, the point cloud having a coordinate matrix for a spatial dimension. The computing unit is set up to replace error entries, in particular all error entries, in the coordinate matrix with a first reference numerical value in order to modify the coordinate matrix. The computing unit is set up to generate an integral image of the modified coordinate matrix and to define an, in particular square, sub-area of the modified coordinate matrix for each entry of the modified coordinate matrix. The processing unit is set up to determine an associated code number for each sub-area as a function of the number of replaced error entries in the sub-area. The computing unit is set up to calculate an average value of entries, in particular all entries, within the respective sub-area based on the integral image and based on the characteristic number of the respective sub-area for each entry of the modified coordinate matrix in order to generate an averaged coordinate matrix. The computing unit is set up to determine a surface normal for at least one of the sampling points based on the averaged coordinate matrix.

According to at least one embodiment of the sensor device, the active optical sensor system is designed as a laser scanner or lidar system.

According to at least one embodiment, the computing unit contains a field-programmable gate array to generate the point cloud, to replace the error entries and / or to generate the integral image of the modified coordinate matrix and / or to determine the characteristic number and / or to calculate the mean value and / or to determine the surface normal.

Further embodiments of the sensor device according to the improved concept result directly from the various embodiments of the method according to the improved concept and vice versa. In particular, the sensor device according to the improved concept can be set up or programmed to carry out a method according to the improved concept or the sensor device carries out a method according to the improved concept.

According to a further independent aspect of the improved concept, a motor vehicle with a sensor device according to the improved concept is specified.

According to a further independent aspect of the improved concept, a computer program with instructions is specified. When the computer program is executed by a sensor device according to the improved concept, in particular by the computing unit of the sensor device, the commands cause the sensor device to carry out a method according to the improved concept.

According to a further independent aspect of the improved concept, a computer-readable storage medium is specified on which a computer program according to the improved concept is stored.

Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown alone in the figures can be used not only in the specified combination, but also in other combinations without departing from the scope of the invention . Embodiments of the invention that are not explicitly shown and explained in the figures, but emerge and can be generated from the explained embodiments by means of separate combinations of features, are therefore also to be regarded as covered and disclosed. Designs and combinations of features are also to be regarded as disclosed, which therefore do not have all the features of an originally formulated independent claim. In addition, designs and combinations of features, in particular through the statements set out above, are to be regarded as disclosed that go beyond the combinations of features set forth in the back references of the claims or differ from them.

• 1 eine schematische Darstellung eines Kraftfahrzeugs mit einer beispielhaften Ausführungsform einer Sensorvorrichtung nach dem verbesserten Konzept;
• 2 eine schematische Darstellung eines aktiven optischen Sensorsystems einer weiteren beispielhaften Ausführungsform einer Sensorvorrichtung nach dem verbesserten Konzept;
• 3 ein Ablaufdiagramm einer beispielhaften Ausführungsform eines Verfahrens nach dem verbesserten Konzept; und
• 4 ein Ablaufdiagramm einer weiteren beispielhaften Ausführungsform eines Verfahrens nach dem verbesserten Konzept.

In the figures show:

• 1 a schematic representation of a motor vehicle with an exemplary embodiment of a sensor device according to the improved concept;
• 2 a schematic representation of an active optical sensor system of a further exemplary embodiment of a sensor device according to the improved concept;
• 3 a flowchart of an exemplary embodiment of a method according to the improved concept; and
• 4th a flowchart of a further exemplary embodiment of a method according to the improved concept.

In 1 is a motor vehicle 1 shown, which is an exemplary embodiment of a sensor device 7th according to the improved concept.

The sensor device 7th contains an active optical sensor system 2 , which is designed, for example, as a lidar system. The sensor device 7th also includes an arithmetic unit 6th that come with the sensor system 2 is connected and can receive sensor data from it. The arithmetic unit 6th in particular contains an FPGA that works together with the sensor system 2 can be designed as an integrated system.

The view in 1 can for example be a top view of the motor vehicle 1 as well as the sensor system 2 and a corresponding roadway in an area 3 of the motor vehicle 1 correspond.

In the neighborhood 3 there are, for example, one or more objects 3a , 3b . The active optical sensor system 2 can emit light pulses, in particular by means of a light source, for example an infrared laser diode, into the environment 3 send out where this from the objects 3a , 3b can be partially reflected and as reflected light pulses 5 , 5 ' possibly partially in the direction of the sensor system 2 can be reflected back. These reflected light components 5 , 5 ' can the sensor system 2 detect by means of several optical detectors and thereby for a multitude of discrete scanning directions based on corresponding time-of-flight measurements associated distances to points on the objects 3a , 3b from which the light pulses are reflected.

In 2 is the sensor system 2 shown schematically to explain a definition of the scanning directions. As in 1 is the sensor system 2 in the left figure in 3 shown in plan view, while the right figure in 3 represents a side view rotated by 90 °.

It can be a reference direction 4th of the sensor system 2 be defined, for example, a central transmission direction within the transmission plane of the sensor system 2 corresponds to. The sensor system 2 has a deflection device in order to deflect the light pulses generated by means of the light source in different directions within the emission plane. For this purpose, the deflection device can, for example, have a rotating mirror, the axis of rotation in particular being oriented perpendicular to the emission plane.

Hit the reflected light pulses 5 on the sensor system 2 , they are also deflected by means of the deflection device. Accordingly, light pulses strike depending on the position of the deflection device, that is to say in particular depending on the angle of rotation of the mirror 5 from different directions onto the optical detectors of the sensor system 2 . The position of the deflection device therefore defines, for example, a horizontal scanning angle α _h .

The optical detectors are arranged, for example, along a direction perpendicular to the emission plane, that is to say, for example, parallel to the axis of rotation of the mirror. Depending on the direction of the incident light pulses 5 can therefore be directed to different of the optical detectors. The various detectors can therefore define _{a vertical scanning angle α v.}

Each combination of a vertical scanning angle α _v and a horizontal scanning angle α _h in particular defines a scanning direction.

The functioning of the sensor device 7th is described below using an exemplary embodiment of a method for analyzing the surroundings of an active optical sensor system 2 explained in more detail according to the improved concept, in particular with reference to FIG 3 and 4th .

In 3 a flow diagram of an exemplary embodiment of a method according to the improved concept is shown.

In step V1 the sensor system scans 2 the environment 3 by determining an associated distance for a large number of scanning directions. Each of the scanning directions is defined by the associated horizontal scanning angle α _h and vertical scanning angle α _v . From the raw sensor data, for example by means of the computing unit 6th , a distance image is generated, that is to say a matrix in which, for example, rows correspond to different vertical scanning angles and columns correspond to different horizontal scanning angles. Entries in the matrix correspond to the associated specific distance.

In step V2 of the method calculates the arithmetic unit 6th an ordered point cloud based on the distance image. The point cloud can be understood as the entirety of three coordinate matrices O _x , O _y , O _z . Each of the coordinate _{matrices O x} , O _y , O _z has the same structure as the distance image, so that rows correspond to different vertical scanning angles and columns correspond to different horizontal scanning angles. The entries in the coordinate matrices O _x , O _y , O _z correspond to the associated Cartesian x, y and z coordinates.

In step V3x generates the arithmetic unit 6th a modified coordinate matrix O _{x, mod} , in that every error entry in the coordinate matrix O _{x is} replaced by zero. For example, steps V3y and V3z modified coordinate matrices O _{y, mod} and O _{z, mod} based on O _y and O _z , are generated.

In addition, the arithmetic unit calculates 6th in the steps V3x , V3y , V3z corresponding integral images I _x , I _y , I _z analogous to equation (2) for the modified coordinate matrices O _{x, mod} , O _{y, mod} , O _{z, mod.}

In step V4 generates the arithmetic unit 6th an error matrix C, where an entry C (m, n) of the error matrix is equal to 1 if an error value has been replaced at a corresponding point in one of the coordinate matrices O _x , O _y , O _z , and otherwise equal to zero. The arithmetic unit 6th generates an integral image I _{C of} the error matrix C according to equation (2).

For each entry in the modified coordinate _{matrix O x, mod} , that is to say for each scanning direction, the arithmetic unit calculates 6th in step V4 based on the integral image I _x, a mean value over a range of a predetermined size r, analogous to equation (5). This is carried out for all entries of the modified coordinate matrix O _{x, mod} , so that an averaged modified coordinate matrix O ' _{x, mod} results.

_{Modified coordinate matrices O 'y, mod} , O' _{z, mod} averaged in an analogous manner are determined for the y, the z direction.

In step V4 can the arithmetic unit 6th in addition, for each averaged sampling point, a difference vector of a right and a left neighboring point, that is to say two horizontal neighboring points, and a difference vector of an upper and a lower neighboring point, that is of two vertical neighboring points.

Based on these difference vectors, the arithmetic unit 6th the corresponding surface normal for the scanning point lying between the horizontal and between the vertical neighboring points 8th (please refer 1 ) is calculated by calculating the cross product of the two vectors.

In the optional step V5 becomes the surface normal 8th possibly rotated by 180 ° so that the resulting surface normals 8th all external normals with regard to the observed environment based on the objects 3a , 3b represent.

In step V6 can the surface normals determined in this way 8th can be used for further analysis steps, for example for image segmentation.

The size r of the averaging area can, for example, be fixed. Alternatively, the size r is adapted adaptively. For example, the computing unit can perform edge detection based on a depth change map in order to identify and localize the position of edges.

In the optional step V8 fits the arithmetic unit 6th the variable r on the basis of the result of the edge detection for each scanning direction in such a way that the area is free of edges, so that the described smoothing or averaging does not take place across edges.

In 4th FIG. 12 is a flow diagram for exemplary processing of sensor data of an active optical sensor system 2 of a motor vehicle 1 using an improved concept method. The car 1 is trained in partially or fully autonomous driving, for example.

In step V9 feature extraction is performed. step V9 can do the steps V1 to V5 and optionally V7 and V8 of the method according to 3 contain.

step V6 out 3 for example, the steps V10 to V14 out 4th include.

In step V10 can the arithmetic unit 6th for example an image segmentation using the surface normals 8th carry out. In step V11 can the arithmetic unit 6th carry out an object recognition based on a result of the image segmentation and in step V12 an object tracking based on a result of the object recognition. In step V13 can the arithmetic unit 6th for example using the surface normals 8th make an estimate of the geometry or shape of the roadway.

In step V14 can the arithmetic unit 6th a path planning for partially or fully autonomous driving based on the results of the steps V13 and V12 carry out.

For performing individual or all calculation steps in the process steps V10 to V14 the computing unit can, for example, be a CPU and / or a control unit of the motor vehicle 1 exhibit.

As described, the improved concept allows the environment of an active optical sensor system and in particular of a motor vehicle to be analyzed by determining surface normals.

The method steps described for determining surface normals can advantageously run on FPGA hardware, for example, so that limited CPU capacity can be saved.

The improved concept uses an ordered point cloud in which defined spatial relationships between adjacent sampling points are used to calculate the surface normals. In order to reduce the influence of noise in the measurement data, an integral image is used for smoothing, with the aid of which mean values over a square data area can be determined essentially by means of additions and a few multiplications. For the calculation of the integral images themselves, only additions are required, so that a particularly efficient implementation, for example on an FPGA, is possible.

By explicitly considering missing entries in the point cloud, the sub-area over which is averaged can be selected to be larger, so that the accuracy of the determination of the surface normals and thus of the analysis is increased.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Non-patent literature cited

• "Adaptive Neighborhood Selection for Real-Time Surface Normal Estimation from Organized Point Cloud Data Using Integral Images", 2012 IEEE / RSJ International Conference on Intelligent Robots and Systems, 2012, Portugal by S. Holzer et al. describes a method for determining surface normals for robotic applications on the basis of camera images [0004]

Claims (15)

A method for analyzing an environment (3) of an active optical sensor system (2), an ordered point cloud of scanning points of the environment (3) being generated by means of the active optical sensor system (2), the point cloud having a coordinate matrix for a spatial dimension, characterized that by means of a computing unit (6) - the coordinate matrix is modified by replacing error entries in the coordinate matrix with a first reference numerical value; an integral image of the modified coordinate matrix is generated; a sub-area of the modified coordinate matrix is defined for each entry of the modified coordinate matrix; an associated code number is determined for each sub-area as a function of the number of replaced error entries in the sub-area; - for each entry of the modified coordinate matrix, an average value of entries within the respective sub-area is calculated based on the integral image and the characteristic number of the respective sub-area in order to generate an averaged coordinate matrix; and - a surface normal (8) is determined for at least one of the sampling points based on the averaged coordinate matrix. Procedure according to Claim 1 , characterized in that by means of the arithmetic unit (6) - an error matrix is generated which has a second reference numerical value at positions which correspond to the error entries; and the characteristic number is determined based on the error matrix. Procedure according to Claim 2 , characterized in that by means of the computing unit (6) - an integral image of the error matrix is generated; and the characteristic number is determined based on the integral image of the error matrix. Method according to one of the Claims 1 to 3 , characterized in that - a distance image of the surroundings (3) is generated by using the active optical sensor system (2) to determine an associated distance for a plurality of scanning directions, each of the scanning directions being defined by an associated horizontal scanning angle (α _h ) and an associated vertical scanning angle (α _v ) is defined; and - the point cloud is generated by means of the computing unit (6) based on the distance image. Method according to one of the Claims 1 to 4th , characterized in that the computing unit (6) is used to perform image segmentation as a function of the surface normal (8). Procedure according to Claim 5 , characterized in that an object classification is carried out by means of the computing unit (6) based on a result of the image segmentation. Method according to one of the Claims 5 or 6th , characterized in that a bounding box for an object (3a, 3b) in the environment (3) is determined by means of the computing unit (6) based on a result of the image segmentation. Method according to one of the Claims 1 to 7th , characterized in that the environment (3) of the active optical sensor system (2) is an environment (3) of a motor vehicle (1) on or in which the active optical sensor system (2) is mounted. A sensor device comprising - an active optical sensor system (2) which is set up to determine an associated distance in each case for a plurality of scanning directions and to generate associated sensor data based on the distance; and - a computing unit (6) which is set up to generate an ordered point cloud of sampling points in the surroundings (3) based on the sensor data, the point cloud having a coordinate matrix for a spatial dimension; characterized in that the computing unit (6) is set up to - replace error entries in the coordinate matrix with a first reference numerical value in order to modify the coordinate matrix; - to generate an integral image of the modified coordinate matrix; - to define a sub-area of the modified coordinate matrix for each entry of the modified coordinate matrix; to determine an associated code number for each sub-area as a function of the number of replaced error entries in the sub-area; to calculate for each entry of the modified coordinate matrix an average value of entries within the respective sub-area based on the integral image and the characteristic number of the respective sub-area in order to generate an averaged coordinate matrix; and to determine a surface normal (8) for at least one of the sampling points based on the averaged coordinate matrix. Sensor device according to Claim 9 characterized in that the computing unit (6) is set up to - generate an error matrix which has a second reference numerical value at positions which correspond to the error entries; and - to determine the key figure based on the error matrix. Sensor device according to Claim 10 , characterized in that the arithmetic unit (6) is set up to - generate an integral image of the error matrix; and to determine the characteristic number based on the integral image of the error matrix. Sensor device according to one of the Claims 9 to 11 , characterized in that the computing unit (6) contains a field-programmable gate array to generate the point cloud, to replace the error entries, to generate the integral image of the modified coordinate matrix, to determine the characteristic number, to calculate the mean value and the surface normal (8 ) to be determined. Motor vehicle with a sensor device (7) according to one of the Claims 9 to 12th . Computer program with commands which, when the computer program is executed by a sensor device (7), according to one of the Claims 9 to 12th the sensor device (7) to cause a method according to one of the Claims 1 to 8th perform. Computer-readable storage medium on which a computer program is based Claim 14 is stored.

Images