DE102018100315A1 - Generating input data for a convolutional neural network - Google Patents

Abstract

The present invention relates to a method for generating input data for a convolutional neural network using at least one camera (3) and at least one range sensor (5, 6), wherein the camera (3) and the range sensor (5, 6) on the motor vehicle (1) that the field of view of the camera (3) and the field of view of the area sensor (5, 6) overlap at least partially, the method comprising the following steps:
Capturing an image frame by the camera (3), the image frame consisting of image data for directions relative to the position of the camera (3) and within the solid angle covered by the camera (3), the directions being represented by coordinates in a camera coordinate system,
simultaneous detection of depth information by the area sensor (5, 6), the depth information consisting of depth data for directions relative to the position of the area sensor (5, 6) and within the space angle covered by the area sensor (5, 6), the directions passing through Coordinates are displayed in a range sensor coordinate system,
Providing a motor vehicle coordinate system related to the camera coordinate system and the area sensor coordinate system by respective sets of translations and rotations given by the position of the camera (3) and the position of the area sensor (5, 6) relative to the origin of the motor vehicle coordinate system, and
Transforming the coordinates in the camera coordinate system and the coordinates in the area sensor coordinate system into coordinates in the motor vehicle coordinate system on the basis of the sets of translations and rotations, thereby obtaining the input data for the convolutional neural network. In this way, a semantic segmentation of objects in an image in Automobile Computer Vision can be improved.

Description

The present invention relates to a method for generating input data for a convolutional neural network using at least one camera and at least one area sensor.

One of the most fundamental problems with automotive computer vision is the semantic segmentation of objects in an image. The segmentation approach refers to the problems of associating each pixel with its corresponding feature class. Recently, there has been a spike in research and design of convolutional neural networks (CNN), supported by increased computational power in computer architectures and the availability of large annotated datasets.

CNNs are very successful in classifying and categorizing tasks, but much of the research is on standard RGB RGB photometric images and does not focus on embedded automotive devices. Automobile hardware must have a low power consumption and thus a low computing power.

In machine learning, a convolutional neural network is a class of deep, feedforward artificial neural networks that has been successfully used to analyze visual images. CNNs use a variation of multilayer perceptrons that are designed to require minimal preprocessing. Convolutional networks have been inspired by biological processes in which the connectivity pattern between neurons is inspired by the organization of the animal visual cortex. Individual cortical neurons respond to stimuli only in a restricted area of the visual field known as the receptive field. The receptive fields of different neurons partially overlap so that they cover the entire visual field.

CNNs use relatively little preprocessing compared to other image classification algorithms. This means that the network learns the filters that have been manually developed in conventional algorithms. This independence of prior knowledge and human effort in feature design is a great advantage. CNNs are used in image and video recognition, recommendation systems and natural language processing.

The article "Multimodal Deep Learning for Robust RGB-D Object Recognition, Andreas Eitel, Jost Tobias Springenberg, Luciano Spinello, Martin Riedmiller, Wolfram Burgard, IEEE / RSJ International Conference on Intelligent Robotics and Systems (IROS), Hamburg, Germany, 2015" proposes an RGB-D architecture for object recognition. This architecture consists of two separate CNN processing streams - one for each modality - that are sequentially combined with a late-fusion network. The focus is on learning with imperfect sensor data, a typical problem in real robotics tasks. For accurate learning, a multi-level training methodology and two key factors for handling depth data with CNNs are introduced. The first is effective encoding of depth information for CNNs, which allows learning without the need for large depth data sets. The second is a data enhancement scheme for robust learning with depth images by degrading them with realistic noise patterns.

From the US 2017/0099200 A1 It is known that data is received that characterizes a request for agent-based calculation of sensor data. The request contains the required trust and latency required to complete the agent-based calculation. Agents to request are determined based on the required trust. Data is transmitted to prompt the particular agents to perform an analysis of the sensor data.

The invention has for its object to provide a way to improve a semantic segmentation of objects in an image in automotive vision computer.

The object is achieved by the subject matter of the independent claims. Preferred embodiments are specified in the subclaims.

• - Erfassen eines Bildrahmens durch eine Kamera, wobei der Bildrahmen aus Bilddaten für Richtungen relativ zur Position der Kamera und innerhalb des durch die Kamera abgedeckten Raumwinkels besteht, wobei die Richtungen durch Koordinaten in einem Kamerakoordinatensystem dargestellt werden,
• - gleichzeitiges Erfassen von Tiefeninformation durch den Bereichssensor, wobei die Tiefeninformation aus Tiefendaten für Richtungen relativ zur Position des Bereichssensors und innerhalb des durch den Bereichssensor abgedeckten Raumwinkels bestehen, wobei die Richtungen durch Koordinaten in einem Bereichssensorkoordinatensystem dargestellt werden,
• - Bereitstellen eines Kraftfahrzeugkoordinatensystems, das mit dem Kamerakoordinatensystem und dem Bereichssensorkoordinatensystem durch entsprechende Sätze von Translationen und Rotationen in Beziehung steht, die durch die Position der Kamera und die Position des Bereichssensors relativ zum Ursprung des Kraftfahrzeugkoordinatensystems gegeben sind, und
• - Transformieren der Koordinaten im Kamerakoordinatensystem und der Koordinaten im Bereichssensorkoordinatensystems in Koordinaten im Kraftfahrzeugkoordinatensystem auf der Basis der Sätze von Translationen und Rotationen, wodurch die Eingabedaten für das konvolutionelle neurale Netzwerk erhalten werden.

The invention therefore provides a method for generating input data for a convolutional neural network using at least one camera and at least one range sensor, wherein the camera and the range sensor are arranged on the motor vehicle such that the field of view of the camera and the field of view of the range sensor overlap at least in part, the method comprising the following steps:

• Capturing an image frame by a camera, the image frame consisting of image data for directions relative to the position of the camera and within the solid angle covered by the camera, the directions being represented by coordinates in a camera coordinate system,
• - simultaneous detection of depth information by the range sensor, the depth information consisting of depth data for directions relative to the position of the range sensor and within the solid angle covered by the range sensor, the directions being represented by coordinates in an area sensor coordinate system,
• Providing a motor vehicle coordinate system related to the camera coordinate system and the area sensor coordinate system through respective sets of translations and rotations given by the position of the camera and the position of the area sensor relative to the origin of the motor vehicle coordinate system, and
• Transforming the coordinates in the camera coordinate system and the coordinates in the area sensor coordinate system into coordinates in the motor vehicle coordinate system on the basis of the sets of translations and rotations, thereby obtaining the input data for the convolutional neural network.

It is thus an essential idea of the invention that the input data for the convolutional neural network have both image data and depth data for common viewing directions relative to the origin of the motor vehicle coordinate system, the directions being represented by coordinates of the common motor vehicle coordinate system serving as a common reference system. In other words, the input data for the convolutional neural network consists of image data and depth data for directions shown in the vehicle coordinate system, although such data was originally represented as data in the coordinate system of the camera and the area sensor, respectively. The transformation of this data into the common automotive coordinate system provides the ability to use data from various sensors / cameras in a common data set that is input to the convolutional neural network. Preferably, the camera continuously captures image frames, and the region sensor continuously captures depth information. Preferably, as a last step of the above-described method, the generated data set consisting of the depth data and the image data is input to the CNN.

• - Darstellen der Koordinaten im Kamerakoordinatensystem durch eine Richtungskosinusmatrix und
• - Darstellen der Koordinaten im Bereichssensorkoordinatensystem durch eine Richtungskosinusmatrix.

According to a preferred embodiment of the invention, the method further comprises the following steps:

• Representing the coordinates in the camera coordinate system by a direction cosine matrix and
• Representing the coordinates in the area sensor coordinate system by a direction cosine matrix.

As known to those skilled in the art, the direction cosine values of a vector are the cosine values of the angles between the vector and the three coordinate axes. Equally meaningful are the contributions of each component of the base to a unit vector in that direction. Directional cosine is an analogous extension of the usual concept of slope to higher dimensions. Therefore, the direction cosine refers to the cosine of the angle between any two vectors. They are used inter alia to form direction cosine matrices representing one set of orthonormal basis vectors with respect to another set, or to represent a known vector in another base.

• - Darstellen der Bilddaten durch einen Farbwert, vorzugsweise durch einen RGB-Wert, für jedes Koordinatentripel der Kosinusmatrix, und
• - Darstellen der Tiefendaten durch einen Abstandswert für jedes Koordinatentripel der Kosinusmatrix.

Preferably, the method further comprises the following steps:

• Representing the image data by a color value, preferably by an RGB value, for each coordinate triplet of the cosine matrix, and
• Representing the depth data by a distance value for each coordinate triplet of the cosine matrix.

In this way, a data set having a color value (as a part of the image frame) and a respective distance value (as a part of a depth map) for a plurality of directions relative to the origin of the motor vehicle coordinate system may be input to the CNN and processed together therein.

Generally, different types of cameras can be used. According to a preferred embodiment of the invention, however, the camera is a fisheye camera with a field of view that is at least 180 °. In general, a single camera may be sufficient for the method according to the invention. However, in accordance with a preferred embodiment of the invention, multiple cameras are used to generate the input data to the convolutional neural network. Preferably, these cameras have different fields of view. More preferably, these cameras cover the entire environment of the motor vehicle.

Further, it is preferable to use a plurality of area sensors for generating the input data to the convolutional neural network. In general, these area sensors can be used by the same type. However, according to a preferred embodiment of the invention, the range sensors have at least two different types of range sensors, preferably at least one LIDAR sensor and at least one ultrasonic sensor. Preferably, these range sensors have different fields of view. More preferably, these range sensors cover the entire environment of the motor vehicle.

The invention also relates to the use of a method described above in a motor vehicle, a sensor assembly for a motor vehicle configured to perform such a method, and a non-transitory computer-readable medium having instructions stored thereon when executed by a processor be cause a sensor assembly of a motor vehicle to carry out such a method.

• 1 schematisch ein Kraftfahrzeug mit einer Sensoranordnung zum Erfassen eines Objekts gemäß einer bevorzugten Ausführungsform der Erfindung;
• 2 schematisch das Kamerakoordinatensystem und das Bereichssensorkoordinatensystem gemäß der bevorzugten Ausführungsform der Erfindung; und
• 3 schematisch das Kraftfahrzeugkoordinatensystem gemäß der bevorzugten Ausführungsform der Erfindung.

Show it:

• 1 schematically a motor vehicle with a sensor arrangement for detecting an object according to a preferred embodiment of the invention;
• 2 schematically the camera coordinate system and the area sensor coordinate system according to the preferred embodiment of the invention; and
• 3 schematically the motor vehicle coordinate system according to the preferred embodiment of the invention.

As in 1 is shown schematically, is in a motor vehicle 1 According to a preferred embodiment of the invention, a sensor arrangement 2 with a camera 3 , an evaluation unit 4 an ultrasonic sensor 5 and a LIDAR sensor 6 provided. As shown by dashed lines, have the camera 3 , the ultrasonic sensor 5 and the LIDAR sensor 6 respective fields of view that overlap. This makes it possible to capture scenes with image data or depth data that can be entered into a convolutional neural network that is in the evaluation unit 4 for classifying objects, such as the person 7 , in front of the motor vehicle 1 is provided.

By using different types of range sensors 5 . 6 ie an ultrasonic sensor 5 and a LIDAR sensor 6 , it is possible to generate multiple input depth maps with RGB image data to use a CNN network that can capture and classify objects. The application here is vehicle sensors, such as the camera 3 , the ultrasonic sensor 5 and the LIDAR sensor 6 to use to generate depth information around a vehicle and to combine this data with environmental view image data. Therefore, such automotive sensors are preferably on all sides of the motor vehicle 1 arranged such that the entire environment of the motor vehicle can be monitored. For the sake of clarity, the present preferred embodiment of the invention focuses only on the three automotive sensors mentioned above as an example.

It is an important aspect of the present preferred embodiment of the invention, the range sensors 5 . 6 ie the ultrasonic sensor 5 and the LIDAR sensor 6 to encode in the same coordinate system as the camera data to produce CNN input data that uses RGB and multi-depth maps together. This input data can then be entered into a convolutional neural network for classification.

As in 2 is shown schematically, each sensor has its own mechanical coordinate system. Here are due to the two-dimensionality of the figure, only the x-axes and the z -Axis represented, ie x _C and z _C for the camera 3 . x _U and z _U for the ultrasonic sensor 5 and x _L and z _L for the LIDAR sensor. Furthermore, as in 3 is shown schematically, a motor vehicle coordinate system as a common reference coordinate system for all vehicle sensors 3 . 5 . 6 Are defined. The motor vehicle coordinate system has its origin ( 0 . 0 . 0 ) in the middle of the front portion of the motor vehicle 1 at street level. With regard to the respective positions of the motor vehicle sensors 3 . 5 . 6 (and all other automotive sensors that may be mounted on the motor vehicle) there is a set of rotations and translations for defining the relationship between each sensor and the vehicle coordinate system. All sensor data may then be transferred to the motor vehicle coordinate system as a common frame of reference and transmitted to the CNN.

In detail, this method according to the present preferred embodiment of the invention is implemented as follows:

Through the camera 3 Image frames are continuously recorded, with the image frames being image data for directions relative to the position of the camera 3 and within of the camera 3 covered spatial angle, wherein the directions are represented by coordinates in the above-described camera coordinate system. At the same time, depth information is provided by the range sensors 5 . 6 ie the ultrasonic sensor 5 and the LIDAR sensor 6 , wherein the depth information is from depth data for directions relative to the positions of the range sensors 5 . 6 and within through the area sensors 5 . 6 Covered solid angle, wherein the directions are represented by coordinates in the coordinate systems of the area sensors.

As described above, there is provided an automotive coordinate system related to the camera coordinate system and the coordinate systems of the area sensors through respective sets of translations and rotations, each by the position of the camera 3 and the positions of the area sensors 5 . 6 given relative to the origin of the motor vehicle coordinate system. Then, the coordinates in the camera coordinate system and the coordinates in the area sensor coordinate systems are transformed based on the sets of translations and rotations in coordinates in the vehicle coordinate system. In this way, the input data for the convolutional neural network is obtained and input to the CNN for object classification.

According to the preferred embodiment of the invention described herein, the coordinates in the camera coordinate system and the coordinates in the range sensor coordinate system are represented by a respective direction cosine matrix. Further, the image data is represented by a color value, i. by an RGB value, for each coordinate triplet of the cosine matrix, and the depth data is represented by a distance value for each coordinate triplet of the cosine matrix.

This way, using image information from the camera 3 together with depth information from different area sensors 5 . 6 A semantic segmentation of objects in an image in automotive computer vision can be significantly improved.

LIST OF REFERENCE NUMBERS

1

motor vehicle

2

sensor arrangement

3

camera

4

evaluation

5

ultrasonic sensor

6

LIDAR sensor

7

person

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 2017/0099200 A1 [0007]

Claims (10)

Method for generating input data for a convolutional neural network using at least one camera (3) and at least one area sensor (5, 6), wherein the camera (3) and the area sensor (5, 6) are arranged on the motor vehicle (1) in that the field of view of the camera (3) and the field of view of the area sensor (5, 6) overlap at least partially, the method comprising the following steps: Capturing an image frame by a camera (3), the image frame consisting of image data for directions relative to the position of the camera (3) and within the solid angle covered by the camera (3), the directions being represented by coordinates in a camera coordinate system; simultaneous detection of depth information by the area sensor (5, 6), the depth information consisting of depth data for directions relative to the position of the area sensor (5, 6) and within the space angle covered by the area sensor (5, 6), the directions passing through Coordinates are displayed in an area sensor coordinate system; Providing a motor vehicle coordinate system related to the camera coordinate system and the area sensor coordinate system by respective sets of translations and rotations given by the position of the camera (3) and the position of the area sensor (5, 6) relative to the origin of the motor vehicle coordinate system; and Transforming the coordinates in the camera coordinate system and the coordinates in the area sensor coordinate system into coordinates in the motor vehicle coordinate system on the basis of the sets of translations and rotations, thereby obtaining the input data for the convolutional neural network. Method according to Claim 1 the method further comprising the steps of: - representing the coordinates in the camera coordinate system by a direction cosine matrix; and - representing the coordinates in the area sensor coordinate system by a direction cosine matrix. Method according to Claim 1 or 2 wherein the method further comprises the steps of: - representing the image data by a color value, preferably by an RGB value, for each coordinate triplet of the cosine matrix; and - representing the depth data by a distance value for each coordinate triplet of the cosine matrix. Method according to one of the preceding claims, wherein the camera (3) is a fisheye camera with a field of view which is at least 180 °. Method according to one of the preceding claims, wherein a plurality of cameras (3) are used to generate the input data for the convolutional neural network. Method according to one of the preceding claims, wherein a plurality of area sensors (5, 6) are used for generating the input data for the convolutional neural network. Method according to Claim 6 wherein the area sensors (5, 6) comprise at least two different types of area sensors, preferably at least one LIDAR sensor (6) and at least one ultrasonic sensor (5). Use of the method according to one of the preceding claims in a motor vehicle (1). Sensor arrangement (2) for a motor vehicle (1), which is configured to perform the method according to one of Claims 1 to 8th perform. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause a sensor assembly (2) of a motor vehicle (1) to perform the method of any one of Claims 1 to 8th perform.

Images