DE102018130821A1 - Method for assessing an environment of a motor vehicle using an artificial neural network with an aggregation unit; Control unit, driver assistance system and computer program product - Google Patents

Abstract

The invention relates to a method for assessing the surroundings of a motor vehicle (1) using an image of a camera system of the motor vehicle (1). In order to enable an improved detection of the surroundings, the following steps are provided: - providing the image of the surroundings from a camera (2) of the motor vehicle (1), - reducing a resolution of the image from an input resolution to a reduced intermediate resolution by an input part ( 4) an artificial neural network and providing a corresponding intermediate image signal (A, B, C) with the intermediate resolution, - reducing a resolution of the image from the intermediate resolution to a further reduced working resolution by the input part (4) of the artificial neural network and providing one corresponding working image signal (D) with the working resolution, - combining the intermediate image signal (A, B, C) and the working image signal (D) into a common evaluation signal by an aggregation unit (5) of the artificial neural network, and - assessing the environment by evaluating the common one Evaluation signals by an evaluator unit (6) of the artificial neural network.

Description

The invention relates to a method for assessing an environment of a motor vehicle on the basis of an image of a camera system of the motor vehicle. The invention also relates to a control unit, a driver assistance system, a computer program product and a computer-readable medium.

Driver assistance systems often require information about an environment of a motor vehicle in order to function. Examples of driver assistance systems are emergency braking assistants, lane keeping assistants and systems that enable autonomous driving / parking. For example, such information can relate to the position of objects in the surroundings of the motor vehicle. Examples of such objects, the position of which may be necessary for the functioning of the driver assistance system, are lane boundaries, curbs, lane markings, other road users (vehicles, pedestrians, cyclists), parked vehicles and / or possible collision partners with whom the motor vehicle could collide during a journey .

For example, camera systems are known for capturing objects in the environment, and such a camera system can comprise one or more cameras. Such a camera can capture the surroundings of the motor vehicle in the form of images, in particular a video signal, which comprises a sequence of individual images (frames). However, the necessary information must first be extracted from these images. A so-called segmentation of images is known, for example, for recognizing objects. Different segments are recognized in the image and examined for their meaning. For this reason, semantic segmentation is often mentioned. Different image areas are thus captured and semantically analyzed.

Semantic segmentation can be done using machine vision, for example. Classic machine vision tools are usually used for machine vision. Examples of this are the evaluation of an optical flow, the detection of gradients or the so-called Hough transformation. Deviating from this, semantic segmentation using artificial neural networks has proven to be advantageous. An artificial neural network is learned to capture different areas in an image and to analyze them semantically. As part of a learning process, the artificial neural network independently learns which image features are of interest and which tools of image processing are to be used for semantic segmentation. Such a learning process can be carried out, for example, using already segmented example images.

It is also known to carry out a so-called depth estimate using the images mentioned above. In the context of such a depth estimation, in particular by means of an artificial neural network, three-dimensional size relationships in the image are inferred on the basis of two-dimensional image features of the image. Such a depth estimate can be used, for example, to recognize curbs or to estimate the distance from foreign objects, in particular third-party vehicles.

The object of the present invention is to enable an improved detection of an environment on the basis of camera images.

This object is achieved according to the invention by the subject matter of the independent claims. Advantageous embodiments with appropriate further developments are the subject of the dependent claims.

• - Bereitstellen des Bildes der Umgebung von einer Kamera des Kraftfahrzeugs,
• - Reduzieren einer Auflösung des Bildes von einer Eingangsauflösung auf eine verringerte Zwischenauflösung durch einen Eingangsteil eines künstlichen neuronalen Netzes und Bereitstellen eines entsprechenden jeweiligen Zwischenbildsignals mit der Zwischenauflösung,
• - Reduzieren einer Auflösung des Bildes von der Zwischenauflösung auf eine nochmals verringerte Arbeitsauflösung durch den Eingangsteil des künstlichen neuronalen Netzes und Bereitstellen eines entsprechenden Arbeitsbildsignals mit der Arbeitsauflösung,
• - Zusammenfassen des Zwischenbildsignals und des Arbeitsbildsignals zu einem gemeinsamen Auswertesignal durch eine Aggregationseinheit des künstlichen neuronalen Netzes, und
• - Beurteilen der Umgebung durch Auswerten des gemeinsamen Auswertesignals durch einen Auswerteeinheit des künstlichen neuronalen Netzes.

A first aspect of the invention relates to a method for assessing an environment of a motor vehicle on the basis of an image of a camera system of the motor vehicle. The process has the following steps:

• Providing the image of the surroundings from a camera of the motor vehicle,
• Reducing a resolution of the image from an input resolution to a reduced intermediate resolution by means of an input part of an artificial neural network and providing a corresponding intermediate image signal with the intermediate resolution,
• Reducing a resolution of the image from the intermediate resolution to a further reduced working resolution by the input part of the artificial neural network and providing a corresponding working image signal with the working resolution,
• - Combining the intermediate image signal and the working image signal into a common evaluation signal by an aggregation unit of the artificial neural network, and
• - Assessment of the environment by evaluating the common evaluation signal by an evaluation unit of the artificial neural network.

For example, the images are provided by the camera of the motor vehicle. The camera can be arranged in the intended position on the motor vehicle or can be arranged. For example, it is the camera around a front camera, a rear camera or a sideways camera of the motor vehicle. Alternatively or additionally, the images can be provided by means of an interface of the camera system and / or an interface of a control unit of the motor vehicle. For example, the control unit has the interface in order to provide the image of the environment for further method steps. The image can be a frame of a video signal from the camera. Accordingly, the present method can be carried out successively for a plurality of frames of the video signal.

The image is then processed through the input part of the artificial neural network. The image is reduced in its resolution in a multi-stage process. In particular, the resolution of the image is initially reduced from an input resolution to the intermediate resolution. The resolution of the image can then be reduced from the intermediate resolution to a working resolution. Several intermediate steps, in particular with a respective intermediate resolution, can be provided in the multi-stage reduction of the resolution. In other words, a number of intermediate resolutions can be provided, the image being reduced in several steps from the input resolution to a number of intermediate resolutions which are in each case becoming smaller. For example, three different intermediate resolutions are provided. Then the image can be successively reduced in its resolution by four reduction stages. In this example, the image is first of all reduced from the input resolution to a first of the intermediate resolutions, then further reduced from the first intermediate resolution to a second of the intermediate resolutions, then further reduced from the second intermediate resolution to a third of the intermediate resolutions and finally from the third intermediate resolution reduced to work resolution.

For example, when the resolution is reduced, image features in the image are extracted simultaneously. Such image features can relate to objects in the area, for example. The input part of the artificial neural network can be taught during a learning process to extract essentially those image features which are relevant for the underlying task, that is to say the assessment of the environment in the present case. This takes place automatically during the learning process based on the training data selected for this. The training data can include, for example, a large number of images of different objects which are then to be recognized during normal operation of the artificial neural network.

The working image signal can include the image of the surroundings in the working resolution. The working picture signal or the working resolution is so called because the picture is made available in the working resolution for further processing. In particular, the image is provided by the input part in the working resolution for the evaluation unit or transmitted to the evaluation unit. The evaluation unit can use the working image signal as an input signal to evaluate the image. In further processing steps, the resolution of the image can be changed by the evaluation unit. The working resolution or the working image signal thus generally represents only a basis for the evaluation following the reduction of the resolution by the evaluation unit.

The input part of the artificial neural network is preferably a so-called CNN encoder. The input part can essentially fold the image. A matrix that defines or derives the respective image is divided into several smaller matrices. This is done in particular in several successive steps. The matrix can, for example, indicate height by width by color of individual (in particular all) pixels of the image. In other words, the matrix can be understood as a digital image or digital representation of the image. Folding can, for example, be carried out using so-called convolutional layers. The matrix can be analyzed using so-called filters. With such a filter, one or more result matrix / result matrices can be formed from the original matrix, the values of which result from a partial area of the original matrix and depending on a size (kernel size), an edge behavior (padding) and a step size of the filter. Alternatively or additionally, the input part of the artificial neural network can comprise one or more aggregation layers (“pooling layer”). Such an aggregation position can summarize the values of several matrix elements. For example, when summarizing, an average value is formed across the several matrix elements ("average pooling") or only the maximum value of the several matrix elements is used ("max pooling"). Overall, the resolution of the image or a corresponding matrix is reduced by the input part. This can be done, for example, by the mechanisms mentioned above or by other mechanisms. Due to the learning process, the artificial neural network is able to reduce the resolution in such a way that relevant image features are extracted for the underlying task, that is to say the detection of the surroundings or the detection of objects. The image features can be passed through the input part of the artificial neural network in the form of a large number of matrices to a subsequent part of the artificial neural network.

The intermediate image signal can be the image in the intermediate resolution. In the case of several intermediate image signals, each of the intermediate image signals can be present in the corresponding intermediate resolution. The intermediate image signal or the intermediate image signals can each comprise a so-called feature map (also referred to with the English technical term “feature map”). In such a feature map, the image features that are extracted when the resolution is reduced in the respective intermediate step will be summarized. Such a feature map can also be provided as part of the working image signal.

The evaluation unit can be a so-called CNN decoder. Alternatively, the evaluation unit can comprise several CNN decoders. The evaluation unit can also be used to increase the resolution. For example, after evaluating the environment or after evaluating the image features, the evaluation unit increases the resolution of the image again, for example to the input resolution. This increase in resolution can take place by means of interpolation, in particular bilinear interpolation. The evaluation unit can be set up to assess the surroundings on the basis of the image features or on the feature maps of the intermediate image signal (s) and the working image signal. In particular, the evaluation unit evaluates the image features or the feature cards accordingly. This evaluation can be learned through the learning process of the artificial neural network. In other words, the evaluation unit can learn to evaluate the image features or the feature cards as part of the teaching process. As part of the teaching process, the evaluation unit can recognize, for example, links between certain image features and certain environmental conditions as prior knowledge. Based on this prior knowledge, similar environmental conditions can then be recognized later.

An idea essential to the invention is that the intermediate image signal or a plurality of intermediate image signals, which are formed in the multi-stage reduction of the resolution of the image in the input part of the artificial neural network, are decoupled or used for further processing in the artificial neural network. For this, however, it is necessary to link the intermediate image signal with the working image signal. In the present case, this is done by means of the aggregation unit. This can also be called an aggregation module. The working image signal is linked by the aggregation unit to one or more intermediate image signals. The working image signal and the one or more intermediate image signals each have a different resolution. For example, in each of the reduction stages, at least partially different image features are extracted from the image. With the aggregation unit, the image features that are extracted in the different reduction levels can be summarized as part of the evaluation signal. The different processing stages of the working image signal and the one or more intermediate image signals can thereby enable an improved detection of the surroundings. In particular, image features that are extracted after the resolution has been reduced several times generally correspond to local image features. In contrast, image features that are extracted after the first or second reduction in resolution generally correspond to more global image features. In other words, as the resolution is reduced in several stages, increasingly more local image features are extracted. The principle of linking the image features of different resolutions enables local and global image features to be connected particularly advantageously. The principle of an aggregation unit is known to the person skilled in the art “Pyramid Scene Parsing Network” by Hengshuang Zhao et al. (https://hszhao.github.io/projects/pspnet/) known. Improvements in accuracy are achieved in this document, although the inputs to the aggregation unit are all taken from the same processing stage.

For example, the environment is assessed using semantic segmentation of the image. In particular, it is provided that a semantic segmentation of the image takes place when the environment is assessed. In other words, the artificial neural network can be trained to semantically segment the image. In the context of such a semantic segmentation, different image areas, in which, for example, different objects and / or background areas in the environment are recorded, are segmented according to their respective meaning.

The meaning content can be determined in relation to a driver assistance function of a driver assistance system of the motor vehicle. For example, segmentation into passable and non-passable surrounding areas can take place. The semantic segmentation can essentially be carried out by the evaluation unit of the artificial neural network. In contrast, necessary for this by the input part Image features are extracted from the image. The aggregation unit allows image features that originate from different processing levels to be combined as part of the evaluation signal. Overall, thanks to the evaluation signal summarized according to the invention, a semantic segmentation can thus be carried out with improved accuracy compared to the prior art.

As an alternative or in addition, it is provided that the environment is assessed by means of a depth estimate. In particular, it is provided that when assessing the surroundings, a depth estimate is made for the surroundings shown in the picture. In other words, the two-dimensional image or two-dimensional image features of the image are used to extrapolate to a depth of image areas. Overall, thanks to the evaluation signal summarized according to the invention, a depth estimate can be carried out with improved accuracy compared to the prior art.

According to a development, it is provided that the evaluation unit comprises a plurality of independent evaluation modules, the evaluation modules carrying out several independent partial evaluations when assessing the environment. The evaluation modules can each have exactly one CNN decoder. The CNN decoders of the evaluation modules can be independent of one another. For example, the semantic segmentation and the depth estimate can each be independent partial evaluations. The respective independent partial evaluations can be carried out based on the same image features or based on the same evaluation signal. In this way, there are synergy effects between the individual partial evaluations, since these at least partially use the same image features. In addition, an accuracy can be increased, since image features that are primarily important for a first of the partial evaluations can also increase the accuracy of a second of the partial evaluations.

According to a further development, it is provided that the summarization or aggregation of the image for each of the evaluation modules is carried out separately by a respective aggregation unit assigned to the corresponding evaluation unit. In other words, each of the plurality of evaluation modules is assigned a respective aggregation unit. The respective partial evaluation by the evaluation modules is carried out, in particular, only on the basis of the evaluation signal of exactly that aggregation unit which is assigned to the corresponding evaluation module. In this way, different aggregation units can be assigned to the different evaluation modules.

In particular, it is provided that a mode of operation of the respective aggregation units is adapted to the partial evaluations of the respective subsequent evaluation module. In other words, the mode of operation of the plurality of aggregation units is in each case adapted to the evaluation module to which the corresponding aggregation unit is assigned. This particular way of working of the devaluation modules can be given by different architecture of the aggregation units. As an alternative or in addition, the respective mode of operation of the aggregation units can be predetermined by the learning process. In other words, a respective, individual mode of operation can be determined for each of the aggregation units as part of a learning process. The aggregation units are taught in particular for the respective partial evaluation of the respective evaluation module to which the corresponding aggregation unit is assigned. As a result, the combination of the intermediate image signal and the working image signal or the evaluation signal resulting therefrom can be adapted particularly well to the respective partial evaluation of the subsequent evaluation module.

According to a further development, the aggregation unit reduces the resolution of the intermediate image signal by combining the intermediate image signal and the working image signal, the respective principle for reducing the resolution differing between the input part and the aggregation unit. In other words, the resolution of the intermediate image signal or the intermediate image signals is further reduced in the aggregation unit. This is particularly necessary if the linkage or aggregation of the working image signal to the intermediate image signal is to take place in a particularly simple manner. In particular, it is provided that the linking takes place by concatenation of the individual signals, that is to say the working image signal and the intermediate image signal. In particular, it may be necessary that the working image signal and the intermediate image signal or the intermediate image signals for linking, aggregate or the concatenation each have the same resolution.

The reduction of the resolution in the aggregation unit is carried out in a different way than in the input part. For example, when the resolution is reduced by the aggregation unit compared to the input part, a larger kernel size ("kernel size") and / or a larger step size ("stride") can be used. In other words, a Dimension by which the resolution is reduced in a reduction step may be larger in the aggregation unit compared to the input part. The larger the kernel size and / or the larger the step size, the greater the reduction in resolution can be in a reduction step. Overall, different image information can be extracted from the image.

Alternatively or additionally, when the resolution is reduced by the input part and by the aggregation unit, a different method for combining individual pixels (“pooling”) can be provided in the context of reducing the resolution. For example, when summarizing by the aggregation unit, an average value over the several matrix elements is formed ("average pooling") and when summarizing through the input part, only the maximum value of the several matrix elements is used ("max pooling").

According to a further development, it is provided that the resolution of the intermediate image signal is reduced by the aggregation unit to a lower resolution than the working resolution and then increased, in particular interpolated, to the working resolution. In other words, the resolution of the intermediate image signal can be reduced by the aggregation unit to a resolution that is less than the working resolution. In a further step, the resolution of the intermediate image signal can be increased to the working resolution by the aggregation unit. For example, the resolution is increased in this way by interpolation, in particular bilinear interpolation. In other words, the resolution of the intermediate image signal can first be reduced by the aggregation unit and then increased again by interpolation. In this way, additional image features can be extracted from the image.

Some embodiments provide that the artificial neural network used is one with an E-Net structure. In other words, the artificial neural network can have an E-Net structure. Such an E-Net structure is characterized in particular by the fact that the input part, in particular the CNN encoder, essentially extracts image features, but the actual evaluation of the image features takes place only later by the evaluation unit, in particular the CNN decoder. It is advantageous here that the input part can be used universally for different partial evaluations of different subsequent evaluation modules.

• - einer Bereitstellungeinheit ausgebildet zum Bereitstellen eines Bildes der Umgebung von einer Kamera des Kraftfahrzeugs,
• - einem Eingangsteil des künstlichen neuronalen Netzes ausgebildet zum Reduzieren einer Auflösung des Bildes von einer Eingangsauflösung auf eine verringerte Zwischenauflösung und Bereitstellen eines entsprechenden Zwischenbildsignals in der Zwischenauflösung sowie ausgebildet zum Reduzieren einer Auflösung des Bildes von der Zwischenauflösung auf eine nochmals verringerte Arbeitsauflösung und Bereitstellen eines entsprechenden Arbeitsbildsignals in der Arbeitsauflösung,
• - einer Aggregationseinheit des künstlichen neuronalen Netzes ausgebildet zum Zusammenfassen des Zwischenbildsignals und des Arbeitsbildsignals zu einem gemeinsamen Auswertesignal, und
• - einer Auswerteeinheit des künstlichen neuronalen Netzes ausgebildet zum Beurteilen der Umgebung durch Auswerten des gemeinsamen Auswertesignals.

A further aspect of the invention relates to a control unit comprising an artificial neural network for assessing an environment of a motor vehicle on the basis of images from a camera system of the motor vehicle

• a provision unit designed to provide an image of the surroundings from a camera of the motor vehicle,
• - An input part of the artificial neural network is designed to reduce a resolution of the image from an input resolution to a reduced intermediate resolution and to provide a corresponding intermediate image signal in the intermediate resolution and is designed to reduce a resolution of the image from the intermediate resolution to a further reduced working resolution and to provide a corresponding working image signal in working resolution,
• an aggregation unit of the artificial neural network designed to combine the intermediate image signal and the working image signal into a common evaluation signal, and
• - An evaluation unit of the artificial neural network designed to assess the environment by evaluating the common evaluation signal.

Providing it can be, for example, an input interface of the control unit and / or a buffer of the control unit assigned to such an input interface. The interface can receive the image from the camera of the motor vehicle and make it available for further processing. The image can be buffered ("buffered") in the buffer. The artificial neural network can be part of the control unit. The control unit according to the invention is in particular designed to carry out the method according to the invention. Accordingly, further developments and advantages of the method according to the invention also apply analogously to the control unit and vice versa.

Another aspect of the invention relates to a driver assistance system for a motor vehicle with a camera for providing an image of an environment of the motor vehicle and the above-mentioned control unit. The camera can be arranged in the intended position on the motor vehicle or can be arranged.

In addition, part of the invention is a motor vehicle which comprises the above-mentioned driver assistance system. The motor vehicle is in particular a motor vehicle, for example a truck or a passenger car. For example, the motor vehicle have an electric drive and / or an internal combustion engine. The camera system, in particular the first and the second camera, of the driver assistance system are arranged on the motor vehicle. Features and advantages that are disclosed in relation to the method also apply to the driver assistance system.

The invention also includes a computer program product with program code means which are stored in a computer-readable medium in order to carry out the method according to the invention when the computer program product is processed on a processor of an electronic control unit.

The computer program product can be stored on a computer-readable medium. Thus, this invention also claims such a computer-readable medium, in particular in the form of a computer-readable floppy disk, CD, DVD, memory card, USB memory unit, or the like, in which program code means are stored in order to carry out the method according to the invention if the program code means are stored in a memory of an electronic Control unit loaded and processed on a processor of the electronic control unit. The readable medium can be volatile (non-volatile) or non-volatile. Such a volatile memory can in particular be formed by a working memory of a microprocessor.

Further features of the invention result from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description and the features and combinations of features mentioned below in the description of the figures and / or shown alone in the figures can be used not only in the combination indicated in each case but also in other combinations without departing from the scope of the invention . Embodiments of the invention are thus also to be regarded as encompassed and disclosed, which are not explicitly shown and explained in the figures, but which emerge from the explanations explained and can be generated by separate combinations of features. Designs and combinations of features are also to be regarded as disclosed, which therefore do not have all the features of an originally formulated independent claim. Furthermore, versions and combinations of features, in particular those explained above, are to be regarded as disclosed, which go beyond or differ from the combinations of features set out in the references of the claims.

• 1 ein Kraftfahrzeug in einer schematischen Vogelperspektive;
• 2 ein Blockschaltbild einer Steuereinheit zum Beurteilen einer Umgebung eines Kraftfahrzeugs; und
• 3 ein Blockschaltbild einer Aggregationseinheit der Steuereinheit.

Show:

• 1 a motor vehicle in a schematic bird's eye view;
• 2nd a block diagram of a control unit for assessing an environment of a motor vehicle; and
• 3rd a block diagram of an aggregation unit of the control unit.

1 shows a motor vehicle 1 in a schematic bird's eye view. The car 1 has a camera system 20th with a camera 2nd on. In the present embodiment, the camera 2nd designed as a front camera. Alternatively, the camera 2nd also be designed as a rear-facing camera or as a laterally aligned camera. The camera system 20th can optionally have several cameras, for example the 4 cameras mentioned above.

Together with a control unit 3rd is the camera 2nd Part of a driver assistance system 21st of the motor vehicle 1 . The driver assistance system 21st is for judging an environment of the motor vehicle 1 set up. The camera does this 2nd the control unit 3rd Images of the area ready. For example, the camera 2nd the control unit 3rd Provide still images or the environment. In the pictures of the camera 2nd the environment is mapped.

The control unit 3rd has an artificial neural network. 2nd shows a block diagram of the control unit 3rd . A staging unit 9 the control unit 3rd is set up to take pictures from the camera system 20th for further processing by the control unit 3rd to provide. For example, the staging unit 9 an input interface of the control unit 3rd . The staging unit 9 can take the pictures from the camera 2nd received and optionally between save.

An image of the surroundings is then through an entrance part 4th of the neural network. The entrance part 4th can for example be designed as a CNN encoder. Through the entrance part 4th an input resolution of the image is reduced. The input resolution can be the resolution with which the image is the input part 4th provided. For example, the input resolution corresponds to a resolution with which the camera 2nd captured the image of the environment. For example, the image has an input resolution of 2048x1024 pixels. In several reduction levels 40 , 41 , 42 , 43 is through the input part 4th reduced the input resolution. For example, the input resolution is first successively reduced to several intermediate resolutions. For example, the resolution of the image is gradually increased from the input resolution to a first intermediate resolution, from the first intermediate resolution to a second intermediate resolution second intermediate resolution to a third intermediate resolution and reduced from the third intermediate resolution to a working resolution. In the present example, each time the resolution is reduced or in each of the reduction levels 40 , 41 , 42 , 43 halved the number of pixels in height and width. The reduction levels 40 , 41 , 42 , 43 of the entrance part 4th can have a kernel size of 2 and a step size of 2, for example. This corresponds, for example, to halving the number of pixels in length and width. Accordingly, the number of pixels each time the resolution is reduced or in each reduction step 40 , 41 , 42 , 43 quartered. The first intermediate resolution is 1024x512 pixels, the second intermediate resolution 512x256 pixels, the third intermediate resolution 256x128 pixels and the working resolution 128x64 pixels.

2nd shows that respective intermediate image signals A , B , C. as well as a working picture signal D to an aggregation unit 5 of the artificial neural network. The intermediate image signals are each based on the different reduction levels 40 , 41 , 42 , 43 from the entrance part 4th provided. In other words, the intermediate image signal relates A the picture after the first reduction stage 40 . The intermediate image signal B concerns the image of the environment after the second reduction stage 41 . The intermediate image signal C. relates to the intermediate image signal after the third reduction stage 42 . The working image signal de relates to the image after the fourth and last reduction stage 43 . Accordingly, the intermediate image signals each have the first, second, third intermediate resolution or the working resolution.

As part of each of the intermediate image signals A , B , C. or the working image signal D respective feature maps can be output. The respective feature cards relate to image features of the image of the surroundings, which are extracted by the input part as part of the respective reduction stage. In other words, the different signals (working image signal and intermediate image signals) A , B , C. , D Image features, the different processing levels of the input part 4th are taken out.

In the aggregation unit 5 become the signals A , B , C. , D connected. 3rd shows a block diagram of the aggregation unit 5 . First, the intermediate image signals A , B , C. a respective reduction level 10th the aggregation unit 5 fed. The reduction levels 10th the aggregation unit 5 can compared to the reduction levels 40 , 41 , 42 , 43 of the entrance part 4th use different algorithms to reduce the resolution. For example, the reduction levels have 10th the aggregation unit 5 a larger value for the kernel size and / or a larger value for the step size than the reduction levels 40 , 41 , 42 , 43 . The reduction levels 10th for the different intermediate signals A , B , C. can be designed differently.

In the present example, the reduction level shows 10th for the intermediate signal A a kernel size of 64 as well as a step size 64 on. The reduction level 10th for the intermediate signal B also has an example kernel size of 64 as well as a step size 64 on. The reduction level 10th for the intermediate signal C. has an example kernel size of 32 as well as a step size 32 on. This results from the respective reduction level 10th If the above-mentioned intermediate resolutions are used, resolutions of 16x8 pixels for the intermediate signal A , 8x4 pixels for the intermediate signal B and 8x4 pixels for the intermediate signal C. . This is less than the working resolution of 128x64 pixels.

Then the aggregation unit 5 respective folding layers 11 ("Convolutional layer") for each of the intermediate signals A , B , C. on. In a further step, a respective corresponding unit 12 a respective normalization, in particular a batch normalization, is carried out. In addition, the aggregation unit includes 5 a respective activation function 13 for each of the intermediate signals A , B , C. . Such an activation function 13 is provided, for example, by means of a unit that is usually referred to with the English technical term “rectified linear unit”, or “ReLu” for short. Subsequently, the resolution is increased by a respective interpolation module 14 . Through the interpolation module 14 becomes the resolution of the intermediate signals A , B , C. each increased to the working resolution. This is done in particular by means of bilinear interpolation.

In a concatenation module 15 are now the intermediate image signals A , B , C. and the working image signal D linked together. The respective feature cards are stacked one behind the other. In other words, the individual feature cards are lined up. This is possible because of the signals A , B , C. , D each have the same resolution. The result of this concatenation is an evaluation signal.

Now referring to 2nd the evaluation signal becomes an evaluation module 6 of the artificial neural network. In the evaluation module 6 For example, an assessment of the environment is carried out on the basis of the image features or the linked feature cards of the evaluation signal.

In the present exemplary embodiment, there are two evaluation modules that are independent of one another 6 , 8th intended. Through the evaluation modules 6 , 8th independent partial evaluations are carried out. For example, the evaluation module 6 a semantic segmentation of the image is carried out. Through the evaluation module 8th a depth estimate of image areas of the image can be carried out independently of this. The independent partial evaluations are both based on the same image features or the same feature cards that are provided by the single input part 4th of the artificial neural network as part of the signals A , B , C. , D to be provided. Each of the evaluation modules 6 , 8th is a respective aggregation unit 5 , 7 assigned. The intermediate image signals A , B , C. through each of the aggregation units 5 , 7 for the subordinate evaluation module 6 , 8th aggregated or linked. The aggregation units 5 , 7 can have the same or a different operating principle. In particular, the aggregation unit 7 are still the same as on the basis of 3rd for the aggregation unit 7 has already been described. In this way, synergies can arise on the one hand and on the other hand the accuracy of both partial evaluations can be improved. This can reduce the necessary computing time while improving the accuracy of the assessment of the environment.

As part of a learning process for the artificial neural network, the two aggregation units can 5 , 7 however, are taught in different ways. In particular, it is possible that the aggregation units 5 , 7 to the partial evaluation of the subsequent evaluation module as part of the learning process 6 , 8th be adjusted. In other words, the signals are linked A , B , C. , D advantageous in each case in a way that is connected to the downstream evaluation module 6 , 8th is adjusted. In this way, the accuracy can be further improved.

Overall, in the common entrance part 4th ("Shared encoder") extracts all image features that are important for both partial evaluations. The advantage of the present approach is that the advantages of the modular structure are combined with the advantages of a multi-scalar approach ("multi-scale"). This combines the advantages of flat layers (“layering”) (simple and local image features with high localization quality) with the advantages of lower layers (complex and global image features due to the depth of the network and the larger receptive field). This fusion of features or image features enables an increase in the accuracy and thus the quality of the individual partial evaluations (“tasks”).

Claims (14)

Method for assessing an environment of a motor vehicle (1) using an image of a camera system of the motor vehicle (1), comprising the steps: - Providing the image of the surroundings from a camera (2) of the motor vehicle (1), - reducing a resolution of the image from an input resolution to a reduced intermediate resolution by an input part (4) of an artificial neural network and providing a corresponding respective intermediate image signal (A, B, C) with the intermediate resolution, Reducing the resolution of the image from the intermediate resolution to a further reduced working resolution by the input part (4) of the artificial neural network and providing a corresponding working image signal (D) with the working resolution, - Combining the intermediate image signal (A, B, C) and the working image signal (D) into a common evaluation signal by an aggregation unit (5) of the artificial neural network, and - Assessment of the environment by evaluating the common evaluation signal by an evaluation unit (6) of the artificial neural network. Procedure according to Claim 1 , characterized in that, when assessing the environment, a semantic segmentation of the image takes place. Method according to one of the preceding claims, characterized in that, when assessing the environment, a depth estimate is carried out for the environment shown in the image. Method according to one of the preceding claims, characterized in that the evaluation unit comprises a plurality of independent evaluation modules (6, 8), with the evaluation modules carrying out several mutually independent partial evaluations when assessing the environment. Procedure according to Claim 4 , characterized in that the combination of the image for each of the evaluation modules (6, 8) is carried out separately by a respective aggregation unit (5, 7) assigned to the respective evaluation unit. Procedure according to Claim 4 or 5 , characterized in that, a mode of operation of the respective aggregation units (5, 7) is adapted to the partial evaluations of the respectively subsequent evaluation module (6, 8). Method according to one of the preceding claims, characterized in that, for the combination of the intermediate image signal (A, B, C) and the working image signal (D), the resolution of the intermediate image signal is reduced by the aggregation unit (5), which is a respective principle to reduce the resolution between the input part and the aggregation unit. Procedure according to Claim 7 , characterized in that, when reducing the resolution by the aggregation unit compared to the input part, a larger kernel size and / or a larger step size is used. Procedure according to Claim 7 or 8th , characterized in that the resolution of the intermediate image signal is reduced by the aggregation unit to a lower resolution than the working resolution and then increased to the working resolution, in particular interpolated. Method according to one of the preceding claims, characterized in that, as the artificial neural network, one with an E-Net structure is used. Control unit (3) comprising an artificial neural network for assessing an environment of a motor vehicle (1) on the basis of images of a camera system of the motor vehicle (1), with - a provision unit designed to provide an image of the surroundings from a camera (2) of the motor vehicle (1), - An input part (4) of the artificial neural network designed to reduce a resolution of the image from an input resolution to a reduced intermediate resolution and to provide a corresponding intermediate image signal (A, B, C) in the intermediate resolution and designed to reduce a resolution of the image from the intermediate resolution on a further reduced working resolution and provision of a corresponding working image signal (D) in the working resolution, - An aggregation unit (5) of the artificial neural network designed to combine the intermediate image signal (A, B, C) and the working image signal (D) into a common evaluation signal, and - An evaluation unit (6) of the artificial neural network designed to assess the environment by evaluating the common evaluation signal. Driver assistance system (21) for a motor vehicle (1), with - a camera (2) for providing an image of an environment of the motor vehicle (1), and - a control unit (3) Claim 11 . Computer program product with program code means which are stored in a computer-readable medium for the method for assessing an environment of a motor vehicle (1) on the basis of images of a camera system (20) of the motor vehicle (1) according to one of the preceding Claims 1 to 10th to be carried out when the computer program product is processed on a processor of an electronic control unit (3). Computer-readable medium, in particular in the form of a computer-readable floppy disk, CD, DVD, memory card, USB memory unit, or the like, in which program code means are stored in order to perform the method for assessing an environment of a motor vehicle (1) on the basis of images from a camera system (20) Motor vehicle (1) according to one of the preceding Claims 1 to 10th to be carried out when the program code means are loaded into a memory of an electronic control unit (3) and processed on a processor of the electronic control unit (3).

Images