FR3089662A1 - Method for recognizing objects such as traffic signs by means of an on-board camera in a motor vehicle - Google Patents

Abstract

The invention relates to a method of recognizing an object by means of a camera on board a motor vehicle, the method comprising the following steps: - capture (100) of a first image by said camera; - presentation (200) of the image to a first neural network, previously trained on a first learning base formed by couples each comprising an image captured by said camera and a second image, corresponding to an image captured simultaneously with the first image by a second camera temporarily implanted so as to have a different angular orientation with respect to the camera, and so that the two images have an overlap area; - generation (300) of a second image corresponding to the first image (300); - presentation (400) of the two images to a second neural network; - classification (500) of the images; - evaluation (600) of the robustness of the classification. Figure for the abstract: Fig. 3

Description

Description

Title of the invention: Method for recognizing objects such as traffic signs by means of an on-board camera in a motor vehicle

The present invention relates to the field of detection and recognition of objects in the environment of a motor vehicle. It relates in particular to a method improving the recognition in real time of traffic signs appearing on images captured by an on-board camera in a motor vehicle.

Motor vehicles are increasingly equipped with sophisticated driver assistance systems, commonly known by the English acronym ADAS (for "Advanced Driver Assistant Systems"), these systems having the objective of improving safety and / or driving comfort.

Some driver assistance systems use imaging devices equipped with so-called intelligent cameras, integrating artificial neural networks for the detection and recognition of objects, such as traffic signs. These neural networks are said to be deep: they have many layers of neurons, and are trained on millions of copies of images before developing performances deemed satisfactory for the desired function. The inventors have found that neural networks that have developed expertise only in vision, such as in the visual sub-domain of detecting traffic signs, do not classify images in the same way as a human being. . In particular, it is possible to artificially generate adverse attacks (known as "adversarial attacks" or "adversarial examples" in English), which, when superimposed on the original image, are undetectable or harmless for the human eye, but which are however likely to mislead the neural network in a radical way, and therefore potentially dangerous.

Such adverse attacks exist in very large numbers, and an example is presented in FIG. 1. FIG. 1 represents an image of a traffic sign, the image being affected by an adverse attack of physical origin, in this case stickers of various shapes and colors affixed to the panel. Such an image is for example capable of being classified by an artificial neural network known as representing a speed limit sign and not a stop sign. Adverse attacks can also be of digital origin, and in particular be specially designed to present, from the point of view of the neural network, all the diag2 signs

[0005]

[0006]

[0007]

[0008]

Nostics of another category of object. Such an adverse attack is likely to result in an incorrect classification of the image, the road sign then being confused with a different object.

These outliers are particularly critical in the case of an autonomous vehicle. In particular, the error caused by the opposing physical attack shown in Figure 1 is worrying for its ease of implementation, since they are simple stickers, and for its potentially disastrous impact in terms of road safety.

The present invention aims to overcome the drawbacks of the state of the art, and more particularly those described above, by proposing a method for recognizing objects located in the environment of a motor vehicle which is robust to images. adversaries, and which makes it possible in particular to avoid errors of interpretation and classification of an image constituting an adverse attack of physical or digital origin.

To this end, it proposes a method of recognizing an object, such as a traffic sign, the object appearing on an image coming from an on-board camera in a motor vehicle, the on-board camera being of monocular type, the method with the following steps:

- capture of an image, hereinafter captured image, by said on-board camera;

presentation of the captured image to a first artificial neural network forming a projector neural network, previously trained on a first learning base formed by pairs of images each comprising a first image captured by said on-board camera and a second image , hereinafter twin image, corresponding to an image as captured simultaneously with the first image by a second camera temporarily installed near this on-board camera so as to have a different angular orientation with respect to the on-board camera, and oriented in such a way so that the images of the second camera have an overlap area with the first image;

- generation by said first neural network of a twin image corresponding to the captured image;

- presentation of the captured image and the corresponding twin image to a second neural network forming a classifier network;

- classification by the second neural network of the captured image and the twin image;

- evaluation of the robustness of the classification of the captured image.

Thus, by implementing a first neural network previously trained with pairs of images, each couple comprising a first image corresponding to an image captured by the monocular camera of a vehicle, and a second image, or twin image, corresponding to an image which would have been captured simultaneously with the first image by a “twin” camera of the on-board camera (that is to say a camera temporarily located near the latter and taking shots from a slightly different angle) , the method according to the invention makes it possible to greatly improve the robustness of the recognition of objects such as traffic signs. Indeed, the inventors have found that these adversary attacks are effective only for a very limited range of range and range of view angles. Thus, a change of angle of view or distance to the object to be recognized makes it possible to obtain the correct classification of this object.

In one embodiment, the classification of the captured image is evaluated as being robust if the classification of the captured image and of the twin image are identical.

In one embodiment, if the classification is evaluated as non-robust, then the previous classification evaluated as robust is taken into account.

In one embodiment, the recovery rate between the twin image and the captured image is greater than or equal to 80%, preferably greater than 90% and for example greater than 95%.

In one embodiment, said first neural network comprises a plurality of convolutional layers and an identical number of deconvolutionary layers arranged symmetrically.

In one embodiment, said first neural network includes direct connections between two symmetrical convolutional and deconvolutionary layers.

In one embodiment, said first neural network is devoid of subsampling layers.

In one embodiment, said first neural network includes a loss function of the mean square error type.

In one embodiment, said first learning base is made up of several tens of thousands of pairs of images.

In one embodiment, said second artificial neural network is previously trained on a second learning base formed by images comprising objects to be recognized, including traffic signs.

In one embodiment, said second neural network has a standard convolutional network architecture, different from that of said first neural network, in particular in that it does not include deconvolutionary layers.

The invention also relates to a computer program product comprising instructions which, when the program is executed by a computer, lead the latter to implement the steps of the method as defined above.

The invention also relates to a motor vehicle equipped with an on-board camera and a computer arranged to implement a method as defined above.

The description of the invention will now be continued with the detailed description of an exemplary embodiment, given below by way of illustration but not limitation, with reference to the accompanying drawings, in which:

[Fig.l]

FIG. 1 illustrates an image of a traffic sign captured by an on-board camera in a motor vehicle, the image presenting an adverse attack of physical origin;

[Fig.2]

FIG. 2 is a diagram of an imaging device intended to be integrated into a driving assistance system for a motor vehicle;

[Fig.3]

Figure 3 is a flowchart detailing the steps of a method according to the invention;

[Fig.4]

FIG. 4 represents an image captured by a camera on board a motor vehicle;

[Fig.5]

Figure 5 shows a twin image of the image of Figure 4;

[Fig.6]

FIG. 6 represents an example of architecture of the first artificial neural network;

[Fig.7]

FIG. 7 represents an example of architecture of the second artificial neural network.

FIG. 2 illustrates an imaging device 1 intended to be integrated into an advanced driving assistance system (ADAS) of a motor vehicle such as an adaptive cruise control.

The imaging device 1 comprises a monocular optical camera 10 on board the vehicle and being provided with a two-dimensional image sensor 11, for example of the CCD type (acronym in English for "Charge-Coupled Device"). translating into French as “charge transfer device”) or CMOS (acronym in English of “Complementary Metal-Oxide Semiconductor” translating into French as “complementary metal oxide semiconductor”).

The camera 10 also includes an optical system 12 associated with the image sensor 11 and being adapted to form on the latter an image of an exterior environment of the vehicle viewed by the camera through an external face of this optical system.

The imaging device 1 also includes a processing module 20 connected to the on-board camera 10 (or integrated therein) and capable of correcting in real time the images captured by the latter when the external face of its optical system has fogging or soiling.

The processing module 20 comprises a computer 21 as well as a storage module 22 comprising non-volatile memory, for example of the EEPROM or FLASH type and random access memory.

The non-volatile memory stores a process for processing in real time the images captured by the on-board camera 10.

According to a preferred embodiment, all of the information contained in this non-volatile memory can be updated by means of communication or means of reading a data medium.

We will now describe in detail in relation to the flow diagram of Figure 3, the different steps of a method according to the invention.

This is initiated with each new capture, by the image sensor 11 of the on-board camera 10, of a captured image I _c (step 100), such as the image shown in FIG. 4. L captured image I _c is likely to represent an adverse attack, that is to say that at least part of the image is likely to include an adverse attack of digital or physical origin, such as for example that represented on Figure 1 (step 100). An adverse attack, as explained above, is likely to result in an erroneous recognition (or classification) of one or more objects appearing on this image, such as for example a traffic sign.

The captured image I _c is then presented to a projector algorithm (step 200) implemented by a first artificial neural network Ri. The architecture of this first neural network R _b or projector network, described below, is advantageously inspired by those described in the articles “Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections” (Mao et al., 30th Conference on Neural Information Processing Systems, Barcelona, Spain, 2016) and “Unsupervised Monocular Depth Estimation with Left-Right Consistency” (Godard et al., Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition CVPR, 2017).

This first neural network R _b as illustrated in FIG. 6, comprises a plurality of convolutional layers C (advantageously between 10 and 15 layers) and an identical number of deconvolutionary layers D arranged symmetrically.

The projector network Ri as a whole has been the subject of an apprenticeship aimed at making it suitable for extracting the characteristics of the captured image which lend themselves best to a correct interpretation of the objects appearing in this image, c that is to say an interpretation in accordance with reality with regard to the object or objects present in the image captured by the camera 2.

The projection operation is encoded in the weights of the connections of the entire network, a priori one cannot point to a layer which would carry out the projection more than another, although the signal approaches the projection sought in the final deconvolution layers.

Each convolutional or deconvolutionary layer is advantageously composed of 64 filters of 3X3 size generating 64 characteristic maps (commonly known under the term "features maps") from the image entering this layer.

As a variant, the number of filters constituting each layer is different (for example equal to 32 or 128). The size of these filters can also be different (for example equal to 5X5, 7X7 or 9X9).

To avoid the loss of error signal inherent in the training of such deep neural networks, direct connections S (commonly known by the English term "skip connections") are provided between two convolutional and deconvolutionary layers symmetrical .

These direct connections advantageously have a pitch equal to three, that is to say that they all extend the three convolutional layers towards the corresponding symmetrical deconvolutionary layers.

They are also bijective for each filter: each unit in the convolutional input channel filter being connected to its equivalent in the deconvolutive output filter (so-called "one-to-one" connectivity).

Each of the convolutional or deconvolutional layers is followed by an activation function advantageously of the ReLU type (acronym in English of "Rectified Linear Unit" translating into French as "Unit Linear Rectified").

This function, also called "non-saturating activation function", gives non-linear properties to the decision function and to the entire network which positively impact on its performance.

This first neural network Ri projector is also advantageously devoid of subsampling layers (commonly known by the English term "pooling").

The input and output images have the same resolution format (1 xhxc), with 1, h and c which respectively represent the width, the height and the number of channels of these images (c being equal to 1 for a monochrome image and 3 for a color image coded in RGB).

This resolution format will advantageously be 800 x 480 x 3 for reasons of practicality (usual format of optical cameras installed in vehicles) and speed of calculations.

This first neural network Ri projector is also previously trained from a first learning base formed by several tens of thousands of pairs of images (advantageously, at least 40,000 and preferably at least 60,000) .

Each pair of images comprises a first image captured by the on-board camera 10 (which may or may not be noisy), and a second image captured simultaneously by a second camera identical to the camera 10, temporarily installed near this camera 10 so as to present a different viewing angle from that of the latter.

The performance of the neural network increasing with the volume and variety of the learning base, an arbitration is therefore required between the cost linked to the acquisition of the images during the rolling phase and the gain linked to the precision. of the two networks.

However, since no labeling of these images is necessary for this learning base (the images obtained by each camera being used for cross-supervision), its cost price is reduced because it consists solely of the costs of rolling and storing images.

In addition, new pairs of images are advantageously created by mirror transformation and inversion from the pairs of images generated during the acquisition phase.

This so-called augmentation technique makes it possible to multiply the number of pairs of images of the training base (and therefore to improve the performance of the neural network) without requiring additional running.

This first projector neural network Ri further comprises a loss function of the mean square error type (commonly known by the English acronym MSE for “Mean Squarred Error”) and making it possible to calculate the classification error on batches originating from of the learning base and comprising for example between 5 and 100 pairs of images.

Thus, given a batch of N image pairs (X _i5 Y _; ):

[Math.l]

MSE = IIF (Xi) -YiII ²

With F (X _; ) which represents the image corrected by the neural network of the image X _; .

The values of the weights of the layers are learned by a stochastic gradient descent algorithm based on the backpropagation of the gradient: the parameters which minimize this loss function are gradually calculated for each layer, starting from the end of the network.

This algorithm is advantageously the ADAM algorithm (Kingma and Ba, "A method for stochastic optimization", 3rd International Conference for Learning Representations, San Diego, 2015), used with a learning rate equal to 10-4.

As a variant, the filters of deconvolutional layers are not obtained by learning and are constituted by the transposed filters of the convolutional layers being symmetrical to them.

This first neural network Ri can also have a different architecture, for example being devoid of deconvolutionary layers.

From the captured image I _c presented to it during step 200, this first neural network Ri will thus generate a twin image Ij (see FIG. 5), this image corresponding to an image taken by a second camera, called twin camera, this camera being used only to constitute the learning base and absent from the imaging device 1 (step 300).

The captured image I _c and the twin image Ij are then presented to a classifier algorithm (step 400) consisting of a second artificial neural network R ₂ whose architecture is different from that described above of the first network neuronal artificial Ri. The architecture of the second neural network is that of a standard convolution deep network, and does not include deconvolutionary layers. FIG. 7 illustrates an exemplary embodiment of the architecture of the second neural network, which comprises convolutional layers C, the last of these layers being integrally connected to a set of integrally connected layers F (generally known by the English designation connected layers ”).

This second neural network R ₂ will perform the classification of both the captured image and the twin image.

This second neural network is for this purpose previously trained from a second learning base also comprising several tens of thousands of images (its size being advantageously identical to that of the first learning base).

The method includes a step of evaluating (600) the robustness of the classification, which consists in comparing the respective classifications of the captured image and of the twin image. When these are identical, the classification will then be considered robust. Otherwise, different strategies can be implemented, such as retaining the previous classification evaluated as robust.

According to other alternative embodiments, the second classifier neural network has an architecture identical to that of the first corrective neural network and projector.

Claims (1)

Claims [Claim 1] Method for recognizing an object, such as a traffic sign, by means of an on-board camera (10) in a motor vehicle, the on-board camera (10) being of monocular type, the method comprising the following steps: - capture (100) of an image, hereinafter image captured (I _c ) by said on-board camera (10); presentation (200) of the captured image (I _c ) to a first artificial neural network (Ri) forming a projector neural network, previously trained on a first learning base formed by pairs of images each comprising a first image captured by said on-board camera (10) and a second image, hereinafter twin image (Ij), corresponding to an image as captured simultaneously with the first image by a second camera temporarily installed near this on-board camera so as to having a different angular orientation relative to the on-board camera, and oriented so that the images of the second camera have an overlap zone with the first image; - generation (300) by said first neural network (RJ of a twin image corresponding to the captured image (300); - presentation (400) of the captured image and of the twin image (Ij) corresponding to a second neural network (R ₂ ) forming a classifier neural network; and - classification (500) by the second neural network (R ₂ ) of the captured image (I _c ) and of the twin image (Ij); - evaluation ( 600) of the robustness of the classification of the captured image. [Claim 2] Method according to claim 1, characterized in that the classification of the captured image is evaluated as being robust if the classification of the captured image and of the twin image are identical. [Claim 3] Method according to the preceding claim, in which, if the classification is evaluated as non-robust, then the previous classification evaluated as robust is taken into account. [Claim 4] Method according to one of the preceding claims, characterized in that said first neural network (RJ comprises a plurality of convolutional layers (C) and an identical number of deconvolutionary layers (D) arranged symmetrically. [Claim 5] Method according to the preceding claim, characterized in that said [Claim 6] [Claim 7] [Claim 8] [Claim 9] [Claim 10] first neural network (RJ comprises direct connections (S) between two symmetrical convolutional (C) and deconvolutive (D) layers. one of the preceding claims, characterized in that said first neural network (RJ is devoid of subsampling layers. Method according to one of the preceding claims, characterized in that the said first neural network (RJ comprises a loss function of the mean square error type. Method according to one of the preceding claims, characterized in that said first learning base is formed by several tens of thousands of pairs of images. Method according to one of the preceding claims, characterized in that said second neural network (R ₂ ) is previously trained on a second learning base formed by images comprising objects to be recognized, including traffic signs. Method according to one of the preceding claims, characterized in that said second neural network (R ₂ ) has a standard convolution deep network architecture, different from that of said first neural network, in particular in that it does not include layers deconvolutionary.