FR3085081A1 - METHOD FOR REAL-TIME CORRECTION OF DAMAGED IMAGES FROM AN ON-BOARD CAMERA WITH DIRT - Google Patents

Abstract

Method for the correction in real time of altered images coming from an on-board camera presenting soiling, successively comprising the steps of capturing an altered image (la) by said on-board camera (100), presentation of said altered image (la) to an artificial neural network previously trained from a learning base formed by pairs of images each comprising a first altered image captured by said on-board camera having dirt and a second clean image captured simultaneously by a second camera devoid of soiling and temporarily located near this on-board camera (200), generation by said neural network of an intermediate corrected image (Ii) whose field of vision corresponds to that of said second camera (300), presentation of said intermediate corrected image (Ii) to a projector algorithm (400), and generation by the latter of a cor image final rigée (If) whose field of vision corresponds to that of said on-board camera (500).

Description

ALTERED IMAGES FROM A CAMERA

lllllllllllllllllllllllllllllllllllllllllllll

Title of invention

Method for correcting altered images in real time from an on-board camera with soiling

Field of the invention

The present invention relates to the field of object detection in the environment of a motor vehicle. It relates in particular to a process for the correction in real time of altered images originating from an on-board camera presenting soiling.

Invention background

Motor vehicles tend to be increasingly equipped with sophisticated driver assistance systems, commonly referred to by the English acronym ADAS for "Advanced Driver Assistant Systems", and intended to improve driving comfort and / or passenger safety.

These driver assistance systems use imaging devices fitted with cameras capable of visualizing the road environment of the vehicle and the external face of the optical system of which is particularly exposed to bad weather and projections (mud, dust, sand , earth, etc ...), or to condensation phenomena which can greatly degrade the quality of the images formed through or by this optical system, to the detriment of the driver's comfort and safety.

There are known such imaging devices comprising an analysis module designed to detect dirt on the external face of the optical system of the camera.

The international application WO2017021377A1 thus discloses an imaging device comprising:

a camera comprising an image sensor as well as an optical system optically associated with the image sensor and comprising a controllable focal lens, said optical system being adapted to form on the image sensor an image of an environment exterior viewed by the camera through an external face of the optical system;

a control module designed to control the liquid lens successively at two distinct focal values, in order to obtain a first clear image of an area of the external environment distant from the external face of the optical system, then a second sharp image of said external face; and

- an analysis module designed to detect an element altering an optical path at the external face of the optical system, by analysis of said second image.

However, this detection system cannot be implemented with monocular cameras.

Also known, for example from Japanese application JP 2001-171491, are devices for washing the external face of the optical system of a camera on board a vehicle.

Such a washing device is equipped with a high-pressure air generation module making it possible to spray high-pressure washing water on the external face of the optical system to clean it and remove dirt therefrom.

However, the use of such a device tends to leave water droplets on the external face of the optical system which also degrades the quality of the images formed through or by this optical system until the water washing spray is completely evaporated and dried.

There are also known, in particular from application FR 2 927 448 A1, image processing methods making it possible to correct the effects of camera shake of the images formed through or by the optical system of a camera.

However, this type of method is not satisfactory for the processing of images altered due to soiling present on the external face of the optical system of the camera.

Subject and summary of the invention

The present invention aims to improve the situation.

To this end, it proposes a method for the correction in real time of altered images coming from an on-board camera presenting soiling, characterized in that it comprises the following steps:

- capture of an image altered by said on-board camera;

presentation of said altered image to a corrective algorithm and projector formed by a first artificial neural network, previously trained from a first learning base formed by pairs of images each comprising a first altered image captured by said on-board camera having dirt and a second clean image captured simultaneously by a second camera devoid of dirt and temporarily installed near this on-board camera so as to present a field of vision which is almost entirely superimposed on that of the latter;

- Generation by said neural network of an intermediate corrected image whose field of vision corresponds to that of said second camera;

- presentation of said intermediate corrected image to a projector algorithm; and

- Generation by said projector algorithm of a final corrected image whose field of vision corresponds to that of said on-board camera.

The method according to the invention makes it possible to obtain a final corrected image very close to that which the on-board camera would have taken in the absence of dirt.

This realism in correction is essentially due to the fact that the altered images of the learning base of said first neural network are constituted by images actually captured by a camera and not by images whose alteration is artificially generated by an algorithm.

According to preferred characteristics of the process according to the invention, taken alone or in combination:

- Said first neural network comprises a plurality of convolutional layers and an identical number of deconvolutionary layers arranged symmetrically;

- Said first neural network includes direct connections between two symmetrical convolutional and deconvolutionary layers;

- Said first neural network is devoid of subsampling layers;

- Said first neural network includes a loss function of the mean square error type;

- Said first learning base is formed of several tens of thousands of said pairs of images;

- Said pairs of images forming said first learning base are pretreated by rectification so that their epipolar lines are horizontal;

said projector algorithm is constituted by a second artificial neural network, previously trained from a second learning base formed by pairs of images each comprising a first clean image captured by said on-board camera devoid of dirt and a second image clean captured simultaneously by said second camera devoid of dirt;

- said second neural network has an architecture identical to that of said first neural network; and or

- Said method comprises, after said step of generating a final corrected image, a step of evaluating the level of soiling of said on-board camera by comparison between this final corrected image and said altered image.

Brief description of the drawings

The description of the invention will now be continued with the detailed description of an exemplary embodiment, given below by way of illustration but not limitation, with reference to the appended drawings, in which:

- Figure 1 is a functional diagram of an imaging device 1 intended to be integrated into an advanced driving assistance system (ADAS) of a motor vehicle;

- Figure 2 shows a flowchart of the method according to the invention for real-time correction of altered images from an on-board camera with dirt;

- Figures 3 to 5 respectively illustrate the altered image captured by the on-board camera, the intermediate corrected image and the final corrected image generated by the method according to the invention; and

- Figure 6 shows the architecture of the first artificial neural network.

detailed description

FIG. 1 illustrates an imaging device 1 intended to be integrated into an advanced driving assistance system (ADAS) of a motor vehicle such as an adaptive cruise control.

The imaging device 1 comprises a monocular optical camera 10 on board the vehicle and being provided with a two-dimensional image sensor 11, for example of the CCD type (acronym in English of "Charge-Coupled Device" translating into French by "charge transfer device") or CMOS (acronym in English of "Complementary Metal-Oxide Semiconductor" translating into French by "complementary metal oxide semiconductor").

The camera 10 also includes an optical system 12 associated with the image sensor 11 and being adapted to form on the latter an image of an external environment of the vehicle viewed by the camera through an external face of this optical system.

The imaging device 1 also includes a processing module 20 connected to the on-board camera 10 (or integrated into the latter) and capable of correcting in real time the images captured by the latter when the external face of its optical system exhibits fogging or soiling.

The processing module 20 comprises a computer 21 as well as a storage module 22 comprising non-volatile memory of the EEPROM or FLASH type and random access memory.

The non-volatile memory stores a process for processing in real time the images captured by the on-board camera 10 and the flowchart of which is shown in FIG. 2.

According to a preferred embodiment, all of the information contained in this non-volatile memory can be updated by means of communication or means of reading a data medium.

We will now describe in detail and in support of the flowchart of Figure 2, the different stages of this process.

This is initiated with each new capture, by the image sensor 11 of the on-board camera 10, of an altered image 1a (see FIG. 3) due to the presence of soiling (water droplets, projections, fogging ) on the external face of the optical system 12 (step 100).

This altered image 1a is then presented to a corrective and projector algorithm (step 200) constituted by a first artificial neural network RN (see FIG. 6) whose architecture described below is inspired by those described in the articles “Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections "(Mao et al., 30th Conference on Neural Information Processing Systems, Barcelona, Spain, 2016) and" Unsupervised Monocular Depth Estimation with Left-Right Consistency "(Godard et al., Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition CVPR, 2017).

This first neural network RN comprises a plurality of convolutional layers C (advantageously between 10 and 15) and an identical number of deconvolutionary layers D arranged symmetrically.

The convolutional layers have learned to extract the characteristics of the altered image which lend themselves best to the reconstruction of a sharp image (see learning procedure below).

The deconvolutionary layers have learned to carry out this reconstruction, in order to obtain a clear image of the scene "projected" on the camera 2.

The two correction / projection operations are encoded in the weights of the connections of the whole network, a priori one cannot point to a layer which would carry out the projection more than another, although the signal approaches the projection sought in the final layers of deconvolution.

Each convolutional or deconvolutionary layer is advantageously composed of 64 filters of size 3 × 3 generating 64 characteristic maps (commonly known under the term “features maps”) from the image entering this layer.

As a variant, the number of filters constituting each layer is different (for example equal to 32 or 128). The size of these filters can also be different (for example equal to 5X5, 7X7 or 9X9).

To avoid the loss of error signal inherent in the training of such deep neural networks, direct connections S (commonly called under the Anglo-Saxon term "skip connections") are provided between two convolutional and deconvolutionary layers.

These direct connections advantageously have a pitch equal to three, that is to say that they extend all the three convolutional layers towards the corresponding symmetrical deconvolutionary layers.

They are also bijective for each filter: each unit in the convolutional input channel filter being connected to its equivalent in the deconvolutive output filter (one-to-one connectivity).

The information passing through these direct connections thus contains more details which helps the deconvolutional layers to generate a more detailed clean version.

Each of the convolutional or deconvolutionary layers is followed by an activation function advantageously of ReLU type (acronym in English of “Rectified Linear Unit” translating into French by “Unit Linear Rectified”).

This function, also called "non-saturation activation function", increases the non-linear properties of the decision function and of the whole network without affecting the receptive fields of the convolutional or deconvolutionary layer.

This first correcting neural network and projector RN is also advantageously devoid of subsampling layers (commonly called by the English term "pooling") because the latter tend to eliminate certain details present in the initial image.

The input and output images have the same resolution format (I xhxc), with I, h and c representing respectively the width, the height and the number of channels of these images (c being equal to 1 for an image monochrome and 3 for a color image coded in RGB).

This resolution format will advantageously be 800 x 480 x 3 for reasons of practicality (usual format of optical cameras installed in vehicles) and speed of calculations.

This first correcting neural network and projector RN is furthermore previously trained from a first learning base formed by several tens of thousands of pairs of images (advantageously at least 40,000 and preferably 60,000).

Each pair of images comprises a first altered image captured by the on-board camera 10, the external face of the optical system 12 of which is dirty, and a second clean image captured simultaneously by a second camera identical to the camera 10, the external face of the system. optical is free of dirt and being temporarily located near this camera 10 so as to present a field of vision which is almost entirely superimposed on that of the latter.

Prior to image acquisition, these differences in soiling between the external faces of the optical systems of the two cameras can be obtained by simple rolling with cameras deactivated, by obstructing the external face of the optical system of the second twin camera but not that of the camera 10.

During the image acquisition phase, taxiing resumes with the two active cameras.

The image pairs from the learning base are preprocessed by rectification so that their epipolar lines are horizontal.

As the performance of the neural network increases with the volume and variety of the learning base, an arbitration is therefore required between the cost linked to the acquisition of images during the rolling phase and the gain linked to the precision of the two networks. .

However, since no labeling of these images is necessary for this learning base (the images obtained by each camera being used for cross-supervision), its cost price is reduced because it consists solely of the running costs and image storage.

Furthermore, new pairs of images are advantageously created by mirror transformation and inversion from the pairs of images generated during the acquisition phase.

This so-called augmentation technique makes it possible to double the number of pairs of images of the training base (and therefore to improve the performance of the neural network) without requiring additional running.

This first correcting neural network and projector RN further comprises a loss function of the quadratic mean error type (commonly known under the English acronym MSE for “Mean Squarred Error”) and making it possible to calculate the classification error on batches originating from the learning base and comprising for example between 5 and 100 pairs of images.

Thus, given a lot of N image pairs (Xi, Yi):

MSE = Σ ^ _{= 1} || F (Xi) - Yill ² with F (Xj) which represents the image corrected by the neural network of the image Xi.

The values of the weights of the layers are learned by a stochastic gradient descent algorithm based on the backpropagation of the gradient: the parameters which minimize this loss function are gradually calculated for each layer, starting from the end of the network.

This algorithm is advantageously the ADAM algorithm (Kingma and Ba, "A method for stochastic optimization", 3rd International Conference for Learning Representations, San Diego, 2015), used with a learning rate equal to 10 ^-4 .

As a variant, the filters of deconvolutive layers are not obtained by learning and are constituted by the transposed filters of the convolutional layers being symmetrical to them.

This first neural network can also have a different architecture, for example being devoid of deconvolutionary layers.

From the corrupted image presented to it during step 200, this first neural network RN will thus generate an intermediate corrected image li (see FIG. 4) whose field of vision corresponds to that of the second camera used only. to constitute the learning base and absent from the imaging device 1 (step 300).

This intermediate corrected image li is then presented to a projector algorithm (step 400) constituted by a second artificial neural network whose architecture is identical to that described above of the first artificial neural network.

This second neural network is previously trained from a second learning base also comprising several tens of thousands of pairs of images (its size being advantageously identical to that of the first learning base).

Each pair of images is formed by a first own image captured by the second camera whose external face of the optical system is free of dirt, and by a second own image captured simultaneously by the camera 10 whose external face of the optical system is also free of dirt.

From the intermediate corrected image presented to it during step 400, this second neural network will thus generate a final corrected image If (see FIG. 5) whose field of vision corresponds to that of the on-board camera 10 ( step 500).

This final corrected image If thus approximates the image that the on-board camera 10 would have captured if the external face of its optical system were not altered.

The comparison of the two images can use a simple mean square error as a measure, a PSNR measure (acronym in English of “Peak Signal to Noise Ratio” translating into French as “signal-to-peak noise ratio”) involving the logarithm of the inverse of the quadratic error, or a structural similarity measure calculated on the basis of the moments (mean and variance) of this image.

Advantageously, the method also includes a final step of evaluating the level of soiling of said on-board camera by comparison between this final corrected image If and the altered image 1a (step 600).

In order to avoid unnecessary calculations, it can also be provided that steps 200 to 600 are only performed, for the following n images captured by the sensor 11 of the on-board camera 10, only if this level of soiling is greater than or equal to one. predetermined threshold value (the value of n corresponding for example to the number of images per second captured by this sensor 11 so that these steps 200 to 500 are at least performed every second).

According to other alternative embodiments, the second projector neural network has an architecture different from that of the first correcting neural network and projector.

According to yet other alternative embodiments, the projector algorithm is not constituted by a neural network. This algorithm can for example consist of an untrained geometric transformation similar to those used in localization and simultaneous mapping applications.

This untrained and pre-calculated geometric transformation can consist of the fundamental matrix between the two images la and, 5 making it possible to calculate for each pixel of image 2 an epipolar line on image 1, along which we will then search the best matched pixel. However, this research requires a measure of similarity between pixels, which is laborious to define and idiosyncratic. "

In general, it is recalled that the present invention is not limited to the embodiments described and shown, but that it encompasses any variant of execution within the reach of the skilled person.

Claims (10)

1. A method of correcting altered images in real time from an on-board camera (10) presenting soiling, characterized in that it comprises the following steps: - Capture of an altered image (la) by said on-board camera (10) (100); - presentation of said altered image (la) to a corrective algorithm and projector formed by a first artificial neural network (RN), previously trained from a first learning base formed by pairs of images each comprising a first image altered captured by said on-board camera (10) having dirt and a second clean image captured simultaneously by a second camera devoid of soiling and temporarily installed near this on-board camera so as to present a field of vision which is almost entirely superimposed on that of this last (200); - Generation by said first neural network (RN) of an intermediate corrected image (li) whose field of vision corresponds to that of said second camera (300); - presentation of said intermediate corrected image (li) to a projector algorithm (400); and - Generation by said projector algorithm of a final corrected image (If) whose field of vision corresponds to that of said on-board camera (10) (500). 2. Method according to claim 1, characterized in that said first neural network (RN) comprises a plurality of convolutional layers (C) and an identical number of deconvolutionary layers (D) arranged symmetrically. 3. Method according to claim 2, characterized in that said first neural network comprises direct connections (S) between two convolutional (C) and deconvolutionary (D) layers. 4. Method according to one of claims 1 to 3, characterized in that said first neural network (RN) is devoid of subsampling layers. 5. Method according to one of claims 1 to 4, characterized in that said first neural network (RN) comprises a loss function of the quadratic mean error type. 6. Method according to one of claims 1 to 5, characterized in that said first learning base is formed of several tens of thousands of said pairs of images. 7. Method according to one of claims 1 to 6, characterized in that said pairs of images forming said first learning base are pretreated by rectification so that their epipolar lines are horizontal. 8. Method according to one of claims 1 to 7, characterized in that said projector algorithm is constituted by a second artificial neural network, previously trained from a second learning base formed by pairs of images each comprising a first clean image captured by said on-board camera (10) free of dirt and a second clean image captured simultaneously by said second camera free of dirt. 9. Method according to claim 8, characterized in that said second neural network has an architecture identical to that of said first neural network (RN). 10. Method according to one of claims 1 to 9, characterized in that it comprises, after said step (500) of generating a final corrected image (If), a step of evaluating the level of soiling of said on-board camera (10) by comparison between this final corrected image (If) and said altered image (la) (600).