EP3717877A2

EP3717877A2 - Method for characterizing samples using neural networks

Info

Publication number: EP3717877A2
Application number: EP18814839.9A
Authority: EP
Inventors: Valeriu Vrabie; Eric Perrin; Sihem MEZGHANI
Original assignee: Critt Mdts; Universite de Reims Champagne Ardenne URCA
Current assignee: Critt Mdts; Universite de Reims Champagne Ardenne URCA
Priority date: 2017-12-01
Filing date: 2018-11-30
Publication date: 2020-10-07
Also published as: JP2021504864A; FR3074596B1; CN111630356A; CA3083079A1; WO2019106179A3; US20200333185A1; US11828652B2; WO2019106179A2; CN111630356B; FR3074596A1

Abstract

The present invention relates to a method for characterizing a sample using a set of spectral images of the sample to be characterized, which have been previously acquired, in particular by infrared thermography or spectral imaging, and to at least one neural network, the method comprising the steps of: - generating at least one volume of values of a parameter observed from said spectral images for a plurality of coordinates of the pixels of the images and a plurality of acquisitions, - extracting at least one set of input data from said data volume, said input data corresponding to the values of the observed parameter for a pixel of the same coordinates according to different acquisitions, wherein at least one transformation function has been applied to said values, - training said at least one neural network using the input data to extract at least one characteristic of the sample to be characterized.

Description

SAMPLE CHARACTERIZATION METHOD USING SAME

NEURON NETWORKS

The present invention relates to methods and devices for characterizing samples from spectral images, in particular acquired by infrared thermography, and using deep neural networks.

Most known surveillance systems, such as for accident prevention, traffic routing or decision making, for example for the detection and / or nondestructive testing of components and / or various phenomena, are based on the use of many sensors and the use of known detection techniques. Infrared, near-infrared (NIR) and ultraviolet (UV) radiation can be used. In particular, far-infrared electromagnetic radiation, also called radiative heat, emitted continuously by any body having a temperature above absolute zero (-273, 15 ° C) is used. Specific detectors allow this radiation to be captured in certain wavelengths and retranscribed in luminance values related to the surface temperature of the object, creating thermal images.

The miniaturization of infrared thermographic cameras, the reduction of their cost of acquisition and the development of computation capabilities of computers have encouraged the use of such cameras as a non-destructive alternative technique in several applications, such as evaluation of damage, fatigue of materials, or thickness estimation of coatings. The possibility of penetrating the coating layer without having any influence on the pigments justifies the use of infrared techniques for the inspection of coating thicknesses such as paint.

This technique has certain limitations, such as high sensitivity to external reflection, emissivity variations, and the use of a heat source as an excitatory source that can not be considered energy-efficient for example because of the use of one or more high-power flashes. This inhomogeneity will directly affect the thermal signature of the target coating observed during both the heating and cooling periods. In order to solve this problem, the temperature distribution during a Thermographic inspection has been investigated and measures to reduce the effects of nonuniform temperature distribution have been suggested, such as the use of an image reconstruction algorithm based on a Fourier transform to inhibit the effect of non-uniform heating. Other methods are used to improve the thermal contrast and overcome these external artifacts, including the use of thermal contrast, absolute thermal contrast, or modified absolute differential contrast. It is also known to use an algorithm based on partial least squares regression to automatically improve the visibility of defects in samples by partially eliminating background noise. Other methods based on the use of higher order statistics or singular value decomposition have been developed. However, these denoising results are not yet optimal.

A perceptron multilayer neural network was used to detect and characterize defects using pulsed infrared thermography. The results show that phase images are less sensitive to noise but an increase in sampling frequency is strongly recommended for this study. The illustration of such results can be found for example in the article "Defect detection in pulsed thermography: a comparison of Kohonen and Perceptron neural networks", by Steve Vallerand et al. Proc. SPIE 3700, Thermosense XXI, March 1999.

It has recently been demonstrated that it is possible to use deep learning algorithms, in English, to classify data, for example images, sounds, or texts, by extracting characteristics to represent data at different levels of abstraction. The use of deep learning makes it possible to obtain robust results and to consider various applications. These algorithms implement models consisting of several supervised layers or not, where the non-linear stages of information processing are hierarchical in nature. The convolutional neuron networks (CNN) are, in known manner, composed of several layers, each layer acting as a filter and resulting in a dimensionality reduction of the data which are then transmitted to the next layer. The layers are composed of neurons, themselves composed of an activation function, a weight and a bias on each of its inputs. As represented in FIG. 1 illustrating the state of the art, in such structures, input data DE are received at sensors S1, S2, S3, each sensor being able to collect the signals representative of a characteristic F (or "feature" in English) of the sample to be analyzed. These signals are transmitted to a neuron network M consisting of multiple layers. This model M is driven by one or more sets of DApp training data, including images, at which F characteristics have been annotated and which are subject to a learning algorithm AApp allowing learning the recognition of said characteristics. An output C is obtained, according to a set of classes C1, C2, C3, data input DE.

These algorithms (especially convolutional neural networks) rely on the learning of many data models, so they are very computationally demanding and require an important database for their learning. The implementation of these algorithms on embedded systems is limited by the computing capacity and by the resources, in particular memory, available. Recently developed systems, for example using a nano-computer of the Raspberry Pi type, are currently capable of analyzing between 2 and 4 frames per second from a video stream. It's still not fast enough to handle video streams satisfactorily.

Application No. CN 6022365 discloses a method of detecting defects on the surface of a material, using a radial basis function (RBF) neuron network and infrared thermography images to create a classifier.

Application CN 10 5760883 describes, in the field of the monitoring of the operating status of a mining equipment, a method for automatically identifying the key components of a conveyor belt, using infrared thermography and a so-called neural network. BP ("back-propagation" in English) to extract the characteristics of the components.

Application CN 10 2621150 relates to a method for identifying damage on an aircraft liner, using a Large Support Vector Machine (SVM) algorithm based on a matrix of gray-scale cooccurrences. a signal characterizing different types of damage to establish a classifier. Fa detection of damage on the coating of the aircraft and damage themselves can be classified and identified, for subsequent maintenance treatment.

Other approaches for performing learning tasks based in particular on software systems and specific DSP ("digital signal processor" in English), or using FPGA ("Programmable Field Gable Array" in English) as presented by example in the article "Optimizing Convolutional Neural Network on DSP" by S. Jagannathan et al, IEEE International Conference on Consumer Electronics 2016, can achieve interesting performance, about three times faster than systems based GPU graphics processors ("Graphie processor unit" in English).

It is known to use pre-trained CNN networks by modifying only their final layers, which makes it possible to propose hybrid networks consisting of a pre-trained CNN network, in particular on a machine with a high computing power, and a supervised classifier to train, for example an SVM algorithm. This makes it possible to reduce the necessary learning base and to envisage its implementation on a low-cost embedded system, for example on a nano-computer of the Raspberry Pi type. The article by Huang FJ et al, "Large-scale learning" "IEEE CVPR 2006, vol 2, pp 4, describes a hybrid network consisting of a convolutional network and an SVM, dedicated to the classification of objects.

International application WO 99/05487 discloses the use of an optical fiber probe and a hybrid neural network to increase the accuracy of tissue lesion analysis.

There remains therefore a need to further improve the estimation reliably of certain sample characteristics from spectral images obtained using a device whose excitation conditions are variable and non-uniform, in particular via a characterization device by infrared thermography.

The invention responds to the need mentioned above by, in one of its aspects, a method of characterizing a sample, using a set of spectral images of the sample to be characterized previously acquired, in particular by infrared thermography. or spectral imaging, and at least one neural network, the method comprising the steps of: generating at least one volume of values D (Nx, Ny, Ne) of a parameter observed from said spectral images, for a plurality of coordinates (x, y) of the pixels N of the images and a plurality of acquisitions Ne,

extracting at least one set of input data D'x, y (Ne) from said data volume D (Nx, Ny, Ne), these input data corresponding to the values of the observed parameter, for a pixel of same coordinates according to different acquisitions Ne, values to which at least one transformation function has been applied,

driving said at least one neural network, in particular at least one layer of the network, by using the input data to extract at least one characteristic from the sample to be characterized, and

using said at least one characteristic extracted by the neural network to classify the input data according to a plurality of classes, each class being representative of at least one characteristic of the sample to be characterized.

Preferably, said at least one neural network, in particular at least one layer of the network, has been previously driven by means of images other than real spectral images (that is to say, derived from real samples). . According to a first variant, said at least one neural network can be trained in advance using images called "natural images", in particular natural images of animals, objects, plants, people. According to a second variant, said at least one neural network can be trained in advance using images called "virtual images", that is to say artificially created images by humans, such as non-limiting examples:

• "virtual spectral images" resulting from simulations / computer creation of virtual sample models,

• "virtual natural images" (images of animals, objects, plants, people) resulting from an artificial creation by Man (computer-retouched photos, photos created at least partially by IA ...) .

According to a third variant, said at least one neural network can be driven in a mixed manner, using natural images and virtual images.

Even more preferentially, only the final layer of the network needs to be driven with the input data. Preferably, the classification is performed by a classifier independent of the neural network. The classifier can be of the wide margin separator (SVM) type. In one variant, the classifier is of the Softamx or RBF type with a Gaussian nucleus. The classification can be done at the level of at least one layer of a perceptron. In one variant, the classification is carried out by the neural network used for the extraction of the characteristics, preferably by the final layer of this network.

Thanks to the use of a hybrid structure comprising a pre-trained network and a classifier, the invention allows a rapid implementation of the system, the database and the learning time being reduced. The implementation of the method according to the invention can thus be done on a low-cost embedded system, for example a nano-computer, also called nano-PC, or a dedicated card, allowing the operation of resource-intensive applications and / or requiring real-time.

The use of an SVM classifier makes it possible to work with large data, making it possible to process many types of data, significantly improve recognition performance, and significantly reduce computation time. .

The robustness of the hybrid architecture according to the invention makes it possible to have a post-processing technique of infrared thermographic data that is slightly sensitive to the non-uniformity of the energy deposition generated by the excitation system and to the measurement conditions. for example, different placements of the spectral image acquisition camera in terms of distances or angles of the lens relative to the sample, or lighting conditions for different acquisitions, depending on the time, the temperature, or the season of the year.

One-dimensional signals

The different acquisitions may correspond to different acquisition times over a predefined acquisition period, particularly in the case of infrared thermography.

In a variant, particularly in the case of spectral imaging, the different acquisitions correspond to acquisitions at different wavelengths, performed at the same time.

Spectral imaging includes multispectral or hyperspectral imagery. Multispectral imaging consists of acquiring a small and limited number of bands discrete, and does not require the use of a spectrometer to analyze the data. Hyperspectral imaging allows the acquisition of a large number of narrow spectral bands through the use of a spectral band separation system such as a spectrometer.

The data volume D advantageously contains P pixels, for each pixel N in the plane x, y correspond to the coordinates (Nx, Ny, Ne), where Ne is the coordinate of the acquisition. This is the same pixel P from one plane to another, recorded at different times or for different wavelengths.

The spectral evolution of a pixel, for example the temporal evolution of temperature, is thus considered as a one-dimensional signal, forming a tuple of values, and used directly for classification.

The input data is advantageously transmitted to the neural network in the form of images representing curves corresponding to the values D'x, y (Ne) of the input data set as a function of the acquisition Ne. This allows the transposition to the one-dimensional signals of deep-learning-type network principles for the study of natural images, including convolution and dimensionality reduction for feature extraction. The fact that the neural network is pre-trained on natural images reduces the number of necessary learning data, while the fact that the neural network is pre-trained on "virtual images" makes it possible to precisely refine the image. desired learning.

For learning, the images can be resized according to the standard dimensions imposed by the neural network used.

The transformation function applied to the values Dx, y (Ne) of the observed parameter can be the identity function, the values remaining unchanged and being used as such by the neural network.

In a variant, said at least one transformation function applied to the values of the observed parameter Dx, y (Ne) is a centering, normalization and / or smoothing function. The spectral responses Dx, y (Ne) can thus each be centered, for example with respect to an average value calculated on all the images having served for the learning of said network, or with respect to the first image resulting from the first acquisition, and / or normalized with respect to their maximum, or with respect to a reference value, corresponding in particular to a predefined wavelength. The smoothing method used may be the so-called "Savitzky and Golay" (SG) method, which consists of approximating on a sliding window of size nor the spectral response segment using a polynomial of degree n, with m inclusive. between 10 and 20 points and n between 1 and 6, for example m = 15 points and n = 4. The smoothing makes it possible to reduce the irregularities and singularities of the responses. Using smoothed spectral responses to compute derivatives avoids artefacts or amplification of derivation noise in the resulting signals.

A calculation function of the first derivative can be applied to the D (Nx, Ny, Ne) values of the observed parameter to obtain the input data (D'x, y (Ne)). In the case where the parameter observed is the temperature of the sample, this calculation makes it possible to take into account the rate of cooling of the sample.

A calculation function of the second derivative can be applied to the values D (Nx, Ny, Ne) of the observed parameter to obtain the input data (D'x, y (Ne)). In the case where the observed parameter is the temperature of the sample, this calculation makes it possible to take into account the acceleration of the cooling of the sample.

Infrared thermography

Preferably, the spectral images used are thermal images acquired by infrared thermography, the parameter observed being the temperature of the sample.

The principle of infrared thermography is based on the measurement of the energy emitted by the surface of a body in a given wavelength interval, corresponding to the temporal acquisition of the thermal radiation in the infrared bands of the electromagnetic spectrum. This energy is transmitted through appropriate optics to a detector. These so-called radiometric systems allow non-contact measurement of surface temperature fields at rates of up to several hundred Hertz for images whose average size is approximately 80000 pixels. The measurement process generally causes only a slight increase in temperature, which does not risk disturbing data acquisition. In a known manner, the detectable temperature variations range from a few tenths to a few tens of degrees Celsius / Kelvin. Thermal excitation of the specimen can be achieved. The numerical analysis of the thermal data is then carried out. The surface of the sample to be characterized may be thermally excited prior to the acquisition of the thermal images, for example by means of a surface excitation device by pulsed illumination such as a flash lamp.

One-dimensional signals and virtual images

In certain variants of the invention, the neural network is driven from virtual images, and preferably via virtual spectral images derived from simulations / computer creation of mathematical models of virtual samples.

In the variant of the invention in which the spectral images used are thermal images acquired by infrared thermography, the parameter observed is the temperature of the sample.

In this case, a virtual sample, computer simulated, may for example be in the form of a bilayer of materials, namely a substrate covered with a coating. However, the skilled person knows how to choose the number of layers of materials (substrate and / or coating) adapted to the intended application.

Said coating can be numerically discretized into voxels, for each of which a curve LT representing the evolution of the temperature over time t can be defined (for example: selected in a database, etc.) or calculated from parameters provided by those skilled in the art to a mathematical model.

A voxel includes all of the layer (s) provided for said coating deposited on the substrate.

This voxel discretization of the coating is particularly advantageous because it is possible to generate as many voxels as desired, which makes it possible to simulate a wide variety of energy distributions in said coating.

Thus, in its preferred variant, said mathematical model implements inter alia a generalized equation EQM having the form below:

T (x, y, t) = F [QE (x, y, t); PS; PR; ER; t; ...] (EQM) for which the function F depends inter alia on at least:

• QE: the amount of energy deposited in a given voxel with plane coordinates (x, y). This value makes it possible to simulate the distribution of the energy deposited in the coating by an idealized thermal excitation means (for example: a flash lamp). Advantageously, it is a pulse response of the "Dirac" type, which will be called LRID _QE . However, the use of other idealized impulse responses may be considered.

PS: a set of physical parameters characterizing the substrate (for example: composition of the material, color of the material, appearance of the material (smooth, matte, etc.), thermal diffusivity, thermal effusivity, etc.) obtained for example via databases communicating with said mathematical model and / or calculated within said model. This set of PS parameters may include PS parameters characterizing a single layer and / or globally characterizing the set of layers constituting said substrate.

• PR: a set of physical parameters characterizing the coating layer (s) deposited on the substrate (for example: composition of the material, color of the material, appearance of the material (smooth, matte, etc.), thermal diffusivity , thermal effusivity, ...). This set of PR parameters may include PR parameters characterizing a single layer and / or generally characterizing the set of layers of said coating. These parameters are obtained for example via databases communicating with said mathematical model and / or calculated within said model,

• ER: a set of parameters characterizing the thickness (s) of the coating. This set of parameters ER can include ER parameters characterizing a single layer and / or generally characterizing at least a subset of layers of said coating, each layer having a specific thickness (identical or different from the other layers).

Of course, the parameters provided to the mathematical model can be of any type: constant or variable, respecting various distributions (uniform, Gaussian, ...). This makes it possible to simulate for example: a heterogeneity of the coloring pigment, a heterogeneity of the surface of the coating, ....

The mathematical model output the ISV _X virtual spectral images that allow, in certain embodiments of the invention, to drive the neural network.

Preferably, the virtual spectral images ISV _X result from the convolution of the curve LT (response resulting from the equation EQM of the model) with a curve of the "impulse response" type LRI, such as for example a "gate function P". This impulse response corresponds to the action of an excitation means real thermal. Of course, the use of other realistic impulse responses, replacing the gate function P can be envisaged.

Furthermore, in advanced variants of said mathematical model, the environmental thermal noise b (x, y, t) can also be taken into account, once the convolution step has been performed, according to for example the equation: [T (x , y, t) * P] + b (x, y, t)

It should be noted that adding noise:

• makes it possible to be as close as possible to reality,

• can improve the classification.

Consequently, in the variants of the invention where the ISV _X virtual spectral images make it possible to drive the neural network, the mathematical model envisaged makes it possible to generate a large variety of thermal responses.

In particular, it is possible to generate a large variety (set) of thermal responses by varying at least the parameters QE, PS and PR for the same order of thickness (s) of layer (s) of coating ER, which allows to take into account the endogenous and exogenous variations for a given scheduling.

Thus, it is possible to:

Generate a set of thermal responses for the same ER coating thickness scheduling,

• group together several sets of responses, within the same class (for example: class 1 corresponds to coating thicknesses ranging from 51 pm to 60 pm, and so on), corresponding to:

o identical sequencing of thicknesses, the other parameters (EQ, PS, PR, ...) being different from one response to the other, and / or

different orders of different thicknesses of coating.

Neural network and feature extraction

The at least one neural network may be a convolutional neural network.

The neural network may include one or more convolutional layers and / or one or more fully connected layers.

In a known manner, each convolutional layer produces an activation of an input image, the first layers extracting basic characteristics, such as the outline, and the upper layers extracting higher level features, such as texture information.

Said at least one characteristic extracted from the input data may be the thickness or thickness range of the sample or portions of the sample, a representative amount of a property of the sample, by a layer of paint, a thickness of an intermediate layer, for example of the Sol Gel type originating from solution-gelling processes, the level of water stress of a plant, the variation of pigmentation of plants, for example leaves, flowers or fruits of plants.

Characterization device

According to another of its aspects, the subject of the invention is also a device for characterizing a sample, in particular by infrared thermography or spectral imaging, comprising:

means for acquiring a set of spectral images of the sample to be characterized, in particular a thermal camera,

a data processing module capable of:

Generating at least one volume of values (D (Nx, Ny, Ne)) of an observed parameter from said spectral images, for a plurality of coordinates (Nx, Ny) of the pixels of the images and a plurality of acquisitions ( Born),

Extracting at least one input data set (D'x, y (Ne)) from said data volume (D (Nx, Ny, Ne)), these input data corresponding to the values of the observed parameter, for a pixel of the same coordinates (x, y) according to different acquisitions (Ne), to which at least one transformation function has been applied, and

an analysis module, comprising at least one neuron network, capable of at least driving said at least one neuron network by using the input data to extract at least one characteristic from the sample to be characterized, and using said characteristic extracted by the neural network to classify the input data according to a plurality of classes, each class being representative of at least one characteristic of the sample to be characterized. Said at least one neural network has preferably been previously trained on images other than spectral images, including natural images of animals, objects, plants, people and / or "virtual images".

The device according to the invention may comprise a classifier independent of the neural network to perform the classification.

The device may comprise a thermal excitation means of the sample to be characterized, in particular a surface excitation device by illumination, preferably a surface excitation device by pulsed illumination such as a flash lamp.

The device may further comprise a decision-making module communicating with the analysis module, and a means of action able to act on the sample, said decision-making module being able to slave said action means retroactively. according to the classification results obtained from said analysis module and to trigger an appropriate action towards the sample. This allows for sample control and reliable tracking, for example in non-destructive testing applications.

The means of action may be a spraying nozzle of a phytosanitary product on crops, capable of spraying a quantity of product suitable for the classification results. The characteristics to be extracted can be the leaf mass, the type of vegetation, or even the type of disease of the plant examined.

The characteristics stated above for the method apply to the device and vice versa.

The characterization device according to the invention can be easily implemented on an embedded system either at low cost, for example a nano-PC for example of the Raspberry PI type, or specific, for example a TX1 type dedicated card of Nvidia®, according to the intended application. Most of the processing can take place in the intelligent embedded system, for example a nano-PC that can be equipped with a high-definition camera.

Control process

Another subject of the invention, according to another of its aspects, is a method for controlling a sample, comprising the step of generating with the device of characterization of a sample as defined above, according to the classification results, information relating to the sample with a view to making a decision to decide on an action to be taken towards the sample to be characterized, and in particular to transmit an instruction of action to a means of action able to implement it.

Computer program product

The invention further relates, in another of its aspects, to a computer program product for implementing the method for characterizing a sample as defined above, using a set of spectral images of the sample. to characterize previously acquired, in particular by infrared thermography or spectral imaging, and at least one neuron network,

the computer program product having a medium and recorded on that medium readable instructions by a processor for when executed:

at least one volume of values (D (Nx, Ny, Ne)) of a parameter observed from said spectral images is generated, for a plurality of coordinates (Nx, Ny) of the pixels of the images and a plurality of acquisitions (Born),

at least one set of input data (D'x, y (Ne)) is extracted from said data volume (D (Nx, Ny, Ne)), these input data corresponding to the values of the observed parameter for a pixel of the same coordinates (x, y) according to different acquisitions (Ne), to which at least one transformation function has been applied,

said at least one neural network is driven using the input data to extract at least one characteristic from the sample to be characterized, and

said at least one characteristic extracted by the neural network is used to classify the input data according to a plurality of classes, each class being representative of at least one characteristic of the sample to be characterized.

Description of figures

The invention will be better understood on reading the following description, non-limiting examples of implementation of the invention, and on examining the appended drawing, in which:

FIG. 1, previously described, illustrates a classification method using a deep learning algorithm according to the prior art, FIG. 2 represents an example of data preparation steps in the method according to the invention, with a view to providing them to at least one neuron network,

FIG. 3 illustrates data classification steps by a neural network in the method according to the invention,

FIG. 4 represents an exemplary device for characterizing a sample according to the invention, and

- Figures 5 to 10 are tables showing the classification performance of the method according to the invention.

FIG. 11 represents the steps of obtaining virtual spectral images for driving the neural network,

detailed description

FIG. 2 shows an example of data preparation steps in the method according to the invention, with a view to providing them to at least one neuron network. In this example, spectral images are acquired by a thermal camera, thus forming so-called thermal images. The parameter observed from these images is the temperature of the surface of the sample E.

From a sample E to be characterized, during a step A, a set of thermal images 2 is acquired over a predefined acquisition period T, using for example an infrared thermal camera.

During a step B, a data volume D (Nx, Ny, Ne) corresponding to the instantaneous temperature values is generated from the thermal images 2, in a frame where Nx and Ny correspond to the coordinates of the pixels N of the images 2 in the directions (x, y) and Ne corresponds to the acquisition, expressed either in number of the image or in time, or wavelength in the case of multispectral or hyperspectral imaging.

A set of one-dimensional input data D'x, y (Ne) is extracted from the data volume D (Nx, Ny, Ne), during a step EP. In the example considered, these input data D'x, y (Ne) correspond to the instantaneous temperature values, for a pixel of the same coordinates (x, y) according to different recording instants Ne, to which at least one function transformation is applied, detailed in the following.

In a step I, IJx images representing curves corresponding to the values D'x, y of the input data set as a function of the recording time Ne are generated for each data set, and are transmitted to a network of neurons R (CNN) convolution in the example. The neural network used in the method according to the invention can nevertheless be of any type.

Advantageously, the method according to the invention further comprises a data pretreatment step PT following the extraction step EP. Preferably, this data preparation step PT consists of applying at least one transformation function to the values Dx, y (Ne), in particular a centering, normalization and / or smoothing function. During a step NL, different functions can be applied to the input data set, for example the identity function forming the set J0 corresponding to the original data set, preprocessed or not, and / or a calculation function of the first derivative forming the game J1, and / or a calculation function of the second derivative forming the game J2.

In the example under consideration, in an application for evaluating the heterogeneity of a paint layer, the sample E to be inspected is a 370x500mm steel metal plate with a layer of paint coating deposited in 4 strips. whose thickness varies from 59 to 95 μm, as can be seen in Figure 2. This sample was placed horizontally against an insulating support to prevent conduction phenomena between the sample and the soil. A total surface thermal excitation of the sample can be performed in order to have a high rate of heating of the entire surface. Preferably and as in the example illustrated, the technique of pulsed infrared thermography is used: a thermal wave is sent to the surface of the sample whose excitation profile is as close as possible to a Dirac pulse. Halogen lamps can be used, but long-term illumination-induced temperature rise can damage the surface of the sample. Alternatively and preferably, several flash lamps generating a large amount of energy in a very short time, positioned at different angles, are used.

In the example considered, the thermal camera records every 5 milliseconds a thermal image, or thermogram, of the front face of the surface of the sample. Following the acquisition of these thermal images, a volume of data is generated as described above. The acquisition period is between ... 0.5 seconds and 2 seconds, being for example equal to 1 seconds. As previously described and as visible in FIG. 3, each pixel of the data volume recorded by the camera is a one-dimensional signal which is represented by an IJx image used as an input of an RNCNN neural network. This RNCNN neuron network has preferably been previously trained on images other than natural spectral images, for example from the "www.image.net" database containing more than 1000 image classes and more than a million images. This neural network may belong to an analysis model MP further comprising a perceptron P, comprising for example an input layer, one or more hidden layers and an output layer. This analysis model MP is initially able to classify images by the neuron network RN then the perceptron P according to classes K (K1, K2 ...).

As described above, the neural network is driven using the input data IJx to extract EF characteristics, corresponding to different thicknesses of paint in the example considered. In the case where the neural network belongs to an MP analysis model comprising a perceptron P, the extraction of the characteristics can be carried out by at least one of the layers of the perceptron.

The extracted characteristics are used to classify the thermal responses according to a plurality of classes C1, C2, each class being representative of a characteristic of the sample to be characterized, here different thicknesses. The classification is, in this example and preferably carried out by a classifier independent of the neural network, of the wide margin separator type (SVM).

In the example considered, at each of the paint coating thicknesses, visible in FIG. 2, is associated a class C1, C2, C3, C4 of different spatial dimensions. Acquisitions on this coating were made at two different times Tl and T2, creating two volumes of data, the first of dimensions 110x611x400, and the second of dimensions 103x631x400. To reconstitute identical measurement conditions between each measurement being difficult, ie to reproduce the same positioning of the samples, the lamps, and the camera, the data volumes do not have the same dimensions, except the temporal dimension since the acquisition period is always the same. As previously described, one-dimensional data sets J0, J1 and J2 have been generated from these volumes. Figure 5 shows the classification performance for the J0 dataset from the second volume, including a set of 8000 signals that were randomly selected (2000 for each class). 70% of the data were randomly selected for the training, corresponding to the "training data set", and the remaining 30% for the classification test, corresponding to the "test data set". It is observed on the diagonal that 97% of class 1, 95% of class 2, 92% of class 3 and 91% of class 4 measurements are well classified and the average accuracy is 93.5%.

Similarly, Figure 6 shows the classification performance for dataset Jl. It is seen by looking at the diagonal that 96% of class 1 measurements, 91% of class 2, 91% of class 1 3 and 87% of those in class 4 are well ranked and the average accuracy is 91.25%.

Figure 7 shows the classification performance for the J2 dataset. It is seen by looking diagonally that 94% of class 1, 86% of class 2, 85% of class 3 and 89% of class 4 are well ranked and the average accuracy is good. is therefore 88.1%. The comparison of the classification results on the 3 sets J0, J1 and J2 shows that the best results are those obtained by taking the normalized and smoothed J0 input data.

The method according to the invention makes it possible to recognize the thermal response of each pixel and to associate it with the right class, in order to reliably find, in the example described, the thickness of each coating strip of the sample.

The table in FIG. 8 shows the independence of the classification results with respect to the energy deposit, ie the verification that a variation in the homogeneity of the energy deposit on the sample has a very small impact on the classification results of each pixel in the data volume. This allows the realization of experimental measurements in which the parameter of uniformity of the deposit of energy on the surface is little annoying.

These classification performances were established for a data set J0 containing 16,000 signals, ie 4000 signals per class, each class comprising 1000 signals acquired with four flash lamps, 1000 signals acquired with three flash lamps, 1000 signals acquired with two flashlights, and 1000 signals acquired with a single flash lamp. Each of the classes thus includes pixels illuminated by a different number of flash lamps. This type of data selection significantly increases the different types of thermal responses depending on the lighting and thus allows the RNCNN network to extract the most significant characteristics possible, increasing the performance of the algorithm and therefore the reliability of classification results.

It can be seen by looking at the diagonal that 97% of Class 1 measurements, 92% of Class 2 measurements, 86% of Class 3 measurements and 91% of Class 4 measurements are well ranked. The average accuracy is 91.5%.

The table in Figure 9 was produced by applying the classifier to 4000 signals different from those used for learning and extracting characteristics by the RNCNN neuron network, but acquired at the same time period T2, for which each class is formed in the same way as when learning, that is to say 1000 signals with 4 flashes, 1000 signals with 3 flashes, 1000 signals with 2 flashes, and 1000 signals with 1 flash. As shown in Figure 9, 93.075% of the Class 1 measurements, 97.65% of the Class 2 measurements, 97.8% of the Class 3 measurements, and 94.8% of the Class 4 measurements are properly classified.

The table in Figure 10 shows the independence of the classification results with respect to the measurement conditions. The realization of two strictly identical measurements spaced over time on the same sample is indeed almost impossible to obtain. The positions of the lamps, the distance from the camera to the sample, the angles of inclination and the rotation of the camera relative to the camera are variable parameters that may affect the classification results.

In this example, the learning data set, acquired at a time period T2, is identical to that used to produce the table of FIG. 9, the classification performance being evaluated by applying the algorithm to 4000 signals. different from the signals used for learning, and acquired at a different time period T1, and for which each class is constituted in the same way as during the learning, that is to say 1000 signals with 4 flashes, 1000 signals with 3 flashes, 1000 signals with 2 flashes, and 1000 signals with 1 flash.

As shown in Figure 10, 98.6% of the Class 1 measurements, 91.65% of the Class 2 measurements, 91.975% of the Class 3 measurements, and 95.65% of the Class 4 measurements are well classified. The classification results of the different pixels belonging to the 4 different classes is very weakly dependent on variations of measurement conditions.

FIG. 11 represents the steps of obtaining virtual images intended for driving the neural network.

In the variant of the invention in which the spectral images used are thermal images acquired by infrared thermography, the parameter observed is the temperature T of the sample.

In this case, a virtual sample 3 may for example be in the form of a stack of materials, namely a substrate 4 (possibly multilayered) covered with a coating 5 (possibly multilayer) discretized in voxels V (x, y), of position (x, y) in the XY plane of sample 3.

Each voxel V (x, y) of this example, visualized along the Z axis includes all of the layer (s) F _zi provided for said coating deposited in the plane XY. In order to alleviate FIG. 11, the layers F _zi , here three in number (F _zi , F _z2 , F _z3 ), are represented only at the level of the gray voxel.

An LT curve representing the evolution of the temperature over time t is associated with each of said voxels V. This curve constitutes the response resulting from the EQM equation implemented within the mathematical model:

T (x, y, t) = F [QE (x, y, t); PS; PR; ER; t; ...] (EQM)

Finally, the mathematical model output the virtual spectral images ISV _X which result from the convolution of the curve LT with a curve of the "impulse response" type LRI, such as a "gate function P", and which are intended for the RN neuron network (CNN) training.

An example of a device 20 for characterizing a sample E according to the invention will now be described with reference to FIG. This device 20 is, in the example considered, a non-destructive control drone by spectral imaging.

The characterization device 20 comprises an acquisition means AQ of a set of spectral images of the sample E, in particular a thermal or spectral camera, scanning all the spectral ranges, for example the infrared, the visible, the near infrared, the middle infrared, the far infrared corresponding to the so-called thermal or thermographic infrared, and any combination of these ranges, especially visible and near infrared. The characterization device 20 also comprises a data processing module MAM capable of generating at least one volume of instantaneous data values D (Nx, Ny, Ne), and extracting at least one set of input data D'x , y (Ne) from said data volume D (Nx, Ny, Ne), as previously described.

In the example considered, the characterization device 20 further comprises an analysis module MP, comprising at least one RNCNN neuron network making it possible to extract characteristics from the input data, the analysis module MP being able to using the extracted characteristics to classify the input data according to a plurality of classes.

The characterization device 20 may comprise a thermal excitation means EX of the sample E to be characterized.

The characterization device 20 may furthermore comprise a decision-making module MD communicating with the analysis module MP and an action means ACT able to act on the sample E. The decision-making module MP is advantageously suitable. to enslave said ACT action means retroactively according to the classification results obtained and to trigger an appropriate action towards the sample E.

The invention is not limited to the examples which have just been described.

One possible application of the invention is the design of an intelligent spraying device as part of smart farming or smart-farming. In such an intelligent sprayer, each spray nozzle, corresponding to the ACT action means, can be associated with an embedded hybrid network according to the invention, for example, on a nano-PC using a low cost camera for example in the field. Visible and / or near-infrared that can detect leaf mass, vegetation type, or even the type of disease of the plant being examined, so as to spray the good phytosanitary product with the amount just needed. Multispectral or hyperspectral imaging can then be used to acquire the images. The plants can be corn, wheat, or vine. Plant growth can also be monitored by the invention and the possible occurrence of a disease can be predicted.

The invention is not limited to a convolutional neural network. Networks such as "Deep Neural Network" (DNN) or "Deep Belief Network" (DBN) type can be envisaged. The processing of several groups of data in parallel can be performed by parallel networks.

The invention can be implemented on any type of hardware, for example a personal computer, a smart phone, a nano-computer, or a dedicated card.

The invention is not limited to applications for the characterization of infrared thermography coatings. Radiation in the visible, near-infrared, mid-infrared, far-infrared, Terahertz or ultraviolet domains could be used. The invention is particularly suitable for non-destructive testing applications, in order to preserve the quality of the samples tested.

The invention can be used in various applications, for example in low-cost smart on-board sensors, or in "fog computing" decentralized infrastructures, in which the objective is to improve efficiency and reduce the amount of transferred data.

The invention can be used in many other fields, such as the military, electrical monitoring, geology, or biology or bioinformatics, particularly to follow manufacturing processes and the quality of materials.

Claims

1. A method for characterizing a sample (E), using a set of spectral images of the sample to be characterized previously acquired, in particular by infrared thermography or spectral imaging, and at least one neural network (RNCNN), the method comprising the steps of:

generating at least one volume of values (D (Nx, Ny, Ne)) of a parameter observed from said spectral images, for a plurality of coordinates (x, y) of the pixels (N) of the images and a plurality of acquisitions (Ne),

extracting at least one set (Jx) of input data (D'x, y (Ne)) from said data volume (D (Nx, Ny, Ne)), these input data corresponding to the values of observed parameter, for a pixel of the same coordinates according to different acquisitions (Ne), values to which at least one transformation function has been applied,

- driving said at least one neural network (RNCNN) using the input data (Jx) to extract at least one characteristic of the sample (E) to be characterized, and

using said at least one characteristic extracted by the neural network (RNCNN) to classify the input data according to a plurality of classes, each class being representative of at least one characteristic of the sample (E) to be characterized .

The method according to claim 1, wherein said at least one neural network (RNCNN) has previously been trained on images other than spectral images, including images of animals, objects, plants, or animals. people.

3. The method of claim 1 or 2, wherein said at least one neural network (RNCNN) has been previously driven on virtual images, including virtual spectral images.

4. Method according to one of claims 1 to 3, wherein said at least one neural network (RNCNN) is a convolutional neuron network (CNN).

5. Method according to any one of claims 1 to 4, wherein the classification is performed by a classifier independent of the neural network.

6. Method according to the preceding claim, wherein the classifier is of the wide margin separator type.

A method according to any one of the preceding claims, wherein the input data (Jx) is transmitted to the neural network (RNCNN) IN the form of images (IJx) representing curves corresponding to the values (D'x , y) of the input data set according to the acquisition (Ne).

The method according to any one of the preceding claims, wherein said at least one transform function applied to the values of the observed parameter (D (Nx, Ny, Ne)) is a centering, normalizing and / or smoothing function. .

The method of any one of claims 1 to 7, wherein said at least one transform function applied to the values of the observed parameter (D (Nx, Ny, Ne)) is the identity function.

The method of any one of claims 1 to 8, wherein a calculation function of the first derivative is applied to the values of the observed parameter (D (Nx, Ny, Ne)) to obtain the input data (D 'x, y (Ne)).

The method according to any one of the preceding claims, wherein a calculation function of the second derivative is applied to the values of the observed parameter (D (Nx, Ny, Ne)) to obtain the input data (D'x y (Ne)).

The method according to any one of the preceding claims, wherein said at least one characteristic extracted from the input data is the thickness or thickness range of the sample (E) or parts of the sample. sample, a representative quantity of a property of the sample, in particular a thickness of paint layer, a thickness of an intermediate layer, the level of water stress of a plant, the variation of pigmentation of plants.

13. A method according to any one of the preceding claims, wherein the spectral images used are thermal images acquired by infrared thermography, the parameter observed being the temperature of the sample (E).

14. Method according to the preceding claim, wherein the surface of the sample (E) to be characterized is thermally excited prior to the acquisition of the thermal images.

15. Device (20) for characterizing a sample (E), in particular by infrared thermography or spectral imaging, comprising:

means for acquiring (AQ) a set of spectral images of the sample (E) to be characterized, in particular a thermal camera, a data processing module (MAM) capable of:

Generating at least one volume of values (D (Nx, Ny, Ne)) of a parameter observed from said spectral images, for a plurality of coordinates (x, y) of the pixels (N) of the images and a plurality of 'acquisitions (Ne),

Extracting at least one input data set (D'x, y (Ne)) from said data volume (D (Nx, Ny, Ne)), these input data corresponding to the values of the observed parameter, for a pixel of the same coordinates according to different acquisitions (Ne), values to which at least one transformation function has been applied, and

an analysis module (MP), comprising at least one neural network (RNCNN), capable of at least driving said at least one neural network using the input data to extract at least one characteristic from the sample (E) to be characterized, and to use said characteristic extracted by the neural network to perform a classification of the input data according to a plurality of classes, each class being representative of a characteristic of the sample (E) to be characterized .

16. Device according to the preceding claim, wherein said at least one neural network (RNCNN) has been previously trained on images other than spectral images, including images of animals, objects, plants, or animals. people.

17. Device according to claim 15 or 16, wherein said at least one neural network (RNCNN) has been previously driven on virtual images, including virtual spectral images.

18. Device according to one of claims 15 to 17, the neural network (RNCNN) comprising one or more convolutional layers and / or one or more fully connected layers.

19. Device according to any one of claims 15 to 18, comprising a classifier independent of the neural network (RNCNN) to perform the classification.

20. Device according to any one of claims 15 to 19, comprising a thermal excitation means (EX) of the sample (E) to be characterized, in particular a surface excitation means by illumination, preferably a surface excitation means by pulsed illumination such as a flash lamp.

21. Device according to any one of claims 15 to 20, further comprising a decision-making module (MD) communicating with the analysis module (MP) and a means of action (ACT) able to act on the sample (E), said decision-making module being able to slave said means of action retroactively according to the classification results obtained from said analysis module and to trigger an appropriate action towards the sample.

22. Device according to the preceding claim, wherein the means of action (ACT) is a spraying nozzle of a phytosanitary product on crops, capable of spraying a quantity of product suitable for the classification results.

23. A method for controlling a sample (E), comprising the step of generating with the device (20) for characterizing a sample according to any one of claims 15 to 22, according to the classification results, information relating to the sample in order to make a decision to decide on an action to be carried out towards the sample to be characterized, and in particular to transmit an instruction to act to a means of action capable of putting it implemented.

24. Computer program product for implementing the method for characterizing a sample (E) according to any one of claims 1 to 14, using a set of spectral images of the sample to be characterized previously acquired, in particular by infrared thermography or spectral imaging, and at least one neural network (RNCNN), the computer program product comprising a support and recorded on this support processor readable instructions for when executed:

at least one volume of values (D (Nx, Ny, Ne)) of a parameter observed from said spectral images is generated, for a plurality of coordinates (x, y) of the pixels (N) of the images and a plurality of acquisitions (Ne),

at least one set of input data (D'x, y (Ne)) is extracted from said data volume (D (Nx, Ny, Ne)), these input data corresponding to the values of the observed parameter for a pixel of the same coordinates according to different acquisitions (Ne), values to which at least one transformation function has been applied, said at least one neuron network (RNCNN) is driven using the input data to extract at least one characteristic from the sample (E) to be characterized, and

said at least one characteristic extracted by the neural network (RNCNN) is used to classify the input data according to a plurality of classes, each class being representative of a characteristic of the sample (E) to be characterized.