FR3105860A1 - System and method for designing a neural network - Google Patents

Abstract

System and method for designing a neural network The present disclosure relates to a method of designing a first neural network (302) for filtering out corrupted samples, the method comprising: providing a second neural network (702); providing a first corrupted sample and a first clean sample to the first neural network (302), and training the first neural network to filter the first corrupted sample, and to generate a second clean sample, based on ground truth provided by the first clean sample; and providing the first clean sample and the first corrupted sample to the second neural network (702), and training the second neural network to corrupt the first clean sample, and generating a second corrupted sample, based on ground truth provided by the first altered sample. Figure for the abstract: Fig. 7

Description

System and method for designing a neural network

This disclosure relates generally to the field of processing systems, and in particular to a system and method for designing a neural network.

Sensors can be used to capture sample data for a wide range of applications. Examples of sensors include two-dimensional (2D) and three-dimensional (3D) cameras that produce data samples in the form of 2D or 3D images or video footage, pressure sensors that produce data samples in the form of pressure readings, depth sensors that produce data samples in the form of distance measurements or depth maps, and microphones that produce sound samples.

Acquisitions from such sensors are almost always affected to a greater or lesser extent by stochastic noise, artifacts and/or missing data. The use of sensor data by automatic algorithms or by human users can be significantly compromised by such alterations. For example, when sensor data is used by automatic decision-making algorithms, such as algorithms to decide whether or not a manufactured part meets required specifications, tampering with sensor data can increase the decision rate. false positives and/or false negatives.

It would therefore be desirable to provide a system and method that can reduce, at least to some extent, the presence of data corruptions in sensor data captured by one or more sensors.

In one aspect, there is provided a method of inspecting objects comprising: capturing one or more samples of an object using one or more sensors; filtering said one or more samples using a neural network to at least partially remove alterations in the samples; and applying an object inspection algorithm to said one or more filtered samples.

According to one embodiment, said one or more sensors comprise at least one camera, said one or more samples comprising at least one image captured by said at least one camera.

According to one embodiment, the object is an artifact, and the object inspection algorithm is arranged to detect one or more anomalies in the artifact, and to classify the object as passing or failing object inspection based on anomaly detection.

According to one embodiment, the method further comprises, before capturing said one or more samples, training the neural network based on one or more training samples and a corresponding ground truth.

According to one embodiment, the training of the neural network comprises the generation of ground truth: by capturing a plurality of samples of the same object using said one or more sensors and/or by using one or more other sensors , while varying one or more parameters of the sensors or other sensors and/or one or more environmental conditions between samples of the same object; and determining, by a processing device, a ground truth associated with the samples based on the plurality of samples of the same object.

According to one embodiment, the method further comprises: capturing a plurality of other samples of another same object using said one or more sensors and/or using one or more other sensors, while varying one or more capture parameters of sensors or other sensors and/or one or more environmental conditions between samples of the other same object; and the ground truth being determined based on the plurality of samples of the same object and the plurality of samples of the other same object.

According to one embodiment, the object is an electronic circuit board.

According to another aspect, an object inspection system is provided comprising: one or more sensors arranged to capture one or more samples of an object; a neural network arranged to filter said one or more samples to at least partially remove alterations; and a processing device arranged to apply an object inspection algorithm to said one or more filtered samples.

According to one embodiment, said one or more sensors comprise at least one camera, said one or more samples comprising at least one image captured by said at least one camera.

According to one embodiment, the object is an artifact, and the object inspection algorithm is arranged to detect one or more anomalies in the artifact, and to classify the object as passing or failing object inspection based on anomaly detection.

According to one embodiment, the object is an electronic circuit board.

According to one embodiment, the neural network is a convolutional neural network, such as a neural network having a UNet structure.

According to another aspect, a training system is provided for the aforementioned neural network, comprising a processing device arranged to carry out training of the neural network on the basis of one or more training samples and of a ground truth corresponding.

According to one embodiment, the learning system further comprises: said one or more sensors and/or one or more other sensors arranged to capture a plurality of samples of the same object, while varying one or more parameters sensors or other sensors and/or one or more environmental conditions between the samples of the same object, the processing device being further arranged to determine a ground truth associated with the samples based on the plurality of samples of the same object.

According to one embodiment, said one or more sensors and/or one or more other sensors are also arranged to capture a plurality of other samples of another same object, while varying one or more parameters of the sensors and/or or one or more environmental conditions between the samples of the other sample object; and the processing device being arranged to determine the ground truth based on the plurality of samples of the same object and the plurality of samples of the other same object.

In another aspect, there is provided a method of designing a first neural network to filter out corrupted samples, the method comprising: providing a second neural network; providing a first corrupted sample and a first clean sample to the first neural network, and training the first neural network to filter the first corrupted sample, and to generate a second clean sample, based on a ground truth provided by the first clean sample; and supplying the first clean sample and the first corrupted sample to the second neural network, and training the second neural network to corrupt the first clean sample, and generating a second corrupted sample, based on a ground truth provided by the first altered sample.

According to one embodiment, the method further comprises: supplying the second altered sample and the second clean sample to the first neural network, and training the first neural network to filter the second altered sample, and to generate a third clean sample, based on a ground truth provided by the second clean sample.

According to one embodiment, the method further comprises: providing the second clean sample and the second altered sample to the second neural network, and training the second neural network to alter the second clean sample, and to generate a third altered sample, based on a ground truth provided by the second altered sample.

According to one embodiment, the method further comprises: generating the first clean sample: by capturing a plurality of samples of the same object using one or more sensors, while varying one or more parameters of the sensors and/or one or more environmental conditions between samples of the same object; and determining, by a processing device, a ground truth corresponding to the first eigensample based on the plurality of samples of the same object.

According to one embodiment, the method further comprises: capturing a plurality of other samples of another same object using said one or more sensors, while varying one or more parameters of the sensors and/or one or more conditions environmental between samples of the other same object; and the ground truth is determined based on the plurality of samples of the same object and the plurality of samples of the other same object.

In another aspect, there is provided a method of inspecting objects comprising: designing a first neural network according to the aforementioned method; capturing one or more samples of an object using said one or more sensors or one or more other sensors; filtering said one or more samples using the first neural network to at least partially remove alterations in the samples; and applying an object inspection algorithm to said one or more filtered samples.

According to one embodiment, said one or more sensors and said one or more other sensors comprise at least one camera, said one or more samples comprising at least one image captured by said at least one camera.

According to one embodiment, the object is an artifact, and the object inspection algorithm is arranged to detect one or more anomalies in the artifact, and to classify the object as passing or failing object inspection based on anomaly detection.

According to one embodiment, the object is an electronic circuit board.

In another aspect, a system is provided for designing a first neural network for filtering out corrupted samples, the system comprising: the first neural network and a second neural network; and a processing device arranged to: supply a first corrupted sample and a first clean sample to the first neural network, the first neural network having been trained to filter the first corrupted sample, and to generate a second clean sample, on the basis of a ground truth provided by the first clean sample; and providing the first clean sample and the first tampered sample to the second neural network, the second neural network being trained to tamper with the first clean sample, and to generate a second tampered sample, based on a ground truth provided by the first altered sample.

According to one embodiment, the processing device is further arranged to: provide the second altered sample and the second clean sample to the first neural network, the first neural network having received training to filter the second altered sample, and generate a third clean sample, based on a ground truth provided by the second clean sample.

According to one embodiment, the processing device is also arranged to: supply the second clean sample and the second altered sample to the second neural network, the second neural network having received training to alter the second clean sample, and generate a third weathered sample, based on a ground truth provided by the second weathered sample.

According to one embodiment, the system further comprises: said one or more sensors arranged to capture a plurality of samples of the same object, while varying one or more sensor capture parameters and/or one or more conditions environmental differences between the samples of the same object, the processing device being further arranged to determine a ground truth corresponding to the first proper sample on the basis of the plurality of samples of the same object.

According to one embodiment, said one or more sensors are further arranged to capture a plurality of other samples of another same object, while varying one or more sensor capture parameters and/or one or more environmental conditions between the samples of the other sample object; and the processing device being arranged to determine the ground truth based on the plurality of samples of the same object and the plurality of samples of the other same object.

According to one embodiment, the first neural network is a convolutional neural network, such as a neural network having a UNet structure.

These characteristics and advantages, as well as others, will be set out in detail in the following description of particular embodiments made on a non-limiting basis in relation to the attached figures, among which:

FIG. 1 partially and schematically represents an embodiment of an optical inspection system intended to inspect electronic circuits;

Figure 2 is a graph showing an example of first pass efficiency in a system like that of Figure 1;

FIG. 3 schematically represents an alteration filter and calculation device of the optical inspection system of FIG. 1 in more detail according to an example embodiment of the present description;

FIG. 4 is a flowchart representing steps in a method for the automatic classification of electronic devices according to an example embodiment of the present description;

FIG. 5 schematically represents a processing system for training a neural network according to an exemplary embodiment of the present description;

FIG. 6 is a flowchart representing steps in a method for training a neural network according to an example embodiment of the present description;

FIG. 7 schematically represents a system for training a pair of neural networks according to an embodiment of the present description;

FIG. 8 is a flowchart representing steps in a method for training a pair of neural networks according to an exemplary embodiment of the present description;

FIG. 9 schematically represents a neural network according to an example embodiment of the present description; And

FIG. 10 schematically represents a neural network according to another exemplary embodiment of the present description.

The same elements have been designated by the same references in the various figures. In particular, the structural and/or functional elements common to the various embodiments may have the same references and may have identical structural, dimensional and material properties.

Unless otherwise specified, when reference is made to two elements connected together, it means directly connected without intermediate elements other than conductors, and when reference is made to two elements coupled together, it means that these two elements can be connected or be linked through one or more other elements.

In the following description, when referring to absolute position qualifiers, such as "front", "rear", "up", "down", "left", "right", etc., or relative, such as the terms "above", "below", "upper", "lower", etc., or to qualifiers of orientation, such as the terms "horizontal", "vertical", etc., it reference is made unless otherwise specified to the orientation of the figures.

Unless specified otherwise, the expressions “about”, “approximately”, “substantially”, and “of the order of” mean to within 10%, preferably within 5%.

FIG. 1 partially and schematically presents an embodiment of an optical inspection system 100, which is for example used for the optical inspection of electronic circuit boards. However, it will be apparent to those skilled in the art that System 100 could be used for optical inspection of a wide variety of types of manufactured objects.

Each electronic circuit board (BOARD) is for example placed on a conveyor 102, which is for example a flat conveyor. The conveyor 102 can move circuit boards in an X direction, for example a horizontal direction, to bring them to an optical inspection region, which will be called scene S, where images of circuit boards can be acquired.

In one example, conveyor 102 comprises an assembly of one or more conveyor belts and one or more rollers, driven by a rotary electric motor 104. Alternatively, conveyor 102 could instead comprise a linear motor arranged to move a carriage supporting the electronic circuit board. The electronic circuit board corresponds for example to a rectangular board having a length and a width varying from 50 mm to 550 mm.

The system 100 comprises for example a system 106 arranged to acquire images of the circuit board. In one example, the image acquisition system 106 comprises an image projection device P comprising at least one projector, a single projector P being represented in FIG. 1. When several projectors P are present, projectors P can be substantially aligned in a direction perpendicular to the X direction.

The acquisition system 106 further comprises, for example, an image acquisition device C comprising at least one digital camera, a single camera C being represented in FIG. 1. When several cameras C are present, cameras C can be substantially aligned, for example, in groups of cameras, preferably in a direction perpendicular to the direction X and/or can be arranged on one side or the other of the projector(s) P.

In one example, the projector P and the camera C are coupled to a control device 108. The control device 108 is for example arranged to control the image acquisition system 106, and also the conveyor 102, by the intermediary of the motor 104. In addition, the control device 108 is for example arranged to produce 2D and/or 3D images on the basis of image data coming from the camera C, and to supply these images to a filter of alterations (CF) 110.

The alterations filter 110 filters, for example, the alterations of the sensor data captured by the image acquisition system 106. According to the embodiments described in more detail below, the alterations filter 110 comprises one or more artificial neural networks. The alterations filter 110 provides filtered sensor data to a computing device 112.

The calculation device 112 is for example arranged to apply an automatic inspection algorithm to the images, and to produce an output signal (P/F) indicating whether the circuit board has passed the inspection successfully (P) or has failed (F). In particular, the inspection algorithm decides, for example, whether a manufactured part complies or not with the required specifications.

According to a variant embodiment, rather than including the projector P, the acquisition system 106 comprises a camera, such as a telecentric camera, arranged vertically aligned on the scene S, for example, at the location of the projector P in FIG. , and coupled to the controller 108.

The optical inspection system 100 can also comprise another acquisition system 114, which for example corresponds to a laser scanner, an altitude detector or a depth map sensor. The control device 108 is for example also arranged to control the acquisition system 114, to receive image and/or depth data captured by the acquisition system 114, and to supply the image or the map corresponding depth to the computing device 112 via the alterations filter 110, so that the inspection can also be based on this image or this depth map.

The purpose of the optical inspection system 100 is to automatically detect if a manufactured object does not meet a desired production quality. For example, in the case of an electronic circuit board, this may involve detecting the presence and/or shape of one or more solder joints, checking the extent of warpage, or checking for other visible items that may indicate a non-compliant object. However, more generally, optical inspection can be used to detect any visible defects in a manufactured object.

Figure 2 is a graph representing, as a function of the number of boards (#boards) examined, an example of the first pass yield (FPY) of an inspection system such as that of Figure 1. In particular, the graph of Figure 2 represents the percentage of circuit boards for which the inspection algorithm does not generate a false positive diagnosis. Note, however, that the graph does not separate false negative diagnoses, which are more difficult to identify.

It was assumed that system 100 was set up for a new series of circuit boards, starting with board 0 (Board 0). The first-pass yield (FPY) is initially at S%, and during a period of fine-tuning FT, parameters of the inspection algorithm and associated equipment are adjusted, causing the first-pass yield to increase and converge, exponentially, to an E% first-pass yield once a K-th board (Board K) has been inspected. After this point, a stable inspection period begins during which the parameters of the inspection algorithm remain fixed, and the first-pass efficiency remains stable and substantially at E%.

It is desirable that the value of E be as close to 100% as possible, since this implies fewer false negative results and thus greater stability. Furthermore, it is also desirable that K be relatively low, i.e. that there be a relatively small number of boards between Board 0 and Board K, since this implies that the fine tuning allows the first pass efficiency to converge quickly from S% to E%, and avoids a too long period during which the algorithm is below the optimal. Similarly, it is also desirable that the difference between E and S be as small as possible, so that production starts under relatively stable conditions, even during convergence to the E% level, and so that the convergence of S to E can occur in a relatively short time frame.

Referring again to Figure 1, an advantage of implementing an taint filter 110 using a neural network is that training of the neural network can be performed before the inspection starts, i.e. before the fine-tuning phase, which allows the initial percentage S of first pass efficiency to be relatively high, and also makes it possible to improve the percentage of long-term stable performance E that can be obtained.

FIG. 3 schematically represents the alterations filter 110 and the calculation device 112 of the optical inspection system 100 of FIG. 1 in more detail according to an example embodiment of the present description.

The alterations filter 110 comprises for example an artificial neural network (NEURAL NETWORK (CF)) 302, and a memory (MEMORY) 304. One or more corrupted samples (CORRUPTED SAMPLE(S)) 306 captured by one or more sensors are for example stored in the memory 304, then are processed by the neural network 302. The neural network 302 for example returns to write in the memory 304 one or more filtered samples (CLEAN SAMPLE(S)) 308.

As known to those skilled in the art, an artificial neural network (hereinafter referred to simply as a "neural network") is a computer system or architecture that is intended to mimic to some extent the neural network organisms found in a human brain. Such neural networks generally include a network of artificial neurons, which are electrical circuits that receive inputs, combine these inputs with their internal state and often with a threshold, and produce an output signal. The outputs of neurons are coupled to the inputs of other neurons by connections, which are often called synapses (their equivalent in the biological brain). These connections are also implemented by electronic circuits, and apply weights to the signals propagated between the output of one or more neurons and the input of another neuron. The function used to generate the input to a neuron, based on the outputs of its predecessor neurons and the connections as a weighted sum, is known as the propagation function.

In some embodiments, neural network 302 is implemented by dedicated hardware, such as by a graphics processing unit (GPU) or other form of ASIC (application specific integrated circuit) that implements neural network architecture. An advantage of such a hardware implementation is that it can be relatively fast during learning and later use.

In alternate embodiments, neural network 302 is implemented by a computer emulation of a neural network. For example, the neural network is implemented by software instructions executed by a processing device like a CPU (central processing unit). In some embodiments, a hybrid implementation may involve portions of the neural network that are implemented in hardware, and other portions of the neural network that are implemented in software.

Regardless of the implementation (hardware and/or software), in some embodiments, the neural network 302 of Figure 3 has an architecture based on a UNet approach, as further described in the publication titled “ U-Net: Convolution Networks for Biomedical Image Segmentation”, O.Ronneberger, P.Fischer and T.Brox, 18 May 2015, Lecture Notes in Computer Science book series, LNCS, volume 9351, the contents of which are incorporated herein by reference in limits permitted by law.

Additionally or instead, neural network 302 includes one or more layers of attention, as described in the publication titled "Self-Attention Generative Adversarial Networks", H.Zhang, I.Goodfellow, D.Metaxas and A .Odena, 14 June 2019, PMLR 2019, the contents of which are also incorporated herein by reference to the extent permitted by law.

Further, neural network 302 may additionally or instead be based on a loss function known as "perceptual loss" as described in more detail in the publication entitled "Perceptual Losses for Real-time Style Transfer". and Super-Resolution”, J.Johnson, A.Alahi, L.Fei-Fei, 26 March 2016, the contents of which are also incorporated herein by reference to the extent permitted by law.

An example neural network 900 is shown in Figure 9, and provides an example implementation of neural network 302.

The neural network 900 includes, for example, neurons 901 formed in several layers.

An input layer 902 corresponds to a first layer of neurons to which input data of an input sample is applied. The example of FIG. 9 represents two neurons 901 in the input layer, but in practice there are for example hundreds or thousands of neurons forming the input layer 902. For example, in the case where the network neuronal 302 is based on samples in the form of images, it comprises for example a number of neurons in the input layer 902 equal to the number of pixels of the input image.

An output layer 904 corresponds to a final layer of neurons from which output data is provided. In the case where the neural network 900 generates output samples of the same format as the input samples, for example images of the same dimensions, the number of neurons 901 forming the output layer 904 is for example equal to the number of neurons forming the input layer 902.

Between the input layer 902 and the output layer 904, there are a number of intermediate layers 906 generally called hidden layers. In the example of Figure 9, there is a single hidden layer 906, although in alternative embodiments there may be several hidden layers. The number of neurons 901 in each hidden layer 906 is for example greater than the number of neurons in the input or output layer 902, 904.

In the case of a single hidden layer 906 as in the example of FIG. 9, each neuron 901 of the hidden layer 906 receives for example an input coming from each neuron 901 of the input layer 902, and each of these connections is a synapse 908, to which an individual synapse weighting parameter φ is applied. These weighting parameters are referenced φ _ij , i being the input neuron index ranging from 1 to M, and j being the neuron index in the hidden layer ranging from 1 to N. In addition, each neuron 901 of the hidden layer 906 for example provides an output signal to each of the output neurons 901, and each of these connections is a synapse 910, to which an individual synapse weighting parameter θ _ij in Figure 9 is applied similarly to the parameters weighting φ _ij . However, for the parameters θ _ij , i is the neuron index in the hidden layer ranging from 1 to N, and j is the neuron index in the output layer ranging from 1 to K.

In the case where there is more than one hidden layer 906, the neurons of a first hidden layer have for example the same inputs as those represented for the hidden layer 906 of FIG. 9, the neurons of a hidden layer final having for example the same outputs as those represented for the hidden layer 906, and each of the other hidden layers receives the outputs coming from each of the neurons of the preceding hidden layer among the hidden layers, and each provides an output to each of the neurons of the next hidden layer among the hidden layers.

The example in Figure 9 corresponds to a "fully connected network", in which all neurons of a given layer are connected to all neurons of the previous and following layers. However, to process relatively large input images, for example 10000 pixels or more, the number of weights to be optimized will become very large. A convolutional neural network (CNN) provides an architectural alternative to the "fully connected network" and is computationally significantly lighter when processing relatively large input images. We will now describe an example of CNN in more detail with reference to Figure 10.

FIG. 10 schematically illustrates an additional convolutional neural network 1000 according to an example embodiment of the present description. The example in Figure 10 is based on the UNet approach involving several convolution layers separated by undersampling and then several other convolution layers separated by oversampling operations, corresponding to the "U" shape.

The neural network 1000 receives for example, on the left of the figure, an input image tile 1002, and generates, on the right of the figure, an output image 1004. In the example of Figure 10, input image 1002 is 512 by 512 pixels, and output image 1004 is the same size as the input image. However, this is only an example and any input and output image size would be possible. The input and output images 1002, 1004 each have, for example, three channels corresponding to color channels. However, in variant embodiments, the neural network 1000 could receive as input and produce as output grayscale images, or depth maps, having only one channel.

Arrows 1006 in Figure 10 represent convolution layers. Each convolution layer 1006 corresponds for example to a certain number of filters. In the example of Figure 10, the input image 1002 is initially filtered by two convolution layers each comprising 16 convolution filters, resulting in images having 16 channels. Each of the convolution filters has a limited number of parameters, such as a 3 by 3 filter having 9 parameters. In one example, a filter based on the Sobel filter kernel could be used, even though the filter parameters are going to be changed during training, and are likely to converge to filters similar to the Sobel filter, but not matching. exactly to this one. The convolution between the input image and the filter is applied to each pixel of the image in the same way, and thus, with relatively few parameters, the image can be completely transformed. Indeed, depending on the parameters of each filter, the filtered output image will be completely different.

In the case of convolution filters having a plurality of parameters, the filter is for example applied to a block of pixels in order to generate each pixel of the resulting image. Despite this, in the example of figure 10, the convolution layers do not cause for example a reduction in the size of the image, since one or more rows or columns of pixels, padded with zeros, are for example added on each edge of the image before the convolution filtering so that an output value can be generated for each pixel of the input image.

Arrows 1008 in FIG. 10 represent the subsampling operations in which blocks of pixels are for example combined by generating an average pixel value of the pixels in the block or by taking the value of the peak pixel among the pixels in the block. In the example in Figure 10, downsampling involves combining two-by-two blocks of pixels resulting in a four-fold reduction in image size. Also, the number of convolution filters applied to each downsampled image is, for example, twice the number applied to the previous image before downsampling. In the example of figure 10, there are four downsampling operations bringing the image size from 512 by 512 to 256 by 256, to 128 by 128, to 64 by 64, and finally to 32 by 32, the number of filters applied to these images increasing from 16 in the case of the 512 by 512 image, up to 256 in the case of the 32 by 32 image. bottom of the "U", and is known as the neck.

Arrows 1010 in FIG. 10 represent oversampling operations in which blocks of pixels are for example duplicated a certain number of times to increase the size of the image. For example, duplicating a pixel once in each direction causes the image size to increase by a factor of 4, since each pixel becomes four pixels. Furthermore, the number of convolution filters applied to each oversampled image is for example equal to half the number of those applied to the previous image before the oversampling. In the example of Figure 10, there are four oversampling operations taking the image size from 32 by 32, to 64 by 64, to 128 by 128, to 256 by 256, and finally to 512 by 512. In addition, the number of filters applied to these images decreases from 256 in the case of the 32 by 32 image, to go down to 16 in the case of the 512 by 512 image.

As shown by arrows 1012 in Fig. 10, in some embodiments, a first convolution operation following each upsampling operation also involves a concatenation of the result of a corresponding downsampling operation. This helps the neural network, for example, to keep edges/contours of the reconstructed image as sharp as in the input image 1002.

As represented by an arrow 1014 in Figure 10, a final convolution layer involves, for example, applying three convolution filters to each pixel in order to reduce and reduce the number of channels to 3.

Of course, Figure 10 is just an example, and there are many variations that could be applied, such as using a different number of convolution layers between each downsampling or oversampling operation. -sampling, using a different number of downsampling and upsampling operations, and/or using a different number of convolution filters in each convolution layer. Further, although an example has been described in which one or more rows and columns padded with zeros are added to the images to avoid image size reduction during convolution filtering, in alternative embodiments these rows and columns additional columns could be populated in a different way, such as with a duplication of their neighboring rows/columns, or no such additional rows/columns could be provided.

As mentioned previously, the implementation of the neural network 900 of Fig. 9 or 1000 of Fig. 10 could be done by dedicated hardware, or by software, or by a hybrid solution involving parts implemented under hardware and software form.

During a training phase, an example of which is described in more detail below, the neural network 302 automatically learns how to filter out corrupted samples in order to remove unwanted noise and other artifacts. In particular, this implies for example the gradual refinement of the various parameters of the various neurons and various connections, such as the memorized states of the neurons, the connections between neurons and the propagation functions, resulting from the weighting parameters φ _ij, θ _ij in the example of Figure 9, or resulting from the parameters of the convolution functions in the example of Figure 10, to generate the neuron inputs based on the output of one or more predecessor neurons. Thus, the trained neural network 302, once trained, is able to remove undesirable sample corruption efficiently.

The calculation device 112 comprises for example a processing device (P) 310, which comprises one or more processors arranged to store instructions in an instruction memory (INSTR MEM) 312. The processing device 310 is for example arranged, under instruction control, to retrieve filtered samples 308 from memory 304, and process those samples based on one or more inspection parameters to determine if one or more test criteria are met. The processing device 310 generates for example an output result in the form of a success or failure message, which is for example outputted (P/F) via an output interface (O /P INTERFACE) 312, which may comprise an operator screen, or a communication interface for providing the result to one or more remote systems. Additionally or instead, the result of each inspection could be stored in memory 304.

In some embodiments, instead of the software implementation shown in Figure 3, the computing device 112 could be implemented at least partially in hardware, for example by one or more ASICs. application) and/or by one or more other artificial neural networks.

FIG. 4 is a flowchart representing steps in a method for the automatic classification of objects manufactured according to an exemplary embodiment of the present description. This method is for example implemented by the alterations filter 110 and the calculation device 112 of FIG. 3.

In a step 402, one or more samples relating to an object to be inspected are captured by one or more sensors. For example, in the case of the optical inspection system 100 of FIG. 1, each sample corresponds to an image or a depth map of a part of a circuit board, each image or depth map comprising a matrix of pixel values, each pixel value providing light intensity, color and/or depth information. However, more generally, said one or more samples could correspond to data samples in the form of pressure readings, samples of sound or vibrations, samples of position, velocity or linear or angular acceleration, etc As with virtually all data samples captured by a sensor, samples are affected by noise and other artifacts originating from the sensor environment, the sensor itself, and/or other sources. Said one or more samples are for example stored in the memory 304 of FIG. 3 in the form of the altered sample(s) 306.

In a step 404, said one or more captured samples are filtered by the alterations filter, and in particular by the neural network 302 having received training. This implies, for example, the modification of one or more values of each sample to at least partially remove a noise component.

In a step 406, one or more inspection algorithms are applied to said one or more filtered samples in order to decide a pass or fail result of the inspection. For example, said one or more inspection algorithms are applied by the computing device 112 of Figure 3. In the case of the optical inspection system 100 of Figure 1, said one or more inspection algorithms decide to a pass or fail for each circuit board passing through the system. However, the same principles could be applied to any manufactured object.

We will now describe a system and a method for carrying out the training of the neural network 302 of FIG. 3, with reference to FIGS. 5 to 8.

FIG. 5 schematically represents a processing system 500 for training a neural network, such as the neural network 302 of FIG. 3, according to an example embodiment of the present description.

The system 500 comprises for example the neural network which has not received training 302' (NEURAL NETWORK (CF)), coupled for example via a bus 502 to a memory 504. The memory 504 stores for example a plurality of samples (SAMPLES) 506 captured by one or more sensors (SENSORS) 508, which are also for example coupled to the bus 502. For example, these sensors 508 are identical or similar sensors to those used to capture samples during an inspection process. In addition, the memory 504 stores for example one or more ground truths (GROUND TRUTH(S)) 510. A processing device (P) 512, which for example comprises one or more processors, is also for example coupled to the bus 502 For example, the processing device 512 is under the control of instructions stored in an instruction memory 514, which is for example a random access memory (RAM).

In operation, in the case where the neural network 302' is implemented by a GPU or other form of ASIC, the part of the bus 502 for communicating with the neural network is for example a PCI Express bus (from the English Peripheral Component Interconnect Express). For example, when an inference is to be made, input data going to the neural network 302' is transferred from the RAM dedicated to the processing device 512, for example a CPU RAM, to a RAM dedicated to the network. neuronal 302', for example a GPU RAM, via the PCI Express bus or the like. The neural network computing units 302' are then, for example, configured to retrieve the input data from the neural network's RAM and to process that input data. The output data generated by the neural network 302′ are then, for example, stored in the RAM dedicated to the neural network, for example in the GPU RAM, then transferred to the RAM dedicated to the processing device 512, for example to the CPU RAM by via the PCI Express or similar bus.

A difficulty in training a neural network to filter out output noise from corrupted samples is that a ground truth, corresponding to a desired result of the filtering, is usually required. However, the known methods for generating a ground truth generally involve the use of one or more sensors capable of giving a significantly greater precision than the sensors of the system. For example, at least ten times greater precision is often used to generate ground truth. A difficulty is that such high precision sensors are expensive and may not always be available.

The present inventor provides an alternative solution for generating ground truth based on samples captured under various conditions, as will be described in more detail now with reference to Figure 6.

FIG. 6 is a flowchart representing steps in a method for training a neural network according to an example embodiment of the present description. This method is for example implemented by the processing system 500 of FIG. 5 in order to carry out the learning of the neural network 302′. In particular, the method is for example implemented by software executed by the processing device 512.

In a step 602, samples of the same object are for example captured multiple times using the sensors 508, while applying M sets of different parameters to the sensors. These captured samples are for example stored in the memory 504 of FIG. 5 in the form of samples 506.

For example, each parameter set defines one or more parameters of the sensors 508, and/or the environment of the sensors and the artifact being captured, parameters that are applied during sample acquisition. For example, each parameter defines a combination of one, some, or all of the following parameters:
- exposure time/sampling time;
- ambient lighting level, for example involving the use of one or more additional parasitic light sources to selectively illuminate the object;
- a level of backlighting of the object;
- the position of the object relative to the sensors, including the altitude of the manufactured object;
- the orientation of the object being sampled, electronic circuit boards or other objects being captured, for example, at 0° or 90°, or any angle in between, relative to the surface of the conveyor 102 of the figure 1 ; And
- in the case where the sensor is a camera having a variable focal length, the selected focal length of the camera.

For example, the number M of sets of different parameters which are applied during the acquisition of samples is at least 2, and can be much higher, for example at least a thousand in certain cases. For example, for some applications, the number of acquisitions could be more than one million. Indeed, one can vary some of the aforementioned parameters for example with relatively small increments over a relatively large range of parameters, and by capturing samples with each combination of these parameters, a relatively large number of different parameter sets can be applied to produce matching samples.

Further, in some embodiments, the sample acquisition is based on a high-fidelity setup of the acquisition system, which is acceptable for generating training data, but which would be unacceptable in a production line since would be too slow. High-fidelity setting involves for example capturing images with relatively low noise, with well-arranged lighting conditions, including for example some steps to automatically adjust exposure time and/or projected intensity, with a well-adjusted focus, comprising for example certain steps for automatically adjusting the focus in order to obtain images with high fidelity and high sharpness, and with a relatively high spatial resolution.

In a step 604, a pseudo-ground truth is for example generated on the basis of the samples captured in step 602.

For example, in some embodiments, and for some aspects, the pseudo ground truth is generated as an average of some or all of the captured samples. Depending on the type of noise, this average could correspond to a usual average, a median or a mode value, or to other more dedicated algorithms based on knowledge of the domain of the final application. Indeed, it would be possible to use more advanced algorithms that are computationally expensive to clean up noisy samples so that a relatively fast neural network is trained to mimic this high-performance processing. A manual operation to improve the quality of the ground truth could also be applied, for example to remove artefacts from a 3D reconstruction.

For example, the position of one or more edges of the captured object can be set in ground truth to a median position, thereby reducing the impact of outliers. In another example, assuming a wide range of illuminations was applied during step 602, the illumination level may be set in ground truth to a medium illumination level.

Other aspects of ground truth can be determined based on an estimated noise distribution resulting from parameter variations in the various samples.

Still other aspects of ground truth can be determined based on one or more detection algorithms. For example, in the case of variations in the focal length between samples, one or more samples having the sharpest focus can be detected using known algorithms, and can be selected as representing the ground truth with respect to the image sharpness.

Pseudo-ground truth, for example, is a pseudo-sample generated based on some or all of the aforementioned approaches to generate the desired result of the accidental filtering operation.

After step 604, in some embodiments steps 602 and 604 are repeated for a new object, to generate another unique ground truth for the new object. For example, the new object is an object made using the same process as the previously captured object. However, due to process tolerances, there will be slight variations. Therefore, the samples captured using the M different sets of parameters will have some variation from the samples captured during the first iteration. Furthermore, the steps 602 and 604 are for example repeated over numerous iterations for numerous new objects, for example for hundreds, thousands or even millions of objects, in order to generate numerous samples and pseudo-truths of corresponding land.

In some embodiments, at least some of the objects based on which pseudo ground truths are generated include examples of both i) good parts that are to be classified as passing the object inspection process ; and ii) defective parts that must be classified as failing in the object inspection process. It is also preferable that the samples used during training, and the corresponding ground truths, include examples of objects passing and failing object inspection. In this way, the objects, and the corresponding samples and ground truths, will reflect what will actually be presented to the object inspection system during use.

In a step 606, the neural network 302 receives, for example, training based on the pseudo-ground truth generated in step 604 during the final iteration. As known to those skilled in the art, training a neural network involves, for example, providing the neural network with many example samples and the corresponding ground truth, so that the neural network learns iteratively how to get ground truth from any sample. In the case of the alterations filter 302, the network is taught to correct the altered samples and to generate a filtered sample approaching the ground truth. In the present case, the ground truth corresponds to the pseudo-ground truth, and the altered samples correspond, for example, to one or more of the samples captured during each iteration of step 602. In addition or instead, at least some of the altered samples used during training may be generated by another neural network, as will now be described in more detail with reference to Figures 7 and 8.

FIG. 7 schematically represents a system 700 for training a pair of neural networks according to an example embodiment of the present description. One of the neural networks must be trained to behave like the alterations filter, and for example corresponds to the neural network 302' of FIG. 5. The other neural network (NEURAL NETWORK (CS)) 702 corresponds for example to an accidental simulator, which simulates the accidental in a sample. In some embodiments, the accidental simulator 702 has the same architecture as that of the accidental filter 302', as previously described. Alternatively, networks 302' and 702 could have different architectures.

The neural network 302' receives, for example, a corrupted sample (CORRUPTED SAMPLE) 704, and is trained to filter this sample to generate a filtered sample (CLEAN SAMPLE) 706. For example, the training of the neural network 302' is based on a ground truth corresponding to a clean sample 706 generated in the previous learning cycle. In addition, the clean sample 706 is also for example provided as input to the neural network 302' to serve as ground truth for a following training cycle of the alterations filter 302'. As known to those skilled in the art, training a neural network based on input samples and ground truths is an iterative process during which the transfer function of the neural network, and in particular the synaptic weights, are adjusted such that the generated result converges to the ground truth based on the minimization of a given cost function. In the present system, the cost function is for example the distance between the filtered sample generated by the neural network 302' and the proper sample 706 providing the ground truth.

The neural network 702 receives for example the own sample 706, and a variable or a random vector (RANDOM), and is trained to generate an altered sample 704 on the basis of a ground truth corresponding to an altered sample 704 generated in the previous learning cycle. The neural network 702 generates, for example, corrupted samples 704 based on the clean sample and the random variable or vector, which makes the generation non-deterministic. The altered sample 704 is also, for example, provided to the neural network 702 to serve as ground truth for a following training cycle of the alterations simulator 702.

We will now describe the operation of the system 700 with reference to Figure 8.

FIG. 8 is a flowchart representing steps in a method of training a pair of neural networks according to an example embodiment of the present description. This method is for example implemented by a pair of neural networks as shown in the system 700 of Figure 7. For example, an architecture similar to that of Figure 5 is used, in which the other neural network 702 is also accessible via bus 502.

In a step 801, two new samples, an altered sample and a clean one, are loaded into the neural networks. For example, the weathered sample corresponds to one or more of the samples captured during an iteration of step 602 of Figure 6, and the clean sample corresponds to the corresponding pseudo-ground truth. Also, in step 801, an index n is for example set to one.

The training of the neural networks is for example carried out in parallel steps 802 and 803. As a variant, these steps could also be carried out sequentially.

Step 802 involves, for example, training the tampering filter 302' to remove the tampering from the tampered sample based on the clean sample, in other words using the clean sample as the truth of ground. For example, this learning is based on the following loss function: where L _F is the loss associated with the impairment filter, F is the impairment filter network, S is the impairment simulation network, R is the random vector or matrix, which for example has a uniform distribution or Gaussian, X are weathered samples from the training dataset, Y is the ground truth associated with the weathered samples X, d is the distance between the samples, and α, β and γ are weights, which can be settled by trial and error. In some embodiments, each of these weights can be set to 1. The distance d represents the difference between the samples. For example, in the case of images, the distance could be the absolute distance L1, also known as the Manhattan distance, or the root mean square distance L2, also known as the Euclidean distance, or any other distance.

Step 803 involves, for example, training the tamper simulator 702 to introduce an tamper to the clean sample based on the tampered sample, in other words using the tampered sample as ground truth. For example, this learning is based on the following loss function: where L _S is the loss associated with the simulation of alterations, δ is a weight, which, like the other weights α, β and γ, can be tuned by trial and error, and could be zero in some cases, and the others parameters are the same as before.

In variant embodiments, identical loss functions could be used during the learning steps 802 and 803. For example, the following common loss function could be used: where L is the loss, and the other parameters are the same as before.

Steps 802 and 803 are followed by steps 804 and 805, respectively.

In step 804, the filtered sample resulting from the filtering applied by the filtering network in step 802 is stored in memory as a new clean sample, replacing for example the clean sample initially loaded in step 801.

In step 805, the tampered sample resulting from the tampering simulation applied by the tamper simulation network in step 803 is stored in memory as a new tampered sample, for example replacing the initially tampered sample. loaded in step 801.

In a step 806 following steps 804 and 805, it is determined for example whether the index n is equal to N, where N is a desired number of iterations of the learning sequence on the basis of the same pair of samples, clean and weathered, originally loaded. For example, N is at least 2, and in some embodiments, N is at least 10, and typically could be greater than 1000.

If the index n is not yet equal to N, then in a step 807, the index n is incremented, and the steps 802 to 805 are repeated, on the basis of the new samples, clean and altered, stored in the previous steps 804 and 805.

Once the index n is equal to N, the next step is step 808, in which it is determined whether other pairs of samples, altered and clean, are available for training. If yes, the method returns for example to step 801, in which the following pair is loaded, and the index n is reset to 1, before repeating N iterations of the learning operations 802 to 805.

When, in step 808, there are no more other pairs to be used for learning, the method for example ends in a step 809 in which learning is considered to be finished.

An advantage of the system and method described in relation to Figures 7 and 8 is that learning can be performed based on a significantly higher number of altered samples than the number that is captured. Indeed, training the alteration simulator to mimic the alterations introduced by the sensor based on random input potentially allows an unlimited number of altered samples to be generated for training.

An advantage of the system and the method described in relation to FIGS. 3 and 4 is that, by using a trained neural network to filter samples coming from sensors, the performance of a quality inspection algorithm applied to the samples can be significantly improved, making it possible to obtain high levels of the initial percentage of first pass efficiency S, and long-term performance represented by the percentage E.

The advantage of the system and of the method described in relation to FIGS. 5 and 6 is that, by determining a ground truth on the basis of multiple samples of the same object captured under variable sensor and environmental conditions, can avoid the cost and/or difficulty of providing a very accurate sensor to generate ground truth.

Various embodiments and variants have been described. The person skilled in the art will understand that certain features of these various embodiments and variations could be combined, and other variations will occur to the person skilled in the art. For example, it will be apparent to those skilled in the art that the techniques described in connection with Figures 3-8 could all be combined in a single embodiment, or be used individually in conjunction with other systems and methods.

Finally, the practical implementation of the embodiments and variants described is within the abilities of those skilled in the art based on the functional indications given above. In particular, the learning of a neural network on the basis of a loss function, such as for example on the basis of the loss functions described in relation to FIG. 8, is within the capabilities of a person skilled in the art.

Claims (15)

A method of designing a first neural network (302) for filtering out corrupted samples, the method comprising:
- providing a second neural network (702);
- providing a first corrupted sample and a first clean sample to the first neural network (302), and training the first neural network to filter the first corrupted sample, and to generate a second clean sample, based on a truth of ground provided by the first clean sample; And
- providing the first clean sample and the first altered sample to the second neural network (702), and training the second neural network to alter the first clean sample, and generating a second altered sample, based on ground truth provided by the first altered sample. A method according to claim 1, further comprising:
- providing the second corrupted sample and the second clean sample to the first neural network (302), and training the first neural network to filter the second corrupted sample, and to generate a third clean sample, based on a truth of ground provided by the second clean sample. A method according to claim 1 or 2, further comprising:
- supplying the second clean sample and the second altered sample to the second neural network (702), and training the second neural network to alter the second clean sample, and to generate a third altered sample, based on a truth of terrain provided by the second weathered sample. A method according to any of claims 1 to 3, further comprising:
- generate the first clean sample:
capturing a plurality of samples of the same object using one or more sensors (106, 114, C), while varying one or more sensor parameters and/or one or more environmental conditions between samples of the same object ; And
determining, by a processing device (512), a ground truth corresponding to the first eigensample based on the plurality of samples of the same object. A method according to claim 4, further comprising:
- capturing a plurality of other samples of another same object using said one or more sensors (106, 114, C), while varying one or more sensor parameters and/or one or more environmental conditions between the samples of the other same object; And
- wherein the ground truth is determined based on the plurality of samples of the same object (BOARD) and the plurality of samples of the other same object. A method of inspecting objects comprising:
- the design of a first neural network (302) according to the method of any one of claims 1 to 5;
- capturing one or more samples of an object (BOARD) using said one or more sensors or one or more other sensors (106, 114, C);
- filtering said one or more samples using the first neural network (302) to at least partially remove alterations in the samples; And
- applying an object inspection algorithm to said one or more filtered samples. A method according to claim 6, wherein said one or more sensors and said one or more other sensors (106, 114, C) comprise at least one camera, said one or more samples comprising at least one image captured by said at least one camera . Method according to claim 6 or 7, in which the object (BOARD) is a manufactured object, and the object inspection algorithm is arranged to detect one or more anomalies in the manufactured object, and to classify the object as passing or failing object inspection based on anomaly detection. A method according to any of claims 1 to 8, wherein the object is an electronic circuit board. A system for designing a first neural network (302) for filtering out corrupted samples, the system comprising:
the first neural network (302) and a second neural network (702); And
a processing device (512) arranged to:
- supplying a first altered sample and a first clean sample to the first neural network, the first neural network being trained to filter the first altered sample, and to generate a second clean sample, based on a ground truth provided by the first clean sample; And
- supplying the first clean sample and the first altered sample to the second neural network, the second neural network being trained to alter the first clean sample, and to generate a second altered sample, based on a ground truth provided by the first altered sample. System according to claim 10, in which the processing device (512) is further arranged to:
- supplying the second corrupted sample and the second clean sample to the first neural network (302), the first neural network having been trained to filter the second corrupted sample, and generating a third clean sample, based on a ground truth provided by the second clean sample. System according to claim 10 or 11, in which the processing device (512) is further arranged to:
- providing the second clean sample and the second tampered sample to the second neural network (702), the second neural network being trained to tamper with the second clean sample, and generating a third tampered sample, based on ground truth provided by the second altered sample. A system according to any of claims 10 to 12, further comprising:
- said one or more sensors (106, 114, C) arranged to capture a plurality of samples of the same object, while varying one or more sensor capture parameters and/or one or more environmental conditions between the samples of the same object, wherein the processing device (512) is further arranged to determine a ground truth corresponding to the first eigensample based on the plurality of samples of the same object. A system according to claim 13, wherein:
- said one or more sensors (106, 114, C) are also arranged to capture a plurality of other samples of another same object, while varying one or more sensor capture parameters and/or one or more environmental conditions between samples of the other sample object; And
- the processing device (512) is arranged to determine the ground truth based on the plurality of samples of the same object and the plurality of samples of the other same object. A method according to any of claims 1 to 9 or a system according to any of claims 10 to 14, wherein the first neural network (302) is a convolutional neural network (1000), such as a neural network having a structure A T.