FR3101175A1 - Product quality analysis method by means of a test bench, computer program product and associated analysis device - Google Patents

Abstract

Process for analyzing the quality of products by means of a test bench, product, computer program and associated analysis device This invention relates to a method for analyzing the quality of products tested using one or more test benches, comprising a learning phase (PA) and a verification phase (PV). The learning phase (PA) includes determining normal behavior for each type of product and updating a learning base. The verification phase (PV) includes the steps of selecting (170) a product to be verified, retrieving a test report relating to this product, and determining (180) a level of confidence in the quality of the product by comparing the test data with reference values, and when the determined confidence level is below a detection threshold, identifying the product as a suspect product. Figure for the abstract: Figure 4

Description

Product quality analysis method using a test bench, computer program product and associated analysis device

The present invention relates to a method for analyzing the quality of products by means of a test bench.

The present invention also relates to an associated computer program product and analysis device.

According to the invention, the products analyzed present assemblies of electronic and/or mechanical and/or optical sub-assemblies.

In a manner known per se, to analyze the quality of such products, test benches are used.

These benches are composed of test equipment managed by a test sequencer implementing a test program.

Thus, when a product is tested by such a bench, the test equipment acquires various measurements relating to the operation of the product.

These measurements are then analyzed by a test computer in order to detect a possible failure of the product.

This analysis generally includes the comparison of measurements with predetermined thresholds. Thus, the measurements are compared one by one with the corresponding thresholds and when at least one of these measurements exceeds the corresponding threshold, the product is then considered suspicious.

Additional tests can then be implemented to determine if it is a false alarm or a faulty product.

However, such a type of analysis proposed by the state of the art is not completely satisfactory.

In particular, often the composition of the products evolves so that the predetermined thresholds are no longer completely adapted to reflect the quality of the products over time.

Similarly, test benches can evolve following breakdowns, preventive maintenance or scheduled maintenance. The evolution of the composition of these benches can lead to a dispersion of the test values.

It is therefore conceivable that this leads to the late and costly removal of manufactured equipment, linked to a difficulty in predicting failures.

The object of the present invention is to remedy this problem and therefore to make it possible to effectively predict the failures of products during their production.

To this end, the subject of the invention is a method for analyzing the quality of products tested by means of one or more test benches, the or each test bench being configured to implement tests on a product and to generate test data following the implementation of these tests, the test data relating to each product being stored in the form of a test report relating to this product;

the method comprising a learning phase and a verification phase;

the learning phase being implemented periodically and for each new implementation, comprising the determination of a normal behavior of each type of product from the test reports relating to each product of this type and the updating of a learning base comprising reference values defining this normal behavior of this type of product;

the verification phase comprising the following steps:

- among the products tested, selection of a product to be checked and recovery of a test report relating to this product;

- determination of a level of confidence in the quality of the product by comparing test data from the test report corresponding to this product with reference values from the learning base in relation to products of the same type as this product, and when the determined confidence level is below a detection threshold, identifying the product as a suspect product.

According to other advantageous aspects of the invention, the method comprises one or more of the following characteristics, taken in isolation or in all technically possible combinations:

- when the product is identified as a suspect product, the verification phase further includes a step of ordering an implementation of additional tests of this product;

- additional tests are performed under different environmental conditions and/or using a different test adapter and/or test bench;

- the step of implementing additional tests comprises a modification of the reference values and/or of the corresponding detection threshold, when the identification of the product as a suspect product has not been confirmed by the additional tests ;

- the method further comprises a feedback phase comprising a step of updating the learning base following returns of products made after a period of exploitation of these products;

- the learning phase implemented for each type of product includes the following steps:

- collection of a test report relating to a product following the implementation of tests in relation to this product and collection of test reports relating to other products of the same type as this product;

- learning on the test data contained in the test reports collected and determination of the reference values by using all of these test data and by analyzing the interactions between these test data; And

- storage of reference values in the learning base.

- the learning phase further comprises a step of analyzing all the test reports relating to products of the same type, said analysis step comprising updating the learning base according to this analysis and being implemented independently of the other steps of the learning phase when these other steps have been implemented at least once;

- the step of learning on the test data from the collected test reports includes determining questionable behavior of the corresponding product type using anomaly detection method or outlier data detection method;

- the questionable behavior of a type of product is characterized by one or more elements chosen from the following group:

- deviation of test data from expected data under normal operating conditions;

- deviation of a test datum from a datum in a temperature environment;

- deviation of the calibrations following the evolution of the composition of the corresponding test bench;

- deviation of test data following changes in the composition of products of this type;

- Deviation of test data following the change of a test adapter of the corresponding test bench;

- each test data corresponds to one of the data types chosen from the group comprising:

- discrete result of a test;

- measurement performed by test equipment of the corresponding test bench;

- information relating to the date and time of the tests;

- information identifying the corresponding test bench and/or the product tested;

- information relating to the composition of the product tested;

- information relating to the composition of the corresponding test bench;

- information relating to the environmental operating conditions of the corresponding test bench and/or of a tested product;

- information relating to a self-test carried out by the corresponding test bench.

The invention also relates to a computer program product comprising software instructions which, when implemented by computer equipment, implement the method as defined above.

The invention also relates to an analysis device comprising technical means configured to implement the analysis method as defined above.

These characteristics and advantages of the invention will appear on reading the following description, given solely by way of non-limiting example, and made with reference to the appended drawings, in which:

- FIG. 1 is a schematic view of an example of a test bench, the test bench comprising a test computer;

- Figure 2 is a detailed schematic view of the test computer of Figure 1;

- FIG. 3 is an analysis device according to the invention, in particular capable of collecting and analyzing test reports produced by the test computer of FIG. 2; And

- FIG. 4 is a flowchart of an analysis method according to the invention, the method being implemented by the analysis device of FIG. 3.

In fact, FIG. 1 shows an example of a test bench 10 capable of producing test reports that can be used by the invention.

This test bench 10 makes it possible to analyze products intended to be delivered, for example, to a customer, as will be explained later.

Each product presents an assembly of electronic and/or mechanical and/or optical subassemblies.

Each product has a product type that defines the set of identical or similar products. The similarity of products of the same type can be defined according to different criteria. These criteria include, for example, the same functionality of the products, their interchangeability, the same functionality of their components, etc.

Thus, for example, products of the same type are identified by the same article code known as the "P/N" ( Part Number ) as well as by other information such as the indices of revision.

A unit product is identified by a serial number known as the "S/N" ( Serial Number ).

Furthermore, it is subsequently considered that the composition of each product is determined by a list of components in which each component is also identified by a unique identifier.

Referring to Figure 1, the test bench 10 includes a test computer 12, a plurality of test equipment 14-1 to 14-N and a test adapter 16.

The test adapter 16, also known by the English term “TUA” (from the English Test Unit Adapter ) has a mechanical and/or electronic and/or optical interface box used to adapt the test equipment 14-1 at 14-N to a product to be tested.

Test Adapter 16 is associated with a particular product type and can be identified by the P/N number associated with that product type. Furthermore, like the products, the test adapter 16 is uniquely identified by an S/N number.

According to a particular embodiment of the invention, the test adapter 16 makes it possible to carry out a self-test of the test bench 10 when no product is connected to it.

In practice, the test bench 10 includes a test adapter 16 for each type of product likely to be tested on the bench and a test adapter 16 to perform a self-test.

Each test equipment 14-1 to 14-N has a module performing one or more measurements, generating one or more test stimuli.

Each test equipment 14-1 to 14-N is controlled by the test computer 12 and is interfaced with the product by means of the test adapter 16.

For example, 14-1 to 14-N test equipment is a voltmeter, signal generator, power supply, oscilloscope, etc.

Test computer 12 is used to control all test equipment 14-1 to 14-N.

Test computer 12 presents a workstation or equivalent that hosts test programs, scripts, and test reports.

An example embodiment of the test computer 12 is shown in Figure 2.

Thus, with reference to this figure, the test computer 12 comprises a storage space 21, a test script extractor 22, a test program 23, hardware and software resources 24, an interface 25 and a report generator testing 26.

The storage space 21 is used in particular to store test scripts and test reports.

Each test script is associated with a product type and presents a list of individual tests needed to test products of that type. This list details how to set up each 14-1 through 14-N test set, what measurement to perform, and what the expected result is.

Each test script is structured for example in a plurality of chapters, each chapter dealing with a subject (analog and digital inputs/outputs, power supply energy conversion, external communication, the heart, etc.).

Each test script is for example identified by the P/N number of the type of product with which it is associated.

Each test report is for example associated with a test script and presents a computer file comprising test results carried out according to this test script for a given product.

The test script extractor 22 makes it possible to extract from the storage space a test script to be executed by the bench 10. To do this, the extractor 22 is for example capable of determining the type of product connected to the test adapter 16 and to extract from the storage space 21 the script associated with this type of product.

The test program 23 presents a test step sequencer capable of executing the test script extracted by the extractor 22.

The test program 23 is for example in the form of an executable file.

Hardware and software resources 24 present all the resources needed to run a script. Thus, these resources comprise for example a processor, a random access memory, connectors, programmable logic circuits, etc.

The interface 25 makes it possible to transmit to the equipment 14-1 to 14-N the commands determined by the test script being executed and to acquire measurements and any other information generated by the measuring equipment 14-1 to 14 -N when testing.

Finally, the test report generator 26 makes it possible to generate for each test script executed a test report on the basis of the data acquired by the interface 25.

The generator 26 also makes it possible to classify these reports in the storage space 21.

FIG. 3 presents an analysis device 30 according to the invention, making it possible to analyze the test reports generated by one or more test benches, such as the test bench 10 described above.

To do this, the analysis device 30 comprises a learning base 41, a processing module 42, a first communication interface 43 and a second communication interface 44.

The analysis device 30 thus takes the form, for example, of a server or a remote computer.

The first communication interface 43 has a communication interface with one or more test benches such as the test bench 10. In particular, this interface 43 makes it possible to connect the analysis device 30 to this or these test benches in order to to retrieve the test reports contained in the corresponding storage spaces 21 .

Advantageously, according to the invention, the first communication interface 43 also makes it possible to transmit to the connected test benches new test scripts following the implementation of an analysis method described below.

The second communication interface 44 allows an administrator to communicate with the analysis device 30 via a man-machine interface 31.

In the exemplary embodiment described, the man-machine interface 31 is in the form of a screen allowing the results of the implementation of the method to be viewed and a keyboard allowing the administrator to enter data in the computer. The man-machine interface 31 can be in any other suitable form, for example in the form of a web interface executable by a remote computer.

The processing module 42 makes it possible to process all the test reports acquired by the first communication interface 43 to generate reference values, as will be explained later.

This processing module 42 is for example in the form of one or more software programs that can be executed by suitable hardware resources.

The learning base 41 is in the form of a single or distributed storage space, making it possible to store all the reference values generated by the processing module 42.

The analysis method implemented by the analysis device 30 will now be explained with reference to FIG. 3 presenting a flowchart of its steps.

In particular, this method comprises a learning phase PA during which the processing module 42 performs learning of the behavior of the products by constituting the learning base 41, a verification phase PV during which the processing module 42 performs verifications of certain products tested in order to determine suspicious products and a feedback phase PR during which the processing module 42 updates the learning base 41 following returns of products made after a period of exploitation of these products.

It is clear that these phases are implemented after the execution of at least one series of tests for at least one type of product for example by the test bench 10. An example of a PT test phase executed by the test bench 10 in relation to a type of product is described below.

During an initial step 110 of the test phase PT, the analysis module 30 of the test computer 12 defines for a given type of product, a plurality of tests to be followed.

These tests are for example determined upstream from a plurality of parameters and are transcribed into a corresponding test script. This script is then saved in storage space 21.

During a following step 120 of the test phase PT, the analysis module 30 of the test computer 12 commands the execution of the tests defined during step 110 for the given type of product.

In particular, during this step 120, the analysis module 30 defines a sample of products on which the tests must be carried out.

Then, the extractor 22 extracts the corresponding test script from the storage space 21 and the test program 23 launches the tests defined by this script.

At each test step, at least one of the devices 14-1 to 14-N is configured in a way defined by the test script and then performs measurements relating to the product currently connected to the test adapter 16.

These measurements can represent discrete values or continuous values and are transmitted to the hardware and software resources 24 notably via the interface 25.

Thus, when it comes to discrete values, the resources 24 check via the test program 23 that the measurements are strictly identical to the expected result. If this is the case for all measurements, the test result is positive. Otherwise, the test result is negative.

When it comes to analog values, the resources 24 check via the test program 23 that the measurements obtained are for example between two expected limits or else below or above an expected limit. If this is the case for all measurements, the test result is positive. Otherwise, the test result is negative.

The expected results and the expected limits present values predetermined upstream for example during the modeling of the products. Failure to meet these expected results/limits signifies product failure.

During a following step 130 of the test phase PT, the resources 24 collect all the test data.

In particular, test data means any data relating to the tests executed by the test program 23 as well as to the conditions under which these tests are executed.

Thus, according to the described embodiment, each test data corresponds to one of the data types chosen from the group comprising:

- test result (positive or negative);

- measurement performed by one of the equipment 14-1 to 14-N;

- information about the date and time of the tests;

- information identifying the test bench 10 and/or the products tested;

- information relating to the composition of the products tested;

- information relating to the composition of the test bench 10;

- information relating to the operating conditions of the test bench 10 and/or of a tested product;

- information relating to a self-test carried out by the test bench 10.

In particular, the identification information comprises for example an identifier of the test bench 10, an identifier of the test adapter 16, an article code P/N or an article series S/N.

The information relating to the composition of the products tested includes in particular a list of components of these products, possibly with their identifier.

Similarly, the information relating to the composition of the test bench 10 includes in particular a list of components of this bench, possibly with identifiers of the test bench 10.

The information relating to the operating conditions of the test bench 10 and/or of a tested product includes, for example, the ambient temperature, the humidity level, etc.

The data collected in relation to a product is then converted into a test report presenting, for example, a JSON type object.

After the test phase PT and in the case of a positive test result, the analysis device 30 implements the learning phase PA for the type of product recently tested.

In particular, during an initial step 140 of this phase PA, the first communication interface 43 retrieves the test report in relation to the product recently tested, for example by the test bench 10 or by any other bench.

During the same step 140, the first communication surface 43 also retrieves the test reports of other products of the same type as said product, when these reports are available. These reports can come from the same test bench or from different test benches.

During a next step 150 of the learning phase PA, the processing module 42 performs learning on the test data from all the test reports collected.

This learning includes in particular the determination of a normal behavior of the products of the type considered at least on a part of the test script and a doubtful behavior on another part of the script, by using for example an algorithm of the Detection of Anomalies type (in English "Outlier Detection") or Detection of Aberrant Data (in English "Novelty Detection").

The questionable behavior of a product can be characterized by one or more elements chosen from the following group:

- Deviation of test data from expected data under normal operating conditions;

- Deviation of test data from data in temperature environment;

- deviation of the calibrations of the test environments over time, following for example the evolution of the composition of the test bench 10;

- deviation of test data over time, for example due to changes in product composition;

- deviation of test data following the change of the test adapter 16.

Then, the processing module 42 determines a set of reference values characterizing the normal behavior of the products of a type considered and the dubious behavior of these products, and stores these reference values in the learning base 41.

These reference values are determined by using all the test data and in particular by analyzing the interactions between these measurement data.

Thus, for example, an interaction or a link between a measurement noting an overconsumption of a product and a constant test datum a change in a component of this product, can be detected at this stage by the processing module 42 .

During a step 160, the processing module 42 retrieves all of the test reports relating to products of the same type and implements learning in a manner analogous to that explained with reference to step 150. After this learning, the processing module 42 updates the learning base 42 and in particular the reference values relating to this type of product.

During this step, the processing module 42 analyzes in particular self-test reports generated by the generator 26 when the test bench 10 was in self-test mode while having the test adapter 16 adapted to the aforementioned type of product.

As illustrated in the flowchart of FIG. 3, step 160 can be implemented independently of the other steps of the learning phase PA and of the test phase PT, when these steps have already been implemented. at least once.

Thus, in practice, step 160 is implemented periodically, for example with a predetermined periodicity.

During a step 170 of the PV verification phase, the processing module 42 chooses a product to be verified, that is to say a tested product whose quality must be verified.

This choice is made for example following the introduction by an administrator of a P/N or S/N number of a product via the man-machine interface 31.

Then, during the same step, the processing module 42 retrieves via the first communication interface 43 the test report relating to the selected product.

During a next step 180 of the PV verification phase, the processing module 42 determines a level of confidence in the quality of the product selected from the test data contained in its test report.

In particular, the confidence level is determined by comparing these test data with reference values from the learning base 41 and relating to the same type of product as the selected product.

The level of confidence thus makes it possible to conclude whether the product has the normal behavior as determined during the PA learning phase or the doubtful behavior as determined during the PA learning phase.

Thus, for example, when this level of confidence is greater than a predetermined detection threshold, the behavior of the product is considered normal. Otherwise, the behavior is considered questionable.

When the behavior of the product is considered doubtful, the next step 190 of the test phase PT is then implemented.

During this step 190, the analysis module 30 commands the execution of additional tests of the product whose behavior is considered doubtful.

These additional tests can for example be determined by the administrator via the man-machine interface 31 following an analysis of the test data generated during the implementation of the previous tests.

Optionally, the additional tests are identical to the previous tests and make it possible to determine the repeatability and the stability of the measurements previously acquired.

In addition, it is possible to carry out the additional tests by choosing a different test adapter 16 and/or by coupling the test bench 10 with a climatic chamber.

In the product considered suspect, it is also possible to swap one or more components with the corresponding components of a reference product whose behavior was considered normal during the same tests.

When the additional tests confirm the questionable behavior of the product, the type of this product is then considered as defective and is removed from the production cycle.

Otherwise, this type of product continues its production cycle. In this case, the administrator has the possibility of modifying the detection threshold and/or at least certain reference values via, for example, the man-machine interface 31 in order to reduce the number of false alarms.

During an optional step 200 of the PV verification phase, the processing module 42 performs additional analyses.

Thus, for example, during this step 200, the processing module 42:

- identifies products that have not been verified; and or

- changes the detection threshold and/or reference values; and or

- obtains performance and quality indicators in order to generate a monitoring table.

At the end of this step, step 160 of the learning phase PA can be implemented again in order to update the learning base.

During an optional step 210, the processing module 42 generates summary data relating to the test results of all the products tested. This data can be presented for example in the form of a table, a diagram or any other suitable form in order to visualize these results.

The PR loopback phase is implemented when a product has been returned after a period of actual exploitation. This period is for example limited by a predetermined threshold expressed in number of years of operation, months of operation, hours of operation, etc. The product is returned following, for example, a breakdown that occurred during this operating period.

During a step 220 of this feedback phase PR, the analysis module 30 implements the tests by executing the script corresponding to the type of this product.

Then, during step 230, the analysis module 30 again compares the test data collected with the reference values from the learning base.

According to this comparison, the analysis module 30 modifies during the following step 240 the reference values by carrying out a new learning process and/or the detection threshold.

If the reference values are modified, the analysis module 30 updates the learning base 41.

In addition, during this step, the administrator modifies via the man-machine interface 31 the test script corresponding to the type of this product, for example by modifying tests already defined in the script or by adding other tests.

It can then be seen that the present invention has a certain number of advantages.

Indeed, the implementation of learning on test data makes it possible to detect suspicious products not by comparing each test measurement with a threshold but by analyzing all the measurements and more generally, all the data of test. This then makes it possible to detect interactions between several test data and the consequences of these interactions.

This makes it possible in particular to detect a deviation of a measurement following a change in the composition of the test bench, its surroundings and in the composition of the product itself.

In addition, the method according to the invention makes it possible to detect defective products during their manufacture and before their commissioning while taking into account returns on products already in service.

This design reduces the number of false alarms during testing and improves the detection of faulty products.

Claims (12)

A method of analyzing the quality of products tested using one or more test benches (10), the or each test bench (10) being configured to implement tests on a product and to generate test following the implementation of these tests, the test data relating to each product being stored in the form of a test report relating to this product;
the method comprising a learning phase (PA) and a verification phase (PV);
the learning phase (PA) being implemented periodically and for each new implementation, comprising the determination of a normal behavior of each type of product from the test reports relating to each product of this type and the implementation updating of a learning base comprising reference values defining this normal behavior of this type of product;
the verification phase (PV) comprising the following steps:
- among the products tested, selection (170) of a product to be checked and recovery of a test report relating to this product;
- determination (180) of a level of confidence in the quality of the product by comparing test data from the test report corresponding to this product with reference values from the learning base in relation to the products of the same type as this product, and when the determined confidence level is below a detection threshold, identifying the product as a suspect product. Method according to claim 1, in which, when the product is identified as a suspect product, the verification phase (PV) further comprises a step (190) of ordering an implementation of additional tests of this product. Method according to Claim 2, in which the additional tests are carried out under different environmental conditions and/or using a different test adapter (16) and/or a different test bench (10). Method according to claim 2 or 3, in which the step (190) of carrying out additional tests comprises a modification of the reference values and/or of the corresponding detection threshold, when the identification of the product as a suspect product has not been confirmed by further testing. Method according to any one of the preceding claims, further comprising a feedback phase (PR) comprising a step (240) of updating the learning base following product returns made after an operating period of these products. Method according to any one of the preceding claims, in which the learning phase (PA) implemented for each type of product comprises the following steps:
- collection (130) of a test report relating to a product following the implementation of tests in relation to this product and collection of test reports relating to other products of the same type as this product;
- learning (140) on the test data contained in the collected test reports and determination of the reference values by using all of these test data and by analyzing interactions between these test data; And
- storage of reference values in the learning base. Method according to claim 6, in which the learning phase (PA) further comprises a step of analyzing (160) all of the test reports relating to products of the same type, said step of analyzing ( 160) comprising the updating of the learning base according to this analysis and being implemented independently of the other steps of the learning phase (PA) when these other steps have been implemented at least once. A method according to any of claims 6 to 7, wherein the step of learning (140) on the test data from the collected test reports comprises determining questionable behavior of the corresponding product type using a anomaly detection method or an outlier detection method. Method according to claim 8, in which the questionable behavior of a type of product is characterized by one or more elements chosen from the following group:
- deviation of a test datum from an expected datum under normal operating conditions;
- deviation of a test datum from a datum in a temperature environment;
- deviation of the calibrations following the evolution of the composition of the corresponding test bench (10);
- deviation of test data following changes in the composition of products of this type;
- deviation of a test datum following the change of a test adapter (16) of the corresponding test bench (10). A method according to any preceding claim, wherein each test data corresponds to one of the data types selected from the group comprising:
- discrete result of a test;
- measurement performed by test equipment (14-1, …, 14-N) of the corresponding test bench (10);
- information relating to the date and time of the tests;
- information identifying the corresponding test bench (10) and/or the product tested;
- information relating to the composition of the product tested;
- information relating to the composition of the corresponding test bench (10);
- information relating to the environmental operating conditions of the corresponding test bench (10) and/or of a tested product;
- information relating to a self-test carried out by the corresponding test bench (10). Computer program product comprising software instructions which, when implemented by computer equipment, implement the method according to any of the preceding claims. Analysis device comprising technical means configured to implement the analysis method according to any one of Claims 1 to 10.