CN114266737A

CN114266737A - Defect detection method, device, equipment and storage medium

Info

Publication number: CN114266737A
Application number: CN202111498951.2A
Authority: CN
Inventors: 李晨阳; 陈想; 罗斌; 陈列; 汪彪
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2021-12-09
Filing date: 2021-12-09
Publication date: 2022-04-01

Abstract

The application provides a defect detection method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring a plurality of first images obtained by shooting a detected object in a plurality of light source directions; performing photometric stereo imaging processing on the plurality of first images to obtain a second image corresponding to the detected object; inputting the plurality of first images into a first defect detection model to obtain a first defect detection result; inputting the second image into a second defect detection model to obtain a second defect detection result; and determining the final defect detection result of the detected object according to the first defect detection result and the second defect detection result. In the scheme provided by the embodiment of the invention, the visible light source and the detected object are taken as the battery as an example, and whether the surface of the battery has defects or not can be accurately detected by combining a plurality of visible light images and two types of luminosity stereo images which are collected in the direction of multiple light sources.

Description

Defect detection method, device, equipment and storage medium

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a defect detection method, apparatus, device, and storage medium.

Background

Industrial defect detection is a challenging task, and the application scenarios thereof cover various application fields such as heavy industry and light industry, for example, industries such as steel, electronics, textile and the like have the need of detecting surface defects of products. Under the task of surface defect detection, the surface defects have different shapes and different positions, so that the detection difficulty is high.

Conventional surface defect detection schemes typically employ visible light imaging of the inspected object and use deep neural networks for defect detection. Different defects have different imaging effects under visible light, so the final detection effect is greatly influenced by the imaging scheme effect.

Disclosure of Invention

The embodiment of the invention provides a defect detection method, a defect detection device, defect detection equipment and a storage medium, which are used for improving the accuracy of a defect detection result.

In a first aspect, an embodiment of the present invention provides a defect detection method, where the method includes:

acquiring a plurality of first images obtained by shooting a detected object in a plurality of light source directions;

performing photometric stereo imaging processing on the plurality of first images to obtain a second image corresponding to the detected object;

inputting the plurality of first images into a first defect detection model to obtain a first defect detection result;

inputting the second image into a second defect detection model to obtain a second defect detection result;

and determining the final defect detection result of the detected object according to the first defect detection result and the second defect detection result.

In a second aspect, an embodiment of the present invention provides a defect detection apparatus, where the apparatus includes:

the acquisition module is used for acquiring a plurality of first images obtained by shooting the detected object in a plurality of light source directions; performing photometric stereo imaging processing on the plurality of first images to obtain a second image corresponding to the detected object;

the detection module is used for inputting the first images into a first defect detection model to obtain a first defect detection result; inputting the second image into a second defect detection model to obtain a second defect detection result;

and the determining module is used for determining the final defect detection result of the detected object according to the first defect detection result and the second defect detection result.

In a third aspect, an embodiment of the present invention provides an electronic device, including: a memory, a processor, a communication interface; wherein the memory has stored thereon executable code which, when executed by the processor, causes the processor to implement at least the defect detection method of the first aspect.

In a fourth aspect, an embodiment of the present invention provides a non-transitory machine-readable storage medium having stored thereon executable code, which when executed by a processor of an electronic device, causes the processor to implement at least the defect detection method according to the first aspect.

In a fifth aspect, an embodiment of the present invention provides a defect detection method, where the method includes:

receiving a request for calling a defect detection service interface by user equipment, wherein the request comprises a plurality of first images obtained by shooting a detected object in a plurality of light source directions;

executing the following steps by utilizing the processing resource corresponding to the defect detection service interface:

In a sixth aspect, an embodiment of the present invention provides a defect detection method, where the method includes:

acquiring a plurality of first images obtained by shooting a detected battery in a plurality of light source directions;

performing photometric stereo imaging processing on the plurality of first images to obtain a second image corresponding to the detected battery;

inputting the first images into a first defect detection model to obtain a first surface defect detection result, wherein the first surface defect detection result is used for reflecting whether the surface of the detected battery has defects;

inputting the second image into a second defect detection model to obtain a second surface defect detection result, wherein the second surface defect detection result is used for reflecting whether the surface of the detected battery has defects or not;

and determining the final surface defect detection result of the detected battery according to the first surface defect detection result and the second surface defect detection result.

In the embodiment of the present invention, in the task of detecting the surface defects of the product, for a detected object, multiple first images may be obtained by shooting the detected object in multiple light source directions, for example, multiple visible light images may be obtained under a visible light source, and then, photometric stereo imaging processing may be performed on the multiple first images to obtain a second image, for example, a normal vector diagram, corresponding to the detected object. And inputting the plurality of first images into a first defect detection model to obtain a first defect detection result, wherein the first defect detection model is obtained by training sample images corresponding to the light source. And inputting the second image into a second defect detection model to obtain a second defect detection result, wherein the second defect detection model is obtained by training a sample image obtained by photometric stereo imaging processing. And finally, determining the final defect detection result of the detected object according to the first defect detection result and the second defect detection result. In the scheme provided by the embodiment of the invention, the visible light source is taken as an example, whether the detected object has surface defects or not can be detected by combining the visible light image and the luminosity three-dimensional image, and a more accurate detection result is obtained.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.

Fig. 1 is a flowchart of a defect detection method according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating different light source configurations according to an embodiment of the present invention;

FIG. 3 is a schematic diagram illustrating an application of a defect detection method according to an embodiment of the present invention;

FIG. 4 is a flowchart of a model training method according to an embodiment of the present invention;

FIG. 5a is a flowchart of a model training method according to an embodiment of the present invention;

FIG. 5b is a schematic diagram of a model training process according to an embodiment of the present invention;

FIG. 6a is a flowchart of a model training method according to an embodiment of the present invention;

FIG. 6b is a schematic diagram of a model training process according to an embodiment of the present invention;

FIG. 7 is a schematic diagram illustrating an application of a defect detection method according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a defect detection apparatus according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an electronic device corresponding to the defect detection apparatus provided in the embodiment shown in fig. 8.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Some embodiments of the invention are described in detail below with reference to the accompanying drawings. The features of the embodiments and examples described below may be combined with each other without conflict between the embodiments. In addition, the sequence of steps in each method embodiment described below is only an example and is not strictly limited.

The defect detection method provided by the embodiment of the invention can be executed by an electronic device, the electronic device can be a server or a user terminal, and the server can be a physical server or a virtual server (virtual machine) of a cloud.

The scheme provided by the embodiment of the invention can be suitable for detecting the defects of the surface of the product, such as scratches, pits, bulges and the like on the surface of the lithium battery. In a conventional inspection scheme, sample images are obtained by visible light imaging of a large number of products, including positive sample images having no defects and negative sample images having defects. Then, training of the neural network model is performed based on the taken sample image, and a model for realizing defect detection (referred to as a defect detection model) is obtained.

However, the conventional model training scheme has some problems as follows:

first, there are differences in the imaging effect of different defects under visible light. The final effect of the computer vision based algorithm scheme is greatly affected by the imaging scheme effect. Because the surface defects have different shapes and heights, clear imaging cannot be performed on all the defects under a single illumination direction, multiple images are often acquired by imaging in multiple light source directions, and multiple images are synthesized for defect detection.

Second, the defect detection rate is low with a small sample size. Some serious defects in the industrial field are difficult to collect a large number of samples, a small number of defect samples and a large number of normal samples are mixed and trained by using a conventional neural network model training method, the neural network model cannot fully learn the defect samples, and finally the defect detection rate is low.

Thirdly, the overall training efficiency of the neural network model is low. Similarly, for the defect of small sample size, the conventional training method needs to be mixed with a large amount of normal samples for training, resulting in a large number of whole samples and finally a long training time. Therefore, it would be very valuable to combine the imaging effects of multiple light source directions while using fewer samples for training to improve the defect detection rate and training efficiency of the model.

Based on the above, the embodiment of the invention provides an object surface defect detection scheme based on visible light and luminosity three-dimensional imaging, and the detection rate of defects is improved by combining the luminosity three-dimensional imaging and the visible light imaging. Meanwhile, a sample balance iterative training mode is used, and the detection rate and the training efficiency of the defect detection model are improved. The visible light image imaging based on the visible light source is only an example, and actually, other types of light sources such as infrared light may be used.

The following examples are provided to illustrate the implementation of this embodiment.

Fig. 1 is a flowchart of a defect detection method according to an embodiment of the present invention, as shown in fig. 1, the method includes the following steps:

101. a plurality of first images obtained by shooting an object to be detected in a plurality of light source directions are acquired.

The object to be detected is an object whose surface is to be detected for defects (flaws), such as a lithium battery, a package can, a component, and the like. For example, in a product production line, after an object to be detected is produced, image acquisition is performed on the object to be detected in different light source directions to obtain the plurality of first images, wherein one first image corresponds to one light source direction.

In practical application, the adopted light source may be a visible light source, an infrared light source, or the like, so that the collected plurality of first images may be a plurality of visible light images or a plurality of infrared images.

As shown in fig. 2, different light source directions refer to the same light source being placed in different orientations with respect to the inspected object. In practical application, a camera can be placed at a certain fixed shooting position, the detected object is kept still, light sources placed in different directions are switched, and each time the light sources are switched, the camera collects a first image. Switching the light sources in different directions means that only one light source in one direction emits light and the other light sources in the other directions are turned off at the same time. Supposing that at the time of T1, the light source A is controlled to emit light, other light sources are turned off, and at the moment, the camera shoots the detected object once to obtain a first image a; then, at the time of T2, controlling the light source B to emit light, turning off the other light sources, and at this time, the camera takes a picture of the detected object once to obtain a first image B; then, at time T3, the light source C is controlled to emit light, and the other light sources are turned off, and at this time, the camera takes a single shot of the object to be detected, and a first image C is obtained.

In the embodiment of the present invention, the use of multi-directional lighting (i.e. multiple light source directions) mainly has the following functions: (1) all the characteristics of the surface of the detected object are usually difficult to clearly observe in a single light source direction, and the detected object is respectively subjected to image acquisition in a plurality of light source directions, so that all the defect characteristics of the surface can be ensured to be observed; (2) photometric stereo imaging requires at least 3 images acquired from light source directions, so another effect of capturing the object under inspection from multiple light source directions is to provide input images to a photometric stereo imaging algorithm to generate a photometric stereo image.

In the process of acquiring the plurality of first images, the detected object and the camera are kept still all the time, and only the direction of the light source is changed, so that the acquired plurality of first images are aligned.

In practical application, a plurality of light source directions can be reasonably designed according to the morphological characteristics of different types of detected objects. For example, for a cubic detected object, a plurality of light source directions can be set to be respectively located in six directions, namely, up, down, left, right, front and back directions of the detected object; for another example, for a cylindrical detected object, a plurality of light source directions may be respectively located in four directions, i.e., up, down, left, and right directions of the detected object.

102. And performing photometric stereo imaging processing on the plurality of first images to obtain a second image corresponding to the detected object.

Photometric stereo imaging is a method for reconstructing the surface of an object, and can reconstruct the normal vector of the surface of the object and the reflectivity of different surface points of the object by using the photometric stereo imaging method, and further, can calculate the relative depth information of different surface points according to the surface normal vector information, so that a normal vector image, an albedo image and a relative depth image of the object can be obtained by using the photometric stereo imaging method. In this embodiment, the plurality of first images are used as input images for photometric stereo imaging processing, and a second image corresponding to the detected object is obtained by performing photometric stereo imaging on the input images, where the second image may be a normal vector image, an albedo image, or a relative depth map.

103. And inputting the plurality of first images into the first defect detection model to obtain a first defect detection result, and inputting the second image into the second defect detection model to obtain a second defect detection result.

The first defect detection model is trained by using a sample image corresponding to the light source (such as a visible light source and an infrared light source), and the second defect detection model is trained by using a sample image obtained through photometric stereo imaging processing. Taking the adopted light source as a visible light source as an example, that is, the sample images adopted for training the first defect detection model are all visible light images, and the sample images adopted for training the second defect detection model are all luminosity stereo images (i.e. images generated by a luminosity stereo imaging method, such as a normal vector diagram and an albedo diagram). The two defect detection models may be trained to identify whether there are defects in the input image and where the defects are located.

After a plurality of first images corresponding to the detected object are obtained, the plurality of first images are respectively input into the trained first defect detection model, the first defect detection model can output a detection result corresponding to each first image, and the detection result is used for indicating whether a defect exists in the corresponding first image and the position of the defect. Similarly, a second image generated based on the plurality of first images is input into a trained second defect detection model, the second defect detection model outputs a detection result corresponding to the second image, and the detection result is used for indicating whether a defect exists in the corresponding second image and the position of the defect.

104. And determining the final defect detection result of the detected object according to the first defect detection result and the second defect detection result.

Optionally, the first defect detection result and the second defect detection result may be fused to finally determine the defect detection result of the detected object.

The first defect detection model outputs defect detection results corresponding to the first images, and therefore the first defect detection results include defect detection results corresponding to the first images.

Optionally, the fusion operation may be: and operation, namely determining that the detected object has the defect if the first defect detection result and the second defect detection result both indicate that the detected object has the defect.

If the defect detection result corresponding to at least one first image in the plurality of first images indicates that the detected object has defects, determining that the first defect detection result indicates that the detected object has defects.

In practical application, in a production factory of a certain product, a detection system for detecting surface defects of the product may be provided, as shown in fig. 3, the detection system may include a detection platform and a management device, the management device may be a server or a PC, a fixing member for fixing a detected object may be provided on the detection platform, a camera for shooting the detected object may be further provided, and light sources may be provided in different directions of the fixing member according to actual requirements. The management device can be in communication connection with the camera and each light source and used for controlling the switching of the light sources and controlling the camera to shoot. The management equipment acquires the plurality of first images respectively shot by the camera, and carries out the processing in the steps to finally obtain a defect detection result, namely marking the corresponding defect position in the image with the detected defect. The process is illustrated schematically in figure 3.

In the above-mentioned scheme provided by the embodiment of the present invention, taking the visible light source as an example, the visible light image and the luminosity stereo image can be combined to detect whether the detected object has surface defects, and compared with the detection of the object surface defects by using only the visible light image acquired in a single illumination direction, the detection result is more accurate. Because the photometric stereo imaging can reconstruct the surface of an object, especially for some defects which can cause the depth of the surface of the object to change obviously, the defects on the surface of the object can be detected more accurately by combining the photometric stereo image. In addition, based on the images collected under the directions of the plurality of light sources, the characteristics of the surface of the object can be observed more comprehensively, and the defects on the surface of the object can be detected more accurately.

Taking the detected object as a battery as an example, in practical application, after the battery is produced, quality detection needs to be performed on the battery to determine whether the battery has defects such as scratches, pits, bulges, and the like on the surface. At this time, for a detected battery, the detection process includes:

inputting a plurality of first images into a first defect detection model to obtain a first surface defect detection result, wherein the first surface defect detection result is used for reflecting whether the surface of the detected battery has defects;

Wherein if the first surface defect detection result indicates that the surface of the inspected battery is defective, and the second surface defect detection result also indicates that the surface of the inspected battery is defective, determining that the final surface defect detection result of the inspected battery is: and has defects.

The use of the first defect detection model and the second defect detection model is described above, and the following training process of the two defect detection models is described below.

Fig. 4 is a flowchart of a model training method according to an embodiment of the present invention, and as shown in fig. 4, the method includes the following steps:

401. acquiring a first positive example sample set and a first negative example sample set for training a first defect detection model, wherein the first positive example sample set comprises a plurality of first sample images of a defect-free first object taken under a plurality of light source directions, the first negative example sample set comprises a plurality of second sample images of a defect second object taken under the plurality of light source directions, and the first object and the second object are any one of a plurality of collected defect-free objects and defective objects respectively.

That is, the first positive example sample set is composed of sample images taken in different light source directions for each of a plurality of objects that are not defective (as positive example sample objects), and the first negative example sample set is composed of sample images taken in different light source directions for each of a plurality of objects that are defective (as negative example sample objects).

402. And acquiring a second positive sample set and a second negative sample set for training a second defect detection model, wherein the second positive sample set comprises a third sample image obtained by performing photometric stereo imaging on the plurality of first sample images, and the second negative sample set comprises a fourth sample image obtained by performing photometric stereo imaging on the plurality of second sample images.

403. A first defect detection model is trained using the first positive example set and the first negative example set.

404. A second defect detection model is trained using the second set of positive examples samples and the second set of negative examples samples.

It is understood that the first defect detection model and the second defect detection model are both used for detecting the defect position existing in the input image, and therefore, for each sample image contained in the first and second normative sample sets, the supervision information is a "defect-free" class label; for each of the sample images contained in the first negative example sample set and the second negative example sample set, the supervisory information is a "defect" category label and a specific defect location, which is usually identified by a rectangular box.

As noted above, some of the more serious deficiencies, it is often difficult to collect a large number of sample objects, resulting in a smaller number of sample images in the negative example sample set, typically much smaller than the number of sample images contained in the corresponding positive example sample set. In order to solve the problems of low defect detection rate and low training efficiency caused by the fact that a training set composed of all positive example sample images and all negative example sample images is directly used for model training in the prior art, the embodiment of the invention provides a sample balance iterative training method.

In summary, the sample equalization iterative training method mainly comprises two stages: the first stage is an initial training stage, and the initial version of the model is obtained by training all negative example sample images and a small number of positive example sample images; the second stage is an iterative training stage, in which the current obtained model is used for reasoning the full quantity of the positive example sample images to obtain false positive example sample images, and the false positive example sample images are added into the positive example sample set used in the first stage to construct a new training data set to train a new model.

Because the models to be trained in the embodiment of the invention comprise the first defect detection model and the second defect detection model, the two defect detection models are both trained by adopting the sample equalization iteration method.

The following describes the training process using the above sample equalization iterative method for the two defect detection models.

Fig. 5a is a flowchart of a model training method according to an embodiment of the present invention, and as shown in fig. 5a, the method includes the following steps:

501. acquiring a first positive example sample set and a first negative example sample set for training a first defect detection model, wherein the first positive example sample set comprises a plurality of first sample images of a defect-free first object taken under a plurality of light source directions, the first negative example sample set comprises a plurality of second sample images of a defect second object taken under the plurality of light source directions, and the first object and the second object are any one of a plurality of collected defect-free objects and defective objects respectively.

In an alternative embodiment, obtaining a first negative sample set for training a first defect detection model includes:

acquiring a plurality of second sample images of a second object taken in a plurality of light source directions;

determining at least one second sample image from the plurality of second sample images according to the defect morphological characteristics corresponding to the second object, wherein the light source direction corresponding to the at least one second sample image corresponds to the defect morphological characteristics;

adding the at least one second sample image to the first negative sample set.

Assuming that N light source directions are preset, where N is greater than 1, M light source directions corresponding to the defect morphological features of the second object may be selected from the defect morphological features, where M is less than N, and M second sample images corresponding to the M light source directions in the acquired N second sample images are added to the first negative sample set.

However, for the input of photometric stereo imaging, one photometric stereo image corresponding to the second object can still be generated using the N second sample images.

In practical applications, the M light source directions corresponding to the defect of the second object may be manually set based on the morphological characteristics of the defect. The morphological feature is embodied as, for example, the location where the defect occurs, the apparent shape of the second object, the shape of the defect, the category of the defect, and so on.

502. According to the number of samples of the first negative sample set, a first positive sample subset matching the number of samples is sampled from the first positive sample set.

503. The first defect detection model is initially trained using a first positive example sample subset and a first negative example sample set.

504. And testing the first defect detection model obtained by the preliminary training by using the first positive example sample set to obtain a second positive example sample subset consisting of the false detection sample images.

505. The second positive example sample subset is updated to the first positive example sample subset, and the first defect detection model is retrained using the updated first positive example sample subset and the first negative example sample set.

In this embodiment, in order to train the first defect detection model, first, a total amount of a first positive example sample set and a first negative example sample set are collected, where the first positive example sample set is composed of sample images of a plurality of defect-free sample objects respectively collected in the directions of a plurality of light sources; the first negative example sample set is composed of sample images of a plurality of defective sample objects respectively acquired under a plurality of light source directions.

The process of training the first defect detection model includes two stages.

The first phase is an initial training phase in which first a first training data set required for this phase is constructed: and sampling a first positive sample subset matched with the sample number from the first positive sample set according to the sample number of the first negative sample set, wherein the first positive sample subset and the first negative sample set form a first training data set of the stage, namely, the first positive sample subset and the first negative sample set are used for carrying out primary training on the first defect detection model. A defective sample object is referred to as a defective sample, and a non-defective sample object is referred to as a normal sample. In practical applications, the number of defect samples that can be collected is small, so the training at this stage is performed using the first negative example sample set, which is the sample image corresponding to the entire number of defect samples. And normal samples are usually more than the number of defect samples, so the normal samples can be sampled in a random sampling mode, and the number of the normal samples in the training data set at this stage is controlled to be matched with the number of the defect samples, for example: the number of normal samples is 2-3 times the number of defective samples. That is to say, the number of normal samples in the first training data set used in this stage is not much greater than the number of defect samples, so as to ensure that the first defect detection model can sufficiently learn the defect samples, and meanwhile, the number of samples included in the first training data set used in this stage constructed in this way is much smaller than that of the originally collected data set (i.e., the data set formed by the first positive sample set and the first negative sample set), so that the model training time can be shortened, and the training efficiency can be improved.

It should be noted that the number of samples in the above-mentioned "sampling the first positive example subset matching the number of samples from the first negative example sample set according to the number of samples in the first positive example sample set" should be understood as the number of the above-mentioned defective samples, i.e., defective sample objects. In practice, a defective sample object may correspond to only one photometric stereo image, i.e. to one image in the first negative example set, and therefore, in this case, the sample number may also be considered as the number of images contained in the first negative example set. In the process of sampling the first positive example sample subset in the first positive example sample set, the sample images corresponding to a target number of normal samples are sampled in the first positive example sample set, where the target number is, for example, 2-3 times of the number of the samples, the normal samples are defect-free sample objects, since in the first normal sample set, one object corresponds to a plurality of sample images (sample images taken respectively from a plurality of light source directions), in practice, therefore, the number of images contained in the first positive example sample set is more than 2-3 times the number of images contained in the first negative example sample set, e.g., if the target number is 2 times the number of the above samples, the light source direction is 4, and the number of the defect samples, i.e., the images, included in the first negative sample set is 30, the number of the images included in the first positive sample set will be: 2 x 4 x 30 ═ 240.

After the first stage of training of the first defect detection model is completed by the first training data set composed of the first positive example sample subset and the first negative example sample set, it is assumed that the first defect detection model obtained at this time is represented as model M1. Thereafter, a second stage of training is performed.

The second phase is an iterative training phase. Due to the fact that the number of normal samples in the first training data set used in the first stage is greatly reduced, the learning of the model to the background is possibly insufficient, the false alarm rate is high, the iterative training stage aims to restrain the false alarm of the model, and the accuracy of the model is improved. Specifically, the currently obtained model M1 is used to predict normal samples that are not added to the first training data set, so as to obtain normal samples that are falsely reported as defective by the model M1, and the sample images corresponding to these falsely detected normal samples are added to the first training data set, so as to construct a new round of second training data set. Then, the newly constructed second training data set is used to train the currently obtained model M1 again, and a new model M2 is obtained through training. And finally, predicting all normal samples which are not added with the second training data set by using the updated model M2, stopping iteration if the false alarm rate is lower than a preset threshold, and otherwise, continuing the iteration process until the false alarm rate is lower than the preset threshold.

As can be seen, in step 504, the first defect detection model obtained by the preliminary training is tested by using the first positive example sample set to obtain the second positive example sample subset composed of the false-detected sample images, wherein the model M1 may be tested by using the sample images corresponding to all the normal samples in the first positive example sample set, or the model M1 may be tested by using only the sample images corresponding to the normal samples in the first positive example sample set that are not added to the first training data set. If at least one sample image of a certain normal sample is considered to contain defect information by the model M1, the normal sample is considered to be a false-reported sample, a plurality of sample images corresponding to the normal sample in the first positive sample set are determined to be false-detected sample images, and the false-detected sample images are added into the second positive sample subset. After testing of a plurality of normal samples, a second positive example sample subset formed by a plurality of sample images corresponding to all the normal samples detected by mistake can be obtained, the second positive example sample subset is updated to the first positive example sample subset, and the model M1 is retrained by using a second training data set formed by the updated first positive example sample subset and the first negative example sample set, so that the model M2 is obtained.

The training process of the first defect detection model described above can refer to the schematic illustration in fig. 5 b.

Fig. 6a is a flowchart of a model training method according to an embodiment of the present invention, and as shown in fig. 6a, the method includes the following steps:

601. and acquiring a second positive sample set and a second negative sample set for training a second defect detection model, wherein the second positive sample set comprises a third sample image obtained by performing photometric stereo imaging on the plurality of first sample images, and the second negative sample set comprises a fourth sample image obtained by performing photometric stereo imaging on the plurality of second sample images.

The process of obtaining the second positive example sample set and the second negative example sample set refers to the related description in the foregoing embodiments. Each sample image used in the training process of the second defect detection model is a photometric stereo image, such as a normal vector diagram.

602. And sampling a third positive sample subset which is matched with the sample number from the second positive sample set according to the sample number of the second negative sample set.

603. The second defect detection model is initially trained using the third positive example sample subset and the second negative example sample set.

604. And testing the second defect detection model obtained by the preliminary training by using a second positive example sample set to obtain a fourth positive example sample subset consisting of the false detection sample images.

605. And updating the third positive example sample subset with the fourth positive example sample subset, and retraining the second defect detection model by using the updated third positive example sample subset and the second negative example sample set.

In this embodiment, a training process of the second defect detection model is also performed by using a sample equalization iterative training method, and includes the two training stages. And will not be described in detail herein. The training process of the second defect detection model may refer to the illustration in fig. 6 b.

In conclusion, the first defect detection model and the second defect detection model are trained in a sample balance iterative training mode, so that the trained models can have good detection performance, and the training efficiency is improved.

As described above, the defect detection method provided by the present invention can be executed in the cloud, and a plurality of computing nodes may be deployed in the cloud, and each computing node has processing resources such as computation and storage. In the cloud, a plurality of computing nodes may be organized to provide a service, and of course, one computing node may also provide one or more services. The way that the cloud provides the service may be to provide a service interface to the outside, and the user calls the service interface to use the corresponding service. The service Interface includes Software Development Kit (SDK), Application Programming Interface (API), and other forms.

According to the scheme provided by the embodiment of the invention, the cloud end can provide a service interface of the defect detection service, and a user calls the service interface through user equipment to trigger a calling request to the cloud end, wherein the request comprises a plurality of first images obtained by shooting the detected object in a plurality of light source directions. The cloud determines the compute nodes that respond to the request, and performs the following steps using processing resources in the compute nodes:

In addition, the model training task described in the foregoing embodiment may also be completed by the computing node in the cloud.

For ease of understanding, the description is illustrative with reference to FIG. 7. The user may invoke a defect detection service interface (API interface in the figure) through user device E1 illustrated in fig. 7, through which a service request containing a plurality of first images is uploaded. In the cloud, as shown in the figure, besides a plurality of computing nodes, a management node E2 running a management and control service is deployed, after receiving a service request sent by user equipment E1, the management node E2 determines a computing node E3 responding to the service request, after receiving a plurality of first images, the computing node E3 executes a photometric stereo imaging step and calls a first defect detection model and a second defect detection model to complete defect detection, and the like, and finally outputs a detection result. The detailed implementation process refers to the description in the foregoing embodiments, and is not repeated herein. Thereafter, the computing node E3 sends the final inspection result to the user equipment E1, and the user equipment E1 displays the final inspection result.

The defect detection apparatus of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these means can each be constructed using commercially available hardware components and by performing the steps taught in this disclosure.

Fig. 8 is a schematic structural diagram of a defect detection apparatus according to an embodiment of the present invention, as shown in fig. 8, the apparatus includes: the device comprises an acquisition module 11, a detection module 12 and a determination module 13.

The acquisition module 11 is configured to acquire a plurality of first images obtained by shooting a detected object in a plurality of light source directions; and performing photometric stereo imaging processing on the plurality of first images to obtain a second image corresponding to the detected object.

The detection module 12 is configured to input the plurality of first images into a first defect detection model to obtain a first defect detection result; and inputting the second image into a second defect detection model to obtain a second defect detection result.

And the determining module 13 is configured to determine a final defect detection result of the detected object according to the first defect detection result and the second defect detection result.

Optionally, the determining module 13 is specifically configured to: and if the first defect detection result and the second defect detection result both indicate that the detected object has defects, determining that the detected object has defects.

Optionally, the apparatus further comprises: a training module, configured to obtain a first positive example sample set and a first negative example sample set for training the first defect detection model; acquiring a second positive sample set and a second negative sample set for training the second defect detection model; training the first defect detection model using the first positive case sample set and the first negative case sample set; training the second defect detection model using the second positive case sample set and the second negative case sample set; the first positive example sample set comprises a plurality of first sample images of a defect-free first object photographed in a plurality of light source directions, the first negative example sample set comprises a plurality of second sample images of a defect second object photographed in the plurality of light source directions, the second positive example sample set comprises a third sample image obtained by performing photometric stereo imaging processing on the plurality of first sample images, the second negative example sample set comprises a fourth sample image obtained by performing photometric stereo imaging processing on the plurality of second sample images, and the first object and the second object are any one of a plurality of collected defect-free objects and a plurality of collected defect-free objects respectively.

Optionally, in the process of training the first defect detection model, the training module is specifically configured to: sampling a first positive sample subset matching the number of samples from the first positive sample set according to the number of samples of the first negative sample set; performing preliminary training on the first defect detection model using the first positive case sample subset and the first negative case sample set; testing the first defect detection model obtained by the preliminary training by using the first positive example sample set to obtain a second positive example sample subset consisting of false-detection sample images; updating the second positive example subset to the first positive example subset; retraining the first defect detection model using the updated first positive case sample subset and the first negative case sample set.

Optionally, in the process of training the second defect detection model, the training module is specifically configured to: sampling a third positive sample subset matching the sample number from the second positive sample set according to the sample number of the second negative sample set; performing preliminary training on the second defect detection model using the third positive case sample subset and the second negative case sample set; testing the second defect detection model obtained by the preliminary training by using the second positive example sample set to obtain a fourth positive example sample subset consisting of the sample images subjected to false detection; updating the third positive sample subset with the fourth positive sample subset; retraining the second defect detection model using the updated third positive example sample set and the second negative example sample set.

Optionally, in the process of training the first defect detection model, the training module is further configured to: acquiring a plurality of second sample images of the second object taken in the light source directions; determining at least one second sample image from the plurality of second sample images according to the defect morphological characteristics corresponding to the second object, wherein the light source direction corresponding to the at least one second sample image corresponds to the defect morphological characteristics; adding the at least one second sample image to the first negative sample set.

The apparatus shown in fig. 8 can perform the steps provided in the foregoing embodiments, and the detailed performing process and technical effects refer to the description in the foregoing embodiments, which are not described herein again.

In one possible design, the structure of the defect detection apparatus shown in fig. 8 can be implemented as an electronic device. As shown in fig. 9, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. Wherein the memory 22 has stored thereon executable code which, when executed by the processor 21, makes the processor 21 at least capable of implementing the defect detection method as provided in the previous embodiments.

Additionally, an embodiment of the present invention provides a non-transitory machine-readable storage medium having stored thereon executable code, which when executed by a processor of an electronic device, causes the processor to at least implement a defect detection method as provided in the foregoing embodiments.

The above described embodiments of the apparatus are merely illustrative, wherein the network elements illustrated as separate components may or may not be physically separate. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course, can also be implemented by a combination of hardware and software. With this understanding in mind, the above-described aspects and portions of the present technology which contribute substantially or in part to the prior art may be embodied in the form of a computer program product, which may be embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including without limitation disk storage, CD-ROM, optical storage, and the like.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A method of defect detection, comprising:

2. The method of claim 1, wherein determining a final defect detection result of the inspected object based on the first defect detection result and the second defect detection result comprises:

and if the first defect detection result and the second defect detection result both indicate that the detected object has defects, determining that the detected object has defects.

3. The method of claim 1, further comprising:

acquiring a first positive sample set and a first negative sample set for training the first defect detection model;

acquiring a second positive sample set and a second negative sample set for training the second defect detection model;

training the first defect detection model using the first positive case sample set and the first negative case sample set;

training the second defect detection model using the second positive case sample set and the second negative case sample set;

the first positive example sample set comprises a plurality of first sample images of a defect-free first object photographed in a plurality of light source directions, the first negative example sample set comprises a plurality of second sample images of a defect second object photographed in the plurality of light source directions, the second positive example sample set comprises a third sample image obtained by performing photometric stereo imaging processing on the plurality of first sample images, the second negative example sample set comprises a fourth sample image obtained by performing photometric stereo imaging processing on the plurality of second sample images, and the first object and the second object are any one of a plurality of collected defect-free objects and a plurality of collected defect-free objects respectively.

4. The method of claim 3, wherein the training the first defect detection model using the first set of positive examples and the first set of negative examples comprises:

sampling a first positive sample subset matching the number of samples from the first positive sample set according to the number of samples of the first negative sample set;

performing preliminary training on the first defect detection model using the first positive case sample subset and the first negative case sample set;

testing the first defect detection model obtained by the preliminary training by using the first positive example sample set to obtain a second positive example sample subset consisting of false-detection sample images;

updating the second positive example subset to the first positive example subset;

retraining the first defect detection model using the updated first positive case sample subset and the first negative case sample set.

5. The method of claim 3, wherein the training the second defect detection model using the second set of positive examples and the second set of negative examples comprises:

sampling a third positive sample subset matching the sample number from the second positive sample set according to the sample number of the second negative sample set;

performing preliminary training on the second defect detection model using the third positive case sample subset and the second negative case sample set;

testing the second defect detection model obtained by the preliminary training by using the second positive example sample set to obtain a fourth positive example sample subset consisting of the sample images subjected to false detection;

updating the third positive sample subset with the fourth positive sample subset;

retraining the second defect detection model using the updated third positive example sample set and the second negative example sample set.

6. The method of claim 3, wherein the obtaining a first negative sample set for training the first defect detection model comprises:

acquiring a plurality of second sample images of the second object taken in the light source directions;

adding the at least one second sample image to the first negative sample set.

7. A method of defect detection, comprising:

8. A defect detection apparatus, comprising:

9. An electronic device, comprising: a memory, a processor, a communication interface; wherein the memory has stored thereon executable code which, when executed by the processor, causes the processor to perform the defect detection method of any of claims 1 to 6.

10. A non-transitory machine-readable storage medium having stored thereon executable code, which when executed by a processor of an electronic device, causes the processor to perform the defect detection method of any one of claims 1 to 6.

11. A method of defect detection, comprising: