CN111681162B - Defect sample generation method and device, electronic equipment and storage medium - Google Patents
Defect sample generation method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111681162B CN111681162B CN202010526870.8A CN202010526870A CN111681162B CN 111681162 B CN111681162 B CN 111681162B CN 202010526870 A CN202010526870 A CN 202010526870A CN 111681162 B CN111681162 B CN 111681162B
- Authority
- CN
- China
- Prior art keywords
- image
- defect
- workpiece
- processed
- style migration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000007547 defect Effects 0.000 title claims abstract description 232
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000003860 storage Methods 0.000 title claims abstract description 10
- 230000005012 migration Effects 0.000 claims abstract description 92
- 238000013508 migration Methods 0.000 claims abstract description 92
- 230000002950 deficient Effects 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims abstract description 23
- 238000012549 training Methods 0.000 claims description 30
- 230000006870 function Effects 0.000 claims description 25
- 238000010586 diagram Methods 0.000 claims description 21
- 230000004913 activation Effects 0.000 claims description 17
- 230000008859 change Effects 0.000 claims description 9
- 238000004891 communication Methods 0.000 claims description 6
- 230000001131 transforming effect Effects 0.000 claims description 5
- 239000000463 material Substances 0.000 abstract description 7
- 238000004519 manufacturing process Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 6
- 238000005070 sampling Methods 0.000 description 5
- 238000013473 artificial intelligence Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000009826 distribution Methods 0.000 description 3
- XEEYBQQBJWHFJM-UHFFFAOYSA-N Iron Chemical compound [Fe] XEEYBQQBJWHFJM-UHFFFAOYSA-N 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005242 forging Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 230000001788 irregular Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 229910000831 Steel Inorganic materials 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 229910052742 iron Inorganic materials 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000000053 physical method Methods 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
Classifications
-
- G06T3/04—
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The application provides a defect sample generation method, a defect sample generation device, electronic equipment and a storage medium. The method comprises the following steps: obtaining a workpiece image to be processed and a workpiece image with defects; the defective workpiece image includes a first defective region; randomly determining a second defect area in the image of the workpiece to be processed; performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; inputting a new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; the generated defects are included in the target image. According to the embodiment of the application, the second defect area is randomly determined in the workpiece image to be processed to simulate the defect, and the position, the shape and the size of the second defect area are random, so that more types of defects can be obtained, and then the authenticity of the simulated defect is improved by using a histogram matching and style migration method. And defects are not required to be manufactured on the workpiece manually, so that manpower and material resources are greatly reduced.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a method and apparatus for generating a defect sample, an electronic device, and a storage medium.
Background
With the rapid development of artificial intelligence technology, artificial intelligence technology represented by deep learning has penetrated to the aspects of society, and taking manufacturing industry as an example, the surface defect detection technology based on artificial intelligence technology is greatly improving the production efficiency and the production quality of manufacturing industry, so that the manufacturing industry in China can improve the product quality competitiveness while reducing the production cost, and promote China to gradually change from manufacturing country to manufacturing country.
When the artificial intelligence technology represented by the neural network is used for surface defect quality inspection, a large number of workpiece samples with defects are usually required to be collected, the neural network can be well fitted and generalized through learning of a large number of workpieces with defects, an intelligent defect detection function is completed, and appearance quality inspection work is replaced by manpower.
But in general defective workpieces are very limited. Therefore, a large number of workpieces with defects are imitated, so that the requirement of a neural network algorithm is met, and finally, a network model with up-to-standard performance can be obtained for production and use.
The existing method for simulating the defects is a manual forging method, wherein the manual forging is to simulate and generate a sample with the defects by using a tool through a physical method. However, this method consumes very much labor and material resources, and has a limited number of imitated defects.
Disclosure of Invention
The embodiment of the application aims to provide a defect sample generation method, device, electronic equipment and storage medium, which are used for solving the problems that a large amount of manpower and material resources are consumed by imitation defects in the prior art, and the number of imitation defects is limited.
In a first aspect, an embodiment of the present application provides a defect sample generating method, including: obtaining a workpiece image to be processed and a workpiece image with defects; wherein the defective workpiece image includes a first defective region; randomly determining a second defect area in the image of the workpiece to be processed; performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defects are included in the target image.
According to the embodiment of the application, the second defect area is randomly determined in the workpiece image to be processed to simulate the defects, and the positions, the shapes and the sizes of the second defect area are random, so that more types of defects can be obtained, then the color and the gray scale of the defect area to be simulated are kept consistent with those of the real defects in a histogram matching mode, the authenticity of the simulated defects is improved, and the texture of the simulated defect area in the workpiece image to be processed is kept consistent with that of the background area in a style migration method. And defects are not required to be manufactured on the workpiece manually, so that manpower and material resources are greatly reduced.
Further, the randomly determining the second defect area in the workpiece image to be processed includes: randomly determining a point from the image of the workpiece to be processed as a datum point; randomly selecting one defect type from preset defect types as a target defect type; transforming the defects corresponding to the target defect types to obtain transformed defects; and taking the datum point as a center point of the transformed defect, and generating the second defect area on the workpiece image to be processed.
The embodiment of the application randomly determines the second defect area in the image of the workpiece to be processed to imitate the defects, and the positions, the shapes and the sizes of the second defect area are random, so that the imitated defects are more diversified.
Further, the histogram matching of the first defect area and the second defect area to obtain a new workpiece image to be processed includes: obtaining new image of workpiece to be processed according to function y≡S [ L' ] +histmatch (S [ L ], H [ M ]); wherein y is the new workpiece image to be processed, S is the workpiece image to be processed, L is the second defect area, L' is the workpiece image with defects, H is the first defect area.
According to the embodiment of the application, the defects formed on the second defect area and the real defects keep consistent in color and gray scale through histogram matching, so that the authenticity of the imitated defects is improved.
Further, before inputting the new workpiece image to be processed into a style migration model for style migration processing, the method further comprises: acquiring a plurality of training samples, wherein the training samples comprise an original workpiece image and a noise image obtained after the original workpiece image is subjected to noise processing; inputting the noise image into a style migration model to be trained, and obtaining a predicted image output by the style migration model to be trained; and optimizing parameters in the style migration model to be trained according to the predicted image and the original workpiece image to obtain a trained style migration model.
According to the embodiment of the application, the style migration model is trained, and the obtained style migration model can process the workpiece image to be processed, so that the imitated defect area in the processed workpiece image to be processed is consistent with the texture of the background area, and the authenticity of the imitated defect sample is improved.
Further, the obtaining a plurality of training samples includes: at least one original workpiece image is obtained, any region on each original workpiece image is scratched, random noise is added to the scratched region, and a plurality of noise images corresponding to the original workpiece images are obtained.
Further, the optimizing parameters in the style migration model to be trained according to the predicted image and the original workpiece image includes: calculating according to the predicted image and the original workpiece image to obtain corresponding content loss, style loss, histogram loss and full-image change loss; and optimizing parameters in the style migration model to be trained according to the content loss, the style loss, the histogram loss and the full graph change loss.
According to the embodiment of the application, the style migration model is optimized from a plurality of factors through content loss, style loss, histogram loss and total graph change loss, so that the obtained style migration model can improve the authenticity of the defect sample obtained by imitation.
Further, the style migration model comprises a first convolution unit, a residual convolution module, a second convolution unit and an up-sampling module which are sequentially connected; wherein the first convolution unit and the second convolution unit each comprise a plurality of convolution modules; the residual convolution module is used for splicing the characteristic diagram output by each convolution module in the first convolution unit and the characteristic diagram output by the residual convolution module and inputting the characteristic diagram into the second convolution unit; the upsampling module includes a tanh activation function.
The embodiment of the application ensures the combination of the front and rear different visual field characteristics through the residual convolution module, and enhances the performance of the convolution network.
In a second aspect, an embodiment of the present application provides a defect sample generating apparatus, including: the image acquisition module is used for acquiring a workpiece image to be processed and a workpiece image with defects; wherein the defective workpiece image includes a first defective region; the region determining module is used for randomly determining a second defect region in the workpiece image to be processed; the histogram matching module is used for performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; the style migration module is used for inputting the new workpiece image to be processed into a style migration model to perform style migration processing, and obtaining a target image output by the style migration model; wherein the generated defects are included in the target image.
In a third aspect, an embodiment of the present application provides an electronic device, including: the device comprises a processor, a memory and a bus, wherein the processor and the memory complete communication with each other through the bus; the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform the method of the first aspect.
In a fourth aspect, embodiments of the present application provide a non-transitory computer readable storage medium comprising: the non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method of the first aspect.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the embodiments of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a defect sample generation method according to an embodiment of the present application;
FIG. 2 is a flow chart of another defect sample generation according to an embodiment of the present application;
FIG. 3 is a structural diagram of a style migration model provided by an embodiment of the present application;
FIG. 4 is a schematic diagram of a training flow of a style migration module according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a device according to an embodiment of the present application;
fig. 6 is a schematic diagram of an entity structure of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.
It can be understood that the style migration model training method and the defect sample generation method provided by the embodiment of the application can be applied to terminal equipment (also called electronic equipment) and a server; the terminal equipment can be a smart phone, a tablet personal computer, a personal digital assistant (Personal Digital Assitant, PDA) and the like; the server may be an application server or a Web server. In addition, the model training method and the defect generating method may be executed by the same terminal device or may be executed by different terminal devices.
In order to facilitate understanding, the technical solution provided by the embodiments of the present application will be described below by taking a terminal device as an execution body as an example, and describing application scenarios of the model training method and the defect generating method provided by the embodiments of the present application.
Fig. 1 is a schematic flow chart of a defect sample generation method according to an embodiment of the present application, as shown in fig. 1, where the method includes:
step 101: obtaining a workpiece image to be processed and a workpiece image with defects; wherein the defective workpiece image includes a first defective area.
The size of the workpiece image to be processed can be the same as or different from the size of the workpiece image with the defects. The image of the workpiece with the defect can be obtained by carrying out image acquisition on a real workpiece with the defect, and can also be generated by the defect generation method. The defective workpiece image may be color or gray image, and if the workpiece image is color image, the workpiece image may be converted into gray image. In selecting the image of the defective workpiece, the first defective area may be a circle, an irregular polygon, a slit, or the like, and the position and size of the first defective area in the workpiece are not limited. The workpiece image to be processed can be obtained by image acquisition of a workpiece without defects, or can be a workpiece image obtained from a network.
Step 102: and randomly determining a second defect area in the image of the workpiece to be processed.
Wherein, a region can be randomly selected as the second defect region in the workpiece image to be processed according to a preset algorithm. It should be noted that random determination means that the position, size and shape of the second defect region in the image of the workpiece to be processed are random. The size and shape of the second defect region may be different from those of the first defect region, and may be the same.
Step 103: and carrying out histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed.
Histogram matching, also called histogram specification, is to transform an image histogram with the histogram of a standard image as a standard, so that the histograms of two images are identical and similar, thereby making the two images have similar hue and contrast. The principle of histogram matching is to equalize both histograms to become the same normalized uniform histogram, and then to perform the inverse operation of equalization on the reference image by taking the uniform histogram as a medium.
According to the embodiment of the application, the gray level histogram of the second defect area is subjected to histogram matching according to the gray level histogram of the first defect area, and the flow is as follows: respectively calculating gray level histograms of the workpiece image to be processed and the workpiece image with the defects, respectively calculating accumulated results on the gray level histograms, and changing the gray level histogram distribution of the second defect area to enable the gray level histogram distribution to be close to the gray level histogram distribution of the first defect area as much as possible, so that a new second defect area is obtained. And removing the second defect area in the workpiece image to be processed, and filling the new second defect area in the original second defect area of the workpiece image to be processed to obtain a new workpiece image to be processed. Wherein the histogram matching can be represented by the following function:
y←S[L']+histmatch(S[L],H[M])
y is the new workpiece image to be processed, S is the workpiece image to be processed, L is the second defect area, L' is the workpiece image with defects, H is the first defect area.
Step 104: inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defects are included in the target image.
The style migration model is obtained through pre-training, and the specific structure and training flow are described in the following embodiments. Inputting a new workpiece image to be processed into a style migration model, processing the input new workpiece image to be processed by the style migration model, and outputting a target image, wherein a defect part in the output target image and the original defect-free workpiece image to be processed can be kept consistent on the whole, so that a imitated defect area can be naturally fused into a background pattern. The style migration model may be represented by the following function:
y'←S[L']+TransferNet(y)[L]
wherein y 'is the new image of the workpiece to be processed, S is the image of the workpiece to be processed, L is the second defect area, and L' is the image of the workpiece to be processed.
FIG. 2 is a flow chart of another generation of defective samples according to an embodiment of the present application, as shown in FIG. 2, a non-defective sample (an image of a workpiece to be processed) is obtained first, and a region, i.e., a second defective region, is randomly set on the non-defective sample. And carrying out histogram matching on the second defect area and the real defects on the workpiece image with the defects to obtain a new workpiece image to be processed. In order to enable textures of defects in the new workpiece image to be processed to be consistent with textures of other areas, the new workpiece image to be processed is input into a style migration model, and the style migration model outputs a target image, so that a generated defect sample is obtained.
According to the embodiment of the application, the second defect area is randomly determined in the workpiece image to be processed to simulate the defects, and the positions, the shapes and the sizes of the second defect area are random, so that more types of defects can be obtained, then the color and the gray scale of the defect area to be simulated are kept consistent with those of the real defects in a histogram matching mode, the authenticity of the simulated defects is improved, and the texture of the simulated defect area in the workpiece image to be processed is kept consistent with that of the background area in a style migration method. And defects are not required to be manufactured on the workpiece manually, so that manpower and material resources are greatly reduced.
On the basis of the foregoing embodiment, the randomly determining, in the image of the workpiece to be processed, a second defect area includes:
randomly determining a point from the image of the workpiece to be processed as a datum point;
randomly selecting one defect type from preset defect types as a target defect type;
transforming the defects corresponding to the target defect types to obtain transformed defects;
and taking the datum point as a center point of the transformed defect, and generating the second defect area on the workpiece image to be processed.
In a specific implementation procedure, defects of various defect types are prestored in the terminal device, for example: circular, irregular polygonal, slit-like, etc. When the second defect area is randomly determined from the workpiece image to be processed, a point can be randomly found on the workpiece image to be processed as a reference point, and the point is also the center point of the second defect area. Then, a defect type is determined from a plurality of defect types as a target defect type, and the defect of the target defect type is transformed, wherein the specific transformation can be stretching, shrinking, rotating, corrugating and the like, so as to obtain the transformed defect.
After the transformed defect is obtained, a second defect region is generated on the workpiece to be processed by taking the datum point as the center point of the transformed defect.
In another implementation process, after the terminal device directly determines the target defect type, the defect corresponding to the target defect type is set at any position of the workpiece to be processed, and then scaling, rotating, corrugating and other transformation operations are performed on the defect, so that an image of the workpiece to be processed with the second defect area is obtained.
The embodiment of the application randomly determines the second defect area in the image of the workpiece to be processed to imitate the defects, and the positions, the shapes and the sizes of the second defect area are random, so that the imitated defects are more diversified.
On the basis of the embodiment, the style migration model comprises a first convolution unit, a residual convolution module, a second convolution unit and an up-sampling module which are sequentially connected; wherein the first convolution unit and the second convolution unit each comprise a plurality of convolution modules;
the residual convolution module is used for splicing the characteristic diagram output by each convolution module in the first convolution unit and the characteristic diagram output by the residual convolution module and inputting the characteristic diagram into the second convolution unit;
the upsampling module includes a tanh activation function.
In a specific implementation process, fig. 3 is a structural diagram of a style migration model provided by an embodiment of the present application, and as shown in fig. 3, a first convolution unit may include three convolution modules, which are a first convolution module, a second convolution module, and a third convolution module, respectively.
The first convolution module comprises a convolution layer with an input channel of 3, an output channel of 32, a convolution kernel of 9 and a step length of 1, an example regularization layer and an activation layer with a Relu function as an activation function; the first convolution module may output a first feature map.
The second convolution module comprises a convolution layer with 32 input channels, 64 output channels, a convolution kernel of 3 and a step length of 2, an example regularization layer and an activation layer with a Relu function as an activation function; the second convolution module may output a second signature.
The third convolution module comprises a convolution layer with an input channel of 64, an output channel of 128, a convolution kernel of 3 and a step length of 2, an example regularization layer and an activation layer with a Relu function as an activation function; the third convolution module may output a third signature.
The input channel of the residual convolution module is 4, the output channels are 128, the convolution kernel is 3, the step length is 1, and the output of the residual convolution module is a fourth characteristic diagram.
The second convolution unit may include three convolution modules, namely a fourth convolution module, a fifth convolution module and a sixth convolution module.
The fourth convolution module comprises a convolution layer with 256 input channels, 128 output channels, 1 convolution kernel and 1 step length, a convolution layer with 128 input channels, 64 output channels, 2 times up-sampling convolution layer with 1 step length, an example regularization layer and an activation layer with a Relu function as an activation function.
The fifth convolution module comprises a convolution layer with 128 input channels, 64 output channels, 1 convolution kernel and 1 step length, a convolution layer with 64 input channels, 32 output channels, 2 times up-sampling convolution layer with 1 step length, an example regularization layer, and an activation layer with a Relu function as an activation function.
The sixth convolution module comprises a convolution layer with an input channel of 64, an output channel of 32, a convolution kernel of 1 and a step length of 1, a convolution layer with an input channel of 64, an output channel of 32, a step length of 1 and a convolution kernel of 9, an example regularization layer, and an activation layer with a Relu function as an activation function.
The upsampling module includes a tanh activation function that is output as a target image.
In the process of processing the input image by using the style migration model, the data input into the fourth convolution module is formed by splicing the first feature map, the second feature map, the third feature map and the fourth feature map. The combination of different visual field characteristics can be ensured, and the performance of the convolution network is enhanced.
Based on the foregoing embodiments, fig. 4 is a schematic diagram of a style migration module training flow provided by the embodiment of the present application, as shown in fig. 4, including:
step 401: and acquiring a plurality of training samples, wherein the training samples comprise an original workpiece image and a noise image obtained after the original workpiece image is subjected to noise processing.
The training sample comprises an original workpiece image and a noise image obtained after the original image is subjected to noise processing. The noise image is the input of the model, and the original workpiece image is the label. The original workpiece image can be obtained by image acquisition of a real non-defective workpiece or can be obtained from the network. The type of the workpiece may be uniform depending on the type of the workpiece in which the defect is actually generated, and may be, for example, a steel plate, an iron plate, or the like, or may be a workpiece of another material or another shape.
The noise image can be obtained by:
a region is randomly determined in the original workpiece image, and it can be understood that the determination manner of the region may be identical to the determination manner of the second defect region in the foregoing embodiment, which is not described herein. After the area is determined, the area is scratched, and random noise is added into the scratched area of the original workpiece image, so that a noise image is obtained. It will be appreciated that one raw workpiece image may generate multiple noise images, such as: the areas with different shapes and sizes can be scratched out from the original workpiece image, and the same noise is added; the areas with different shapes and sizes can be removed from the original workpiece image, and different types of noise can be added; areas with different shapes and sizes can be scratched out from the original workpiece image, and different types of noise can be added.
Step 402: and inputting the noise image into a style migration model to be trained, and obtaining a predicted image output by the style migration model to be trained.
Step 403: and optimizing parameters in the style migration model to be trained according to the predicted image and the original workpiece image to obtain a trained style migration model.
In a specific implementation process, model loss corresponding to the style migration model is calculated according to a predicted image output by the style migration model and a corresponding original workpiece image, the model loss comprises content loss, style loss, histogram loss and full-graph change loss, and total loss can be obtained through each loss value and corresponding weight. Parameters in the style migration model are optimized through a back propagation method by utilizing the total loss.
It should be noted that, training of the style migration model may be achieved through multiple iterations, and in addition, when judging whether the model training meets the training end condition, the first model may be verified by using the test sample, where the test sample is obtained in a manner consistent with the training sample, and also includes the original workpiece image and the noise image. The first model is a model obtained by performing one-round training optimization on the style migration model by utilizing a plurality of training samples; specifically, the terminal equipment inputs a touch mode in a test sample into a first model, and processes an input noise image by using the first model to obtain a corresponding predicted image; and calculating the accuracy rate according to the original workpiece image in the test sample and the predicted image output by the first model, and when the predicted accuracy rate is larger than a preset threshold value, considering that the model performance of the first model can meet the requirement, and generating a style migration model according to the parameters and the model structure of the first model.
It should be understood that the above preset threshold may be set according to practical situations, and the preset threshold is not specifically limited in the embodiment of the present application.
In addition, when judging whether the style migration model meets the training ending condition, the iteration number can be preset, and when the iteration number in the training process reaches the preset iteration number, the training can be stopped.
It should be appreciated that the number of iterations may be set based on historical experience, as embodiments of the present application are not specifically limited.
According to the embodiment of the application, the style migration model is trained, and the obtained style migration model can process the workpiece image to be processed, so that the imitated defect area in the processed workpiece image to be processed is consistent with the texture of the background area, and the authenticity of the imitated defect sample is improved.
Fig. 5 is a schematic structural diagram of an apparatus provided in an embodiment of the present application, where the apparatus may be a module, a program segment, or a code on an electronic device. It should be understood that the apparatus corresponds to the embodiment of the method of fig. 1 described above, and is capable of performing the steps involved in the embodiment of the method of fig. 1, and specific functions of the apparatus may be referred to in the foregoing description, and detailed descriptions thereof are omitted herein as appropriate to avoid redundancy. The device comprises: an image obtaining module 501, a region determining module 502, a histogram matching module 503 and a style migration module 504, wherein:
the image obtaining module 501 is used for obtaining an image of a workpiece to be processed and an image of a workpiece with a defect; wherein the defective workpiece image includes a first defective region; the area determining module 502 is configured to randomly determine a second defect area in the image of the workpiece to be processed; the histogram matching module 503 is configured to perform histogram matching on the first defect area and the second defect area, so as to obtain a new workpiece image to be processed; the style migration module 504 is configured to input the new workpiece image to be processed into a style migration model for style migration processing, so as to obtain a target image output by the style migration model; wherein the generated defects are included in the target image.
On the basis of the above embodiment, the area determining module 502 is specifically configured to:
randomly determining a point from the image of the workpiece to be processed as a datum point;
randomly selecting one defect type from preset defect types as a target defect type;
transforming the defects corresponding to the target defect types to obtain transformed defects;
and taking the datum point as a center point of the transformed defect, and generating the second defect area on the workpiece image to be processed.
On the basis of the above embodiment, the histogram matching module 503 is specifically configured to:
obtaining new image of workpiece to be processed according to function y≡S [ L' ] +histmatch (S [ L ], H [ M ]);
wherein y is the new workpiece image to be processed, S is the workpiece image to be processed, L is the second defect area, L' is the workpiece image with defects, H is the first defect area.
On the basis of the above embodiment, the apparatus further includes a training module for:
acquiring a plurality of training samples, wherein the training samples comprise an original workpiece image and a noise image obtained after the original workpiece image is subjected to noise processing;
inputting the noise image into a style migration model to be trained, and obtaining a predicted image output by the style migration model to be trained;
and optimizing parameters in the style migration model to be trained according to the predicted image and the original workpiece image to obtain a trained style migration model.
Based on the above embodiment, the training module is specifically configured to:
at least one original workpiece image is obtained, any area on each original workpiece image is scratched, random noise is added to the subtracted area, and a plurality of noise images corresponding to the original workpiece images are obtained.
Based on the above embodiment, the training module is specifically configured to:
calculating according to the predicted image and the original workpiece image to obtain corresponding content loss, style loss, histogram loss and full-image change loss;
and optimizing parameters in the style migration model to be trained according to the content loss, the style loss, the histogram loss and the full graph change loss.
On the basis of the embodiment, the style migration model comprises a first convolution unit, a residual convolution module, a second convolution unit and an up-sampling module which are sequentially connected; wherein the first convolution unit and the second convolution unit each comprise a plurality of convolution modules;
the residual convolution module is used for splicing the characteristic diagram output by each convolution module in the first convolution unit and the characteristic diagram output by the residual convolution module and inputting the characteristic diagram into the second convolution unit;
the upsampling module includes a tanh activation function.
In summary, in the embodiment of the present application, the second defect area is randomly determined in the image of the workpiece to be processed to simulate the defect, so that the position, the shape and the size of the second defect area are random, more types of defects can be obtained, then the color and the gray scale of the defect area to be simulated are consistent with those of the real defect by using a histogram matching mode, the authenticity of the simulated defect is improved, and the texture of the simulated defect area in the image of the workpiece to be processed is consistent with that of the background area by using a style migration method. And defects are not required to be manufactured on the workpiece manually, so that manpower and material resources are greatly reduced.
Fig. 6 is a schematic diagram of an entity structure of an electronic device according to an embodiment of the present application, as shown in fig. 6, where the electronic device includes: a processor (processor) 601, a memory (memory) 602, and a bus 603; wherein, the liquid crystal display device comprises a liquid crystal display device,
the processor 601 and the memory 602 perform communication with each other through the bus 603;
the processor 601 is configured to invoke program instructions in the memory 602 to perform the methods provided in the above method embodiments, for example, including: obtaining a workpiece image to be processed and a workpiece image with defects; wherein the defective workpiece image includes a first defective region; randomly determining a second defect area in the image of the workpiece to be processed; performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defects are included in the target image.
The processor 601 may be an integrated circuit chip having signal processing capabilities. The processor 601 may be a general-purpose processor including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but may also be a Digital Signal Processor (DSP), application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. Which may implement or perform the various methods, steps, and logical blocks disclosed in embodiments of the application. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The Memory 602 may include, but is not limited to, random access Memory (Random Access Memory, RAM), read Only Memory (ROM), programmable Read Only Memory (Programmable Read-Only Memory, PROM), erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable Read Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), and the like.
The present embodiment discloses a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, are capable of performing the methods provided by the above-described method embodiments, for example comprising: obtaining a workpiece image to be processed and a workpiece image with defects; wherein the defective workpiece image includes a first defective region; randomly determining a second defect area in the image of the workpiece to be processed; performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defects are included in the target image.
The present embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause a computer to perform the methods provided by the above-described method embodiments, for example, including: obtaining a workpiece image to be processed and a workpiece image with defects; wherein the defective workpiece image includes a first defective region; randomly determining a second defect area in the image of the workpiece to be processed; performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defects are included in the target image.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
Further, the units described as separate units may or may not be physically separate, and units displayed as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
Furthermore, functional modules in various embodiments of the present application may be integrated together to form a single portion, or each module may exist alone, or two or more modules may be integrated to form a single portion.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and variations will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (9)
1. A defect sample generation method, comprising:
obtaining a workpiece image to be processed and a workpiece image with defects; wherein the defective workpiece image includes a first defective region;
randomly determining a second defect area in the image of the workpiece to be processed;
performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed;
inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the target image includes the generated defect;
wherein the randomly determining the second defect area in the workpiece image to be processed includes:
randomly determining a point from the image of the workpiece to be processed as a datum point;
randomly selecting one defect type from preset defect types as a target defect type;
transforming the defects corresponding to the target defect types to obtain transformed defects;
and taking the datum point as a center point of the transformed defect, and generating the second defect area on the workpiece image to be processed.
2. The method of claim 1, wherein histogram matching the first defect region with the second defect region to obtain a new image of the workpiece to be processed comprises:
obtaining new image of workpiece to be processed according to function y≡S [ L' ] +histmatch (S [ L ], H [ M ]);
wherein y is the new workpiece image to be processed, S is the workpiece image to be processed, L is the second defect area, L' is the workpiece image with defects, H is the first defect area.
3. The method of claim 1, wherein prior to entering the new workpiece image to be processed into a style migration model for style migration processing, the method further comprises:
acquiring a plurality of training samples, wherein the training samples comprise an original workpiece image and a noise image obtained after the original workpiece image is subjected to noise processing;
inputting the noise image into a style migration model to be trained, and obtaining a predicted image output by the style migration model to be trained;
and optimizing parameters in the style migration model to be trained according to the predicted image and the original workpiece image to obtain a trained style migration model.
4. The method of claim 3, wherein the obtaining a plurality of training samples comprises:
at least one original workpiece image is obtained, any area on each original workpiece image is scratched, random noise is added to the subtracted area, and a plurality of noise images corresponding to the original workpiece images are obtained.
5. A method according to claim 3, wherein said optimizing parameters in the style migration model to be trained from the predicted image and the raw workpiece image comprises:
calculating according to the predicted image and the original workpiece image to obtain corresponding content loss, style loss, histogram loss and full-image change loss;
and optimizing parameters in the style migration model to be trained according to the content loss, the style loss, the histogram loss and the full graph change loss.
6. The method of any one of claims 1-5, wherein the style migration model comprises a first convolution unit, a residual convolution module, a second convolution unit, and an upsampling module connected in sequence; wherein the first convolution unit and the second convolution unit each comprise a plurality of convolution modules;
the residual convolution module is used for splicing the characteristic diagram output by each convolution module in the first convolution unit and the characteristic diagram output by the residual convolution module and inputting the characteristic diagram into the second convolution unit;
the upsampling module includes a tanh activation function.
7. A defect sample generating apparatus, comprising:
the image acquisition module is used for acquiring a workpiece image to be processed and a workpiece image with defects; wherein the defective workpiece image includes a first defective region;
the region determining module is used for randomly determining a second defect region in the workpiece image to be processed;
the histogram matching module is used for performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed;
the style migration module is used for inputting the new workpiece image to be processed into a style migration model to perform style migration processing, and obtaining a target image output by the style migration model; wherein the target image includes the generated defect;
the region determining module is specifically used for randomly determining a point from the workpiece image to be processed as a reference point;
randomly selecting one defect type from preset defect types as a target defect type;
transforming the defects corresponding to the target defect types to obtain transformed defects;
and taking the datum point as a center point of the transformed defect, and generating the second defect area on the workpiece image to be processed.
8. An electronic device, comprising: a processor, a memory, and a bus, wherein,
the processor and the memory complete communication with each other through the bus;
the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform the method of any of claims 1-6.
9. A non-transitory computer readable storage medium storing computer instructions which, when executed by a computer, cause the computer to perform the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010526870.8A CN111681162B (en) | 2020-06-09 | 2020-06-09 | Defect sample generation method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010526870.8A CN111681162B (en) | 2020-06-09 | 2020-06-09 | Defect sample generation method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111681162A CN111681162A (en) | 2020-09-18 |
| CN111681162B true CN111681162B (en) | 2023-09-01 |
Family
ID=72435297
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010526870.8A Active CN111681162B (en) | 2020-06-09 | 2020-06-09 | Defect sample generation method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111681162B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111862096B (en) * | 2020-09-23 | 2021-06-18 | 平安科技（深圳）有限公司 | Image segmentation method and device, electronic equipment and storage medium |
| CN112884758B (en) * | 2021-03-12 | 2023-01-10 | 国网四川省电力公司电力科学研究院 | Defect insulator sample generation method and system based on style migration method |
| CN113808011B (en) * | 2021-09-30 | 2023-08-11 | 深圳万兴软件有限公司 | Style migration method and device based on feature fusion and related components thereof |
| CN114332086B (en) * | 2022-03-14 | 2022-05-13 | 启东市固德防水布有限公司 | Textile defect detection method and system based on style migration and artificial intelligence |
| CN116030038B (en) * | 2023-02-23 | 2023-06-20 | 季华实验室 | Unsupervised OLED defect detection method based on defect generation |
| CN115861312B (en) * | 2023-02-24 | 2023-05-26 | 季华实验室 | OLED dry film defect detection method based on style migration positive sample generation |
| CN116091873B (en) * | 2023-04-10 | 2023-11-28 | 宁德时代新能源科技股份有限公司 | Image generation method, device, electronic equipment and storage medium |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2043593A1 (en) * | 1990-06-12 | 1991-12-13 | Henry Spalding Baird | Methods and apparatus for image analysis |
| CN104091327A (en) * | 2014-06-19 | 2014-10-08 | 华南理工大学 | Method and system for generating dendritic shrinkage porosity defect simulation image of casting |
| CN107358636A (en) * | 2017-06-16 | 2017-11-17 | 华南理工大学 | A kind of rarefaction defect image generating method based on textures synthesis |
| CN108447054A (en) * | 2018-03-22 | 2018-08-24 | 北京木业邦科技有限公司 | Defects in timber sample acquiring method, device, electronic equipment and storage medium |
| CN109190620A (en) * | 2018-09-03 | 2019-01-11 | 苏州科达科技股份有限公司 | License plate sample generating method, system, equipment and storage medium |
| CN109800697A (en) * | 2019-01-09 | 2019-05-24 | 国网浙江省电力有限公司舟山供电公司 | Transformer target detection and open defect recognition methods based on VGG-net Style Transfer |
| CN110675359A (en) * | 2019-06-29 | 2020-01-10 | 创新奇智(南京)科技有限公司 | Defect sample generation method and system for steel coil surface and electronic equipment |
| CN110689477A (en) * | 2019-09-07 | 2020-01-14 | 创新奇智(重庆)科技有限公司 | Universal flaw image simulation method |
| CN110706308A (en) * | 2019-09-07 | 2020-01-17 | 创新奇智(成都)科技有限公司 | GAN-based steel coil end face edge loss artificial sample generation method |
| WO2020048242A1 (en) * | 2018-09-04 | 2020-03-12 | 阿里巴巴集团控股有限公司 | Method and apparatus for generating vehicle damage image based on gan network |
| CN111127309A (en) * | 2019-12-12 | 2020-05-08 | 杭州格像科技有限公司 | Portrait style transfer model training method, portrait style transfer method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9857953B2 (en) * | 2015-11-17 | 2018-01-02 | Adobe Systems Incorporated | Image color and tone style transfer |
-
2020
- 2020-06-09 CN CN202010526870.8A patent/CN111681162B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2043593A1 (en) * | 1990-06-12 | 1991-12-13 | Henry Spalding Baird | Methods and apparatus for image analysis |
| CN104091327A (en) * | 2014-06-19 | 2014-10-08 | 华南理工大学 | Method and system for generating dendritic shrinkage porosity defect simulation image of casting |
| CN107358636A (en) * | 2017-06-16 | 2017-11-17 | 华南理工大学 | A kind of rarefaction defect image generating method based on textures synthesis |
| CN108447054A (en) * | 2018-03-22 | 2018-08-24 | 北京木业邦科技有限公司 | Defects in timber sample acquiring method, device, electronic equipment and storage medium |
| CN109190620A (en) * | 2018-09-03 | 2019-01-11 | 苏州科达科技股份有限公司 | License plate sample generating method, system, equipment and storage medium |
| WO2020048242A1 (en) * | 2018-09-04 | 2020-03-12 | 阿里巴巴集团控股有限公司 | Method and apparatus for generating vehicle damage image based on gan network |
| CN109800697A (en) * | 2019-01-09 | 2019-05-24 | 国网浙江省电力有限公司舟山供电公司 | Transformer target detection and open defect recognition methods based on VGG-net Style Transfer |
| CN110675359A (en) * | 2019-06-29 | 2020-01-10 | 创新奇智(南京)科技有限公司 | Defect sample generation method and system for steel coil surface and electronic equipment |
| CN110689477A (en) * | 2019-09-07 | 2020-01-14 | 创新奇智(重庆)科技有限公司 | Universal flaw image simulation method |
| CN110706308A (en) * | 2019-09-07 | 2020-01-17 | 创新奇智(成都)科技有限公司 | GAN-based steel coil end face edge loss artificial sample generation method |
| CN111127309A (en) * | 2019-12-12 | 2020-05-08 | 杭州格像科技有限公司 | Portrait style transfer model training method, portrait style transfer method and device |
Non-Patent Citations (1)
| Title |
|---|
| 基于卷积神经网络的主变压器外观缺陷检测方法;位一鸣;童力;罗麟;杨珊;;浙江电力(04);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111681162A (en) | 2020-09-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111681162B (en) | Defect sample generation method and device, electronic equipment and storage medium | |
| CN105608708A (en) | Block plane fitting-based image binaryzation method and device | |
| CN112070727B (en) | Metal surface defect detection method based on machine learning | |
| CN110895795A (en) | Improved semantic image inpainting model method | |
| CN110969600A (en) | Product defect detection method and device, electronic equipment and storage medium | |
| CN110909615A (en) | Target detection method based on multi-scale input mixed perception neural network | |
| CN115984662B (en) | Multi-mode data pre-training and identifying method, device, equipment and medium | |
| CN114596290A (en) | Defect detection method, defect detection device, storage medium, and program product | |
| CN111325728B (en) | Product defect detection method, device, equipment and storage medium | |
| CN114581646A (en) | Text recognition method and device, electronic equipment and storage medium | |
| CN114359269A (en) | Virtual food box defect generation method and system based on neural network | |
| CN110807428A (en) | Coal sample identification method and device, server and storage medium | |
| CN112200766A (en) | Industrial product surface defect detection method based on area-associated neural network | |
| CN113592021B (en) | Stereo matching method based on deformable and depth separable convolution | |
| CN113034432B (en) | Product defect detection method, system, device and storage medium | |
| CN117392042A (en) | Defect detection method, defect detection apparatus, and storage medium | |
| CN114548250A (en) | Mobile phone appearance detection method and device based on data analysis | |
| CN113642667A (en) | Enhancement strategy determination method and device, electronic equipment and storage medium | |
| CN113052798A (en) | Screen aging detection model training method and screen aging detection method | |
| CN109409424B (en) | Appearance defect detection model modeling method and device | |
| CN115270645B (en) | Design method and system based on ERNIE-DPCNN model | |
| CN113286053B (en) | File scanning method, equipment, medium and product | |
| CN115493532B (en) | Measuring system, method and medium for measuring area of element to be measured on surface of plate | |
| CN117333740B (en) | Defect image sample generation method and device based on stable diffusion model | |
| CN107784662B (en) | Image target significance measurement method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |