CN116167923B - Sample expansion method and sample expansion device for x-ray image - Google Patents
Sample expansion method and sample expansion device for x-ray image Download PDFInfo
- Publication number
- CN116167923B CN116167923B CN202310458772.9A CN202310458772A CN116167923B CN 116167923 B CN116167923 B CN 116167923B CN 202310458772 A CN202310458772 A CN 202310458772A CN 116167923 B CN116167923 B CN 116167923B
- Authority
- CN
- China
- Prior art keywords
- image
- flaw
- content
- shared
- generator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000012549 training Methods 0.000 claims abstract description 40
- 230000006870 function Effects 0.000 claims description 58
- 230000004913 activation Effects 0.000 claims description 20
- 230000002950 deficient Effects 0.000 claims description 11
- 238000013507 mapping Methods 0.000 claims description 6
- 238000004806 packaging method and process Methods 0.000 claims description 4
- 230000003416 augmentation Effects 0.000 claims description 3
- 230000003042 antagnostic effect Effects 0.000 claims description 2
- 230000008485 antagonism Effects 0.000 abstract description 2
- 230000007547 defect Effects 0.000 description 7
- 230000011218 segmentation Effects 0.000 description 7
- 238000001514 detection method Methods 0.000 description 5
- 238000009826 distribution Methods 0.000 description 5
- 238000005070 sampling Methods 0.000 description 5
- 239000000463 material Substances 0.000 description 4
- 238000004321 preservation Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000013519 translation Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000005489 elastic deformation Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000000513 principal component analysis Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000007306 turnover Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30148—Semiconductor; IC; Wafer
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Analysing Materials By The Use Of Radiation (AREA)
Abstract
The invention provides a sample expansion method and a sample expansion device of an x-ray image, wherein the method adopts a twice cross reconstruction mode, the first stage uses the coding characteristics of a normal image to realize one-time generation, the discriminator is used for realizing the antagonism training in one stage, the second stage still uses a cross transfer mode, but a content discriminator is not used, an unsupervised unpaired data set is realized by adding a cross circulation consistency loss function, and an x-ray pseudo image which cannot be distinguished from true or false by the discriminator can be generated. The invention can realize flaw generation of the X-ray image without paired tag data sets, meanwhile, the content and detail of the image are encoded, the content characteristics and semantic characteristics of the sample are decoded, a weight sharing encoder is used for extracting shared potential codes in an x domain and a y domain, and a content discriminator is used for distinguishing domain membership of the shared code characteristics, so that the diversity of flaw code characteristics can be increased.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a sample expansion method for an x-ray image and a sample expansion device for an x-ray image.
Background
The image processing technology is widely applied to the defect detection field, and the problems of low manual detection rate, omission and the like are effectively solved by utilizing the strong learning ability of the deep neural network, so that the investment of labor cost is reduced, and the performance of a product is improved.
However, a large number of flaw samples are needed when the detection model based on the depth network is trained, the yield is very high in the industrial production process, the flaw samples are generated in a small probability time, and the occurrence of flaw features has certain randomness, so that the data volume of the original samples is insufficient.
In order to improve the quantity and diversity of flaw samples, in the related technology, rotation, filling, translation, interpolation, overturning, gaussian noise, turnover, cutting, geometric transformation, scaling, elastic deformation and principal component analysis dithering and combination thereof are generally adopted. There are also sample expansion methods based on deep learning, which require a large number of paired data sets, and acquisition of paired data sets is very difficult, and this also goes against the data expansion initiative.
Disclosure of Invention
In order to solve the above-mentioned problems, a first object of the present invention is to provide a sample expansion method for x-ray images.
A second object of the present invention is to provide a sample expansion device for x-ray images.
The technical scheme adopted by the invention is as follows:
an embodiment of the first aspect of the present invention provides a method for expanding a sample of an x-ray image, including the steps of:
acquiring a training dataset of x-ray images of an IC (Integrated Circuit ) package module, the training dataset comprising: normal images and defective images; scaling the training data set; the normal image and the flaw image after scaling are respectively input into a first content encoder and a second content encoder with shared weights, the first content encoder and the second content encoder judge membership degrees of feature encoding domains through a content discriminator so as to map the contents of the normal image and the flaw image into a first shared feature space through the first content encoder and the second content encoder, and the first shared feature space comprises: first signature coding Z for x-domain sharing x1 First signature code Z shared with y domain y1 The method comprises the steps of carrying out a first treatment on the surface of the Inputting the scaled flaw image into a first detail encoder to generate a first detail feature code; encoding the first detail featuresFirst signature code Z shared with the x domain x1 Input to a first generator G y1 In order to generate a normal pseudo-graph, the y-domain is sharedFeature code Z y1 Input to the second generator Gx 1 To generate a flaw pseudo-graph; inputting the flaw image and the normal pseudo-image into a first discriminator for resistance training to obtain a normal pseudo-image, and inputting the normal image and the flaw pseudo-image into a second discriminator for resistance training to obtain a flaw pseudo-image; inputting the primary normal pseudo-graph into a second detail encoder to generate a second detail feature code +.>Third and fourth content encoders for sharing the primary flaw-pseudograph and the primary normal-pseudograph input weights to map contents of the primary flaw-pseudograph and the primary normal-pseudograph to a second shared feature space by the third and fourth content encoders, the second shared feature space comprising: second signature encoding Z for x-domain sharing x2 Second signature code Z shared with y domain y2 The method comprises the steps of carrying out a first treatment on the surface of the Encoding the second detail feature +.>Second signature code Z shared with the x domain x2 Input to a third generator G y2 In order to generate a reconstructed flaw pseudo-graph, encoding a second feature shared by the y-domains into Z y2 Input to fourth generator Gx 2 Generating a reconstructed normal pseudo-graph, wherein the reconstructed flaw pseudo-graph and the flaw image adopt a cross-loop consistency loss function to calculate a loss value, and the reconstructed normal pseudo-graph and the normal image cross-loop consistency loss function calculate a loss value; sample expansion of the x-ray image is performed using the reconstructed flaw artifact.
The sample expansion method of the x-ray image provided by the invention can also have the following additional technical characteristics:
according to an embodiment of the present invention, the first content encoder, the second content encoder, the third content encoder, the fourth content encoder include: a full convolution network of two downsampled residual modules, the first detail encoder or the second detail encoder comprising: a convolution layer, a residual module and a full connection layer.
According to one embodiment of the invention, the content discriminator comprises: three layers of 5 x 5 downsampled convolution modules, one layer of 3 x 3 convolved layers and a LeakyReLU (leakage rectifying linear unit function) activation function.
According to one embodiment of the invention, the content discriminator uses a cross entropy loss function to determine the membership of the feature codes.
According to one embodiment of the invention, the first generator G y1 Or the third generator G y2 Comprising the following steps: two residual modules, two upsampling modules and a convolution module, the upsampling modules comprising: bilinear interpolation, convolution layer, BN (batch normalization) layer and ReLU (rectified linear unit function) activation functions.
According to one embodiment of the invention, the second generator Gx 1 Or the fourth generator Gx 2 Comprising the following steps: two upsampling modules, one convolution module and Tanh (hyperbolic tangent function) activation function.
According to one embodiment of the invention, the first discriminator and the second discriminator calculate the loss value using a cross entropy loss function when performing the resistance training.
An embodiment of a second aspect of the present invention proposes a sample expansion device of an x-ray image, comprising: the acquisition module is used for acquiring a training data set formed by the x-ray images of the IC packaging module, and the training data set comprises: normal images and defective images; the scaling module is used for scaling the training data set; the first content encoder and the second content encoder are used for mapping the content of the normal image and the flaw image to a first shared feature space, and judging the membership degree of a feature encoding domain through a content discriminator, wherein the first shared feature space comprises: first signature coding Z for x-domain sharing x1 And y domain sharingIs encoded with the first characteristic of Z y1 The method comprises the steps of carrying out a first treatment on the surface of the A first detail encoder for generating a first detail feature code from the scaled defective imageFirst generator G y1 The first generator G y1 For encoding +/according to said first detail feature>First feature code shared with the x domain +.>Generating a normal pseudo graph; second generator Gx 1 The second generator Gx 1 Encoding Z according to a first characteristic shared by the y domains y1 Generating a flaw pseudo-graph; the first discriminator is used for performing contrast training according to the flaw image and the normal pseudo-graph to acquire a normal pseudo-graph; a second discriminator for performing an antagonistic training based on the normal image and the flaw artifact to obtain a flaw artifact; a second detail encoder for generating a second detail feature code +_based on the primary normal pseudo-graph>And a third content encoder and a fourth content encoder for weight sharing for mapping contents of the primary flaw artifact and the primary normal artifact to a second shared feature space, the second shared feature space comprising: second signature encoding Z for x-domain sharing x2 Second signature code Z shared with y domain y2 The method comprises the steps of carrying out a first treatment on the surface of the Third generator G y2 The third generator G y2 For encoding +_according to said second detail feature>Second signature code Z shared with the x domain x2 Generating a reconstruction flaw pseudo-graph; fourth generationGx device 2 The fourth generator Gx 2 Encoding Z according to a second characteristic shared by the y domains y2 Generating a reconstructed normal pseudo-graph, wherein the reconstructed flaw pseudo-graph and the flaw image adopt a cross-circulation consistency loss function to calculate a loss value, and the reconstructed normal pseudo-graph and the normal image adopt an L1 loss function to calculate a loss value; and the expansion module is used for carrying out sample expansion of the x-ray image by adopting the reconstructed flaw pseudo-image.
The sample expansion device for x-ray images provided by the invention can also have the following additional technical characteristics:
according to an embodiment of the present invention, the first content encoder, the second content encoder, the third content encoder, the fourth content encoder include: a full convolution network of two downsampled residual modules, the first detail encoder or the second detail encoder comprising: a convolution layer, a residual module and a full connection layer.
According to one embodiment of the invention, the content discriminator comprises: three layers of 5×5 downsampling convolution modules, one layer of 3×3 convolution layers and a LeakyReLU activation function, wherein the content discriminator adopts a cross entropy loss function to judge the membership degree of feature coding, and the first generator G y1 Or the third generator G y2 Comprising the following steps: two residual modules, two upsampling modules and a convolution module, the upsampling modules comprising: bilinear interpolation, convolution layer, BN layer, and ReLU activation functions; the second generator Gx 1 Or the fourth generator Gx 2 Comprising the following steps: two upsampling modules, a convolution module and a Tanh activation function.
The invention has the beneficial effects that:
the invention can realize flaw generation of the X-ray image without paired tag data sets, meanwhile, the content and detail of the image are encoded, the content characteristics and semantic characteristics of the sample are decoded, a weight sharing encoder is used for extracting shared potential codes in an x domain and a y domain, and a content discriminator is used for distinguishing domain membership of the shared code characteristics, so that the diversity of flaw code characteristics can be increased.
Drawings
FIG. 1 is a flow chart of a method of sample augmentation of an x-ray image according to one embodiment of the present invention;
FIG. 2 is a block schematic diagram of a sample expansion device for x-ray images according to one embodiment of the invention;
FIG. 3 is a comparison of an input picture, a primary pseudogram, and a reconstructed pseudogram after a sample expansion method of an x-ray image employing an embodiment of the present invention;
FIG. 4 is a schematic view of a sample generated after a sample expansion method of an x-ray image employing an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
FIG. 1 is a flow chart of a method for sample augmentation of an x-ray image according to one embodiment of the present invention, as shown in FIG. 1, comprising the steps of:
s1, acquiring a training data set formed by an x-ray image of an IC packaging module, wherein the training data set comprises: normal images and defective images.
Specifically, for some IC package modules, defects mainly exist in the internal module, and a surface image of the IC package module cannot be directly obtained to perform substantial defect detection, so that an x-ray image of the IC package module needs to be obtained, when a training data set is constructed, the x-ray image is adopted, the training data set comprises a normal image and a defect image, classification of the defect image comprises wire breakage, cold joint and the like.
S2, scaling the training data set.
S3, respectively inputting the normal image and the flaw image subjected to scaling treatment into a first content encoder and a second content encoder with shared weights, judging membership degrees of feature encoding domains by the first content encoder and the second content encoder through a content discriminator so as to map the contents of the normal image and the flaw image into a first shared feature space through the first content encoder and the second content encoder, wherein the first shared feature space comprises: first signature coding Z for x-domain sharing x1 First signature code Z shared with y domain y1 。
In an embodiment of the present invention, the content discriminator may include: three layers of 5×5 downsampled convolution blocks, one layer of 3×3 convolution layers, and a LeakyReLU activation function.
Specifically, as shown in fig. 2, the input size of the training data set is scaled to 128×128, and then the shared feature codes (Z x1 ，Z y1 ) To map normal and flaw image content to a shared feature space, discrete representation learning based on contrast domain adaptation is introduced to learn richer texture and semantic features. When the content features of the image are mapped to the shared feature space by weight sharing the first content encoder and the second content encoder, the same high-level mapping function cannot guarantee that the same content is represented as two domains encoding the same information, and therefore, a content discriminator is addedContent discriminator->The domain membership of the feature codes of the normal image and the flaw image can be distinguished through the cross entropy loss function, and the cross entropy loss function is used for optimizing the domain membership. Furthermore, the content discriminator cannot be used to distinguish between the characteristics encoded by the content encoder that generated the domain. Optimization function of content discriminator->Can be expressed as:
representing the expectations of the x-domain +.>Representing the desire of the y-field, +.>Is a content discriminator.
S4, inputting the scaled flaw image into a first detail encoder to generate a first detail feature code
S5, coding the first detail featuresFirst signature code Z shared with x domain x1 Input to a first generator G y1 In order to generate a normal pseudo-graph, the first characteristic shared by the y domain is encoded with Z y1 Input to the second generator Gx 1 To generate a flaw artifact.
S6, inputting the flaw image and the normal pseudo-image into a first discriminatorPerforming contrast training to obtain a normal pseudo-graph, and inputting the normal image and the flaw pseudo-graph into a second discriminator +>And training the row antagonism to obtain a flaw pseudo graph.
Specifically, as shown in fig. 2, the x-domain shared first feature code Z acquired in step S2 is encoded x1 And the first detailed feature code of the flaw image acquired in the step S4Into a first generator G y1 In step S3, the y domain obtained in step S3 is shared with the first feature code Z y1 Into a second generator Gx 1 。
In an embodiment of the invention, a first generator G y1 And a second generator Gx 1 Or the input data is different, and therefore the structure is also different, the first generator G y1 As a y-domain generator, a second generator Gx 1 Is an x-domain generator. First generator G y1 Comprising the following steps: two residual error modules, two upsampling modules and a convolution module, the upsampling modules comprising: bilinear interpolation, convolution layer, BN layer, and ReLU activation functions. First detail feature encoding of input flaw imageFirst signature encoding Z shared with x-domain x1 And splicing the residual error modules to form primary feature codes, forming secondary coding features by the primary feature codes through the up-sampling module, and the like, and generating a normal pseudo graph by the final output four feature codes through an activation function.
Second generator Gx 1 Comprising the following steps: two upsampling modules, a convolution module and a Tanh activation function. The input feature code is up-sampled twice and convolved to generate a final feature code, and finally a defect pseudo-graph is generated through a Tanh activation function
Specifically, the normal artifact and the defect artifact generated in step S5 need to pass through two discriminators (i.e., the first discriminator described aboveAnd a second discriminator->) Performing contrast training to improve the authenticity of flaw sample generation, wherein the discriminator is in a multi-scale form and consists of a group of convolution layers which gradually downsampled, and downsamples the input pseudo-graph by 64 times, 32 times and 16 times respectively, and the two discriminators output the true graph and the pseudo-graph by adopting the same full convolution mode as PathGan through multi-scale full convolutionThe cross entropy loss function calculates the final loss value, and finally optimizes the discriminator, encoder and generator using a backward gradient propagation algorithm.
For the defective image domain (y domain), the countermeasures against loss are defined(resistance loss of the first discriminator), defining resistance loss for the normal image domain (x domain)>(loss of resistance of the second discriminator),
i.e. optimized using a cross entropy loss function, the formula is:
indicating loss of resistance of the second discriminator, < >>Representing a second discriminator, I x Representing normal pseudo-graph, x representing x-domain variable, y representing y-domain variable, ++>Representing the expectations of a normal image with a sampling distribution of x-P (x), the +.>Indicating the desire for a defective image with a sampling distribution of x-P (y), I y Representing a flaw pseudo-graph->Indicating loss of resistance of the first discriminator, < >>The normal image with a sampling distribution of x to P (x) and the defective image with a sampling distribution of y to P (y) are shown.
The steps S1-S6 are forward cross translation stages, the first feature code Z shared by the y domain y1 Through G x1 Generating a flaw pseudo-graph, first detail feature encodingFirst signature code Z shared with x domain x1 After being spliced, pass through G y1 The normal pseudo-graph is generated by spelling, the generated pseudo-graph may change the characteristics in the original graph and generate some interference characteristics irrelevant to the image content, and for this purpose, the invention also performs the backward reconstruction stage of S7-S8.
S7, inputting the primary normal pseudo-graph into a second detail encoder to generate a second detail feature codeThird and fourth content encoders for inputting weights for the primary flaw-pseudo-graph and the primary normal-pseudo-graph to map contents of the primary flaw-pseudo-graph and the primary normal-pseudo-graph to a second shared feature space by the third and fourth content encoders, the second shared feature space including: second signature encoding Z for x-domain sharing y2 Second signature code Z shared with y domain y2 。
The second detail encoder has the same structure as the first detail encoder, the third content encoder has the same structure as the first content encoder, and the fourth content encoder has the same structure as the second content encoder.
S8, coding the second detail featuresSecond signature code Z shared with x domain x2 Input to a third generator G y2 In order to generate a reconstructed flaw pseudo-graph, the second characteristic shared by the y domain is encoded with Z y2 Input to fourth generator Gx 2 To generate a reconstructed normal pseudo-graph, wherein the reconstructed flaw pseudo-graphAnd calculating loss values of the images and the flaw images by adopting a cross-loop consistency loss function, and reconstructing a normal pseudo image and a normal image cross-loop consistency loss function to calculate the loss values.
Third generator G y2 And a first generator G y1 Is the same in structure, the fourth generator Gx 2 And a second generator Gx 1 The structure of (2) is the same.
Specifically, as shown in fig. 2, a pseudo-graph generated in the forward cross-translation stage undergoes a cross-translation, but the content is encoded and the generated pseudo-graph is identified without using a corresponding identifier, and after the normal and flaw pseudo-graphs are respectively encoded, the encoded characteristics are subjected to G y2 And Gx 2 The method comprises the steps of respectively generating a reconstruction flaw pseudo-image and a reconstruction normal pseudo-image, wherein the whole process is called cross-loop consistency, a loss value between the secondarily generated pseudo-image and an input image is calculated by using a cross-loop consistency loss function (such as an L1 loss function), and the process is used for keeping the content of the reconstructed image to be the same as that of the original image and only carrying out transformation on fine features.
S9, performing sample expansion of the x-ray image by using the reconstructed flaw pseudo-image.
On the basis of the above steps, the super-parameter setting of the present invention is described in detail, the input size is 128×128, two content encoders, one detail encoder, and two generators, one content discriminator are optimized by using 6 adam optimizers, respectively, wherein the beta value is set to 0.5, the weight attenuation coefficient is 0.0001, the initial learning rate size is set to 0.0001, the learning rate attenuation period is 200 epochs, and since the x-ray acquired image is a gray scale map, the number of input and output channels is set to 1, wherein the weight coefficient of the KL divergence loss is 0.01, and the cross-cycle consistency loss function weight coefficient is 10.
Fig. 3 is a comparison diagram of an input picture, a primary pseudo-image and a reconstructed pseudo-image after the sample expansion method of the x-ray image is adopted, wherein the upper part of the picture in fig. 3 sequentially represents a normal image, a primary flaw pseudo-image and a reconstructed normal pseudo-image, and the lower part of the picture in fig. 3 sequentially represents a flaw image, a primary normal pseudo-image and a reconstructed flaw pseudo-image. As can be seen from fig. 3, the detail features of the image generated after the secondary reconstruction are more clear.
Fig. 4 is a schematic view of a sample generated by the sample expansion method of the x-ray image according to the present invention.
The segmentation effect after sample expansion by adopting the expansion method provided by the invention is obtained through a relevant specific experiment, and the segmentation effect of the same segmentation model after different sample expansion methods is compared in table 1.
TABLE 1
| Sample expansion method | Segmentation model | Picture IS | Dicescore |
| None | U-net | 1.29 | 0.971 |
| CycleGan | U-net | 0.93 | 0.977 |
| OURs | U-net | 1.17 | 0.985 |
Wherein None represents not performing sample expansion, cycleGan represents performing sample expansion by CycleGan, and OURs represents performing sample expansion by using the sample expansion method of the image proposed by the invention. U-net represents the segmentation model, IS (Inception Score) represents the distance of the distribution of the generated picture from the real image, the Dice score is the harmonic mean of the precision and recall, this score is the detection score at the pixel level, which when applied to a binary segmentation task evaluates the degree of overlap between the predicted value a and the real value B. As can be seen from table 1, the segmentation effect is significantly improved after the sample expansion is performed by the sample expansion method of the image according to the present invention.
In summary, according to the sample expansion method of the x-ray image in the embodiment of the invention, flaw generation of the x-ray image can be realized without a pair of tag data sets, meanwhile, content and style of the image are encoded through discrete representation of self-adaptation of a domain of reactance, content characteristics and semantic characteristics of the sample are decoded, shared potential codes in an x domain and a y domain are extracted by using a weight shared encoder, domain membership of the shared coding characteristics is distinguished by a content discriminator, and diversity of flaw coding characteristics is increased. In order to realize the preservation of flaw samples and original image contents, a twice cross reconstruction mode is adopted, the coding features of normal images are used for realizing one-time generation in the first stage, the discriminator is used for realizing traditional resistance loss in the one-time stage, the cross transmission mode is still used in the second stage, but a content discriminator is not used, and finally, the unsupervised unpaired data set training and original image content preservation are realized by adding a cross circulation consistency loss function.
Corresponding to the sample expansion method of the X-ray image, the invention also provides a sample expansion device of the X-ray image.
FIG. 2 is a block schematic diagram of a sample expansion apparatus for x-ray images according to one embodiment of the invention, as shown in FIG. 2, the apparatus comprising: acquisition module, scaling module, weight sharing first and second content encoder, first detail encoder, first lifetimeAdult G y1 A second generator Gx 1 A first discriminator, a second detail encoder, a third content encoder and a fourth content encoder for weight sharing, a third generator G y2 Fourth generator Gx 2 And an expansion module.
The acquisition module is used for acquiring a training data set formed by the x-ray images of the IC packaging module, and the training data set comprises: normal images and defective images; the scaling module is used for scaling the training data set; the first content encoder and the second content encoder are used for mapping the content of the normal image and the flaw image to a first shared feature space, and the first content encoder and the second content encoder judge the membership degree of the feature encoding domain through the content discriminator, and the first shared feature space comprises: first signature coding Z for x-domain sharing x1 First signature code Z shared with y domain y1 The method comprises the steps of carrying out a first treatment on the surface of the The first detail encoder is used for generating a first detail characteristic code according to the flaw image after the scaling treatmentFirst generator G y1 For coding according to the first detail feature->First feature code shared with x-domain +.>Generating a normal pseudo graph; second generator Gx 1 A second generator Gx 1 First feature encoding Z for sharing according to y-domain y1 Generating a flaw pseudo-graph; the first discriminator is used for performing contrast training according to the flaw image and the normal pseudo-graph to acquire a normal pseudo-graph; the second discriminator is used for performing contrast training according to the normal image and the flaw pseudo-graph to acquire a flaw pseudo-graph; the second detail encoder is used for generating second detail feature codes according to the primary normal pseudo-graph>Third content encoder and fourth content encoder with shared weights, weight sharingThe shared third content encoder and fourth content encoder are configured to map the content of the primary flaw-artifact and the primary normal-artifact to a second shared feature space, the second shared feature space comprising: second signature encoding Z for x-domain sharing x2 Second signature code Z shared with y domain y2 The method comprises the steps of carrying out a first treatment on the surface of the Third generator G y2 For coding according to the second detail feature->Second signature code Z shared with x domain x2 Generating a reconstruction flaw pseudo-graph; fourth generator Gx 2 Second feature encoding Z for sharing according to y-domain y2 Generating a reconstructed normal pseudo-graph, wherein the reconstructed flaw pseudo-graph and the flaw image adopt a cross-circulation consistency loss function to calculate a loss value, and the reconstructed normal pseudo-graph and the reconstructed normal image adopt an L1 loss function to calculate the loss value; and the expansion module is used for carrying out sample expansion of the x-ray image by adopting the reconstructed flaw pseudo-image.
According to an embodiment of the present invention, the first content encoder, the second content encoder, the third content encoder, the fourth content encoder include: the full convolution network composed of the residual error modules of the two downsampled, the first detail encoder or the second detail encoder comprises: a convolution layer, a residual module and a full connection layer.
According to one embodiment of the present invention, a content discriminator includes: three layers of 5×5 downsampling convolution modules, one layer of 3×3 convolution layers and a LeakyReLU activation function, wherein the content discriminator adopts a cross entropy loss function to judge domain membership degree of feature coding, and a first generator G y1 Or a third generator G y2 Comprising the following steps: two residual error modules, two upsampling modules and a convolution module, the upsampling modules comprising: bilinear interpolation, convolution layer, BN layer, and ReLU activation functions; second generator Gx 1 Or fourth generator Gx 2 Comprising the following steps: two upsampling modules, a convolution module and a Tanh activation function.
According to one embodiment of the invention, the first discriminator and the second discriminator calculate the loss value using a cross entropy loss function when performing the resistance training.
In summary, the sample expansion device of the x-ray image needs to pair the tag data sets to realize flaw generation of the x-ray image, meanwhile, the content and style of the image are encoded through discrete representation of self-adaptation of the anti-domain, the content features and the semantic features of the sample are decoded, the shared potential codes in the x-domain and the y-domain are extracted by using the encoder of weight sharing, the domain membership degree of the shared coding features of the content discriminator is added, and the diversity of flaw coding features is increased. In order to realize the preservation of flaw samples and original image contents, a twice cross reconstruction mode is adopted, the coding features of normal images are used for realizing one-time generation in the first stage, the discriminator is used for realizing traditional resistance loss in the one-time stage, the cross transmission mode is still used in the second stage, but a content discriminator is not used, and finally, the unsupervised unpaired data set training and original image content preservation are realized by adding a cross circulation consistency loss function.
In the description of the present invention, the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. The meaning of "a plurality of" is two or more, unless specifically defined otherwise.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily for the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction. In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and additional implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order from that shown or discussed, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the embodiments of the present invention.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As with the other embodiments, if implemented in hardware, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
Those of ordinary skill in the art will appreciate that all or a portion of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, where the program may be stored in a computer readable storage medium, and where the program, when executed, includes one or a combination of the steps of the method embodiments.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing module, or each unit may exist alone physically, or two or more units may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product.
The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like. While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (10)
1. A method for sample expansion of an x-ray image, comprising the steps of:
acquiring a training dataset of x-ray images of an IC package module, the training dataset comprising: normal images and defective images;
scaling the training data set;
the normal image and the flaw image after scaling are respectively input into a first content encoder and a second content encoder with shared weights, the first content encoder and the second content encoder judge membership degrees of feature encoding domains through a content discriminator so as to map the contents of the normal image and the flaw image into a first shared feature space through the first content encoder and the second content encoder, and the first shared feature space comprises: first signature coding Z for x-domain sharing x1 First signature code Z shared with y domain y1 ；
Inputting the scaled flaw image into a first detail encoder to generate a first detail feature code
Encoding the first detail featuresFirst signature code Z shared with the x domain x1 Input to a first generator G y1 In order to generate a normal pseudo-graph, the first characteristic shared by the y domains is encoded with Z y1 Input to the second generator Gx 1 To generate a flaw pseudo-graph;
inputting the flaw image and the normal pseudo-image into a first discriminator for resistance training to obtain a normal pseudo-image, and inputting the normal image and the flaw pseudo-image into a second discriminator for resistance training to obtain a flaw pseudo-image;
inputting the primary normal pseudo-graph into a second detail encoder to generate a second detail feature codeInputting the primary flaw-pseudograph and the primary normal-pseudograph into a third content encoder and a fourth content encoder for weight sharing, respectively, to map the contents of the primary flaw-pseudograph and the primary normal-pseudograph to a second shared feature space through the third content encoder and the fourth content encoder, the second shared feature space comprising: second signature encoding Z for x-domain sharing x2 Second signature code Z shared with y domain y2 ；
Encoding the second detail featuresSecond signature code Z shared with the x domain x2 Input to a third generator G y2 In order to generate a reconstructed flaw pseudo-graph, encoding a second feature shared by the y-domains into Z y2 Input to fourth generator Gx 2 In order to generate a reconstructed normal artifact, wherein the reconstructed flaw artifact and the flaw image calculate a loss value using a cross-loop consistency loss function, the reconstructed normal artifact and the normalCalculating a loss value by adopting a cross-loop consistency loss function;
sample expansion of the x-ray image is performed using the reconstructed flaw artifact.
2. The method of claim 1, wherein the first content encoder, the second content encoder, the third content encoder, and the fourth content encoder comprise: a full convolution network of two downsampled residual modules, the first detail encoder or the second detail encoder comprising: a convolution layer, a residual module and a full connection layer.
3. The method of sample augmentation of an x-ray image of claim 1, wherein the content discriminator comprises: three layers of 5×5 downsampled convolution blocks, one layer of 3×3 convolution layers, and a LeakyReLU activation function.
4. The method of claim 3, wherein the content discriminator uses a cross entropy loss function to determine domain membership of feature codes.
5. The method for sample extension of an x-ray image according to claim 1, wherein the first generator G y1 Or the third generator G y2 Comprising the following steps: two residual modules, two upsampling modules and a convolution module, the upsampling modules comprising: bilinear interpolation, convolution layer, BN layer, and ReLU activation functions.
6. The method of sample expansion of an x-ray image of claim 1, wherein the second generator Gx 1 Or the fourth generator Gx 2 Comprising the following steps: two upsampling modules, a convolution module and a Tanh activation function.
7. The method of claim 1, wherein the first discriminator and the second discriminator calculate a loss value using a cross entropy loss function when performing an contrast training.
8. A sample expansion device for an x-ray image, comprising:
the acquisition module is used for acquiring a training data set formed by the x-ray images of the IC packaging module, and the training data set comprises: normal images and defective images;
the scaling module is used for scaling the training data set;
the first content encoder and the second content encoder are used for mapping the content of the normal image and the flaw image to a first shared feature space, and judging the membership degree of a feature encoding domain through a content discriminator, wherein the first shared feature space comprises: first signature coding Z for x-domain sharing x1 First signature code Z shared with y domain y1 ；
A first detail encoder for generating a first detail feature code from the scaled defective image
First generator G y1 The first generator G y1 For encoding according to said first detail featuresFirst signature code Z shared with the x domain x1 Generating a normal pseudo graph;
second generator Gx 1 The second generator Gx 1 Encoding Z according to a first characteristic shared by the y domains y1 Generating a flaw pseudo-graph;
the first discriminator is used for performing contrast training according to the flaw image and the normal pseudo-graph to acquire a normal pseudo-graph;
a second discriminator for performing an antagonistic training based on the normal image and the flaw artifact to obtain a flaw artifact;
a second detail encoder for generating a second detail feature code from the primary normal pseudo-graph
And a third content encoder and a fourth content encoder for weight sharing for mapping contents of the primary flaw-artifact and the primary normal-artifact, respectively, to a second shared feature space comprising: second signature encoding Z for x-domain sharing x2 Second signature code Z shared with y domain y2 ；
Third generator G y2 The third generator G y2 For encoding according to said second detail featuresSecond signature code Z shared with the x domain x2 Generating a reconstruction flaw pseudo-graph;
fourth generator Gx 2 The fourth generator Gx 2 Encoding Z according to a second characteristic shared by the y domains y2 Generating a reconstructed normal pseudo-graph, wherein the reconstructed flaw pseudo-graph and the flaw image adopt a cross-circulation consistency loss function to calculate a loss value, and the reconstructed normal pseudo-graph and the normal image adopt the cross-circulation consistency loss function to calculate the loss value;
and the expansion module is used for carrying out sample expansion of the x-ray image by adopting the reconstructed flaw pseudo-image.
9. The sample expansion device of claim 8, wherein the first content encoder, the second content encoder, the third content encoder, the fourth content encoder comprise: a full convolution network of two downsampled residual modules, the first detail encoder or the second detail encoder comprising: a convolution layer, a residual module and a full connection layer.
10. The sample expansion device of x-ray images of claim 8, wherein the content discriminator comprises: three layers of 5×5 downsampling convolution modules, one layer of 3×3 convolution layers and a LeakyReLU activation function, wherein the content discriminator adopts a cross entropy loss function to judge domain membership of feature codes, and the first generator G y1 Or the third generator G y2 Comprising the following steps: two residual modules, two upsampling modules and a convolution module, the upsampling modules comprising: bilinear interpolation, convolution layer, BN layer, and ReLU activation functions; the second generator Gx 1 Or the fourth generator Gx 2 Comprising the following steps: two upsampling modules, a convolution module and a Tanh activation function.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310458772.9A CN116167923B (en) | 2023-04-26 | 2023-04-26 | Sample expansion method and sample expansion device for x-ray image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310458772.9A CN116167923B (en) | 2023-04-26 | 2023-04-26 | Sample expansion method and sample expansion device for x-ray image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116167923A CN116167923A (en) | 2023-05-26 |
| CN116167923B true CN116167923B (en) | 2023-07-28 |
Family
ID=86414974
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310458772.9A Active CN116167923B (en) | 2023-04-26 | 2023-04-26 | Sample expansion method and sample expansion device for x-ray image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116167923B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111126433A (en) * | 2019-11-19 | 2020-05-08 | 佛山市南海区广工大数控装备协同创新研究院 | Positive and negative sample data balancing method in factory PCB defect detection |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3916635B1 (en) * | 2020-05-26 | 2023-05-10 | Fujitsu Limited | Defect detection method and apparatus |
| CN114943673A (en) * | 2021-02-09 | 2022-08-26 | 富泰华工业（深圳）有限公司 | Defect image generation method and device, electronic equipment and storage medium |
| CN115205616A (en) * | 2022-05-28 | 2022-10-18 | 浙江工业大学 | Cloth flaw data enhancement method based on generation countermeasure network |
| CN115409823A (en) * | 2022-09-06 | 2022-11-29 | 长沙理工大学 | Solar screen defect detection method based on generative countermeasure network |
-
2023
- 2023-04-26 CN CN202310458772.9A patent/CN116167923B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111126433A (en) * | 2019-11-19 | 2020-05-08 | 佛山市南海区广工大数控装备协同创新研究院 | Positive and negative sample data balancing method in factory PCB defect detection |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116167923A (en) | 2023-05-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111476719B (en) | Image processing method, device, computer equipment and storage medium | |
| CN112446890A (en) | Melanoma segmentation method based on void convolution and multi-scale fusion | |
| Couturier et al. | Image denoising using a deep encoder-decoder network with skip connections | |
| CN115222630A (en) | Image generation method, and training method and device of image denoising model | |
| CN114429151B (en) | Method and system for identifying and reconstructing magnetotelluric signals based on depth residual error network | |
| CN111899203B (en) | Real image generation method based on label graph under unsupervised training and storage medium | |
| CN115512368B (en) | Cross-modal semantic generation image model and method | |
| CN113344110B (en) | Fuzzy image classification method based on super-resolution reconstruction | |
| CN116958163A (en) | Multi-organ and/or focus medical image segmentation method and device | |
| CN117058160A (en) | Three-dimensional medical image segmentation method and system based on self-adaptive feature fusion network | |
| CN116994044A (en) | Construction method of image anomaly detection model based on mask multi-mode generation countermeasure network | |
| Liu et al. | Facial image inpainting using multi-level generative network | |
| CN116167923B (en) | Sample expansion method and sample expansion device for x-ray image | |
| CN114494387A (en) | Data set network generation model and fog map generation method | |
| CN112633123B (en) | Heterogeneous remote sensing image change detection method and device based on deep learning | |
| CN113298097A (en) | Feature point extraction method and device based on convolutional neural network and storage medium | |
| CN117593187A (en) | Remote sensing image super-resolution reconstruction method based on meta-learning and transducer | |
| CN117217997A (en) | Remote sensing image super-resolution method based on context perception edge enhancement | |
| CN112598575A (en) | Image information fusion and super-resolution reconstruction method based on feature processing | |
| CN117197470A (en) | Polyp segmentation method, device and medium based on colonoscope image | |
| US12112524B2 (en) | Image augmentation method, electronic device and readable storage medium | |
| CN114692831B (en) | Method for calculating variable convolution kernel for variable resolution, storage medium | |
| Zhao et al. | Generating Counterfactual Images: Towards a C2C-VAE Approach. | |
| CN113177938B (en) | Method and device for segmenting brain glioma based on circular convolution kernel and related components | |
| CN112346126B (en) | Method, device, equipment and readable storage medium for identifying low-order faults |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |