CN110363781B - Molten pool contour detection method based on deep neural network - Google Patents
Molten pool contour detection method based on deep neural network Download PDFInfo
- Publication number
- CN110363781B CN110363781B CN201910581281.7A CN201910581281A CN110363781B CN 110363781 B CN110363781 B CN 110363781B CN 201910581281 A CN201910581281 A CN 201910581281A CN 110363781 B CN110363781 B CN 110363781B
- Authority
- CN
- China
- Prior art keywords
- molten pool
- image
- network
- original
- data set
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/12—Edge-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/155—Segmentation; Edge detection involving morphological operators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention discloses a molten pool contour detection method based on a deep neural network, which comprises the steps of collecting a molten pool image during welding, cutting out redundant background to obtain an original molten pool image set; making a segmentation mark sample, and forming a molten pool picture mark data set with an original molten pool image set; training by utilizing a generated countermeasure network DCGAN to generate an image similar to an original molten pool image, and forming an expanded molten pool image marking data set with the molten pool image marking data set; and (4) performing data augmentation on colorimetry and morphology, putting the data augmentation into a semantic segmentation network for training, and extracting the outline of the molten pool. According to the method, the network model has better generalization capability while saving manpower and time, and the segmentation precision of the weak edge area of the molten pool is better improved.
Description
Technical Field
The invention belongs to the field of machine vision, and particularly relates to a molten pool contour detection method based on a deep neural network.
Background
Automatic welding greatly liberates manpower and is widely applied to intelligent production. The computer vision is used for extracting and analyzing the outline of the molten pool, which is beneficial to detecting and controlling the automatic welding process in real time and ensuring the quality of the welding product. In the traditional method, Canny operators, CV active contours and the like are adopted for edge detection, however, molten pool images are influenced by welding processes and materials in the actual situation, the situations that the gray distribution is uneven and the molten pool images are interfered by arc light easily occur, and at the moment, the traditional method is difficult to extract accurate and complete closed molten pool contours.
Disclosure of Invention
The invention aims to provide a molten pool contour detection method based on a deep neural network.
The technical solution for realizing the purpose of the invention is as follows: a molten pool contour detection method based on a deep neural network comprises the following steps:
step 1, establishing a molten pool visual sensing system, collecting a molten pool image during welding, and cutting out redundant background to obtain an original molten pool image set;
step 2, manufacturing a corresponding segmentation mark sample based on the molten pool image obtained in the step 1, and forming a molten pool image mark data set with the original molten pool image set;
step 3, training by using a generation countermeasure network DCGAN based on the molten pool picture marking data set obtained in the step 2, generating an image similar to the original molten pool image, and forming an expanded molten pool picture marking data set with the molten pool picture marking data set;
step 4, performing data augmentation on colorimetry and morphology on the expanded molten pool image marking data set, and putting the expanded molten pool image marking data set into a semantic segmentation network for training;
and 5, extracting the outline of the molten pool by using the semantic network model obtained by training in the step 4.
Compared with the prior art, the invention has the following remarkable advantages: 1) by utilizing the method for generating the similar images of the countermeasure network and controlling the data amplification strength of the random number, under the condition that the quantity of the data set is not enough to achieve the precision of the segmentation task, the network model has better generalization capability while saving labor and time; 2) the characteristics of different scales of the residual error network are fused, so that the segmentation precision of the weak edge area of the molten pool is better improved.
Drawings
FIG. 1 is a schematic diagram of a visual molten pool sensing system established by the present invention.
FIG. 2 is an image of a molten pool after the window of the present invention has been cropped.
FIG. 3 is a schematic view of a weld puddle image and a corresponding marked sample according to the present invention.
Fig. 4 shows generated images during the DCGAN training process of the present invention, wherein (a) is generated image with an epoch-10, (b) is generated image with an epoch-120, and (c) is generated image with an epoch-300.
FIG. 5 is a generated image after segmentation and screening in accordance with the present invention.
Fig. 6 is a schematic diagram of a mark sample corresponding to the generated image and the original image according to the present invention.
Fig. 7 is a schematic structural diagram of a Res-Seg network according to the present invention.
FIG. 8 is a flow chart of data augmentation according to the present invention.
FIG. 9 is a flow chart of the method of the present invention
FIG. 10 is a graph showing the test results of the present invention, wherein (a) is the test result of Canny, (b) is the test result of CV, (c) is the test result of ENet, (d) is the test result of ResNet50, and (e) is the test result of the algorithm of the present invention.
Detailed Description
The invention is further illustrated by the following examples in conjunction with the accompanying drawings.
A molten pool contour detection method based on a deep neural network comprises the following steps:
step 1, establishing a molten pool visual sensing system, collecting a molten pool image during welding, and cutting out redundant background to obtain an original molten pool image set;
the established molten pool vision sensing system is shown in figure 1, and because a large arc exists at the contact part of a welding gun and a workpiece under the TIG welding process, in order to inhibit the adverse effect of the arc on the outline detection, a 660nm optical filter is arranged in front of a camera lens to obtain a clear high-quality image. Because only image information of the area around the molten pool is concerned by the invention, background information is redundant, and the less the background information is, the better the background information is, therefore, the collected molten pool color image (original image) is subjected to window cropping by taking the molten pool area as an important point. After collecting 1920 × 1200 weld pool images, 400 × 400 window cropping is performed according to the weld pool area, and the image after cropping is shown in fig. 2.
Step 2, manufacturing a corresponding segmentation mark sample based on the molten pool image obtained in the step 1, and forming a molten pool image mark data set with the original molten pool image set;
when the corresponding division mark sample is created, the molten pool area to be divided is filled with pixels having a grayscale of 255, and the background area is filled with pixels having a grayscale of 0, as shown in fig. 3.
Step 3, training by utilizing a generation countermeasure network DCGAN based on the molten pool picture marking data set obtained in the step 2, generating an image similar to the original molten pool image, and forming an expanded molten pool picture marking data set with the molten pool picture marking data set;
firstly, images of an original molten pool image set are spliced into one image according to a set Batch Size (such as Batch Size 4), and the image is sequentially sent into a generation countermeasure network DCGAN. Every 100 times of training of the Size of the Batch Size, the pictures in the original data set are tested, the test result is shown in fig. 4, and it can be seen that in the training process, along with the increase of the number of training rounds (epoch), the generated pictures are gradually clear and are closer to the original molten pool image. After training, the images of the original molten pool image set are tested by using the generated network model, and a set number (for example, 100) of generated maps spliced by the Size of Batch Size are generated. And then cutting the generated picture into a single molten pool picture, and reserving the generated picture with better quality after screening, as shown in figure 5. Finally, as shown in fig. 6, the newly generated melt pool image is searched for the corresponding original image and the split mark sample (label), and the expanded melt pool image mark data set is formed together. This manner of data replenishment increases the sample size of the data set in a manner that saves labor and time (without the need for remanufacturing redundant markers).
Step 4, performing data augmentation on colorimetry and morphology on the expanded molten pool image marking data set, and putting the expanded molten pool image marking data set into a semantic segmentation network for training;
a designed semantic segmentation network Res-Seg is shown in FIG. 7, the network evolves based on a residual error network ResNet-50, firstly, the ResNet-50 network is used for carrying out layer-by-layer progressive convolution operation on input to obtain feature maps with different scales, and if a segmentation result with the same size as that of an input picture is obtained, the feature maps obtained by convolution need to be sampled to the size of an original picture, and a loss function is calculated with a marked sample;
as shown in FIG. 7, feature f based on artwork 1/32 size 1/32 After the upsampling operation, the restored picture is simply the last scale convolutionFeatures in the convolution kernel in a layer are used only to determine the segmentation result, which is relatively flat. Thus, the present invention employs a forward iteration of first f 1/32 Upsampling to restore feature f with size of 1/16 scale of original image 1/16 Is then convolved with f output in the convolution process 1/16 Additive fusion is performed. The fused features are subjected to the same operation as the feature f of 1/8 original image scale in the convolution process 1/8 Performing addition fusion, and up-sampling the obtained features to the original image size, wherein the operation is shown in the formula (1), so that the features of the lower layer and the upper layer are fully fused and output, and the target segmentation precision is effectively improved;
where D represents the upsampling operation.
In addition, the network loss function is designed as shown in equation (2):
wherein E is a softmax function, i and j determine whether the pixel is located in the foreground area f to be segmented g Middle or background area b g In, y ij And representing the binary predicted value of the pixel, wherein a is the pixel ratio of the background to the image, and b is the pixel ratio of the foreground to the image.
Before the data set is put into a Res-Seg network for training, data augmentation is carried out on the molten pool image and the corresponding marked sample, and the steps are shown in figure 8, and the operation of rotating, cutting and other morphological changes, color adjustment and the like are introduced into the molten pool image. The requirement of the generalization capability of the model makes these operations need to have the capability of adjusting the intensity (the angle of the picture rotation, the size of the brightness adjustment, etc.), and the concept of random number control is introduced here, and before each time the picture is transmitted into the network for training, the intensity of the aforementioned operation is adjusted according to the generated random number and then transmitted into the network. Therefore, the trained network model is more robust. The augmentation steps are as follows:
a. setting a maximum rotation angle, a maximum scaling value (scale) and a maximum cropping length and width value;
b. generating a random floating point number M between 0 and 1, multiplying the random floating point number M with the maximum rotation angle, the maximum scale value and the maximum cutting length and width value by taking the M as a reference to generate a random floating point number for controlling form change, and controlling rotation, scaling and cutting operations; multiplying the color image by scale by using 2M-1 as a reference to generate a random floating point number for controlling color change, and controlling brightness, saturation, contrast, sharpness and Gaussian blur operation;
c. and b, loading the molten pool image of the expanded molten pool image marking data set and the corresponding label thereof, executing rotation and scaling operation, judging whether to perform cutting and color conversion operation according to the positive and negative of the random floating point number for controlling color change in the step b, if so, not performing cutting and color conversion operation, otherwise, performing cutting and color conversion on the image and the marking sample.
And 5, extracting the outline of the molten pool by using the semantic network model obtained by training in the step 4.
Examples
In order to verify the effectiveness of the scheme of the invention, a network model obtained by training is utilized to test a molten pool image in a test set, and the comparison result of the segmentation result, which is obtained by contouring and then superposed on an original color image, and a traditional algorithm is shown in fig. 9. It can be seen that the contour line extracted by the method is smoother and closer to the real boundary of the molten pool than the contour line extracted by the traditional contour line extraction method or the traditional semantic segmentation network.
Calculating the segmentation accuracy, P, of the target and background according to equation (3) ii Representing correctly classified pixels, P ij (i ≠ j) represents the misclassified pixels, k represents the total number of classes:
the segmentation accuracy of the method of the present invention and the conventional method is shown in table 1. The table shows that the segmentation precision of the method is improved compared with that of the traditional method, and the mode of supplementing the data set and fusing the multilayer characteristics of the residual error network based on the generation of the countermeasure network is favorable for improving the segmentation task precision.
TABLE 1 segmentation accuracy comparison Table of object and background
In order to verify the robustness of the network model, tests were also performed on data outside the test set, and the test results are shown in table 2. The accuracy of the invention can reach 92%, which improves the accuracy of about 2 percentage points on the original ResNet-50 basic network and improves the accuracy of about 7 percentage points on the original ResNet-101 basic network, wherein one important reason is that: the ResNet-101 network has too many layers, a complex structure and too many parameters, so that the network is over-fitted on training data, and the training model cannot show a good effect on unknown data except the training data.
TABLE 2 comparison Table of contour extraction accuracy
Claims (5)
1. A molten pool contour detection method based on a deep neural network is characterized by comprising the following steps:
step 1, establishing a molten pool visual sensing system, collecting a molten pool image during welding, and cutting out redundant background to obtain an original molten pool image set;
step 2, manufacturing a corresponding segmentation mark sample based on the molten pool image obtained in the step 1, and forming a molten pool image mark data set with the original molten pool image set;
step 3, training by using a generation countermeasure network DCGAN based on the molten pool picture marking data set obtained in the step 2, generating an image similar to the original molten pool image, and forming an expanded molten pool picture marking data set with the molten pool picture marking data set;
step 4, performing data augmentation on the expanded molten pool image marking data set in terms of color and morphology, and putting the expanded molten pool image marking data set into a semantic segmentation network for training;
step 5, extracting the outline of the molten pool by utilizing the semantic network model obtained by training in the step 4;
in the step 4, a semantic segmentation network is evolved based on a residual error network ResNet-50, firstly, the ResNet-50 network is utilized to carry out layer-by-layer progressive convolution operation on input to obtain feature maps with different scales, then the feature maps with the same size of the original image are obtained according to the operation of the formula (1), and a loss function is calculated by the feature maps and the marked sample;
wherein D represents an upsampling operation, f 1/32 Feature based on original 1/32 size, f, representing convolution output 1/16 Features based on the original 1/16 size that represent the convolution output;
the loss function design of the semantic segmentation network is shown as the formula (2):
wherein E is a softmax function, i and j determine whether the pixel is located in the foreground area f to be segmented g Middle or background area b g In, y ij And representing the binary predicted value of the pixel, wherein a is the pixel ratio of the background to the image, and b is the pixel ratio of the foreground to the image.
2. The molten pool profile detection method based on the deep neural network as claimed in claim 1, wherein in step 1, in order to suppress adverse effects of arc on profile detection, a 660nm filter is arranged in front of a camera lens to acquire a clear high-quality image, considering that a large arc exists at a contact part of a welding gun and a workpiece under a TIG (tungsten inert gas) welding process.
3. The molten pool profile detection method based on the deep neural network as claimed in claim 1, wherein in step 2, when the segmentation marker sample is produced, the molten pool area to be segmented is filled with pixels with a gray scale of 255, and the background area is filled with pixels with a gray scale of 0.
4. The molten pool profile detection method based on the deep neural network as claimed in claim 1, wherein in step 3, images of an original molten pool image set are firstly spliced according to a set batch size and are sequentially sent into a generation countermeasure network DCGAN; then, after training is finished, testing the images of the original molten pool image set by using the generated network model to generate a generated image spliced according to batch size; cutting the generated drawing into a single molten pool picture, and screening the molten pool picture meeting the conditions; and finally, searching the corresponding original image and the segmentation mark sample according to the newly generated molten pool image to jointly form an expanded molten pool image mark data set.
5. The molten pool profile detection method based on the deep neural network as claimed in claim 1, wherein in step 4, before the data set is put into a Res-Seg network for training, the molten pool image and the corresponding marked sample are subjected to data augmentation through morphological change and color adjustment, and the specific steps are as follows:
a. setting a maximum rotation angle, a maximum scaling value and a maximum cutting length and width value;
b. generating a random floating point number M between 0 and 1, multiplying the random floating point number M with the maximum rotation angle, the maximum scaling ratio value and the maximum cutting length and width value by taking the M as a reference to generate a random floating point number for controlling form change, and controlling rotation, scaling and cutting operations; multiplying the maximum scaling value by 2 × M-1 to generate a random floating point number for controlling color change and control brightness, saturation, contrast, sharpness and Gaussian blur operation;
c. and b, loading the molten pool image of the expanded molten pool image marking data set and the corresponding segmentation marking sample thereof, executing rotation and scaling operation, judging whether to perform cutting and color conversion operation according to the positive and negative of the random floating point number for controlling color change in the step b, if so, not performing cutting and color conversion operation, otherwise, performing cutting and color conversion on the image and the marking sample.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910581281.7A CN110363781B (en) | 2019-06-29 | 2019-06-29 | Molten pool contour detection method based on deep neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910581281.7A CN110363781B (en) | 2019-06-29 | 2019-06-29 | Molten pool contour detection method based on deep neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110363781A CN110363781A (en) | 2019-10-22 |
| CN110363781B true CN110363781B (en) | 2022-08-12 |
Family
ID=68215976
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910581281.7A Active CN110363781B (en) | 2019-06-29 | 2019-06-29 | Molten pool contour detection method based on deep neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110363781B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102220823B1 (en) * | 2019-12-31 | 2021-02-26 | 한국과학기술원 | Method and apparatus for estimating depth of molecular pull during 3d processing, and 3d printing system having the same |
| CN111429423B (en) * | 2020-03-19 | 2022-04-08 | 南通大学 | Multi-spectral welding image identification method based on quaternion multi-degree-of-freedom neurons |
| CN111932539B (en) * | 2020-10-13 | 2021-02-02 | 南京知谱光电科技有限公司 | Molten pool image and depth residual error network-based height and penetration collaborative prediction method |
| CN113077423B (en) * | 2021-03-22 | 2023-06-09 | 中国人民解放军空军工程大学 | Laser selective melting pool image analysis system based on convolutional neural network |
| CN112906833B (en) * | 2021-05-08 | 2021-08-17 | 武汉大学 | Plasma energy deposition image identification method based on full convolution neural network |
| CN113343996B (en) * | 2021-05-11 | 2022-10-11 | 武汉大学 | Method for calculating directional energy deposition area of plasma based on deep convolutional network |
| CN113554587A (en) * | 2021-05-31 | 2021-10-26 | 江苏大学 | Molten pool image geometric feature extraction method and system based on deep learning |
| CN113240668B (en) * | 2021-06-08 | 2024-04-16 | 南京师范大学 | Image digital feature distribution-based generated molten pool image quality evaluation method |
| CN113673529A (en) * | 2021-08-16 | 2021-11-19 | 连城凯克斯科技有限公司 | Semantic segmentation model training method, silicon fusion state detection method and electronic equipment |
| CN115424129B (en) * | 2022-10-13 | 2023-08-11 | 哈尔滨市科佳通用机电股份有限公司 | Abnormality detection method and abnormality detection system for wallboard damage |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8422769B2 (en) * | 2010-03-05 | 2013-04-16 | Microsoft Corporation | Image segmentation using reduced foreground training data |
| CN105243355A (en) * | 2015-09-09 | 2016-01-13 | 大连理工大学 | Event-driven remote wireless coalbed methane well station abnormal scene safety monitoring method |
-
2019
- 2019-06-29 CN CN201910581281.7A patent/CN110363781B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110363781A (en) | 2019-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110363781B (en) | Molten pool contour detection method based on deep neural network | |
| CN109859171B (en) | Automatic floor defect detection method based on computer vision and deep learning | |
| CN110717872B (en) | Method and system for extracting characteristic points of V-shaped welding seam image under laser-assisted positioning | |
| CN105571502A (en) | Measuring method of weld gap in friction-stir welding | |
| CN111833306A (en) | Defect detection method and model training method for defect detection | |
| CN111932572B (en) | Aluminum alloy molten pool contour extraction method | |
| CN116597391B (en) | Synchronous on-line monitoring method for weld surface morphology and penetration state | |
| CN113379740A (en) | VPPAW fusion in-situ real-time monitoring system based on perforation molten pool image and deep learning | |
| CN110599453A (en) | Panel defect detection method and device based on image fusion and equipment terminal | |
| CN114926407A (en) | Steel surface defect detection system based on deep learning | |
| CN112419237A (en) | Automobile clutch master cylinder groove surface defect detection method based on deep learning | |
| CN114155372A (en) | Deep learning-based structured light weld curve identification and fitting method | |
| CN115719332A (en) | Welding quality detection method | |
| CN114743102A (en) | Furniture board oriented flaw detection method, system and device | |
| CN106770318A (en) | Weld defect real-time detection method based on improved background subtraction | |
| CN117094975A (en) | Method and device for detecting surface defects of steel and electronic equipment | |
| CN109558877B (en) | KCF-based offshore target tracking algorithm | |
| CN103743750A (en) | Method for generating distribution diagram of surface damage of heavy calibre optical element | |
| CN111523411B (en) | Synthetic aperture imaging method based on semantic patching | |
| CN113506240A (en) | Metallographic phase crystal boundary extraction and communication method based on deep learning | |
| CN117252815A (en) | Industrial part defect detection method, system, equipment and storage medium based on 2D-3D multi-mode image | |
| CN108705689B (en) | Automatic tool setting process method for repairing surface microdefects of large-caliber precise optical crystal | |
| CN106600606A (en) | Ship painting profile detection method based on image segmentation | |
| Rill-García et al. | Syncrack: Improving Pavement and Concrete Crack Detection Through Synthetic Data Generation | |
| CN115578362A (en) | Defect detection method and device for electrode coating, electronic device and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |