CN111402255B

CN111402255B - Side die failure detection method and system based on deep learning

Info

Publication number: CN111402255B
Application number: CN202010280045.4A
Authority: CN
Inventors: 张承瑞; 曹斌
Original assignee: Shandong University
Current assignee: Shandong University
Priority date: 2020-04-10
Filing date: 2020-04-10
Publication date: 2022-09-06
Anticipated expiration: 2040-04-10
Also published as: CN111402255A

Abstract

The invention provides a method and a system for detecting side die failure based on deep learning, which relate to the field of machine material detection, and are used for acquiring multi-angle images of a side die and preprocessing the images to obtain spliced images; obtaining a failure area of the side die by using the trained neural network model and taking the spliced image as input; the failure detection of all side forms after using and before using in the component production line can be realized, whether the side form can continue to use is judged, the condition of artifical hourglass inspection, false retrieval has been avoided, because the unqualified condition of product that the side form warp and cause takes place in having avoided prefabricated reinforced concrete component production process.

Description

Side die failure detection method and system based on deep learning

Technical Field

The disclosure relates to the field of machine material detection, in particular to a method and a system for detecting side die failure based on deep learning.

Background

The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.

In the modern building industry, the assembly type building obtains more and more favor with its advantage of safety, environmental protection, swift efficient, has driven the development of prefabricated reinforced concrete member processing industry simultaneously, need use the side forms in prefabricated reinforced concrete member production process, and the quality of side forms has decided the quality of the prefabricated reinforced concrete member of final production.

The inventor finds that if the side forms used are bent, abnormal deformation of the concrete member is caused, so that the final product is not in accordance with production requirements; after the maintenance of the prefabricated part is finished, a formwork removal process is performed, the purpose is to remove the side formwork from the maintained prefabricated part, the damage to the side formwork caused by the formwork removal process is increased, the failure probability of the side formwork is increased, and therefore corresponding failure detection is required before the side formwork is put in storage, however, a system for detecting whether the side formwork fails is not provided at present, and manual screening is mainly used; the manual screening is mainly based on experience to carry out subjective judgment, accurate identification is difficult to realize, misjudgment of side die failure is caused, the detection and screening efficiency is low, and the existing requirements for side die failure detection are difficult to meet.

Disclosure of Invention

The purpose of the disclosure is to provide a method and a system for detecting the failure of a side form based on deep learning, which can match with a conveying device to realize rapid and continuous failure detection of the side form and meet the requirements of failure detection of the side form by acquiring multi-angle image information of the side form, performing corresponding splicing processing and inputting a trained neural network model, determining a failure region and judging whether the side form fails or not.

The first purpose of the present disclosure is to provide a side mode failure detection method based on deep learning, which adopts the following technical scheme:

acquiring a multi-angle image of a side die and preprocessing the multi-angle image to obtain a spliced image;

obtaining a failure area of the side die by using the trained neural network model and taking the spliced image as input;

the neural network model is obtained based on the training of the failure side mode image and the normal side mode image.

And further, acquiring a plurality of images of the side surface and the top surface of each side die and splicing the images to obtain continuous and complete image information of the side surface and the top surface of the side die.

Further, the training of the neural network model specifically includes:

collecting side die images, including failure side die images and normal side die images;

carrying out data enhancement on the acquired image, expanding the data volume and establishing an image data set;

and framing and marking the failure area on the acquired failure side formwork image.

Further, the processing of the stitched image by using the neural network model specifically includes:

performing feature extraction on the spliced images to generate a group of feature maps;

defining a plurality of frame selection marks on the feature map as candidate regions, and comparing and judging the candidate regions with the feature map of the neural network model to reduce the number of the frame selection marks;

and performing ROI operation on the candidate region and the feature map of the neural network model to finally obtain a frame selection mark containing a failure region, and judging whether the side die fails.

Further, the failure of the side forms is the structural deformation of the side forms.

The second purpose of the present disclosure is to provide a side mode failure detection system based on deep learning, which adopts the following technical scheme:

the preprocessing module is configured to acquire multi-angle images of the side forms and perform image splicing;

and the data processing module is used for acquiring the spliced images and processing the spliced images by using the trained neural network model to obtain the failure area of the side die.

The third objective of the present disclosure is to provide a side mode failure detection system based on deep learning, which utilizes the side mode failure detection method based on deep learning as described above and adopts the following technical solutions:

a conveying device configured to carry and convey the sideform through the image acquisition device;

the image acquisition device is configured to acquire a multi-angle image of the side die to be detected and send the multi-angle image to the processor;

and the processor acquires multi-angle image data and executes the steps in the side mode failure detection method based on deep learning.

Further, still include:

the communication device is configured to acquire multi-angle image data of the image acquisition device and transmit the multi-angle image data to the processor;

and the manipulator is configured to grab the detected side forms and store the side forms respectively according to whether the side forms are invalid or not.

A fourth object of the present disclosure is to provide a computer-readable storage medium, on which a program is stored, which when executed by a processor, implements the steps in the deep learning-based side mode failure detection method as described above.

A fifth object of the present disclosure is to provide an electronic device, which includes a memory, a processor and a program stored in the memory and executable on the processor, and the processor executes the program to implement the steps in the method for detecting side mode failure based on deep learning as described above.

Compared with the prior art, the utility model has the advantages and positive effects that:

(1) the failure detection of all side forms in the component production line after and before use can be realized, whether the side forms can be used continuously is judged, the conditions of manual omission and false detection are avoided, and the condition that products are unqualified due to the deformation of the side forms in the production process of the prefabricated reinforced concrete component is avoided;

(2) the side forms are detected based on the deep learning algorithm, so that the system has self-learning capacity, the obtained data are more and more along with the increase of the detection times, the deep learning algorithm continuously performs self-learning and self-perfection through the accumulation of the data, the obtained result is more accurate, and stable and accurate judgment is finally realized;

(3) adopt the image acquisition device of conveying mechanism cooperation multi-angle, can carry out continuous transport and image acquisition to the side forms, guarantee side forms testing process continuity and detection efficiency, satisfy the demand of current component prefabrication work.

Drawings

The accompanying drawings, which are included to provide a further understanding of the disclosure, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure and are not to limit the disclosure.

Fig. 1 is a schematic overall structure diagram of a failure detection system in embodiment 3 of the present disclosure;

fig. 2 is a schematic flow chart of side mode failure detection in

embodiments

1, 2, and 3 of the present disclosure.

In the figure, 1, a first industrial camera, 2, a second industrial camera, 3, a camera support, 4, a conveyer belt, 5, a side die with a built-in RFID electronic chip, 6, an RFID reader, 7 and a mechanical arm.

Detailed Description

It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present disclosure. As used herein, the singular forms "a", "an", and/or "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof;

for convenience of description, the words "up", "down", "left" and "right" in this disclosure, if any, merely indicate that the directions of movement are consistent with those of the figures themselves, and are not limiting in structure, but merely facilitate the description of the invention and simplify the description, rather than indicate or imply that the referenced device or element must have a particular orientation, be constructed and operated in a particular orientation, and thus should not be construed as limiting the present disclosure.

As introduced in the background art, there is no system for detecting whether the side forms are failed in the prior art, which mainly relies on manual screening; the manual screening is mainly based on experience to carry out subjective judgment, accurate identification is difficult to realize, misjudgment of the side form failure is caused, the detection screening efficiency is low, the existing requirement for side form failure detection is difficult to meet, and aiming at the problems, the method and the system for detecting the side form failure based on deep learning are provided.

Example 1

In an exemplary embodiment of the present disclosure, as shown in fig. 2, a method for detecting a side mode failure based on deep learning is provided.

The method comprises the following steps:

establishing a neural network model, training, and collecting side mode images, including failure side mode images and normal side mode images;

framing and marking the failure area on the collected failure side die image;

obtaining a trained neural network model;

acquiring a plurality of images of the side surface and the top surface of each side die and splicing the images to obtain continuous and complete image information of the side surface and the top surface of the side die;

and obtaining the failure area of the side die by using the trained neural network model and taking the spliced image as input.

Specifically, the process of establishing the image data set in the neural network model comprises the following steps:

1) a large number of side mode images under the actual operation scene are collected, wherein the side mode images comprise failure side mode images and normal side mode images.

2) Data enhancement is performed on the acquired images to expand the data volume. In this embodiment, the enhancement mode may be flipping, rotating, scaling, cropping, translating, and noise.

3) And marking the failure region of the side die, wherein in the embodiment, the failure region is framed by a rectangular frame on the acquired image by adopting LabelImg software with an open source.

The judgment of whether the side model is invalid or not uses a Fast-RCNN target detection network, wherein the Fast-RCNN comprises three parts, the first part is a feature extraction network, the second part is target candidate region extraction, and the third part is a Fast-RCNN detector, and the method specifically comprises the following steps:

1) the stitched image is input into a feature extraction network, namely a Convolutional Neural Network (CNN), wherein ResNet101 is selected to generate a set of feature maps with specific sizes.

2) And (3) the generated feature map defines a plurality of rectangular frames as candidate regions at each pixel position according to different sizes and length ratios through a candidate region extraction network (RPN), and then the candidate regions are compared with labeled rectangular frames on the images marked in the image data set trained in advance to carry out classification judgment on whether the candidate regions are foreground or not and regression correction on the positions of the frames, so that the number of the candidate rectangular frames is reduced. Wherein the RPN is a fully convolutional neural network. The network input is extracted as an n x n window of the feature map, for each window there are k target candidate regions, called k anchors, the model uses a 3 x 3 window, so k is typically 9.

3) Performing ROI operation on the feature map by using a frame of the candidate region, mapping candidate rectangular frames with different scales screened by the extraction network back to the original image to generate feature maps with the same size, then classifying all classes of the feature maps, selecting a softmax classification network by the classification network, and performing coordinate regression on the frame position to finally obtain the rectangular frame containing the failure region.

And after the detection of the side die is completed, classifying the side die according to whether the side die fails or not.

The side forms are detected based on the deep learning algorithm, so that the system has self-learning capacity, the obtained data are more and more along with the increase of the detection times, the deep learning algorithm continuously self-learns and self-perfects through data accumulation, the obtained result is more accurate, and stable and accurate judgment is finally realized.

Example 2

In another exemplary embodiment of the present disclosure, a deep learning based side mode failure detection system is provided as shown in fig. 2.

and the data processing module is used for acquiring the spliced images and processing the images by using the trained neural network model to obtain the failure area of the side form.

The working method of the side form failure detection system refers to the side form failure detection method in embodiment 1, and is not described herein again.

Example 3

In another exemplary embodiment of the present application, as shown in fig. 1 and fig. 2, a side mode failure detection system based on deep learning is provided, which uses the detection method as described in embodiment 1.

As shown in fig. 1, the device mainly comprises a conveying device, an image acquisition device, a processor and a manipulator; wherein, the processor is internally integrated with an image processing module and a deep learning module;

the conveying device is used for conveying the side molds, the specific structure of the conveying device can select the existing conveying mechanisms such as a conveying belt, a driving roller and the like, and the conveying device is provided with a corresponding driving mechanism and a corresponding speed regulating mechanism which are respectively used for driving the whole conveying device to convey the side molds and regulating the speed of the moving side molds; in the present embodiment, a conveyor belt mechanism is selected;

of course, also can select other transport modes according to actual demand, when selecting transport mode, the side forms that will guarantee to carry can be more convenient acquire the image by image acquisition device, reduce sheltering from in transportation process, improve its comprehensive degree that acquires the side forms image.

in this embodiment, the image capturing device includes a first industrial camera 1, a second industrial camera 2, and a camera support 3, the first industrial camera 1 is perpendicular to the conveyor belt 4 and fixed on the camera support 3 at an angle relative to the center line of the conveyor belt 4, the view range includes the width of the entire conveyor belt 4 in the width direction, the second industrial camera 2 is fixed on the camera support 3 at an angle for horizontally shooting the conveyor belt 4, and the lower limit of the view range is the conveyor belt 4;

in order to ensure that the side forms can be accurately identified and shot in the conveying process, an identification structure is preset in the side forms, corresponding identification equipment is configured on a camera support, when the identification equipment is triggered, the side forms are judged to reach the shooting position, and a camera is started;

in the embodiment, the RFID reader 6 is selected to be matched with an RFID electronic chip for identification, wherein the RFID electronic chip is preset in the side die 5; the RFID reader can read the passing side forms with the built-in RFID electronic chips, and the camera support is arranged behind the RFID reader;

the two

cameras

1 and 2 start photographing simultaneously when the RFID reader 6 reads that the side forms 5 with the built-in RFID electronic chips pass by, the conveyer belt 4 takes one photo at each walking camera view range, photographing is stopped when the walking distance is larger than the length of the side forms 5, and the length of the side forms 5 is recorded by the built-in RFID electronic chips. And the collected pictures are transmitted to the side die image processing module for processing.

Adopt the image acquisition device of conveying mechanism cooperation multi-angle, can carry out continuous transport and image acquisition to the side forms, guarantee side forms testing process continuity and detection efficiency, satisfy the demand of current component prefabrication work.

The image processing module correspondingly splices the pictures, and then the deep learning module analyzes the obtained data and carries out failure detection judgment on the side forms based on the neural network model;

specifically, the steps in the side mode failure detection method based on deep learning as described in embodiment 1 may be performed.

Further, the method also comprises the following steps: the communication device is configured to acquire multi-angle image data of the image acquisition device and transmit the multi-angle image data to the processor;

In this embodiment, the communication device may select a wireless communication module, or may select other communication modes, such as wired communication, to ensure stability of data transmission.

The manipulator is an existing commodity manipulator, the tail end actuating mechanism of the manipulator is matched with a structure capable of stably grabbing the side forms, data are obtained from a processor, and if the side forms fail, the side forms are placed in a waste material area by the mechanical arm 7 to wait for processing; if the side die is not invalid, the side die is put into a warehouse by the mechanical arm 7 and is continuously used.

The failure detection of all side forms after using and before using in the component production line can be realized, whether the side form can continue to use is judged, the condition of artifical hourglass inspection, false retrieval has been avoided, because the unqualified condition of product that the side form warp and cause takes place in having avoided prefabricated reinforced concrete component production process.

Example 4

In another embodiment of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and the program, when executed by a processor, implements the steps in the side mode failure detection method based on deep learning according to embodiment 1.

Example 5

In another embodiment of the present disclosure, an electronic device is provided, which includes a memory, a processor, and a program stored in the memory and executable on the processor, and the processor executes the program to implement the steps in the side mode failure detection method based on deep learning according to embodiment 1.

The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure, and various modifications and changes may be made to the present disclosure by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. The side mode failure detection method based on deep learning is characterized by comprising the following steps:

the neural network model is obtained based on the training of the failure side mode image and the normal side mode image;

acquiring a plurality of images of the side surface and the top surface of each side die and splicing the images to obtain continuous and complete image information of the side surface and the top surface of the side die; the processing of the spliced image by using the neural network model specifically comprises the following steps:

and performing ROI operation on the candidate area and the feature map of the neural network model, mapping candidate rectangular frames with different scales screened by the extraction network back to the original image to generate feature maps with the same size, classifying all the categories of the feature maps, selecting a softmax classification network by the classification network, performing coordinate regression on the position of a frame, finally obtaining a framing mark containing a failure area, and judging whether the side model fails or not.

2. The deep learning-based side mode failure detection method of claim 1, wherein the training of the neural network model specifically comprises:

3. The deep learning-based side form failure detection method of claim 1, wherein the side form failure is structural deformation of the side form.

4. Side mode failure detection system based on degree of depth study, its characterized in that includes:

the preprocessing module is configured to acquire multi-angle images of the side forms and perform image splicing; acquiring a plurality of images of the side surface and the top surface of each side die and splicing the images to obtain continuous and complete image information of the side surface and the top surface of the side die; the data processing module is used for acquiring the spliced images and processing the spliced images by using the trained neural network model to obtain a failure area of the side form; the processing of the spliced image by using the neural network model specifically comprises the following steps:

and performing ROI operation on the candidate area and the feature map of the neural network model, mapping candidate rectangular frames with different scales screened by the extraction network back to the original image to generate feature maps with the same size, classifying all classes of the feature maps, selecting a softmax classification network by the classification network, regressing coordinates of frame positions, finally obtaining a framing mark containing a failure area, and judging whether the side model fails.

5. Side mode failure detection system based on degree of depth study, its characterized in that includes:

a processor for acquiring multi-angle image data and executing the steps of the side mode failure detection method based on deep learning according to any one of claims 1-3.

6. The deep learning-based side mode failure detection system of claim 5, further comprising:

7. A computer-readable storage medium, on which a program is stored, which, when being executed by a processor, carries out the steps of the deep learning based side mode failure detection method according to any one of claims 1 to 3.

8. An electronic device comprising a memory, a processor and a program stored on the memory and executable on the processor, wherein the processor implements the steps of the deep learning based side mode failure detection method according to any one of claims 1-3 when executing the program.