CN114913370A

CN114913370A - State automatic detection method and device based on deep learning and morphology fusion

Info

Publication number: CN114913370A
Application number: CN202210487272.3A
Authority: CN
Inventors: 张益辉; 张颖; 吴灏; 傅伯雄; 王丽华; 董璇; 李炀; 苏克
Original assignee: State Grid Corp of China SGCC; Shijiazhuang Power Supply Co of State Grid Hebei Electric Power Co Ltd
Current assignee: State Grid Corp of China SGCC; Shijiazhuang Power Supply Co of State Grid Hebei Electric Power Co Ltd
Priority date: 2022-05-06
Filing date: 2022-05-06
Publication date: 2022-08-16

Abstract

The invention relates to a state automatic detection method and a state automatic detection device based on deep learning and morphology fusion; firstly, carrying out target positioning and initial type analysis on an image frame of a protective pressing plate, and automatically identifying acquired images of a type I protective pressing plate and a type II protective pressing plate to obtain an identification state; extracting image characteristics by using a characteristic parameter extraction method to obtain characteristic parameters; and fusing the recognition state and the characteristic state based on the fusion information to determine the final detection state of the type I and type II protective pressing plates. The state automatic detection method and device based on deep learning and morphology fusion provided by the invention can realize effective detection of states of different types of protection pressing plates, are accurate in identification, high in applicability and good in robustness, are beneficial to monitoring and controlling the real-time state of the protection pressing plates, are efficient, practical, objective and accurate in scheme, and can be used for assisting in inspection of the protection pressing plates and electric power in an electric power system, so that the safety of the electric power system is better ensured.

Description

State automatic detection method and device based on deep learning and morphology fusion

Technical Field

The invention relates to the field of image analysis and processing, in particular to a state automatic detection algorithm method and device based on deep learning and morphology fusion.

Background

The protection pressing plate is also called as a protection connecting sheet, is used for protecting bridges and ties of the power device, which are connected with external wiring, and is related to the function of protection and whether an action outlet can normally play a role. In a power system, a relay protection system cannot be separated from normal operation and maintenance, and a protection pressing plate is an important equipment component for ensuring the normal operation of the power system and is an important part in relay protection routing inspection. Therefore, the method which is convenient, rapid, accurate and appropriate to select is of great significance to the safety performance evaluation of the power system when the state of the protection pressing plate is monitored and patrolled. However, the traditional manual inspection has large workload, and the long-time repetitive work easily causes visual fatigue, so that the efficiency is low, and even misoperation occurs. With the rapid development of society, power systems are also developing towards the trend of intellectualization and informatization. Therefore, the artificial intelligence method is adopted for carrying out auxiliary detection on the protective pressing plate, so that the artificial pressure is reduced, and the reliability of the system is improved.

In recent years, image processing techniques are increasingly used for automatic detection of the state of the protective platen. For example, in the prior art, the OTSU algorithm is improved, that is, the influence of a shadow area caused by illumination is eliminated by threshold processing, and the identification of the state of the pressing plate is realized by using a Graham-based minimum circumscribed rectangle algorithm; or aiming at highlight interference caused by reflection of the protection screen cabinet, detecting a highlight area by utilizing a two-dimensional maximum class, eliminating highlight by improving a sparse algorithm, and identifying the state by using a minimum circumscribed rectangle; and the state of the pressing plate and the corresponding character label are automatically identified based on an image processing technology, and the identification accuracy is obtained by acquiring the row number and the column number of the pressing plate and comparing the row number and the column number with the position information of the pressing plate in the database. Chinese patent CN111915509A discloses a protection pressing plate state identification method based on image processing shadow removal optimization, which comprises the steps of graying a color image of a protection pressing plate, converting the color image into a gray image, and then enhancing contrast and binarizing the gray image to eliminate a shadow area. And obtaining a convex hull of each protection pressure plate switch through a Graham algorithm principle, and then connecting the convex hulls into a rectangle through a minimum external rectangle principle to obtain the rectangular area. And setting a threshold value for the rectangular area, judging that the rectangular area is thrown out if the rectangular area is larger than the threshold value, and otherwise, judging that the rectangular area is thrown in. The method can effectively reduce the influence of shadow interference, and has low requirement on the quality of the acquired image and strong robustness; the influence of shadow interference can be effectively reduced, and the running state of the pressing plate in the image can be accurately identified.

In the state recognition of the protective pressing plate, the traditional image processing method and the deep learning both realize a better recognition result. However, the traditional image processing algorithm is easily affected by environmental factors, such as illumination shadow and the like, which bring difficulty to identification. Meanwhile, in the prior art, the state of only one type of protection pressing plate is generally recognized, while the relay protection pressing plates in the actual operation environment are frequently of various types, and automatic state detection needs to be performed on various types of protection pressing plates.

Therefore, in order to meet the automatic identification requirements of various types of protection pressing plates, the identification and inspection processes are more visual, efficient and low in cost, a new identification mode is urgently needed for carrying out protection pressing plate and power inspection on the power system, and the safety performance of the power system is better guaranteed.

Disclosure of Invention

In order to accurately identify the types and states of different protection pressing plates, the invention provides an automatic protection pressing plate state detection method based on a YOLOv5 deep learning network and color space and morphology fusion characteristics, and the effective identification of the states of two protection pressing plates is realized.

The technical scheme adopted by the invention is as follows:

a state automatic detection method based on deep learning and morphology fusion is used for automatically detecting a protective pressing plate and is characterized by comprising the following steps: s01, carrying out target positioning and initial type analysis on the collected image frames of the protective pressing plate, and acquiring the corresponding types of the target protective pressing plate in each image frame, wherein the types are respectively marked as type I and type II; s02, preprocessing the image collected by the I-type protective pressing plate, and automatically identifying the working state of the I-type protective pressing plate by using a deep learning target network to obtain a first identification state; s03, extracting image characteristics of the I-type protective pressing plate by using a cubic spline difference value and color space characteristic parameter extraction method to obtain a first characteristic parameter; s04, detecting a first characteristic state corresponding to the protective pressing plate based on the first characteristic parameter, fusing the first identification state and the first characteristic state based on the fusion information, and determining the final detection state of the I-type protective pressing plate; s05, automatically identifying the state of the II-type protective pressing plate by using an improved deep learning network to obtain a second identification state; s06, extracting image characteristics of the II-type protective pressing plate by a characteristic parameter extraction method based on color space and morphological fusion characteristics to obtain second characteristic parameters; and S07, determining the final detection state of the II type protective pressure plate.

Further, a method of determining a final detection state of the type II protective pressing plate of S07: and detecting a second characteristic state corresponding to the II-type protective pressing plate based on the second characteristic parameter, and fusing the second identification state and the second characteristic state based on the fusion information.

Preferably, step S01 includes:

firstly, sobel edge detection is carried out on an input protection pressing plate image frame, after edge information of a picture is obtained, maximum and minimum filtering processing is carried out on the picture, meanwhile, after binary information of the picture is obtained through global threshold processing, a small connected domain is removed according to the area of the connected domain, each protection pressing plate object in the picture is obtained through a connected domain external rectangle, and finally, linear detection is carried out on each object to judge whether a target protection pressing plate belongs to the type I or the type II.

Preferably, the preprocessing of the collected image of the I-type protective platen in step S02 includes:

and respectively extracting sobel operators in the x direction, the y direction and the 135-degree direction of the images, combining the three images in proportion to obtain a final characteristic image after gradient processing, and finally using the final characteristic image as the input of a target network for identification.

Preferably, step S03 includes:

s301, firstly, using an image resolution reduction method based on cubic spline interpolation for an original image;

s302, preprocessing the image of the protective pressing plate by utilizing a histogram equalization method based on contrast ratio limiting self-adaption;

s303, after median filtering is carried out on the image, the image is segmented based on HSV color space, and the initial segmentation image obtained after three-channel binarization images are superposed is subjected to morphological processing to obtain a final binarization segmentation result;

s304, extracting shape characteristic parameters of the protective pressing plate by using a connected domain-based external rectangular parameter extraction method, so as to optimize and supplement the recognition result of deep learning.

Preferably, step S04 includes:

s401, detecting a first characteristic state corresponding to the protective pressing plate based on the first characteristic parameter;

obtaining first characteristic parameters h, w, S1 and S2 of the I-type protective pressure plate through the previous steps, wherein h and w are respectively the length and the width of a circumscribed rectangular frame of the protective pressure plate, S1 and S2 are respectively characteristic values obtained according to the height h and the width w, S1 is the area of the circumscribed rectangle of the protective pressure plate, S2 is the average value of the areas of all detected circumscribed rectangles of the protective pressure plate, and the formula (5) to (6) shows that:

S1＝h*w (5)

then, judging the state A of the I-type protection pressing plate by a single parameter threshold judgment method, wherein the selection formula is shown as the formula (7):

according to the automatic selection of the parameter range, two states A corresponding to the I-type protective pressing plate can be obtained ₁ 、A ₂ Wherein A is ₁ In the input state, A ₂ Is in an exit state;

s402, automatically detecting the state of the protective pressing plate based on the fusion state information;

the target protection pressing plate has rectangular frame information A after the first characteristic parameter is extracted _r (x, y, h, w) while there is a rectangular area B of the target protective platen in the first recognition state result _r ；

A is to be _r And B _r Performing intersection judgment, if A _r And B _r If the intersection is empty, the final state of the I-type protection pressing plate is A _i (ii) a If A is _r And B _r If the intersection is not empty, the final state of the I-type protective pressing plate is B _i And finally obtaining the final detection state of all objects in the image of the protective pressing plate as E, wherein the final detection state is shown as the formula (8):

preferably, the step S05 of automatically recognizing the type II protective platen status using the modified deep learning network includes:

in order to suppress interference of the complex background of the type II protective platen, a channel and space convolution block attention model, a channel attention module and a space attention module were introduced behind the CSP module of the YOLOv5 network model.

Preferably, step S06 includes:

s601, firstly, automatically detecting the highlight area of an original image based on pixel difference, and repairing the highlight area of the image by using a fast marching algorithm;

s602, then, realizing the protection pressing plate foreground segmentation by using a watershed algorithm;

s603, carrying out image segmentation based on HSV color space, and carrying out morphological processing on an initial segmentation image obtained after three-channel binarization images are overlapped to obtain a final binarization segmentation result;

s604, extracting the characteristic parameters of the pressing plate switch by using a pressing plate shape characteristic parameter extraction method based on the connected domain external rectangle, so as to optimize and supplement the classification result of deep learning.

Preferably, step S07 includes:

s701, on the basis of the obtained second characteristic parameter, obtaining a second characteristic state A' by a parameter characteristic-based automatic detection method for the state of the protective pressing plate;

after the width w and the height h of the second characteristic parameter are input, multi-parameter calculation is firstly carried out to obtain characteristic values a, S3 and S4, wherein a is the ratio of h to w, S3 is the area of the circumscribed rectangle, S4 is the mean value of the areas of the circumscribed rectangles of all the detection objects, and the formulas (14) to (16) are shown as follows:

S3＝h*w (15)

then, removing small connected domains in the binary region through connected domain selection based on a multi-parameter threshold, and finally judging a second characteristic state A' of the II-type protection pressing plate through a single-parameter threshold judgment algorithm, wherein a judgment formula is shown as a formula (17):

according to the automatic selection of the parameter range, three states A of the II type protective pressing plate state can be obtained ₁ ′、A ₂ ′、A ₃ ', wherein A ₁ ' is an Exit State, A ₂ ' in a Standby State, A ₃ ' is a charging state;

s702, automatically detecting the pressing plate state of the second characteristic state A ' and the second identification state B ' based on the fusion state information to obtain a final II-type protective pressing plate state E ';

according to the rectangular position information A of the rectangular frame obtained from the obtained second characteristic parameter _r ' (x, y, h, w) while there may be a rectangular area B of the target protective platen in the second recognition state result _r ', A _r ' and B _r ' performing intersection judgment;

if A _r ' and B _r If the intersection of' is empty, then the final state A is _i '; if A _r ' and B _r If the intersection of' is not empty, then final state B _i 'the final detected state of all type II protective platen objects in the obtained image is E', as shown in equation (18):

a state automatic detection device based on deep learning and morphology fusion, which is a detection device based on module units corresponding to the steps of the state automatic detection method in any one of claims 1-9 and is used for carrying out state automatic detection on a type I protective pressing plate and a type II protective pressing plate.

The invention discloses a state automatic detection method based on deep learning and morphology fusion, which effectively combines a YOLOV5 deep learning network with a color space and morphology fusion characteristic for automatically detecting the type and the state of a protective pressing plate, and comprises the following steps:

firstly, carrying out target positioning and initial type analysis on collected image frames of the protection pressing plate, and acquiring the corresponding types of the target protection pressing plate in each image frame, wherein the types are respectively marked as type I and type II.

The invention provides an automatic detection method for the state of an I-type protection pressing plate fused with a deep learning network for an input I-type protection pressing plate image. After an I-type protective pressing plate image is input, firstly, a first characteristic parameter is obtained through a characteristic parameter extraction method based on cubic spline difference and color space characteristics, and a first characteristic state A is obtained through a state automatic detection method based on the first parameter characteristics; meanwhile, carrying out state recognition on the protective pressing plate by using a YOLOv5m deep learning network to obtain a first recognition state B; and finally, automatically detecting the pressing plate state of the second characteristic state A and the second identification state B based on the fusion state information to obtain a final I-type protection pressing plate state E.

In order to realize more accurate measurement on the state of the II-type protective pressing plate, the invention provides an automatic detection method for the state of the II-type protective pressing plate based on the fusion characteristics of improved YOLOV5 and color space and morphology. Firstly, improving a yolov5 basic network, introducing a space and channel convolution attention model, enhancing the significance of a fault target to be detected, and detecting the state of a pressing plate to obtain a second recognition state B'; secondly, obtaining a second characteristic parameter by a protection pressing plate characteristic parameter extraction method based on color space and morphological fusion characteristics, and obtaining a second characteristic state A' by a protection pressing plate state automatic detection method based on parameter characteristics; and then, providing a II-type pressing plate state automatic detection method fusing the deep learning network, and automatically detecting the pressing plate state of the second characteristic state A ' and the second identification state B ' based on the fusion state information to obtain a final II-type protective pressing plate state E '.

On the other hand, the invention also provides a state automatic detection device based on deep learning and morphology fusion, which is a detection device formed by module units corresponding to any one of the above state automatic detection method steps and is used for automatically detecting the type and the state of the protective pressing plate.

In summary, due to the adoption of the technical scheme, the invention has the beneficial effects that:

the state automatic detection method and device based on deep learning and morphology fusion provided by the invention can realize effective detection of states of different types of protection pressing plates, are accurate in identification, high in applicability and good in robustness, are beneficial to monitoring and controlling the real-time state of the protection pressing plates, are efficient, practical, objective and accurate in scheme, and can be used for assisting in routing inspection of the protection pressing plates and electric power in an electric power system, so that the safety performance of the electric power system is better ensured.

Drawings

Fig. 1 is a flowchart of a method for automatically detecting a state of a protective pressing plate according to an embodiment of the present invention.

FIG. 2- (a) is a view of type I protective pressboard;

FIG. 2- (b) type II protective pressing plate diagram.

Fig. 3 is a flowchart of a method for automatically detecting the type of a protective pressure plate based on shape characteristics according to an embodiment of the present invention.

FIG. 4- (a) is a view showing the retreated state of the type I protective presser plate;

FIG. 4- (b) is a view showing a state where the type I protective presser is put in.

Fig. 5 is a schematic diagram of a result of gradient processing performed by the sobel operator according to the embodiment of the present invention.

Fig. 6 is a diagram of the yolov5 network structure according to the embodiment of the present invention.

Fig. 7 is a schematic diagram of a detection result of the state of the type i protective pressing plate based on YOLOv5 according to an embodiment of the present invention.

Fig. 8 is a flowchart of extracting characteristic parameters of an i-type protective pressing plate based on cubic spline difference and color space characteristics according to an embodiment of the present invention.

Fig. 9- (a) is a diagram of the histogram equalization effect of the type ii platen switch according to the embodiment of the present invention: original drawing;

fig. 9- (b) is a diagram of the histogram equalization effect of the type ii platen switch according to the embodiment of the present invention: and (4) performing I-type protection pressing plate graph after processing based on cubic spline difference.

Fig. 10- (a) is a schematic view of the hole filling effect provided by the embodiment of the invention: pre-filled graph;

fig. 10- (b) is a schematic diagram of the hole filling effect provided by the embodiment of the invention: and (4) filling the figure.

Fig. 11 is a binarized image of the protective platen after morphological processing according to an embodiment of the present invention.

Fig. 12 is a schematic diagram of the shape characteristic parameters of the protective pressure plate provided by the embodiment of the invention.

Fig. 13- (a) is a schematic diagram of classifying states of type ii protective pressing plates according to an embodiment of the present invention: a throw-in state diagram;

fig. 13- (b) is a schematic diagram of classifying states of type ii protective pressing plates according to an embodiment of the present invention: a standby state diagram;

fig. 13- (c) is a schematic diagram of classifying states of type ii protective pressing plates according to an embodiment of the present invention: the state diagram is exited.

Fig. 14 is a diagram of an improved YOLOv5 network model provided by an embodiment of the present invention.

Fig. 15 is a flow chart of extracting characteristic parameters of a type ii protective pressing plate based on color space and morphological fusion characteristics according to an embodiment of the present invention.

FIG. 16- (a) is a diagram illustrating a highlight restoration effect of a pressing plate switch according to an embodiment of the present invention;

fig. 16- (b) is a highlight restoration effect diagram of the pressing plate switch provided by the embodiment of the invention, namely an effect diagram after highlight restoration of the pressing plate switch.

Fig. 17- (a) is a diagram of a foreground segmentation effect of the pressure plate switch provided by the embodiment of the present invention: switching an original drawing by a pressing plate;

fig. 17- (b) is a diagram of a foreground segmentation effect of the pressure plate switch provided by the embodiment of the present invention: and (5) dividing the foreground of the pressing plate switch into effect graphs.

Fig. 18 is the binarized image obtained by switching the pressing plate after the morphological processing according to the embodiment of the present invention.

Fig. 19 is a parameter schematic diagram of a method for extracting shape characteristic parameters of a protective pressing plate based on a connected domain circumscribed rectangle according to an embodiment of the present invention.

Fig. 20 is a schematic diagram showing the automatic detection result of the pressure plate state based on the fusion state information: (a) a pressure plate state automatic detection result graph based on the parameter characteristics; (b) a type II protection pressing plate state detection result graph based on a YOLOv5m network; (c) and (4) a protection pressing plate state automatic detection result graph based on the fusion state information.

FIG. 21- (a) is a schematic diagram showing the test results of the protective pressing plate provided by the present invention based on various network status tests, wherein the test result of the type I protective pressing plate is shown in the figure;

FIG. 21- (b) is a schematic diagram of the test results of the protective pressing plate provided by the present invention based on the detection of various network states, i.e., a type II protective pressing plate detection result diagram.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In the following embodiments, a type i protective pressing plate and a type ii protective pressing plate are selected as target protective pressing plates, so as to describe a specific scheme in detail, in other embodiments, the target protective pressing plates may also be other types of target protective pressing plates, or non-target protective pressing plates in similar scenes, which are only analysis objects targeted by the technical scheme of the present invention, and it is to be applied to the technical scheme of the present invention and solve corresponding technical problems.

Example 1

Referring to the overall flow chart shown in fig. 1, the present embodiment is a method for automatically detecting states based on deep learning and morphology fusion, which effectively combines the YOLOV5 deep learning network with color space and morphology fusion features for automatically detecting the type and state of a protective pressing plate, and comprises the following steps,

s01, carrying out target positioning and initial type analysis on the collected image frames of the protective pressing plate, and acquiring the corresponding types of the target protective pressing plate in each image frame, wherein the types are respectively marked as type I and type II;

before the state detection of the protective pressing plate, the type of the protective pressing plate needs to be detected. The embodiment provides a method for automatically detecting the type of a protection pressing plate based on shape characteristics, so as to realize the type detection of the protection pressing plate, and lay a foundation for the subsequent state detection steps of different protection pressing plates of each type.

The types of protective pressure plates targeted in this embodiment are two common types: forms I and II, as shown in FIGS. 2(a) and (b), respectively.

It can be seen that the type I protective press plate is directly in the shape of a rectangular structure, while the type II protective press plate comprises an upper circular structure and a lower circular structure and a square structure for connecting the two circular structures. The two types have different structural characteristics, and the methods for detecting the states of the two types are determined to be different in weight.

The type detection of the type I and type II protective press plates is performed by using an automatic detection method of the type of the protective press plates based on shape characteristics, as shown in fig. 3.

Firstly, conducting sobel edge detection on an input protection pressing plate image frame, conducting maximum and minimum filtering processing on the image after obtaining edge information of the image, meanwhile, removing a small connected domain according to the area of the connected domain after obtaining binary information of the image through global threshold processing, and then obtaining each protection pressing plate object in the image through a connected domain external rectangle. And finally, judging whether the target protection pressing plate belongs to the type I or the type II by carrying out linear detection on each object.

S02, preprocessing the image collected by the I-type protective pressing plate, and automatically identifying the working state of the I-type protective pressing plate by using a deep learning target network to obtain a first identification state;

the state of the type i protective pressing plate is divided into two types, i.e., "withdrawing" and "throwing", as shown in fig. 4(a) and (b), respectively.

Preprocessing is carried out on collected image frames of the I-type protective pressing plate, mainly aiming at the state characteristics of the I-type protective pressing plate, a Sobel operator is adopted to carry out gradient processing on the images firstly, and processed characteristic images are obtained. Because the exit state of the I-type protective pressing plate has an angle of 45 degrees in the reverse direction, in the embodiment, sobel operator extraction in the x direction, the y direction and the 135-degree direction is respectively carried out on the image, and then the three images are combined in proportion to obtain the final characteristic image after gradient processing. And finally, taking the final characteristic image as the input of a target network for recognition.

The sub-images in the three directions and the final feature image after merging are shown in fig. 5(a) - (d), respectively.

In this embodiment, a YOLOv5 target detection network is adopted as a target network, and mainly includes four parts: the network general block diagram of the input end, the trunk network (Backbone), the detection Neck (Neck) and the Prediction layer (Prediction) is shown in fig. 6.

After the preprocessed final characteristic image enters an input end, the YOLOv5 target detection network enriches a data set by adopting Mosaic data enhancement; meanwhile, the data is subjected to self-adaptive anchor frame and the picture is subjected to self-adaptive scaling, so that the algorithm speed is improved.

The preprocessed image data set enters a backbone network and mainly comprises Focus, CBL and CSP1-X, SSP4 modules. The CBL layer is the smallest component in YOLOV and consists of conv (convolutional layer), BN (batch normalization layer), and leak-relu activation functions.

And slicing the data set in a Focus structure, and then entering concat (splicing layer) to improve the convolution speed. And after the characteristics of the input data set are extracted by the convolution layer and the CBL layer, the Bn layer normalizes the structure and finally enters the convolution of the next layer through an activation function. The data set is further processed by CSP1-x to optimize gradient information in the network. CSP1-x uses CSPNet (Cross Stage Partial network) to divide the input into two parts, one part is convoluted by the residual error component for x times, the other part is directly convoluted, and finally the two parts are spliced. The output data is further processed by SPP (spatial pyramid pooling), and the input with different sizes is converted into the output with the same size by convolution and down sampling of the maximum pooling layer. The detection Neck (hack) is the fusion part of the network, combining and passing the features to the prediction layer. In the Neck layer, the network adopts a structure that FPN is combined with PAN. The top-down FPN transfers and fuses the high-level feature information through up-sampling; the PAN from bottom to top conveys the location features, which are fused, enhancing the ability of the network to fuse features. Meanwhile, a CSP2-x structure is added into the Neck network for feature fusion. And finally, at the output end, the network adopts the GIOU _ Loss as a Loss function and screens the target frame through non-maximum value inhibition.

An example of a state detection for a type i protective platen using the YOLOv5m target recognition network is finally implemented as shown in fig. 7.

S03, extracting image characteristics from the I-type protective pressing plate by using a characteristic parameter extraction method of cubic spline difference and color space characteristics to obtain a first characteristic parameter;

because the protection pressing plate images are all shot by mobile phones of workers in the transformer substation, the illumination in a main control room of the transformer substation is uneven, and the image quality is greatly influenced by conditions such as illumination, distance, shooting angle and the like, the pressing plate images collected in the experimental application scene have interference of a large amount of highlights and shadows, and the deep learning network is not comprehensive enough for learning pictures in various states, so that the condition of inaccurate classification can occur. In order to obtain a more accurate platen segmentation result, the embodiment further provides a rectangular protective platen feature parameter extraction method with color space and morphology fusion features on the basis of a deep learning network segmentation result, and optimizes the deep learning classification result of the protective platen.

The general flow chart of the method for extracting image features from an I-type protective pressing plate by using the cubic spline difference and the feature parameter extraction method of color space features is shown in fig. 8:

s301, firstly, reducing the image resolution of the original image based on cubic spline interpolation, thereby reducing the operation amount and improving the image processing efficiency under the condition of not influencing subsequent image segmentation and characteristic parameter extraction;

because the actual service environment of the protective pressing plate has a unified standard, and the background of the picture of the protective pressing plate is single. According to the state identification requirement of the protection pressing plate, the foreground and the background of the picture need to be segmented, and then the state feature extraction of the protection pressing plate is realized. In the process of extracting characteristic parameters of the pressing plate, the requirement on the image precision is not high. Therefore, in order to increase the operation speed, reduce the amount of operation, and improve the recognition efficiency, an input original image is first processed by an image resolution reduction method based on cubic spline interpolation.

And fitting the discrete points by using cubic spline interpolation, wherein the curve fitted by the cubic spline interpolation method is more practical than the curve fitted by using a 3-time line connection method. The corresponding interpolation algorithm implementation flow is shown below.

Assume that there are n +1 data nodes (x) ₀ ,y ₀ )，(x ₁ ,y ₁ )，(x ₂ ,y ₂ )，…，(x _n ,y _n )，

a. Calculating step length h _i ＝x _i+1 -x _i (i＝0,1,…,n-1)；

b. Substituting the data nodes and the specified head end point conditions into a matrix equation;

c. solving the matrix equation to obtain a second derivative value h _i ；

d. Calculating coefficients of the spline curve: a is _i ＝y _i ；

Wherein i is 0,1, …, n-1;

e. in each subinterval x _i ≤x≤x _i+1 In, create an equation

g _i (x)＝a _i +b _i (x-x _i )+c _i (x-x _i ) ² +d _i (x-x _i ) ³ (4)。

S302, preprocessing the image of the protective pressing plate by utilizing a histogram equalization method based on contrast ratio limitation self-adaption, wherein the step can improve the difference between foreground and background pixels of a large highlight or shadow area and reduce interference for the subsequent image segmentation step;

because the pressing plate picture is shot by the mobile phone, the difference of the background color of the collected image is large, and some highlight or shadow areas exist, so that the subsequent color space conversion and image foreground segmentation are seriously interfered. Therefore, histogram equalization of an image in image processing is very critical for subsequent platen state detection.

In order to distinguish the foreground protective pressing plate and the background as much as possible, the input image is divided into sub-regions and then is subjected to contrast stretching, in order to prevent a large highlight or shadow region from generating great influence on the overall image quality, a threshold value of histogram distribution is set, and the distribution exceeding the threshold value is dispersed to probability density distribution in a 'uniform' mode, so that the increase of a conversion function (cumulative histogram) is limited. In order to reduce the discontinuity between each sub-region, each sub-region is connected by using a bilinear interpolation algorithm, and finally, the preprocessing of the protection pressure plate image is realized, as shown in fig. 9.

because the colors of the I-type protection pressing plate switch are red, yellow and white, the extraction of color characteristics can realize the foreground segmentation of the pressing plate switch area. Therefore, in the embodiment, the image segmentation method based on the HSV color space is utilized to convert the image into the HSV space and then perform the threshold processing. After threshold processing, three channels of red, yellow and blue are superposed to display a complete binarization processing effect.

And S304, finally, extracting shape characteristic parameters of the protective pressing plate by using a connected domain-based external rectangular parameter extraction method, so that the method is used for optimizing and supplementing the classification result of deep learning.

Due to the fact that a large amount of interference exists in the protection pressing plate image after binarization processing, for example, small isolated points, the edge of a type I protection pressing plate switch is discontinuous, center loss is caused by reflection of light at the center position of the protection pressing plate, and the like, the subsequent feature extraction of the protection pressing plate is greatly influenced. Therefore, morphological processing is required for the connected component, so as to obtain an effective foreground segmentation image.

According to the method, firstly, the fracture edges of the rectangular protective pressing plate in the binary image are connected through corrosion and expansion, then the hole filling is carried out on the image, the area integrity of a switch is ensured, and finally, the isolated point removing operation is carried out on the image, so that the binary image with completely separated foreground and background is obtained.

The effect of the hole filling process is shown in figure 10.

After the binarization threshold processing is performed on the image of the protective pressing plate, it can be found that some smaller noise points exist in the segmented image, and the smaller noise points need to be removed, so that the feature extraction is performed on the image in the future.

In order to remove noise points, contour information in the binarized image is extracted, and areas with small areas are filled with black, and the effect after black filling is shown in fig. 11.

After the segmentation area of the image of the protection pressing plate is obtained, the external rectangle of the connected domain needs to be extracted from the protection pressing plate. And obtaining the initial coordinates x and y of the upper left corner of the circumscribed rectangle of each connected domain, the width w and the height h of the circumscribed rectangle and the pixel number of each connected domain according to the 8-connected neighborhood of each pixel point.

The I-type protective pressing plate has two states, namely 'input' and 'exit'. Therefore, after the image of the protective pressing plate is segmented, the area characteristics of the external rectangle of the communicated region are extracted, and the partial area which obviously does not meet the requirement is removed by setting the threshold parameter, so that the representation of two states can be realized. The corresponding shape characteristic parameter diagrams of the two states are shown in fig. 12(a) - (b), respectively.

And S04, detecting a first characteristic state corresponding to the protective pressing plate based on the first characteristic parameter, fusing the first identification state and the first characteristic state based on the fusion information, and determining the final detection state of the I-type protective pressing plate.

For an input I-type protective pressing plate image, firstly, obtaining a first characteristic parameter through a characteristic parameter extraction method based on cubic spline difference and color space characteristics, and obtaining a first characteristic state A through a state automatic detection method based on the first parameter characteristics; meanwhile, carrying out state recognition on the protective pressing plate by using a modified YOLOv5m deep learning network to obtain a first recognition state B; and finally, the first identification state and the first characteristic state are subjected to a pressing plate state automatic detection method based on fusion state information to obtain a final protection pressing plate detection state.

S401, detecting a first characteristic state A corresponding to the protective pressing plate based on the first characteristic parameter;

first characteristic parameters h, w, S and S of the protection pressing plate can be obtained through the steps, wherein h and w are the length and the width of a rectangular frame externally connected with the protection pressing plate respectively, and S and S are characteristic values obtained according to the height h and the width w respectively. Wherein S is the area of the external rectangle of the protection pressing plate, and S is the average value of the areas of the external rectangles of all the detected protection pressing plates, as shown in the formula (5-6).

s＝h*w (5)

according to the automatic selection of the parameter range, two states A corresponding to the I-type protective pressing plate can be obtained ₁ 、A ₂ Wherein A is ₁ In an "input" state, A ₂ An "exit" state.

S402, automatically detecting the pressing plate state based on the fusion state information;

in order to more effectively detect the state of the I-type protection pressing plate, the embodiment provides an automatic detection method for the state of the I-type protection pressing plate, which is integrated with a deep learning network. After an I-type protective pressing plate image is input, firstly, a first characteristic parameter is obtained through a protective pressing plate characteristic parameter extraction method based on color space and morphological fusion characteristics, and a first characteristic state A is obtained through a protective pressing plate state automatic detection method based on parameter characteristics; meanwhile, carrying out state recognition on the protective pressing plate by using a YOLOv5m deep learning network to obtain a first recognition state B; and finally, automatically detecting the pressing plate state of the second characteristic state A and the second identification state B based on the fusion state information to obtain a final I-type protection pressing plate state E.

After the first characteristic parameter extraction is completed, the target type I protective pressing plate has a rectangular frame information Ar (x, y, h, w), and a rectangular area Br of the target protective pressing plate may exist in the first identification state result. A is to be _r And B _r Performing intersection judgment, if A _r And B _r The intersection of the protective pressing plates is empty, the final state of the I-type protective pressing plate is A _i (ii) a If A _r And B _r If the intersection is not empty, the final state of the I-type protective pressing plate is B _i And finally obtaining the final detection state of all objects in the image of the protective pressing plate as E, wherein the final detection state is shown as the formula (8):

thus, the final type I protective pressing plate state E can be obtained.

For the II type protective pressing plate, compared with the I type protective pressing plate, the geometric shape of the II type protective pressing plate is more complicated with the shooting background. The type II protective platen has three states, i.e., "throw-in", "standby", and "throw-in" as shown in fig. 13(a) to (b).

In order to achieve more accurate measurement of the state of the type II protective pressing plate, the present embodiment provides an automatic detection method for the state of the type II protective pressing plate based on the fusion characteristics of the improved YOLOV5 and the color space and morphology. Firstly, improving on a yolov5 basic network, introducing a space and channel convolution attention model, enhancing the significance of a fault target to be detected, and detecting the state of a pressing plate; secondly, providing a II-type protective pressing plate characteristic parameter extraction algorithm based on color space and morphological fusion characteristics to obtain the characteristic parameters of the protective pressing plate; and then, providing a II-type pressure plate state automatic detection method fusing a deep learning network, namely obtaining the state of the protection pressure plate by utilizing a pressure plate state automatic detection algorithm based on parameter characteristics, and fusing state information of the state of the protection pressure plate and the state of a deep learning result to realize the identification of the II-type protection pressure plate state.

S05, automatically identifying the state of the II-type protective pressing plate by using an improved deep learning network to obtain a second identification state;

in order to suppress the interference of the complex background of the type II protective pressing plate, the embodiment introduces a spatial and channel convolution attention model on the basis of the YOLOv5 identification network to enhance the significance of the target to be detected.

As shown in fig. 14, a channel and space convolution block attention model, a channel attention module and a space attention module are introduced behind a CSP module of the YOLOv5 network model, so that the extracted features are more refined, and the expressive power of the target recognition model is improved.

The input feature map F (H multiplied by W multiplied by C) is respectively subjected to global maximum pooling and global average pooling based on width and height dimensions to obtain two 1 multiplied by C feature maps, and then the two feature maps are respectively sent into a two-layer neural network (MLP), the number of neurons in the first layer is C/r (r is a reduction rate), an activation function is Relu, the number of neurons in the second layer is C, and the two-layer neural network is shared. And then, summing the features output by the MLP, and then performing sigmoid activation operation to generate a final channel attention feature, namely M _ c. And finally, performing element mode multiplication operation on the M _ c and the input feature graph F to generate the input features required by the space attention module. And taking the feature map output by the channel attention module as an input feature map of the module. Firstly, performing global maximum pooling and global average pooling based on channels to obtain two H multiplied by W multiplied by 1 feature maps, then performing channel splicing operation on the 2 feature maps based on the channels, and then performing 7 multiplied by 7 convolution operation to reduce the dimension into 1 channel, namely H multiplied by W multiplied by 1. And generating a spatial attention feature, namely M _ s, through sigmoid. And finally, multiplying the characteristics with the input characteristics of the module to obtain the finally generated characteristics.

S06, extracting image characteristics of the II-type protective pressing plate by a characteristic parameter extraction method based on color space and morphological fusion characteristics to obtain second characteristic parameters;

because the protection pressing plate images are all shot by mobile phones of substation workers, the illumination in a main control room of the substation is not uniform, and the image quality is greatly influenced by conditions such as illumination, distance, shooting angles and the like, the pressing plate images acquired by the experimental application scene have interference of a large amount of highlights and shadows, and the deep learning network is not comprehensive enough for learning pictures in various states, so that the condition of inaccurate classification can occur. In order to obtain a more accurate state identification result, the embodiment further provides a method for extracting the characteristic parameters of the type ii protective pressing plate with color space and morphology fusion characteristics, which is used for optimizing the deep learning classification result of the protective pressing plate.

The overall flow is shown in fig. 15, and the method for extracting the characteristic parameters of the type ii protective pressing plate based on the color space and morphological fusion characteristics comprises the following steps:

because the images of the protective pressing plate are all shot by the mobile phone and are influenced by shooting equipment and the field environment, the collected images are greatly interfered by reflection, shadow and the like. Therefore, the detection and repair of the highlight region on the protective platen image first in the preprocessing is extremely critical for the subsequent protective platen state detection.

The difference between the highlight position of the protective pressing plate image and the pixel value of the surrounding adjacent area is large, and the difference between the pixel values in the gray level image is obvious. In order to reduce the amount of computation and increase the image processing speed, a threshold range is set for the grayscale image of the platen first, and a binary mask image of a highlight area is created, thereby automatically detecting the highlight area of the platen image.

And after the highlight area is detected, realizing the processing of protecting the highlight area of the pressing plate image by using an image restoration method based on a fast moving method according to the same position relation between the mask image and the highlight area of the original image.

The fast moving method starts to repair the boundary of the non-zero area of the mask image, continuously and dynamically updates the boundary of the repair area, gradually moves into the area, repairs the adjacent pixels of each target pixel, and fills all the content in the boundary. When each target pixel is repaired, that pixel is replaced by the normalized weighted sum of all known pixels in the neighborhood. The corresponding weight setting mechanism is:

more weight is given to pixels close to the repaired point, to the normal close to the boundary and to pixels located on the boundary contour.

Once a pixel is repaired, it will move to the next nearest pixel using the fast marching method.

The weight function used in this step is shown in equation (9).

W (p, q) is a weight function for defining the weight of each pixel in the field. The calculation formula of w (p, q) is shown in formula (10).

w(p,q)＝dir(p,q)·dst(p,q)·lev(p,q)(10)

The values of the direction factor dir (p, q), the geometric distance factor dst (p, q) and the horizontal set distance factor lev (p, q) of each target pixel point are respectively calculated by equations (11) - (13):

wherein d is ₀ And T ₀ Respectively, a distance parameter and a level set parameter, and the general value is 1. The direction factor dir (p, q) ensures that the closer to the normal direction the more

The contribution of the pixel point of (1) to the p point is maximum; the geometric distance factor dst (p, q) ensures that pixel points closer to p point contribute more to p point; the level set distance factor lev (p, q) ensures that the closer to the contour line of the region to be repaired passing through the point p, the more the contribution of the known pixel point to the point p.

Fig. 16 shows the result of the protective platen image after the automatic detection in the highlight area and the image restoration.

S602, secondly, protecting the pressing plate foreground segmentation by using a watershed algorithm, wherein the step can avoid the interference of shadows in the image of the protecting pressing plate and realize a better foreground segmentation effect;

after highlight restoration processing is performed on an original pressing plate image, in order to extract a pressing plate switch area, foreground segmentation needs to be performed on the image. Because the images of the protective pressing plate are all shot and collected by the mobile phone, the lighting condition is poor, and the quality of the images is poor and the noise is more due to the limitation of the performance of the mobile phone. After the traditional filtering methods such as median, mean, gaussian and the like are used for denoising, the image quality still cannot meet the subsequent processing requirements. Based on the watershed algorithm, the background of the image of the protective pressing plate with high noise and low image quality can be changed into black, so that a good foreground segmentation effect is achieved.

The watershed algorithm is mainly applied to extracting approximately consistent nodular targets from a background, so that an area with small gray level change is represented. Because the transformer protection clamp plate in the central control room of transformer substation all sets up on the iron sheet cabinet, the background colour is comparatively unified, mainly has the shadow interference that the illumination inequality brought after highlight restoration handles. Therefore, in the embodiment, a foreground segmentation algorithm based on a watershed algorithm is used, a random seed is selected from the background, and starting from the seed, edge points are searched for according to pixel gradient transformation within an autonomously set gray threshold. All the points meeting the threshold are regarded as the same reservoir, all the pixel points in the reservoir are marked as black, and the colors of other areas are kept unchanged, so that a good foreground segmentation effect is achieved, and the foreground segmentation effect is shown in fig. 17.

due to the influence of factors such as uneven illumination in the main control room of the transformer substation, other contents in the shot picture and the like, some pressing plate images still have more interference of irrelevant areas after being divided by the foreground. The colors of the press plate switch are red, yellow and blue, and the extraction of the color characteristics can further realize the segmentation of the press plate switch area. Therefore, in the embodiment, the image segmentation method based on the HSV color space is utilized to convert the image into the HSV space and then perform the threshold processing.

After HSV space conversion is carried out on the pressing plate switch picture, a primary segmentation image is obtained, but the problems that switch edges are not communicated, the filling of a switch central area is incomplete, isolated noise points exist in the background and the like exist in the segmentation image, and great interference is brought to the subsequent pressing plate shape characteristic parameter extraction. Aiming at the problems, the algorithm uses a morphological processing method, firstly, the binary image is processed through corrosion and expansion, broken edges in the binary image are removed, and then hole filling and isolated point removing are carried out on the image, so that the binary image with completely separated foreground and background is obtained.

Through expansion and corrosion, the edges of the switch can be communicated, and fine peaks can be removed; in the isolated point removing step, firstly, extracting outline information in the binarized image, and filling a region with a smaller area into black; and in the hole filling step, a pixel point set in a closed circle formed by eight connected lattices in the binary image is processed, and the inner area of the connected area is completely changed into white. The obtained binarized image after the morphological processing is shown in fig. 18.

And S604, finally, extracting the characteristic parameters of the pressing plate switch by using the provided pressing plate shape characteristic parameter extraction method based on the connected domain circumscribed rectangle, thereby optimizing and supplementing the classification result of deep learning.

After the segmentation area of the pressing plate switch image is obtained, the circumscribed rectangle of the connected domain needs to be extracted from the pressing plate switch. The coordinates (x, y) of the upper left corner initial position of the circumscribed rectangle of each connected domain, the height h and the width w of the circumscribed rectangle are obtained according to the 8-connected neighborhood of each pixel point, and a parameter schematic diagram is shown in fig. 19.

And S07, detecting a second characteristic state corresponding to the II-type protective pressing plate based on the second characteristic parameter, fusing the second identification state and the second characteristic state based on the fusion information, and determining the final detection state of the II-type protective pressing plate.

In order to improve the detection result of the type ii protection pressing plate state, the present embodiment provides an automatic detection method of the type ii protection pressing plate state that merges with a deep learning network. After inputting a II-type protective pressing plate image, firstly obtaining a second characteristic parameter by a protective pressing plate characteristic parameter extraction method based on color space and morphological fusion characteristics, and obtaining a second characteristic state A' by a protective pressing plate state automatic detection method based on parameter characteristics; meanwhile, carrying out state recognition on the protection pressing plate by using an improved YOLOv5m deep learning network to obtain a second recognition state B'; and finally, automatically detecting the pressing plate state of the second characteristic state A ' and the second identification state B ' based on the fusion state information to obtain a final II-type protective pressing plate state E '.

The overall flow chart is shown in fig. 14, and includes:

s701, on the basis of the obtained second characteristic parameter, obtaining a second characteristic state A' by a parameter characteristic-based automatic detection method for the state of the protective pressing plate; meanwhile, carrying out state recognition on the protection pressing plate by using a YOLOv5m deep learning network to obtain a state B;

after the width w and the height h of the second characteristic parameter are input, multi-parameter calculation is firstly carried out to obtain characteristic values a, S and S. Wherein a is the ratio of h to w, S is the area of the circumscribed rectangle, and S is the mean value of the areas of the circumscribed rectangles of all the detection objects, as shown in formulas (14) to (16).

s＝h*w (15)

And then removing small connected domains in the binary region through connected domain selection based on a multi-parameter threshold, and finally judging a second characteristic state A' of the II-type protection pressing plate through a single-parameter threshold judgment algorithm, wherein the judgment formula is shown as a formula (17).

According to the automatic selection of the parameter range, three states A of the II type protective pressing plate state can be obtained ₁ ′、A ₂ ′、A ₃ ', wherein A ₁ Is in an "exit" state, A ₂ Is in a "standby" state, A ₃ ' is in the "throw-in" state.

S702, automatically detecting the pressing plate state of the second characteristic state A ' and the second identification state B ' based on the fusion state information to obtain a final II-type protective pressing plate state E '.

According to the obtained second characteristic parameterRectangular position information A of the rectangular frame _r ' (x, y, h, w) while there may be a rectangular area B of the target protective platen in the second recognition state result _r '. A is to be _r ' and B _r ' make intersection judgment. If A _r ' and B _r If the intersection of' is empty, then the final state A is _i '; if A _r ' and B _r If the intersection of' is not empty, then final state B _i 'the final detection state of all the type II protective platen objects in the obtained image is E', as equation (18).

The final test state of the type II protective platen is shown in fig. 21.

Example 2

The present embodiment is a state automatic detection device based on deep learning and morphology fusion, and is characterized in that the device is a detection device based on module units corresponding to the steps of the state automatic detection method in any one of the foregoing embodiments, and is used for automatically detecting the type and state of a protective pressure plate.

Example 3

To better illustrate the excellent effects of the solution provided by the present invention, the present example is illustrated by experimental test results. The experimental video data are obtained by shooting in national power grid xxx, and experimenters shoot 380 protection pressing plate pictures in different environments and time, 104 pictures of the I-type protection pressing plate, and 216 pictures of the II-type protection pressing plate.

(1) I-type protection pressing plate state detection experiment result and analysis

In the experiment, the I-type protective pressing plate is subjected to state recognition through a YOLOv5m network. The type II protective press plate had 104 pictures in total, 74 of the pictures were used for network training, and the remaining 30 pictures were tested, with the test results shown in table 1.

Table 1 type 2 protective platen status detection results based on YOLOv5m

As can be seen from table 1, of 1089 type i protective platen objects, 309 "input" states and 780 "exit" states all achieve correct detection of the target state, and the accuracy rate reaches 100%.

(2) Type II protection pressing plate state detection experiment result and analysis

In the experiment, after the primary state detection is carried out on the protection pressing plate through the YOLOv5m network, the detection accuracy is further improved by adding a II-type protection pressing plate state automatic detection algorithm fused with the deep learning network.

2.3.1 results of a model II protective platen Condition testing experiment based on YOLOv5m

The experiment performed a preliminary status check of the protective platen over the YOLOv5m network. The type II protective pressing plate has 216 pictures, and each picture is provided with a plurality of protective pressing plates. 151 pictures are used for network training, and the remaining 65 pictures are tested, and the detection results are shown in table 2.

TABLE 2 type II protective platen Condition test results based on YOLOv5m

As can be seen from table x, there were 664 of 1709 type ii protective platens in the "standby" state, 596 in the "exit" state, and 449 in the "throw-in" state. Wherein the status detection accuracy for "exit" is 0.9161, which is slightly lower than the accuracy for "standby" and "throw-in" 0.9246 and 0.9955; meanwhile, the missing rate of 0.0772 of exit is slightly higher than that of standby and input.

(3) Type II protection pressing plate state detection experiment result based on fusion characteristics of YOLOv5m and color space and morphology

After the automatic detection method for the state of the type II protection pressing plate integrated with the deep learning network is added, the detection result of the state of the type II protection pressing plate is shown in table 3.

Table 3 type II protective pressing plate state detection experiment results based on fusion characteristics of YOLOv5m and color space and morphology

As can be seen from Table 3, the experiment effectively improves the accuracy of state detection and reduces the omission factor. Wherein the detection accuracy of the protection pressing plate in the standby state is improved by 2.87%, and the omission factor is reduced by 2.4%; the detection accuracy of the protection pressing plate in the exit state is improved by 5.37 percent, and the omission factor is reduced by 4.87 percent.

(4) Comparison with the prior art

In this embodiment, the results of the YOLOv5 experiment in this embodiment are compared with the classic fast-RCNN deep learning network and ssd (single shot multi-box detector) deep learning network, and the accuracy of each network is shown in fig. 21.

As can be seen from FIG. 21, the state detection results of the two types of protective pressing plates by using YOLOv5 are superior to those of the fast-RCNN and the SSD deep learning network. Performing state recognition on the I-type protective pressing plate by using a Fater-RCNN, wherein the 'input' and 'exit' states obtain the accuracy of 20.06 percent and 66.54 percent; the SSD network is used for carrying out state recognition on the type I protective pressing plate, wherein the recognition accuracy rates of 'input' and 'exit' are respectively 63.11% and 86.02%. The Fater-RCNN is used for carrying out state recognition on the II type protective pressing plate, and the 'input', 'exit' and 'standby' respectively obtain the accuracy rates of 71.06%, 66.78% and 48.34%; the SSD network is used for carrying out state recognition on the type II protective pressing plate, and the 'input', 'exit' and 'standby' respectively obtain the accuracy rates of 82.98%, 83.39% and 84.41%. Compared with the far-RCNN deep learning network and the SSD deep learning network, the YOLOv5 has higher detection accuracy.

The invention provides a state automatic detection scheme based on deep learning and morphology fusion aiming at the current situation of detection of a protective pressing plate, which is used for carrying out state detection on two types of protective pressing plates. Firstly, classification of two types of protective pressing plates is realized through automatic detection of the types of the protective pressing plates based on shape characteristics; secondly, a YOLOv5m deep learning network is used for realizing the state detection of the I-type protective pressing plate, so that higher accuracy is obtained; and then, a II-type protection pressing plate state automatic detection algorithm based on the fusion characteristics of YOLOv5, color space and morphology is provided to realize the state detection of the II-type protection pressing plate, and the accuracy rate of the three states of standby, quitting and putting-in is more than 95%. Experiments show that the method provided by the invention can realize effective detection of the states of different types of protection pressing plates, has accurate identification, high applicability and better robustness, and is beneficial to monitoring and controlling the real-time state of the protection pressing plates.

All of the features disclosed in this specification, or all of the steps in any method or process so disclosed, may be combined in any combination, except combinations of features and/or steps that are mutually exclusive.

Claims

1. A state automatic detection method based on deep learning and morphology fusion is used for automatically detecting a protective pressing plate and is characterized by comprising the following steps: s01, carrying out target positioning and initial type analysis on the collected image frames of the protective pressing plate, and acquiring the corresponding types of the target protective pressing plate in each image frame, wherein the types are respectively marked as type I and type II; s02, preprocessing the image collected by the I-type protective pressing plate, and automatically identifying the working state of the I-type protective pressing plate by using a deep learning target network to obtain a first identification state; s03, extracting image characteristics of the I-type protective pressing plate by using a cubic spline difference value and color space characteristic parameter extraction method to obtain a first characteristic parameter; s04, detecting a first characteristic state corresponding to the protective pressing plate based on the first characteristic parameter, fusing the first identification state and the first characteristic state based on the fusion information, and determining the final detection state of the I-type protective pressing plate; s05, automatically identifying the state of the II-type protective pressing plate by using an improved deep learning network to obtain a second identification state; s06, extracting image characteristics of the II-type protective pressing plate by a characteristic parameter extraction method based on color space and morphological fusion characteristics to obtain second characteristic parameters; and S07, determining the final detection state of the II type protective pressure plate.

2. The method for automatically detecting the state based on the deep learning and morphology fusion as claimed in claim 1, wherein the step S07 is a method for determining the final detection state of the type II protective pressing plate: and detecting a second characteristic state corresponding to the II-type protective pressing plate based on the second characteristic parameter, and fusing the second identification state and the second characteristic state based on the fusion information.

3. The method for automatically detecting states based on deep learning and morphology fusion as claimed in claim 2, wherein the step S01 includes:

4. The method for automatically detecting the state based on the deep learning and morphology fusion of claim 3, wherein the preprocessing of the collected image of the type I protective pressing plate in the step S02 comprises:

5. The method for automatically detecting states based on deep learning and morphology fusion as claimed in claim 2, wherein the step S03 includes:

6. The method for automatically detecting states based on deep learning and morphology fusion as claimed in claim 5, wherein the step S04 includes:

S1＝h*w (5)

according to the automatic selection of the parameter range, two states A corresponding to the I-type protective pressing plate can be obtained ₁ 、A ₂ Wherein A is ₁ Is input intoState A ₂ Is in an exit state;

A is to be _r And B _r Performing intersection judgment, if A _r And B _r The intersection of the protective pressing plates is empty, the final state of the I-type protective pressing plate is A _i (ii) a If A _r And B _r If the intersection is not empty, the final state of the I-type protective pressing plate is B _i And finally obtaining the final detection state of all objects in the image of the protective pressing plate as E, wherein the final detection state is shown as the formula (8):

7. the method for automatically detecting the state based on the deep learning and morphology fusion of claim 1, wherein the step S05 of automatically identifying the state of the type II protective pressing plate by using the improved deep learning network comprises:

in order to suppress the interference of the complex background of the type II protective platen, a channel and space convolution block attention model, a channel attention module and a space attention module are introduced behind the CSP module of the YOLOv5 network model.

8. The method for automatically detecting states based on deep learning and morphology fusion as claimed in claim 7, wherein the step S06 includes:

9. The method for automatically detecting states based on deep learning and morphology fusion as claimed in claim 1, wherein the step S07 includes:

S3＝h*w (15)

automatically selected according to parameter range, canThree states A of II type protection pressing plate states are obtained ₁ ′、A ₂ ′、A ₃ ', wherein A ₁ ' is an Exit State, A ₂ ' is in Standby State, A ₃ ' is a charging state;

if A _r ' and B _r If the intersection of' is empty, then final state A _i '; if A _r ' and B _r If the intersection of' is not empty, then final state B _i 'the final detected state of all type II protective platen objects in the obtained image is E', as shown in equation (18):

10. a state automatic detection device based on deep learning and morphology fusion is characterized in that the device is a detection device formed by module units corresponding to the steps of the state automatic detection method in any one of claims 1 to 9 and is used for carrying out state automatic detection on a type I protective pressing plate and a type II protective pressing plate.