CN116229570B - Aloft work personnel behavior situation identification method based on machine vision - Google Patents
Aloft work personnel behavior situation identification method based on machine vision Download PDFInfo
- Publication number
- CN116229570B CN116229570B CN202310147997.2A CN202310147997A CN116229570B CN 116229570 B CN116229570 B CN 116229570B CN 202310147997 A CN202310147997 A CN 202310147997A CN 116229570 B CN116229570 B CN 116229570B
- Authority
- CN
- China
- Prior art keywords
- safety belt
- unit
- frame
- deep learning
- safety
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000013135 deep learning Methods 0.000 claims abstract description 37
- 230000006399 behavior Effects 0.000 claims abstract description 19
- 230000006870 function Effects 0.000 claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 10
- 230000000877 morphologic effect Effects 0.000 claims abstract description 8
- 230000004927 fusion Effects 0.000 claims description 23
- 238000011176 pooling Methods 0.000 claims description 22
- 238000012549 training Methods 0.000 claims description 10
- 238000013526 transfer learning Methods 0.000 claims description 7
- 230000008878 coupling Effects 0.000 claims description 5
- 238000010168 coupling process Methods 0.000 claims description 5
- 238000005859 coupling reaction Methods 0.000 claims description 5
- 238000005457 optimization Methods 0.000 claims description 3
- 238000001514 detection method Methods 0.000 abstract description 6
- 238000005516 engineering process Methods 0.000 abstract description 6
- 238000000605 extraction Methods 0.000 abstract description 5
- 238000004737 colorimetric analysis Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 3
- 238000013508 migration Methods 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 241000238102 Scylla Species 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000009430 construction management Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Biophysics (AREA)
- Social Psychology (AREA)
- Medical Informatics (AREA)
- Psychiatry (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Human Computer Interaction (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
- Emergency Lowering Means (AREA)
Abstract
The invention discloses a machine vision-based high-altitude operation personnel behavior situation identification method, which comprises the following steps: s1, identifying an operator, a safety helmet, a safety belt and a safety belt hook in a real-time picture; s2, judging whether the safety helmet and the safety belt are worn correctly, if so, entering a step S3; otherwise, early warning output is carried out; s3, judging whether the hanging mode of the safety belt is correct, if so, entering a step S4; otherwise, early warning output is carried out; s4, judging whether the state of the safety belt hook is normal, if so, judging that the behavior situation of the overhead working personnel is normal; otherwise, early warning output is carried out. On the basis of deep learning target detection, the invention integrates scene recognition technology, cross-over ratio, logic judgment function, colorimetry space target extraction and morphological processing technology, realizes the recognition of illegal operation behavior of power operation in a specific scene, and carries out corresponding early warning output, thereby effectively improving the supervision convenience of overhead operation.
Description
Technical Field
The invention relates to the field of safety construction management, in particular to a machine vision-based high-altitude operation personnel behavior situation identification method.
Background
The power grid operation is often in dangerous environments such as high altitude, high voltage and the like, and the environments often threaten the safety of power operators, so a series of regulations are formulated in the industry to standardize the operation behaviors of the constructors. However, there are often actions such as not wearing a helmet, not wearing a safety belt, not using a hook, etc. for convenience of a constructor. To ensure the safety of constructors to a certain extent, constructors usually send patrol personnel to perform the investigation. But the overhead operation is higher from the ground, and ground personnel are difficult to supervise the overhead personnel through naked eyes, and personnel tour back and forth also waste human resources, so that the overhead operation safety supervision is difficult to become a hidden trouble of the industry.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides the recognition method for the behavior situation of the aerial work personnel based on the machine vision, which solves the problem of difficult safety supervision of the aerial work.
In order to achieve the aim of the invention, the invention adopts the following technical scheme:
the invention provides a machine vision-based high-altitude operator behavior situation identification method, which comprises the following steps:
s1, acquiring a real-time picture, and identifying an operator, a safety helmet, a safety belt and a safety belt hook in the real-time picture through a deep learning network;
s2, judging whether the safety helmet and the safety belt are worn correctly, if so, entering a step S3; otherwise, early warning output is carried out;
s3, judging whether the hanging mode of the safety belt is correct, if so, entering a step S4; otherwise, early warning output is carried out;
s4, judging whether the state of the safety belt hook is normal, if so, judging that the behavior situation of the overhead working personnel is normal; otherwise, early warning output is carried out.
Further, the method for constructing the deep learning network in step S1 includes:
replacing an SPPF module of the yolov5 network with an SPPFCSPC module, replacing a coupling head of the yolov5 network with a decoupling head, and optimizing by adopting a SIOU loss function; the SPPFCSPC module comprises a first convolution unit and a second convolution unit which are connected in parallel; the second convolution unit is sequentially connected with a third convolution unit and a fourth convolution unit, and the output end of the fourth convolution unit is respectively connected with the first maximum pooling unit and the first fusion unit; the output end of the first maximum pooling unit is respectively connected with the input ends of the second maximum pooling unit and the first fusion unit; the output end of the second maximum pooling unit is respectively connected with the input ends of the third maximum pooling unit and the first fusion unit; the output end of the third maximum pooling unit is connected with the input end of the first fusion unit; the output end of the first fusion unit is sequentially connected with a fifth convolution unit and a sixth convolution unit; the output end of the sixth convolution unit and the output end of the first convolution unit are respectively connected with the second fusion unit; the output end of the second fusion unit is connected with a seventh convolution unit; the input ends of the first convolution unit and the second convolution unit are jointly used as the input end of the SPPFCSPC module; the output end of the seventh convolution unit is the output end of the SPPFCSPC module.
Further, the specific method for optimizing by using the SIOU loss function comprises the following steps:
a1, taking an image containing a safety helmet, a safety belt and/or a safety belt hook as a training sample, and acquiring a real frame of the training sample; inputting training samples into a deep learning network to obtain a prediction frame;
a2, according to the formula:
acquiring an angle loss lambda of the deep learning network; wherein C is h The height difference between the true frame center point and the predicted frame center point; pi is the circumference ratio; sigma is the distance between the center points of the real frame and the predicted frame;the center coordinates of the real frames; />The center point coordinates of the prediction frame are obtained; max (·) represents taking the maximum value; min (·) represents taking the minimum value;
a3, according to the formula:
obtaining the distance loss delta of the deep learning network; where γ=2- Λ, ρ x And ρ y As an intermediate parameter ρ t ＝{ρ x ,ρ y }；X w And X h The width and the height of the minimum circumscribed rectangle of the real frame and the prediction frame are respectively;
a4, according to the formula:
acquiring shape loss omega of the deep learning network; wherein e is a natural constant; w, h, w gt And h gt The width and the height of the prediction frame and the real frame are respectively; θ is a constant; w (W) w And W is h Are all intermediate parameters, w t ＝{W w ,W h }；
A5, according to the formula:
obtaining comprehensive Loss value Loss of deep learning network SIOU The method comprises the steps of carrying out a first treatment on the surface of the The IOU is the ratio of the intersection set and the union set of the prediction frame and the real frame;
a6, according to the comprehensive Loss value Loss of the deep learning network SIOU And optimizing parameters of the deep learning network to finish optimization of the deep learning network.
Further, the specific method for judging whether the safety helmet and the safety belt are worn correctly in the step S2 is as follows:
acquiring the region R (x p ) Region R (x) of the helmet h ) And a region R (x) s ) According to the formula:
obtaining the correct discrimination value IOU of whether the safety helmet and the safety belt are worn X The method comprises the steps of carrying out a first treatment on the surface of the Wherein n represents the intersection; u represents the denoise set;
judging IOU X Whether the safety helmet is larger than a corresponding threshold value or not, if so, judging that the safety helmet and the safety belt are worn correctly; otherwise, early warning output is carried out.
Further, the specific method for judging whether the hanging mode of the safety belt is correct in the step S3 includes the following sub-steps:
s3-1, acquiring an identification frame containing a safety belt, and independently extracting pictures in the identification frame;
s3-2, performing binarization processing on the extracted picture to obtain a binary image;
s3-3, calculating the center coordinates (X 0 ,Y 0 ) And from the center coordinatesCoordinates (X of the belt pixel furthest apart e ,Y e )；
S3-4, according to the formula:
T＝|y1-y2|/4
acquiring a judging threshold T of a hanging mode of the safety belt; wherein y1 and y2 are the ordinate of the upper and lower boundaries of the seat belt identification frame, respectively;
s3-5, if Y 0 -Y e <-T is established, then it is determined as high hanging abutment, i.e. safe; if Y 0 -Y e >And T, judging that the vehicle is low in hanging and high in use, namely early warning output.
Further, the specific method of step S3-2 comprises the following sub-steps:
s3-2-1, obtaining a dot r of a cubic sphere of an RBG color space where the safety belt is located i And radius R 0 And RBG component a thereof i The method comprises the steps of carrying out a first treatment on the surface of the Wherein i=1, 2,3;
s3-2-2, according to the formula:
performing color slicing on the extracted picture to obtain RBG component b after the color slicing i ；
S3-2-3, the method of representing white by 1 and black by 0 will be represented by RBG component b i Binarizing the formed image to obtain a binary image after color slicing;
s3-2-4, removing burr holes of the binary image after the color slicing through morphological opening operation to obtain the binary image.
Further, the specific method for judging whether the state of the safety belt hook is normal in the step S4 is as follows:
constructing a safety belt hook data set through self-timer to perform transfer learning on the yolov5 network, and identifying the safety belt hook state by adopting the yolov5 network after transfer learning; the safety belt hook state comprises normal, thin wire hanging, oblique wire hanging, unclosed buckle and buckle tying line 5;
and when the state of the safety belt hook is that a thin wire is hung, oblique lines are hung, the buckle is not closed or the buckle is tied, early warning output is carried out.
The beneficial effects of the invention are as follows: on the basis of deep learning target detection, the invention integrates scene recognition technology, cross-over ratio, logic judgment function, colorimetry space target extraction and morphological processing technology, realizes the recognition of illegal operation behavior of power operation in a specific scene, and carries out corresponding early warning output, thereby effectively improving the supervision convenience of overhead operation.
Drawings
FIG. 1 is a schematic flow chart of the method;
FIG. 2 is a schematic diagram of a deep learning network;
FIG. 3 is a schematic view of SPPFCSPC structure;
FIG. 4 is a schematic diagram of a decoupling head configuration;
FIG. 5 is a schematic diagram of recognition results of a deep learning network according to an embodiment;
FIG. 6 is a belt identification chart in an embodiment;
FIG. 7 is a schematic diagram of a color slice binarization process according to an embodiment;
FIG. 8 is a schematic diagram of the morphological open operation according to the embodiment;
FIG. 9 is a diagram showing the result of identifying the coordinates of the belt region according to the embodiment;
FIG. 10 is a schematic diagram of early warning output in an embodiment;
fig. 11 is a schematic diagram of the recognition result being safe in the embodiment.
Detailed Description
The following description of the embodiments of the present invention is provided to facilitate understanding of the present invention by those skilled in the art, but it should be understood that the present invention is not limited to the scope of the embodiments, and all the inventions which make use of the inventive concept are protected by the spirit and scope of the present invention as defined and defined in the appended claims to those skilled in the art.
As shown in fig. 1, the method for identifying the behavior situation of the aerial working personnel based on machine vision comprises the following steps:
s1, acquiring a real-time picture, and identifying an operator, a safety helmet, a safety belt and a safety belt hook in the real-time picture through a deep learning network;
s2, judging whether the safety helmet and the safety belt are worn correctly, if so, entering a step S3; otherwise, early warning output is carried out;
s3, judging whether the hanging mode of the safety belt is correct, if so, entering a step S4; otherwise, early warning output is carried out;
s4, judging whether the state of the safety belt hook is normal, if so, judging that the behavior situation of the overhead working personnel is normal; otherwise, early warning output is carried out.
The method for constructing the deep learning network in the step S1 comprises the following steps: replacing an SPPF module of the yolov5 network with an SPPFCSPC module, replacing a coupling head of the yolov5 network with a decoupling head, and optimizing by adopting a SIOU loss function; the SPPFCSPC module comprises a first convolution unit and a second convolution unit which are connected in parallel; the second convolution unit is sequentially connected with a third convolution unit and a fourth convolution unit, and the output end of the fourth convolution unit is respectively connected with the first maximum pooling unit and the first fusion unit; the output end of the first maximum pooling unit is respectively connected with the input ends of the second maximum pooling unit and the first fusion unit; the output end of the second maximum pooling unit is respectively connected with the input ends of the third maximum pooling unit and the first fusion unit; the output end of the third maximum pooling unit is connected with the input end of the first fusion unit; the output end of the first fusion unit is sequentially connected with a fifth convolution unit and a sixth convolution unit; the output end of the sixth convolution unit and the output end of the first convolution unit are respectively connected with the second fusion unit; the output end of the second fusion unit is connected with a seventh convolution unit; the input ends of the first convolution unit and the second convolution unit are jointly used as the input end of the SPPFCSPC module; the output end of the seventh convolution unit is the output end of the SPPFCSPC module.
The specific method for optimizing by adopting the SIOU loss function comprises the following steps:
a1, taking an image containing a safety helmet, a safety belt and/or a safety belt hook as a training sample, and acquiring a real frame of the training sample; inputting training samples into a deep learning network to obtain a prediction frame;
a2, according to the formula:
acquiring an angle loss lambda of the deep learning network; wherein C is h The height difference between the true frame center point and the predicted frame center point; pi is the circumference ratio; sigma is the distance between the center points of the real frame and the predicted frame;the center coordinates of the real frames; />The center point coordinates of the prediction frame are obtained; max (·) represents taking the maximum value; min (·) represents taking the minimum value;
a3, according to the formula:
obtaining the distance loss delta of the deep learning network; where γ=2- Λ, ρ x And ρ y As an intermediate parameter ρ t ＝{ρ x ,ρ y }；X w And X h The width and the height of the minimum circumscribed rectangle of the real frame and the prediction frame are respectively;
a4, according to the formula:
acquiring shape loss omega of the deep learning network; wherein e is a natural constant; w, h, w gt And h gt The width and the height of the prediction frame and the real frame are respectively; θ is a constant, the degree of attention to shape loss is controlled, and in order to avoid excessively paying attention to shape loss, the movement of the prediction frame is reduced, and θ=2; w (W) w And W is h Are all intermediate parameters, w t ＝{W w ,W h }；
A5, according to the formula:
obtaining comprehensive Loss value Loss of deep learning network SIOU The method comprises the steps of carrying out a first treatment on the surface of the The IOU is the ratio of the intersection set and the union set of the prediction frame and the real frame;
a6, according to the comprehensive Loss value Loss of the deep learning network SIOU And optimizing parameters of the deep learning network to finish optimization of the deep learning network.
The specific method for judging whether the safety helmet and the safety belt are worn correctly in the step S2 is as follows: acquiring the region R (x p ) Region R (x) of the helmet h ) And a region R (x) s )，According to the formula:
obtaining the correct discrimination value IOU of whether the safety helmet and the safety belt are worn X The method comprises the steps of carrying out a first treatment on the surface of the Wherein n represents the intersection; u represents the denoise set;
judging IOU X Whether the safety helmet is larger than a corresponding threshold value or not, if so, judging that the safety helmet and the safety belt are worn correctly; otherwise, early warning output is carried out. If IOU X =0, then this indicates that there is an object that is not being worn correctly (e.g., is put aside).
The specific method for judging whether the hanging mode of the safety belt is correct in the step S3 comprises the following substeps:
s3-1, acquiring an identification frame containing a safety belt, and independently extracting pictures in the identification frame;
s3-2, performing binarization processing on the extracted picture to obtain a binary image;
s3-3, calculating the center coordinates (X 0 ,Y 0 ) And coordinates (X) of the belt pixel farthest from the center coordinates e ,Y e )；
S3-4, according to the formula:
T＝|y1-y2|/4
acquiring a judging threshold T of a hanging mode of the safety belt; wherein y1 and y2 are the ordinate of the upper and lower boundaries of the seat belt identification frame, respectively;
s3-5, if Y 0 -Y e <-T is established, then it is determined as high hanging abutment, i.e. safe; if Y 0 -Y e >And T, judging that the vehicle is low in hanging and high in use, namely early warning output.
The specific method of the step S3-2 comprises the following substeps:
s3-2-1, obtaining a dot r of a cubic sphere of an RBG color space where the safety belt is located i And radius R 0 And RBG component a thereof i The method comprises the steps of carrying out a first treatment on the surface of the Wherein i=1, 2,3;
s3-2-2, according to the formula:
performing color slicing on the extracted picture to obtain RBG component b after the color slicing i ；
S3-2-3, the method of representing white by 1 and black by 0 will be represented by RBG component b i Binarizing the formed image to obtain a binary image after color slicing;
s3-2-4, removing burr holes of the binary image after the color slicing through morphological opening operation to obtain the binary image.
The specific method for judging whether the state of the safety belt hook is normal in the step S4 is as follows:
constructing a safety belt hook data set through self-timer to perform transfer learning on the yolov5 network, and identifying the safety belt hook state by adopting the yolov5 network after transfer learning; the safety belt hook state comprises normal, thin wire hanging, oblique wire hanging, unclosed buckle and buckle tying line 5;
and when the state of the safety belt hook is that a thin wire is hung, oblique lines are hung, the buckle is not closed or the buckle is tied, early warning output is carried out.
In a specific implementation process, the deep learning network of the method takes yolov5 as a main network, improves the yolov5 network, improves the main network, the coupling head structure and the loss function of the yolov5 network, and realizes accurate identification of targets such as people, safety belts, safety belt hooks, safety caps and the like in an electric power operation scene. The network takes yolov5 as a main network, the feature extraction part of the main network is used for designing SPPFCSPC to replace SPPF of the original network, the receptive field of the main network is enlarged, the extraction capacity of the model to deep important features is improved, and the feature loss caused in the network feature processing process is reduced. And secondly, a decoupling head is used for replacing a coupling head structure in the original network, the confidence coefficient and the regression frame are realized separately by decoupling operation, and the negative influence caused by conflict between classification and regression tasks is relieved, so that the network detection precision is improved and the network convergence is accelerated. Finally, a SIOU (Scylla Intersection Over Union) loss function is introduced, and vector angles between the real frame and the predicted frame are further considered, so that the loss function is optimized, and the convergence speed of the model is increased. Optimized Yolov5 (deep learning network) is shown in fig. 2. Wherein the backup: a backbone network; neg: a bottleneck network; coupled Head: decoupling the detection head; CBS: and the convolution processing module is used for executing convolution, batch normalization and activation function operation on the input feature map. C3_x: the backbone network residual characteristic learning module comprises 3 standard convolution layers and x residual modules. SPPFCSPC: and a rapid spatial pyramid pooling module. Upsample: and an up-sampling module. Concat: and the feature fusion module is used for superposing two feature layers with the same size and adding the channel numbers. C3_1_F: and a neck network characteristic learning module. Coupled head: the head module is decoupled.
The method aims at the problem that more target detail information is lost due to the fact that the SPPF module repeatedly uses the pooling layer, and the target with a small individual hook is regarded as a background and cannot be detected finally. The method modifies the last SPPF module of the YOLOv5s backhaul, and the modified SPPF module is SPPFCSPC, and the structure is shown in figure 3. The SPPFCSPC module inputs a 512 x 20 feature map, serially inputs 3 convolution kernels of 5 x 5 size, and performs three convolution operations. Finally, the result is combined with the 1×1 convolution operation data, resulting in a feature map of size 512×20×20. SPPFCSPC can effectively increase receptive field through maximum pooling, so that the algorithm is suitable for images with different resolutions, and multi-scale target information can be obtained under the condition that the feature mapping size is kept unchanged. Maxpool2d in fig. 3: and (5) a maximum pooling module. Where k is the size of the window, k=5, and the size of the window is 5×5; s is the window moving step length; p is an edge fill value; conv: and a convolution module. k is the convolution kernel size, s is the step size, and p is the edge fill value. Concat: and the fusion module is used for superposing two characteristic layers with the same size channels and adding the channel numbers.
The decoupling Head (Decoupled Head) structure is shown in fig. 4. For the input feature layer, the decoupling head can utilize 1×1 convolution to reduce the dimension of the input feature layer, then 2 3×3 convolutions are respectively used in classification and regression branches, and parallel channels respectively perform object classification and target frame coordinate regression tasks. The Cls, the Reg and the Obj 3 outputs can be obtained through processing, wherein the Cls is the type corresponding to the target frame, the Reg is the position information of the target frame, and the Obj is whether each feature point contains an object. And fusing the 3 output values to obtain final prediction information. The decoupling operation is implemented by separating the confidence coefficient from the regression frame, and the negative influence caused by conflict between classification and regression tasks is relieved while the computational complexity is increased to a bit, so that the aims of improving the network detection precision and accelerating the network convergence are achieved.
And identifying the above targets by adopting optimized yolo v5, and recording the area of the identified targets, wherein the identification result is shown in fig. 5. And adding an identification frame for each identification target by the optimized yolo v5, and giving a corresponding discrimination value.
In one embodiment of the present invention, a picture in an identification frame including a seat belt is extracted, resulting in a picture as shown in fig. 6. The picture is subjected to color slice binarization to obtain the picture shown in fig. 7, and morphological open operation processing is further performed to obtain a binary image shown in fig. 8, wherein the identification result of the safety belt region coordinates in the binary image is shown in fig. 9. If the hanging mode of the safety belt is the low hanging high use mode as shown in fig. 10, early warning output is performed. If the fastening system of the seat belt is for high fastening as shown in fig. 11, the seat belt is determined to be safe.
Aiming at the problem of insufficient hook data quantity, the identification of the hook state carries out transfer learning on the basis of a yolov5 network, and a SCUT-hook data set is built by self-timer. The recognition network adopts the trained yolov5 to carry out migration learning. And (3) the migration learning freezes all convolution layers of the pre-training network, and only the top layer partial parameters are trained and fine-tuned by utilizing the collected and marked wearing equipment images. The parameters can be used for identifying the safety belt hook after being adjusted.
In summary, on the basis of deep learning target detection, the invention integrates scene recognition technology, cross-correlation ratio, logic judgment function, colorimetry space target extraction and morphological processing technology, realizes the recognition of illegal operation behaviors of power operation in a specific scene, carries out corresponding early warning output, and effectively improves the supervision convenience of overhead operation.
Claims (5)
1. The method for identifying the behavior situation of the aerial working personnel based on the machine vision is characterized by comprising the following steps of:
s1, acquiring a real-time picture, and identifying an operator, a safety helmet, a safety belt and a safety belt hook in the real-time picture through a deep learning network;
s2, judging whether the safety helmet and the safety belt are worn correctly, if so, entering a step S3; otherwise, early warning output is carried out;
s3, judging whether the hanging mode of the safety belt is correct, if so, entering a step S4; otherwise, early warning output is carried out;
s4, judging whether the state of the safety belt hook is normal, if so, judging that the behavior situation of the overhead working personnel is normal; otherwise, early warning output is carried out;
the specific method for judging whether the safety helmet and the safety belt are worn correctly in the step S2 is as follows:
acquiring the region R (x p ) Region R (x) of the helmet h ) And a region R (x) s ) According to the formula:
obtaining the correct discrimination value IOU of whether the safety helmet and the safety belt are worn X The method comprises the steps of carrying out a first treatment on the surface of the Wherein n represents the intersection; u represents the denoise set;
judging IOU X Whether the safety helmet is larger than a corresponding threshold value or not, if so, judging that the safety helmet and the safety belt are worn correctly; otherwise, early warning output is carried out;
the method for constructing the deep learning network in the step S1 comprises the following steps:
replacing an SPPF module of the yolov5 network with an SPPFCSPC module, replacing a coupling head of the yolov5 network with a decoupling head, and optimizing by adopting a SIOU loss function; the SPPFCSPC module comprises a first convolution unit and a second convolution unit which are connected in parallel; the second convolution unit is sequentially connected with a third convolution unit and a fourth convolution unit, and the output end of the fourth convolution unit is respectively connected with the first maximum pooling unit and the first fusion unit; the output end of the first maximum pooling unit is respectively connected with the input ends of the second maximum pooling unit and the first fusion unit; the output end of the second maximum pooling unit is respectively connected with the input ends of the third maximum pooling unit and the first fusion unit; the output end of the third maximum pooling unit is connected with the input end of the first fusion unit; the output end of the first fusion unit is sequentially connected with a fifth convolution unit and a sixth convolution unit; the output end of the sixth convolution unit and the output end of the first convolution unit are respectively connected with the second fusion unit; the output end of the second fusion unit is connected with a seventh convolution unit; the input ends of the first convolution unit and the second convolution unit are jointly used as the input end of the SPPFCSPC module; the output end of the seventh convolution unit is the output end of the SPPFCSPC module.
2. The machine vision-based aerial work personnel behavior situation recognition method according to claim 1, wherein the specific method for optimizing by using the SIOU loss function comprises the following steps:
a1, taking an image containing a safety helmet, a safety belt and/or a safety belt hook as a training sample, and acquiring a real frame of the training sample; inputting training samples into a deep learning network to obtain a prediction frame;
a2, according to the formula:
acquiring an angle loss lambda of the deep learning network; wherein C is h The height difference between the true frame center point and the predicted frame center point; pi is the circumference ratio; sigma is the distance between the center points of the real frame and the predicted frame;the center coordinates of the real frames; />The center point coordinates of the prediction frame are obtained; max (·) represents taking the maximum value; min (·) represents taking the minimum value;
a3, according to the formula:
obtaining the distance loss delta of the deep learning network; where γ=2- Λ, ρ x And ρ y As an intermediate parameter ρ t ＝{ρ x ,ρ y }；X w And X h The width and the height of the minimum circumscribed rectangle of the real frame and the prediction frame are respectively;
a4, according to the formula:
acquiring shape loss omega of the deep learning network; wherein e is a natural constant; w, h, w gt And h gt The width and the height of the prediction frame and the real frame are respectively; θ is a constant; w (W) w And W is h Are all intermediate parameters, w t ＝{W w ,W h }；
A5, according to the formula:
obtaining comprehensive Loss value Loss of deep learning network SIOU The method comprises the steps of carrying out a first treatment on the surface of the The IOU is the ratio of the intersection set and the union set of the prediction frame and the real frame;
a6, according to the comprehensive Loss value Loss of the deep learning network SIOU And optimizing parameters of the deep learning network to finish optimization of the deep learning network.
3. The machine vision-based high-altitude operation personnel behavior situation recognition method according to claim 1, wherein the specific method for judging whether the hanging mode of the safety belt is correct in the step S3 comprises the following sub-steps:
s3-1, acquiring an identification frame containing a safety belt, and independently extracting pictures in the identification frame;
s3-2, performing binarization processing on the extracted picture to obtain a binary image;
s3-3, calculating the center coordinates (X 0 ,Y 0 ) And coordinates (X) of the belt pixel farthest from the center coordinates e ,Y e )；
S3-4, according to the formula:
T＝|y1-y2|/4
acquiring a judging threshold T of a hanging mode of the safety belt; wherein y1 and y2 are the ordinate of the upper and lower boundaries of the seat belt identification frame, respectively;
s3-5, ifY 0 -Y e <-T is established, then it is determined as high hanging abutment, i.e. safe; if Y 0 -Y e >And T, judging that the vehicle is low in hanging and high in use, namely early warning output.
4. The machine vision-based aerial working personnel behavior situation recognition method of claim 3, wherein the specific method of step S3-2 comprises the following sub-steps:
s3-2-1, obtaining a dot r of a cubic sphere of an RBG color space where the safety belt is located i And radius R 0 And RBG component a thereof i The method comprises the steps of carrying out a first treatment on the surface of the Wherein i=1, 2,3;
s3-2-2, according to the formula:
performing color slicing on the extracted picture to obtain RBG component b after the color slicing i ；
S3-2-3, the method of representing white by 1 and black by 0 will be represented by RBG component b i Binarizing the formed image to obtain a binary image after color slicing;
s3-2-4, removing burr holes of the binary image after the color slicing through morphological opening operation to obtain the binary image.
5. The machine vision-based high-altitude operation personnel behavior situation recognition method according to claim 1, wherein the specific method for judging whether the state of the safety belt hook is normal in the step S4 is as follows:
constructing a safety belt hook data set through self-timer to perform transfer learning on the yolov5 network, and identifying the safety belt hook state by adopting the yolov5 network after transfer learning; the safety belt hook state comprises normal, thin wire hanging, oblique wire hanging, unclosed buckle and buckle tying line 5;
and when the state of the safety belt hook is that a thin wire is hung, oblique lines are hung, the buckle is not closed or the buckle is tied, early warning output is carried out.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310147997.2A CN116229570B (en) | 2023-02-21 | 2023-02-21 | Aloft work personnel behavior situation identification method based on machine vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310147997.2A CN116229570B (en) | 2023-02-21 | 2023-02-21 | Aloft work personnel behavior situation identification method based on machine vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116229570A CN116229570A (en) | 2023-06-06 |
| CN116229570B true CN116229570B (en) | 2024-01-23 |
Family
ID=86576416
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310147997.2A Active CN116229570B (en) | 2023-02-21 | 2023-02-21 | Aloft work personnel behavior situation identification method based on machine vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116229570B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117235650B (en) * | 2023-11-13 | 2024-02-13 | 国网浙江省电力有限公司温州供电公司 | Method, device, equipment and medium for detecting high-altitude operation state |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111160440A (en) * | 2019-12-24 | 2020-05-15 | 广东省智能制造研究所 | Helmet wearing detection method and device based on deep learning |
| CN111931573A (en) * | 2020-07-07 | 2020-11-13 | 南京南瑞信息通信科技有限公司 | Helmet detection and early warning method based on YOLO evolution deep learning model |
| CN112016502A (en) * | 2020-09-04 | 2020-12-01 | 平安国际智慧城市科技股份有限公司 | Safety belt detection method and device, computer equipment and storage medium |
| KR20210006722A (en) * | 2019-07-09 | 2021-01-19 | 주식회사 케이티 | Apparatus, method and computer program for determining whether safety equipment is worn |
| CN112784821A (en) * | 2021-03-06 | 2021-05-11 | 深圳市安比智慧科技有限公司 | Building site behavior safety detection and identification method and system based on YOLOv5 |
| CN113553977A (en) * | 2021-07-30 | 2021-10-26 | 国电汉川发电有限公司 | Improved YOLO V5-based safety helmet detection method and system |
| CN113688865A (en) * | 2021-07-19 | 2021-11-23 | 青岛科技大学 | Helmet wearing identification detection based on improved Yolo v3 |
| CN113822383A (en) * | 2021-11-23 | 2021-12-21 | 北京中超伟业信息安全技术股份有限公司 | Unmanned aerial vehicle detection method and system based on multi-domain attention mechanism |
| CN115565232A (en) * | 2022-10-24 | 2023-01-03 | 广东电网有限责任公司广州供电局 | Power distribution room switch cabinet face part identification method based on improved YOLOv5 algorithm |
| CN115690675A (en) * | 2022-10-12 | 2023-02-03 | 大连海洋大学 | ESB-YOLO model cultured fish shoal detection method based on channel non-dimensionality reduction attention mechanism and improved YOLOv5 |
-
2023
- 2023-02-21 CN CN202310147997.2A patent/CN116229570B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20210006722A (en) * | 2019-07-09 | 2021-01-19 | 주식회사 케이티 | Apparatus, method and computer program for determining whether safety equipment is worn |
| CN111160440A (en) * | 2019-12-24 | 2020-05-15 | 广东省智能制造研究所 | Helmet wearing detection method and device based on deep learning |
| CN111931573A (en) * | 2020-07-07 | 2020-11-13 | 南京南瑞信息通信科技有限公司 | Helmet detection and early warning method based on YOLO evolution deep learning model |
| CN112016502A (en) * | 2020-09-04 | 2020-12-01 | 平安国际智慧城市科技股份有限公司 | Safety belt detection method and device, computer equipment and storage medium |
| CN112784821A (en) * | 2021-03-06 | 2021-05-11 | 深圳市安比智慧科技有限公司 | Building site behavior safety detection and identification method and system based on YOLOv5 |
| CN113688865A (en) * | 2021-07-19 | 2021-11-23 | 青岛科技大学 | Helmet wearing identification detection based on improved Yolo v3 |
| CN113553977A (en) * | 2021-07-30 | 2021-10-26 | 国电汉川发电有限公司 | Improved YOLO V5-based safety helmet detection method and system |
| CN113822383A (en) * | 2021-11-23 | 2021-12-21 | 北京中超伟业信息安全技术股份有限公司 | Unmanned aerial vehicle detection method and system based on multi-domain attention mechanism |
| CN115690675A (en) * | 2022-10-12 | 2023-02-03 | 大连海洋大学 | ESB-YOLO model cultured fish shoal detection method based on channel non-dimensionality reduction attention mechanism and improved YOLOv5 |
| CN115565232A (en) * | 2022-10-24 | 2023-01-03 | 广东电网有限责任公司广州供电局 | Power distribution room switch cabinet face part identification method based on improved YOLOv5 algorithm |
Non-Patent Citations (2)
| Title |
|---|
| Safety Helmet Detection Based on YOLOv5;Fangbo Zhou等;《2021 IEEE International Conference on Power Electronics, Computer Applications (ICPECA)》;第6-11页 * |
| 基于YOLO的安全帽检测方法;林俊等;《计算机系统应用》;第174-179页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116229570A (en) | 2023-06-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113553977B (en) | Improved YOLO V5-based safety helmet detection method and system | |
| CN109684959B (en) | Video gesture recognition method and device based on skin color detection and deep learning | |
| CN111145165A (en) | Rubber seal ring surface defect detection method based on machine vision | |
| CN113553979B (en) | Safety clothing detection method and system based on improved YOLO V5 | |
| CN112115775B (en) | Smoke sucking behavior detection method based on computer vision under monitoring scene | |
| CN112183482A (en) | Dangerous driving behavior recognition method, device and system and readable storage medium | |
| CN1932847A (en) | Method for detecting colour image human face under complex background | |
| CN116229570B (en) | Aloft work personnel behavior situation identification method based on machine vision | |
| CN110929593A (en) | Real-time significance pedestrian detection method based on detail distinguishing and distinguishing | |
| CN106909884B (en) | Hand region detection method and device based on layered structure and deformable part model | |
| CN111553214B (en) | Method and system for detecting smoking behavior of driver | |
| CN110057820B (en) | Method, system and storage medium for on-line detection of chlorine-hydrogen ratio of hydrogen chloride synthesis furnace | |
| CN112733914B (en) | Underwater target visual identification classification method based on support vector machine | |
| CN114331986A (en) | Dam crack identification and measurement method based on unmanned aerial vehicle vision | |
| CN108564034A (en) | The detection method of operating handset behavior in a kind of driver drives vehicle | |
| CN113158850A (en) | Ship driver fatigue detection method and system based on deep learning | |
| CN112101260A (en) | Method, device, equipment and storage medium for identifying safety belt of operator | |
| CN116665011A (en) | Coal flow foreign matter identification method for coal mine belt conveyor based on machine vision | |
| Reddy et al. | A Deep Learning Model for Traffic Sign Detection and Recognition using Convolution Neural Network | |
| Wali et al. | Shape matching and color segmentation based traffic sign detection system | |
| CN114241542A (en) | Face recognition method based on image stitching | |
| CN113128476A (en) | Low-power consumption real-time helmet detection method based on computer vision target detection | |
| CN117456358A (en) | Method for detecting plant diseases and insect pests based on YOLOv5 neural network | |
| CN114092572A (en) | Clothing color analysis method, system, storage medium and computer equipment | |
| CN113052234A (en) | Jade classification method based on image features and deep learning technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |