CN114821806A - Method and device for determining behavior of operator, electronic equipment and storage medium - Google Patents
Method and device for determining behavior of operator, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114821806A CN114821806A CN202210555206.5A CN202210555206A CN114821806A CN 114821806 A CN114821806 A CN 114821806A CN 202210555206 A CN202210555206 A CN 202210555206A CN 114821806 A CN114821806 A CN 114821806A
- Authority
- CN
- China
- Prior art keywords
- determining
- operator
- behavior
- human body
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
Abstract
The embodiment of the invention discloses a method and a device for determining the behavior of an operator, electronic equipment and a storage medium. The method comprises the following steps: collecting operation images of an operator in an electric power construction site in the operation process; determining an incidence relation between human key points of an operator in the operation image and each human key point based on a human posture recognition model established in advance; and determining the operation behaviors of the operators corresponding to the human key points and the association relation, comparing the operation behaviors with preset violation behaviors, and determining whether the violation behaviors exist in the operators. According to the technical scheme of the embodiment of the invention, the violation behavior can be determined through the human posture recognition model, the operation image does not need to be checked by a safety supervisor, the resource waste is reduced, and the accuracy and the effectiveness of determining the violation behavior are improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of computer vision, in particular to a method and a device for determining behavior of an operator, electronic equipment and a storage medium.
Background
At present, in the power grid inspection process in China, a large number of operating personnel are still required to carry out operations such as equipment maintenance and hidden danger elimination on the site of power equipment. The electric power operation site belongs to a high-risk environment, and has numerous personnel and equipment, so that in order to improve the safety standard of power grid operation and maintenance, the personnel behaviors in the electric power construction scene need to be monitored in real time, the violation behaviors of incorrectly wearing safety helmets, not wearing tools and the like on the construction site can be timely and accurately found, early warning is timely carried out, and the operation safety in the monitoring range is guaranteed.
In the prior art, usually, the operation image information of an operator in an operation site needs to be acquired, and a safety supervisor determines whether the current operator has illegal behaviors such as incorrect wearing of a safety helmet, no wearing of a tool and the like in a construction site by checking the operation image information of the operator. However, the prior art method consumes a large amount of human resources; and the violation behaviors are confirmed by subjective cognition to make mistakes, so that the accuracy and the effectiveness of the determining process are reduced.
Disclosure of Invention
The embodiment of the invention provides a method and a device for determining behaviors of an operator, electronic equipment and a storage medium, which are used for determining illegal behaviors through a human body posture recognition model without checking operation images by a safety supervisor, so that resource waste is reduced, and the accuracy and the effectiveness of determining the illegal behaviors are improved.
In a first aspect, an embodiment of the present invention provides a method for determining an operator behavior, including:
collecting operation images of an operator in an electric power construction site in the operation process;
determining an incidence relation between human key points of the operating personnel and each human key point in the operating image based on a pre-established human posture recognition model;
and determining the operation behavior of the operator corresponding to the human body key point and the incidence relation, comparing the operation behavior with a preset violation behavior, and determining whether the operator has the violation behavior.
In a second aspect, an embodiment of the present invention further provides an operator behavior determination apparatus, where the apparatus includes:
the acquisition operation image module is used for acquiring operation images of operation processes of operators in the electric power construction site;
the human body key point determining module is used for determining the association relationship between the human body key points of the operating personnel and the human body key points in the operating image based on a human body posture recognition model established in advance;
and the operation behavior determining module is used for determining the operation behaviors of the operators corresponding to the human body key points and the incidence relation, comparing the operation behaviors with preset violation behaviors and determining whether the violation behaviors exist in the operators.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining the behavior of the operator provided by any of the embodiments of the present invention.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method for determining the behavior of the operator according to any embodiment of the present invention.
According to the method for determining the behavior of the operating personnel, provided by the embodiment of the invention, the human body posture recognition model is constructed, the operation behavior of the operating personnel is automatically recognized through the human body posture recognition model, manual monitoring is not needed, a large amount of human resources are saved, the accuracy of violation behavior determination is improved, the potential safety hazard of operation is eliminated, and the operation and maintenance safety is improved; and the behavior of the operating personnel is analyzed, and when the illegal behavior of the operating personnel is determined, the warning information is used for prompting, so that the safety problem can be found in time, and the potential safety hazard in the working process of the power grid is solved.
In addition, the operator behavior determination device, the electronic device and the storage medium provided by the invention correspond to the method, and have the same beneficial effects.
Drawings
In order to illustrate the embodiments of the present invention more clearly, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of an operator behavior determination method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another method for determining operator behavior according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a human body key point and an association relationship provided in an embodiment of the present invention;
fig. 4 is a block diagram of an operator behavior determination apparatus according to an embodiment of the present invention;
fig. 5 is a structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
It should be further noted that, for the convenience of description, only some but not all of the relevant aspects of the present invention are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Example one
Fig. 1 is a flowchart of an operator behavior determination method according to an embodiment of the present invention. The method may be executed by an operator behavior determination apparatus, the apparatus may be implemented by software and/or hardware, and may be configured in a terminal and/or a server to implement the operator behavior determination method in the embodiment of the present invention.
As shown in fig. 1, the method of the embodiment may specifically include:
s101, collecting operation images of an operation process of an operator in an electric power construction site.
In specific implementation, an image collector installed on an electric power construction site can be used for collecting operation images of an operation process of an operator. And determining whether the behavior of the operator is standard or not by analyzing the behavior and the action of the operator in the operation image. For example, two or more job images may be captured at the same time and for the same job scenario.
Specifically, in order to improve the pixel quality of the operation image, enhance the characteristics of the target to be detected, optimize the detection performance and avoid the over-fitting, the operation image can be subjected to preprocessing operations such as convolution calculation, linear rectification function calculation, pooling operation and the like.
Optionally, after collecting the operation image of the operation process of the operator in the electric power construction site, the method further includes: and performing convolution calculation, linear rectification function calculation and pooling operation on the operation graph, and updating the operation graph into the graph obtained after the operation.
In the preprocessing operation, the job image is input to a convolution layer with a convolution kernel size of 3 × 3, and the adjustment of the pixel value of the job image is realized through convolution calculation. And calculating the first image obtained after the convolution calculation through a linear rectification function to generate a second image. The linear rectification function can enable the pixel value of the pixel position of which the pixel value is smaller than the preset threshold value in the image to be set to be 0, so that the network sparsity of the image is improved, and overfitting is avoided while the detection performance is improved. And inputting the second image into a pooling layer with the size of 2 x 2, and outputting the preprocessed feature map, so that the image is filtered, and pixels with strong expression capability in the image are highlighted. Further, in order to enhance the characteristic effect of the characteristic map, the output characteristic map may still be input to the convolution layer, the linear rectification function, and the pooling layer again for processing, which is not limited in the embodiment of the present invention. For example, the pre-processing operation may go through a total of 12 convolutional layers, 12 linear rectification functions, and 3 pooling layers.
S102, determining the association relationship between the human key points of the operating personnel in the operating image and each human key point based on a pre-established human posture recognition model.
In a specific implementation, the human body key points of the operator in the operation image can be identified based on a human body posture identification model established in advance. Illustratively, the human body key points may include a nose, neck, right shoulder, right elbow, right hand, left shoulder, left elbow, left hand, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, and left ear. Further, the association relationship between the human key points of the operator in the operation image can be determined. It should be noted that the association relationship is a position relationship and a connection relationship of each human body key point in the operation image, and the current posture of the operator can be determined based on the human body key points and the association relationship.
Optionally, determining, based on a pre-established human body posture recognition model, an association relationship between a human body key point of an operator in the operation image and each human body key point, including: inputting the operation images into a human body posture recognition model, and generating a first number of confidence level images and a second number of affinity field images corresponding to the operation images; determining human body key points of the operating personnel in the operating image based on the first number of confidence level images; based on the second number of affinity field images, an association between the body keypoints is determined.
In particular, the first number may correspond to the number of human key points that need to be determined. Illustratively, the first number may be 18, each corresponding to 18 human key points. Each human body key point can correspond to a confidence image, and the probability degree of each pixel position as the corresponding key point can be reflected by the confidence value represented by each pixel position in the confidence image.
For example, in the confidence image corresponding to the nose, each pixel position corresponds to a confidence value, the confidence value can reflect the possible degree that the information reflected by the pixel position of the working image is the nose of the working person, and the higher the confidence value is, the higher the possible degree is. Therefore, the human key points of the operator and the pixel positions of the human key points in the working image can be determined based on the confidence images. Furthermore, the confidence image can include a work background image besides the image corresponding to each human body key point so as to embody the work background information of the power site.
S103, determining the operation behaviors of the operators corresponding to the key points and the incidence relation of the human body, comparing the operation behaviors with preset violation behaviors, and determining whether the violation behaviors exist in the operators.
In specific implementation, the behavior characteristics of the operator can be determined by combining the human body key points and the incidence relation between the human body key points, and the action posture of the operator in the operation image can be determined and given based on the behavior characteristics. For example, based on the association relationship between the human body key points and the human body key points, if the behavior characteristic of the operator is that the left elbow, the left hand and the left shoulder are at the same height, the action posture of the operator can be determined as the left arm lifting.
Optionally, determining the operation behavior of the operator corresponding to the human body key point and the association relationship includes: and determining the operation background information of the corresponding electric power construction site in the operation image, and determining the operation behavior of the operator based on the operation background information, the human body key points and the association relation.
Specifically, based on the key points and the association relationship of the human body, the current action posture of the operator can be determined, for example, the action postures of leg lifting, squatting, standing, lying, hand lifting and the like. It is not possible to accurately judge whether the worker is performing normal task operations or violations from the action posture alone. In order to improve the accuracy of the determination process, the operation background information of the corresponding electric power construction site in the operation image can be determined through the information output by the human body posture recognition model. And determining the current operation behavior of the operator based on the operation background information and the current action posture of the operator. Determining whether the operation behavior of the operator is in an illegal behavior range set in the behavior specification, and if so, indicating the current illegal operation of the operator; if not, the current normal work of the operator is indicated, and no illegal action is caused.
For example, when the current action of the operator is determined as the leg raising action, the leg raising action has no violation property; when the leg part is located right above the safety fence when the leg of the operator is lifted by combining the operation background information, the operator can be determined to cross the fence, and the current behavior of the operator can be determined to be the violation behavior.
Optionally, determining the operation behavior of the operator based on the operation background information, the human body key points and the association relationship includes: determining whether the job context information includes job related information; and if so, determining the operation behavior of the operator based on the operation associated information and the key points and the association relation.
The job related information comprises equipment information and/or job identification information; the equipment information comprises information such as power equipment, cables and fences in the transformer substation, and the operation identification information comprises information such as work safety prompts and safety signs. When the device information and/or the job identification information are included in the job background information, the device information and/or the job identification information need to be combined to accurately judge whether the job behavior of the worker violates the rule. When the job background information does not include the job related information, it is indicated that the specific content of the job background information does not affect the judgment of the behavior of the operator. For example, in a case where only environment information such as sky and ground is included in the work context information, it may be determined that the work context information does not include the work related information.
Furthermore, when the operation background information does not include the operation associated information, the operation behavior of the operator can be determined directly based on the human body key points and the association relation, and the efficiency of determining the operation behavior of the operator is improved.
The method for determining the behavior of the operator, provided by the embodiment of the invention, comprises the steps of collecting an operation image of the operator in an operation process in an electric power construction site; determining the incidence relation between the human body key points of the operating personnel in the operating image and each human body key point through a pre-established human body posture recognition model; and determining the operation behaviors of the operators corresponding to the key points and the incidence relation of the human body, comparing the operation behaviors with preset violation behaviors, and determining whether the violation behaviors exist in the operators. The embodiment of the invention realizes the violation determination through the human body posture recognition model, does not need a safety supervisor to check the operation image, reduces the resource waste, and improves the accuracy and the effectiveness of the violation determination.
Example two
Fig. 2 is a flowchart of another method for determining an operator behavior according to an embodiment of the present invention. The present embodiment is optimized based on the above technical solutions. Optionally, before determining the association relationship between the human body key points of the operator in the working image and each human body key point, the method further includes: acquiring a sample image of at least one operator in the operation process to form a sample data set; marking the operators in each sample image of the sample data set in advance; and training the convolutional neural network based on the marked sample images, and establishing a human body posture recognition model based on a training result. Optionally, after comparing the job behavior with a preset violation behavior and determining whether the violation behavior exists for the operator, the method further includes: and when the fact that the illegal behaviors exist in the operating personnel is determined, warning information containing the illegal behaviors is generated and sent to the safety personnel terminal to prompt. The same or corresponding terms as those in the above embodiments are not explained in detail herein.
As shown in fig. 2, the method of the embodiment may specifically include:
s201, obtaining a sample image of at least one operator in the operation process, forming a sample data set, and marking the operator in each sample image of the sample data set in advance.
Optionally, obtaining a sample image of a work process of at least one worker includes: acquiring a historical monitoring video, and performing frame extraction operation on the historical monitoring video according to a preset time interval; and determining each frame image obtained by the frame extraction operation as a sample image.
In specific implementation, a sample image of the working process of at least one operator can be acquired by extracting frames from the historical monitoring video at preset time intervals. Specifically, the preset time interval may be set to 1 hour or one day, and two or more video frames may be extracted from the monitoring video at the same time in the frame extraction process. Furthermore, the extracted video frames can be compared, and based on the similarity between the video frames, the video frame with the larger similarity difference between the video frames is selected as a sample image, and the sample data set is formed by the sample images. Furthermore, key points and association relations of the human body in each sample image in the sample data set can be marked.
For example, when a sample image is obtained, in order to improve the diversity of the sample image, video frames containing workers of different genders, ages, work types and body types can be extracted from a historical monitoring video to serve as the sample image. Furthermore, the video frames of different weather and different scenes can be extracted as sample images in consideration of the background diversity of the sample images, so that the obtained sample images are enriched, and the convolutional neural network can be trained better.
Optionally, forming a sample data set includes: performing transformation processing on at least one acquired sample image; and forming a sample data set based on each transformed image and the sample image obtained after the transformation processing. Wherein the conversion process includes at least one of a scaling process, a contrast adjustment process, a color dithering process, and a noise addition process. Specifically, in order to increase the diversity of the sample data set, the sample images can be transformed, and the sample data set is composed of the transformed images and the sample images, so that the purpose of expanding the sample data set is achieved.
S202, training the convolutional neural network based on the marked sample images, and establishing a human body posture recognition model based on the training result.
Specifically, each marked sample image is input into the convolutional neural network, and the convolutional neural network is trained based on the marking information of each artificial key point and the incidence relation in the sample image and the output result of the convolutional neural network. And constructing a human body posture recognition model based on each generation parameter in the convolutional neural network obtained through training.
On the basis of the above embodiment, training the convolutional neural network based on each labeled sample image, and the specific step of establishing the human body posture recognition model based on the training result may include: inputting the sample image into a convolutional neural network, and outputting the recognition result of the convolutional neural network on the posture of the operator in the sample image; comparing the identification result with the mark in the sample image, adjusting the network parameter of the convolutional neural network based on the comparison result, and updating the convolutional neural network; and repeatedly inputting the sample image into the convolutional neural network, comparing the obtained recognition result with the mark, and adjusting the grid parameters of the neural network until the current convolutional neural network meets the preset condition, ending the training process of the convolutional neural network, and determining the current convolutional neural network as the human body posture recognition model.
Specifically, for each sample image, the sample image marked with the mark may be input into a convolutional neural network, and the convolutional neural network identifies the posture of the operator in the sample image to obtain an identification result. Illustratively, the recognition result includes human key points of the operator and an association relationship between the human key points. And comparing the recognition result with the mark in the sample image based on the recognition result, so as to adjust the network parameter of the convolutional neural network, so that the recognition result of the convolutional neural network on the sample image after the network parameter is adjusted can be closer to the mark of the sample image, and the accuracy of the trained convolutional neural network is improved.
Further, the above operations may be repeated, and the network parameters of the convolutional neural network are continuously adjusted until the current training process reaches the preset conditions, and the training of the convolutional neural network may be stopped, and the current convolutional neural network is determined as the human body posture recognition model.
For example, the preset condition may be that the number of times of repeated execution reaches a preset number threshold, and a person skilled in the art may determine the preset number threshold according to an actual application situation, which is not limited to the embodiment of the present invention. Further, the preset condition may further include that the number of the marks inconsistent with the current recognition result is smaller than a preset number threshold, that is, if the recognition effect of the current convolutional neural network is closer to the real situation of the operator in the sample image, the training may be stopped.
And S203, collecting operation images of the operation process of the operators in the electric power construction site.
S204, determining the association relationship between the human key points of the operating personnel in the operating image and each human key point based on the pre-established human posture recognition model.
S205, determining the operation behaviors of the operators corresponding to the key points and the incidence relation of the human body, comparing the operation behaviors with preset violation behaviors, and determining whether the violation behaviors exist in the operators.
According to the method for determining the behavior of the operating personnel, the human body posture recognition model is constructed, the operation behavior of the operating personnel is automatically recognized through the human body posture recognition model, manual monitoring is not needed, a large amount of human resources are saved, the accuracy of violation behavior determination is improved, the potential safety hazard of operation is eliminated, and the operation and maintenance safety is improved. Training the human body posture recognition model through the sample data set, so that the violation behavior can be determined through the human body posture recognition model;
and S206, when the fact that the operator has the violation is determined, generating warning information containing the violation, and sending the warning information to the safety personnel terminal for prompting.
Optionally, when it is determined that the operator has an illegal behavior, the operator and the security personnel need to be prompted in time to avoid operation danger due to the illegal behavior. Specifically, warning information including the violation can be generated and sent to the safety personnel terminal and/or the operation personnel terminal to prompt the operation personnel and enable the safety personnel to find out the dangerous situation in time. Further, the warning information may be sent to the operator terminal or the safety personnel terminal based on pre-stored contact information of the operator or the safety personnel, such as a telephone number, a mailbox address, and the like.
According to the method for analyzing the behavior of the operating personnel, when the fact that the operating personnel violate the behaviors is determined, the warning information is used for prompting, so that the safety problem can be found in time, and the potential safety hazard in the working process of a power grid is solved.
EXAMPLE III
The embodiment corresponding to the method for determining the behavior of the operator is described in detail above, and in order to make the technical solution of the method further clear to those skilled in the art, a specific application scenario is given below.
In specific implementation, after the operation image to be detected is input to the human body posture identification model, the operation image to be detected can be divided into a first branch for determining key points of a human body and a second branch for determining an association relation. The first branch output result contains feature maps of 19 channels, wherein the feature maps contain the confidence degrees of all key points, and the confidence degrees are respectively represented by 18 human body key point feature maps and a background feature map. The second branch can output a feature map of 38 channels for embodying the affinity field, i.e. the association relationship between the key points of the human body.
Specifically, the two branches respectively repeat 4 times of combination operations consisting of convolution calculation and linear rectification function operation, and perform one convolution operation respectively, so that the first branch outputs feature maps of 19 channels, the second branch outputs feature maps of 38 channels, and the feature maps obtained by the two branches are subjected to splicing operation to generate a unique target feature map. In order to improve the accuracy of the output feature map, the above steps can be repeated by taking the target feature map as an input item, and a person skilled in the art can determine the repetition times according to actual needs. And determining the human body key points and the association relation of the operating personnel based on the finally obtained spliced characteristic diagram.
FIG. 3 is a schematic diagram of a human body key point and an association relationship provided in an embodiment of the present invention; as shown in fig. 3, the human body key points are connected based on the association relationship, so that the operation posture of the operator can be reflected, and the operation behavior of the operator can be determined.
For example, the violation of crossing a guardrail can be taken as an example, and the process of determining the violation can be described. And determining key point coordinates of each human body key point according to the human body key points and the association relation determined by the human body posture recognition model, determining that the right ankle or the left ankle is positioned above the guard rail based on the operation background information when the right ankle height coordinate of the operator is larger than the left knee coordinate or the left ankle height coordinate is larger than the right knee coordinate, and if so, determining that the operator is crossing the guard rail currently and performing illegal operation.
When determining that the operating personnel has the violation, in order to avoid the operating danger caused by the violation, the operating personnel and the safety personnel need to be prompted in time. Specifically, warning information including the violation can be generated and sent to the safety personnel terminal and/or the operation personnel terminal to prompt the operation personnel and enable the safety personnel to find out the dangerous situation in time. Further, the warning information may be sent to the operator terminal or the safety personnel terminal based on pre-stored contact information of the operator or the safety personnel, such as a telephone number, a mailbox address, and the like.
In this embodiment, a method for determining a behavior of an operator is provided, where a human body posture recognition model is constructed, an association relationship between a human body key point of the operator and each human body key point in an operation image is determined, and an operation behavior of the operator is determined according to the human body key point and the association relationship, so as to determine whether the operation behavior violates a rule. The problem that whether violation behaviors exist or not cannot be accurately determined in the power grid production and inspection process is solved, the safety standard of power grid operation and maintenance is improved, and the personnel behaviors in the power construction scene are safely supervised. Meanwhile, the behavior analysis method of the operating personnel is provided, whether the behavior of the operating personnel violates is analyzed by combining the operation background and the action of the operating personnel, if the behavior violates, warning information is generated for prompting, the problem that the behavior violates cannot be found in time is solved, the safety supervision personnel do not need to check the operation image, the resource waste is reduced, and the accuracy and the effectiveness for determining the behavior violates are improved.
Example four
Fig. 4 is a block diagram of an operator behavior determination apparatus according to an embodiment of the present invention, which is configured to execute the operator behavior determination method according to any of the embodiments described above. The present invention is not limited to the above embodiments, and the details of the embodiments of the operator behavior determination apparatus are not described in detail. The device may specifically comprise:
the acquisition operation image module 10 is used for acquiring operation images of the operation process of operators in the electric power construction site;
the human body key point determining module 11 is used for determining the association relationship between the human body key points of the operating personnel in the operating image and each human body key point based on a human body posture recognition model established in advance;
and the operation behavior determining module 12 is configured to determine an operation behavior of the operator corresponding to the human body key point and the association relationship, compare the operation behavior with a preset violation behavior, and determine whether the violation behavior exists in the operator.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the apparatus further includes:
the human body posture recognition model training module is used for acquiring at least one sample image of the operation process of the operator before determining the incidence relation between the human body key points of the operator in the operation image and each human body key point to form a sample data set; marking the operators in each sample image of the sample data set in advance; and training the convolutional neural network based on the marked sample images, and establishing a human body posture recognition model based on a training result.
On the basis of any optional technical scheme in the embodiment of the present invention, optionally, the training human body posture recognition model module includes:
the conversion processing unit is used for performing conversion processing on the acquired at least one sample image; forming a sample data set based on each transformed image and the sample image obtained after the transformation processing; wherein the conversion process includes at least one of a scaling process, a contrast adjustment process, a color dithering process, and a noise addition process.
On the basis of any optional technical scheme in the embodiment of the present invention, optionally, the training human body posture recognition model module includes:
an input sample image unit, configured to input the sample image into the convolutional neural network, and output a recognition result of the convolutional neural network on a posture of the operator in the sample image;
the updating convolutional neural network unit is used for comparing the identification result with the mark in the sample image, adjusting the network parameter of the convolutional neural network based on the comparison result and updating the convolutional neural network;
and the training process ending unit is used for repeatedly executing the operations of inputting the sample image into the convolutional neural network, comparing the obtained recognition result with the mark and adjusting the grid parameters of the neural network until the current training process meets the preset condition, ending the training process of the convolutional neural network and determining the current convolutional neural network as the human body posture recognition model.
On the basis of any optional technical scheme in the embodiment of the present invention, optionally, the preset condition includes that the number of times of repeated execution reaches a preset number threshold.
On the basis of any optional technical scheme in the embodiment of the present invention, optionally, the training human body posture recognition model module includes:
the system comprises an acquisition historical monitoring video unit, a frame extraction unit and a frame extraction unit, wherein the acquisition historical monitoring video unit is used for acquiring a historical monitoring video and carrying out frame extraction on the historical monitoring video according to a preset time interval; and determining each frame image obtained by the frame extraction operation as the sample image.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the module 11 for determining human body key points includes:
generating affinity field images, inputting the operation images into the human body posture recognition model, and generating a first number of confidence level images and a second number of affinity field images corresponding to the operation images; determining human body key points of the operating personnel in the operating image based on the first number of confidence level images; and determining the association relation among the key points of the human body based on the second number of the affinity field images.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the apparatus further includes:
and the updating module is used for performing convolution calculation, linear rectification function calculation and pooling operation on the operation graph after acquiring the operation image of the operation process of the operator in the electric power construction site, and updating the operation graph into the graph obtained after the operation.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the apparatus further includes:
and the prompting module is used for generating warning information containing the violation and sending the warning information to the safety personnel terminal for prompting when the violation is determined to exist in the operation personnel after the operation behavior is compared with the preset violation behavior and whether the violation behavior exists in the operation personnel is determined.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the determining job behavior module 12 includes:
and the operation background information determining module is used for determining operation background information of a corresponding electric power construction site in the operation image and determining operation behaviors of the operators based on the operation background information, the human key points and the association relation.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the determining job behavior module 12 includes:
determining whether a job related information unit is included for determining whether job related information is included in the job context information; the job related information comprises equipment information and/or job identification information; and if so, determining the operation behavior of the operator based on the operation associated information, the key point and the association relation.
On the basis of any optional technical solution in the embodiment of the present invention, optionally, the determining job behavior module 12 further includes:
and the operation behavior determining unit is used for determining the operation behavior of the operator based on the human key point and the association relation when the operation background information does not comprise operation associated information.
The device for determining the behavior of the operator, provided by the embodiment of the invention, can realize the following method: collecting operation images of an operator in an electric power construction site in the operation process; determining an incidence relation between human key points of an operator in the operation image and each human key point based on a human posture recognition model established in advance; and determining the operation behaviors of the operators corresponding to the key points and the incidence relation of the human body, comparing the operation behaviors with preset violation behaviors, and determining whether the violation behaviors exist in the operators. The embodiment of the invention realizes the violation determination through the human body posture recognition model, does not need a safety supervisor to check the operation image, reduces the resource waste, and improves the accuracy and the effectiveness of the violation determination.
It should be noted that, in the embodiment of the operator behavior determination apparatus, the included units and modules are merely divided according to the functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
EXAMPLE five
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. FIG. 5 illustrates a block diagram of an exemplary electronic device 20 suitable for use in implementing embodiments of the present invention. The illustrated electronic device 20 is merely an example and should not be used to limit the functionality or scope of embodiments of the present invention.
As shown in fig. 5, the electronic device 20 is embodied in the form of a general purpose computing device. The components of the electronic device 20 may include, but are not limited to: one or more processors or processing units 201, a system memory 202, and a bus 203 that couples the various system components (including the system memory 202 and the processing unit 201).
The system memory 202 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)204 and/or cache memory 205. The electronic device 20 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, the storage system 206 may be used to read from and write to non-removable, nonvolatile magnetic media. A magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 203 by one or more data media interfaces. Memory 202 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 208 having a set (at least one) of program modules 207 may be stored, for example, in the memory 202, such program modules 207 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which may comprise an implementation of a network environment. Program modules 207 generally perform the functions and/or methodologies of embodiments of the present invention as described herein.
The electronic device 20 may also communicate with one or more external devices 209 (e.g., keyboard, pointing device, display 210, etc.), with one or more devices that enable a user to interact with the electronic device 20, and/or with any devices (e.g., network card, modem, etc.) that enable the electronic device 20 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 211. Also, the electronic device 20 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet) via the network adapter 212. As shown, the network adapter 212 communicates with other modules of the electronic device 20 over the bus 203. It should be understood that other hardware and/or software modules may be used in conjunction with electronic device 20, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The processing unit 201 executes various functional applications and data processing by running a program stored in the system memory 202.
The electronic equipment provided by the invention can realize the following method: collecting operation images of an operator in an electric power construction site in the operation process; determining an incidence relation between human key points of an operator in the operation image and each human key point based on a human posture recognition model established in advance; and determining the operation behaviors of the operators corresponding to the key points and the incidence relation of the human body, comparing the operation behaviors with preset violation behaviors, and determining whether the violation behaviors exist in the operators. The embodiment of the invention realizes the violation determination through the human body posture recognition model, does not need a safety supervisor to check the operation image, reduces the resource waste, and improves the accuracy and the effectiveness of the violation determination.
Example six
An embodiment of the present invention provides a storage medium containing computer-executable instructions, which when executed by a computer processor, perform a method for determining a behavior of a worker, the method comprising:
collecting operation images of an operator in an electric power construction site in the operation process; determining an incidence relation between human key points of an operator in the operation image and each human key point based on a human posture recognition model established in advance; and determining the operation behaviors of the operators corresponding to the key points and the incidence relation of the human body, comparing the operation behaviors with preset violation behaviors, and determining whether the violation behaviors exist in the operators. The embodiment of the invention realizes the violation determination through the human body posture recognition model, does not need a safety supervisor to check the operation image, reduces the resource waste, and improves the accuracy and the effectiveness of the violation determination.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the operations of the method described above, and may also execute the relevant operations in the method for determining the behavior of the operator provided by any embodiment of the present invention.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing description is only exemplary of the invention and that the principles of the technology may be employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (15)
1. A method of determining an operator's behavior, comprising:
collecting operation images of an operator in an electric power construction site in the operation process;
determining an incidence relation between human key points of the operating personnel and each human key point in the operating image based on a pre-established human posture recognition model;
and determining the operation behavior of the operator corresponding to the human body key point and the incidence relation, comparing the operation behavior with a preset violation behavior, and determining whether the operator has the violation behavior.
2. The method according to claim 1, further comprising, prior to said determining an association between human keypoints and each of said human keypoints for the worker in the work image:
acquiring a sample image of at least one operator in the operation process to form a sample data set;
marking the operators in each sample image of the sample data set in advance;
training a convolutional neural network based on each marked sample image, and establishing the human body posture recognition model based on the training result.
3. The method of claim 2, wherein said composing the sample data set comprises:
carrying out transformation processing on at least one acquired sample image;
forming the sample data set based on each transformed image obtained after transformation processing and the sample image;
wherein the transformation process includes at least one of a scaling process, an adjustment contrast process, a color dithering process, and an increase noise process.
4. The method of claim 2, wherein training a convolutional neural network based on each labeled sample image, and establishing the human body posture recognition model based on the training result comprises:
inputting the sample image into the convolutional neural network, and outputting the recognition result of the convolutional neural network on the posture of the operator in the sample image;
comparing the identification result with the mark in the sample image, adjusting the network parameter of the convolutional neural network based on the comparison result, and updating the convolutional neural network;
and repeatedly inputting the sample image into a convolutional neural network, comparing the obtained recognition result with the mark, and adjusting the grid parameters of the neural network until the current training process meets the preset condition, ending the training process of the convolutional neural network, and determining the current convolutional neural network as the human body posture recognition model.
5. The method of claim 4, wherein the predetermined condition comprises a number of iterations reaching a predetermined threshold number of iterations.
6. The method of claim 2, wherein said obtaining a sample image of a work process of at least one worker comprises:
acquiring a historical monitoring video, and performing frame extraction operation on the historical monitoring video according to a preset time interval;
and determining each frame image obtained by the frame extraction operation as the sample image.
7. The method according to claim 1, wherein the determining an association relationship between the human key points of the operator and each human key point in the operation image based on a human gesture recognition model established in advance comprises:
inputting the operation images into the human body posture recognition model, and generating a first number of confidence level images and a second number of affinity field images corresponding to the operation images;
determining human key points of the operator in the operation image based on the first number of the confidence level images;
and determining the incidence relation among the key points of the human body based on the second number of the affinity field images.
8. The method of claim 1, further comprising, after said collecting job images of a worker's work process in an electrical power job site:
and performing convolution calculation, linear rectification function calculation and pooling operation on the operation graph, and updating the operation graph into a graph obtained after operation.
9. The method according to claim 1, wherein after comparing the job behavior with a preset violation behavior and determining whether the job personnel has a violation behavior, the method further comprises:
and when the fact that the operator has the illegal behavior is determined, generating warning information containing the illegal behavior, and sending the warning information to a safety personnel terminal for prompting.
10. The method according to claim 1, wherein the determining the working behavior of the operator corresponding to the human body key point and the association comprises:
and determining the operation background information of the corresponding electric power construction site in the operation image, and determining the operation behavior of the operator based on the operation background information, the human key points and the association relation.
11. The method of claim 10, wherein the determining the work activity of the worker based on the work context information, the human keypoints, and the incidence relation comprises:
determining whether the job context information comprises job related information; the job related information comprises equipment information and/or job identification information;
and if so, determining the operation behavior of the operator based on the operation associated information, the key point and the association relation.
12. The method of claim 11, further comprising:
and if not, determining the operation behavior of the operator based on the human body key point and the incidence relation.
13. An operator behavior determination device, comprising:
the acquisition operation image module is used for acquiring operation images of operation processes of operators in the electric power construction site;
the human body key point determining module is used for determining the association relationship between the human body key points of the operating personnel and the human body key points in the operating image based on a human body posture recognition model established in advance;
and the operation behavior determining module is used for determining the operation behaviors of the operators corresponding to the human body key points and the incidence relation, comparing the operation behaviors with preset violation behaviors and determining whether the violation behaviors exist in the operators.
14. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of determining the behavior of a worker as claimed in any one of claims 1 to 12.
15. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a method for determining a behavior of a worker as claimed in any one of claims 1 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210555206.5A CN114821806A (en) | 2022-05-19 | 2022-05-19 | Method and device for determining behavior of operator, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210555206.5A CN114821806A (en) | 2022-05-19 | 2022-05-19 | Method and device for determining behavior of operator, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114821806A true CN114821806A (en) | 2022-07-29 |
Family
ID=82518122
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210555206.5A Pending CN114821806A (en) | 2022-05-19 | 2022-05-19 | Method and device for determining behavior of operator, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114821806A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115983824A (en) * | 2022-12-13 | 2023-04-18 | 苏州德拓联合智能系统科技有限公司 | Public equipment operation management method and system based on health diagnosis |
-
2022
- 2022-05-19 CN CN202210555206.5A patent/CN114821806A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115983824A (en) * | 2022-12-13 | 2023-04-18 | 苏州德拓联合智能系统科技有限公司 | Public equipment operation management method and system based on health diagnosis |
| CN115983824B (en) * | 2022-12-13 | 2024-03-15 | 苏州德拓联合智能系统科技有限公司 | Public equipment operation management method and system based on health diagnosis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109858367B (en) | Visual automatic detection method and system for worker through supporting unsafe behaviors | |
| CN112347916B (en) | Video image analysis-based power field operation safety monitoring method and device | |
| CN114155614B (en) | Method and system for identifying anti-violation behavior of operation site | |
| CN112769908A (en) | Cooperative control method and device and electronic equipment | |
| CN106251240A (en) | Power transmission network method for early warning based on big data | |
| CN114665608B (en) | Intelligent sensing inspection system and method for transformer substation | |
| CN112949457A (en) | Maintenance method, device and system based on augmented reality technology | |
| CN115376130A (en) | Method, device, equipment and medium for generating electric power construction site identification model | |
| US20200394600A1 (en) | Fire protection inspection method, fire protection inspection device, and remote device | |
| CN115393566A (en) | Fault identification and early warning method and device for power equipment, storage medium and equipment | |
| CN113343998A (en) | Reading monitoring system and method for electric power mechanical meter, computer equipment and application | |
| CN114821806A (en) | Method and device for determining behavior of operator, electronic equipment and storage medium | |
| CN113743725B (en) | Ascending qualification detection method and system for ascending operators based on edge calculation | |
| CN114565312A (en) | Power transmission and transformation project refined construction management method based on BIM | |
| CN114358464A (en) | Operation risk assessment method and device, electronic equipment and storage medium | |
| CN114120086A (en) | Pavement disease recognition method, image processing model training method, device and electronic equipment | |
| CN114881665A (en) | Method and system for identifying electricity stealing suspected user based on target identification algorithm | |
| CN106355187A (en) | Application of visual information to electrical equipment monitoring | |
| CN112749735A (en) | Converter tapping steel flow identification method, system, medium and terminal based on deep learning | |
| CN115860979A (en) | Artificial intelligence management system for field operation of power grid | |
| CN115471911A (en) | Power grid operation field violation action identification method and system | |
| CN115909212A (en) | Real-time early warning method for typical violation behaviors of power operation | |
| CN115641607A (en) | Method, device, equipment and storage medium for detecting wearing behavior of power construction site operator | |
| CN114758296A (en) | Power grid equipment remote monitoring method and system based on VR technology | |
| CN115249281A (en) | Image occlusion and model training method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |