CN111079722B

CN111079722B - Hoisting process personnel safety monitoring method and system

Info

Publication number: CN111079722B
Application number: CN202010205214.8A
Authority: CN
Inventors: 董梅; 张亮; 胡辉; 宋杰
Original assignee: Hangzhou Ruhr Technology Co Ltd
Current assignee: Hangzhou Ruhr Technology Co Ltd
Priority date: 2020-03-23
Filing date: 2020-03-23
Publication date: 2023-01-03
Anticipated expiration: 2040-03-23
Also published as: CN111079722A

Abstract

The invention discloses a method and a system for monitoring personnel safety in a hoisting process, wherein the method comprises the following steps: constructing a hoisting field model marked with obstacles; training to generate workers and hoisting an object detection network; continuously detecting workers who correctly wear the safety helmet, workers who incorrectly wear the safety helmet and hoisted objects; when a worker is detected but the hoisting field does not allow people to enter, and/or a worker who does not correctly wear the safety helmet is detected, an alarm is triggered; otherwise, judging whether relative motion exists between the worker and the hoisting object; if yes, predicting the path of a worker by combining a hoisting site model and utilizing a sliding window method; predicting a path of a hoisted object by utilizing an ant colony algorithm based on the hoisting site model; and judging whether the distance between the worker and the hoisting object at a certain moment is smaller than a preset distance, and if so, triggering to alarm. The invention can comprehensively and timely identify the danger in the hoisting process and can effectively prevent the occurrence of hoisting accidents.

Description

Hoisting process personnel safety monitoring method and system

Technical Field

The invention relates to the technical field of image processing, in particular to a method and a system for monitoring personnel safety in a hoisting process.

Background

With the increasing population of towns, the demand on high-rise and super high-rise buildings is increasing, and in order to improve the mechanization and industrialization of construction, the tower crane is widely applied to the construction process of the high-rise and super high-rise buildings. Safety accidents occurring in the hoisting process still emerge endlessly, and the existing hoisting process monitoring system mainly considers the safety state of a crane and the collision condition with surrounding structures. However, the most serious lifting accidents are often related to people, the consequences of the accidents are serious, and the proportion of serious injuries and death is high.

Existing hoist process personnel safety monitoring primarily uses digital cameras to determine the location of construction workers, or statically placed and dynamically moving cameras to track workers on a construction site. In addition, the prior art proposes a tracking method based on machine learning for tracking workers on a construction site. However, the method only tracks workers, cannot identify whether the hoisting process is safe, and cannot give an early warning to the workers.

The invention patent application with publication number CN 109019335A discloses a hoisting safety distance detection method for deep learning of a base hand, which comprises the following steps: acquiring an image around a hoisting object in a tower crane structure through a camera; marking workers and hoisting objects in the obtained image to manufacture a data set; training the data set by using a faster R-CNN in deep learning; identifying and positioning workers and hoisted objects in the images by using the trained detection model; and calculating the pixel distance between the worker and the hoisted object in the image according to the positioning information in the detection result, and converting the pixel distance between the worker and the hoisted object into the actual distance between the worker and the vertical projection point of the hoisted object according to the height between the hoisted object and the camera, the real length of the hoisted object and the pixel length, thereby realizing the monitoring of the hoisting safety distance.

Although the application can improve the safety of hoist and mount in-process through the distance detection between workman and the hoist and mount object, but the dangerous identification mode to hoist and mount process is single, just carries out safety precaution when detecting that the distance between workman and hoist and mount object is less than a definite value, can not effectively prevent the emergence of hoist and mount accident. Therefore, how to realize comprehensive and timely danger identification in the hoisting process and effectively prevent hoisting accidents from happening is a problem to be solved urgently in the field.

Disclosure of Invention

The invention aims to provide a lifting process personnel safety monitoring and system aiming at the defects of the prior art. The invention can comprehensively and timely identify the danger in the hoisting process and can effectively prevent the occurrence of hoisting accidents.

In order to achieve the purpose, the invention adopts the following technical scheme:

a hoisting process personnel safety monitoring method comprises the following steps:

s1, collecting images of a hoisting field, identifying obstacles in the hoisting field, and constructing a hoisting field model marked with the obstacles;

s2, training to generate a Faster R-CNN worker and a hoisted object detection network;

s3, continuously detecting workers who correctly wear the safety helmet, workers who do not correctly wear the safety helmet and hoisted objects by using the frame images in the video shot in the hoisting field by using the detection network until the workers who correctly wear the safety helmet and/or the workers who do not correctly wear the safety helmet are detected;

s4, when a worker is detected but the hoisting site does not allow people to enter, and/or a worker who does not correctly wear the safety helmet is detected, triggering an alarm; otherwise, executing step S5;

s5, judging whether relative motion exists between the worker and the hoisting object or not based on the distance difference between the worker and the hoisting object in the front frame image and the rear frame image; if yes, executing step S6;

s6, training and generating an LSTM worker path prediction model based on the detected historical path of the worker, and predicting the worker path by combining the lifting site model marked with the barrier and utilizing a sliding window method;

s7, predicting a path of the hoisting object by using an ant colony algorithm based on the detected historical path of the hoisting object and the hoisting field model marked with the obstacle;

s8, judging whether the distance between the worker and the hoisting object at a certain moment is smaller than a preset distance in the predicted path of the worker and the hoisting object, and if yes, triggering an alarm.

Further, the step S1 includes:

dividing each collected hoisting field image into two-dimensional image grids in a transverse and longitudinal equal division mode, and integrating a plurality of two-dimensional image grids to generate a three-dimensional grid model; and identifying the obstacles through a convolutional neural network, determining the positions of the obstacles in the three-dimensional grid model, identifying the obstacles in the three-dimensional grid model, and generating a hoisting field model with the obstacles.

Further, the step S5 includes:

judgment of

If the relative movement does not exist, the worker and the hoisting object do not move relatively; wherein,

、

respectively the position coordinates of a worker in the front frame image and the back frame image,

、

respectively are the position coordinates of the hoisted object in the front and back two frames of images,

is a preset distance threshold.

Further, the step S6 includes:

grouping historical path data of workers, wherein each group comprises continuous M +1 position data, taking the former M position data as the input of an LSTM network, predicting the M +1 th data, calculating a loss function of an LSTM worker path prediction model, iterating, optimizing and updating the LSTM worker path prediction model, and training to generate the LSTM worker path prediction model; and predicting the position at the (N + 1) th moment by adopting the position data from the (N-M) th to the (N), judging whether the predicted position belongs to an obstacle position set, if so, excluding the position for predicting the position again, replacing the position data at the (N-M) th moment after predicting the position of the worker at the (N + 1) th moment, and predicting the position at the (N + 2) th moment by adopting the data from the (N-M + 1) th to the (N + 1) th moments.

Further, the step S7 includes:

determining the positions of all barriers in a hoisting field, the starting point of the hoisting object and the position of the target point, and initializing parameters, wherein the parameters comprise maximum iteration times, information elicitation factors, expected elicitation factors and the number of ants, and the elicitation factors at the barriers are 0; generating ants with preset number at the starting position of the hoisting object at equal time intervals, selecting the next position node by each ant according to the state transition probability, and adding the nodes which pass by the ants into a taboo table; updating path pheromones according to the position transfer process of ants, and updating state transfer probability; and continuously executing the operation to select the next position node until the ant reaches the target position, updating the pheromone concentration and the iteration times, and outputting an optimal path when the iteration times reach the preset maximum iteration times.

The invention also provides a system for monitoring the personnel safety in the hoisting process, which comprises:

the hoisting field model building module is used for acquiring a hoisting field image, identifying obstacles in a hoisting field and building a hoisting field model marked with the obstacles;

the detection network generation module is used for training and generating a Faster R-CNN worker and hoisting an object detection network;

the detection module is used for continuously detecting workers who correctly wear the safety helmet, workers who incorrectly wear the safety helmet and hoisted objects by using the frame images in the video shot in the hoisting field by using the detection network until the workers who correctly wear the safety helmet and/or the workers who incorrectly wear the safety helmet are detected;

the first alarm module is used for triggering an alarm when a worker is detected but the hoisting site does not allow people to enter and/or a worker who does not correctly wear the safety helmet is detected; otherwise, calling a relative motion detection module;

the relative motion detection module is used for judging whether relative motion exists between the worker and the hoisting object or not based on the distance difference between the worker and the hoisting object in the front frame image and the rear frame image; if yes, calling a first path prediction module;

the first path prediction module is used for training and generating an LSTM worker path prediction model based on the detected historical path of the worker, and predicting the worker path by combining the hoisting site model marked with the obstacle and utilizing a sliding window method;

the second path prediction module is used for predicting a path of the hoisting object by utilizing an ant colony algorithm based on the detected historical path of the hoisting object and the hoisting field model marked with the obstacle;

and the second alarm module is used for judging whether the distance between the worker and the hoisting object at a certain moment is smaller than the preset distance in the predicted path of the worker and the hoisting object, and if so, triggering alarm.

Further, the hoisting site model building module comprises:

Further, the relative motion detection module includes:

judgment of

If the vertical movement is not established, the worker and the hoisting object move relatively, otherwise, the worker does not move relatively; wherein,

、

、

is a preset distance threshold.

Further, the first path prediction module comprises:

grouping historical path data of workers, wherein each group comprises continuous M +1 position data, taking the former M position data as the input of an LSTM network, predicting the M +1 th data, calculating a loss function of an LSTM worker path prediction model, iterating, optimizing and updating the LSTM worker path prediction model, and training to generate the LSTM worker path prediction model; and predicting the position at the (N + 1) th moment by adopting the position data from the (N-M) th to the (N) th moments, judging whether the predicted position belongs to an obstacle position set, if so, excluding the position for predicting the position again, replacing the position data at the (N-M) th moment by the position data at the (N-M + 1) th moment after predicting the position of the worker at the (N + 1) th moment, and predicting the position at the (N + 2) th moment by adopting the data from the (N-M + 1) th to the (N + 1) th moments.

Further, the second path prediction module comprises:

determining the positions of all barriers in a hoisting field, the starting point of the hoisting object and the position of the target point, and initializing parameters, wherein the parameters comprise maximum iteration times, information elicitation factors, expected elicitation factors and the number of ants, and the elicitation factors at the barriers are 0; generating ants with preset number at the starting position of the hoisted object at equal time intervals, selecting nodes at the next position by each ant according to the state transition probability, and adding the nodes which have already walked into a taboo table; updating path pheromone according to the position transfer process of ants and updating state transfer probability; and continuously executing the operation to select the next position node until the ant reaches the target position, updating the pheromone concentration and the iteration times, and outputting an optimal path when the iteration times reach the preset maximum iteration times.

Compared with the prior art, the invention has the following advantages:

(1) According to the invention, paths of workers and hoisting objects are predicted, personnel safety accidents possibly occurring in the hoisting process are found in advance, safety early warning is timely carried out on personnel in the hoisting field, and the occurrence of hoisting accidents is effectively prevented;

(2) The invention identifies unsafe behaviors such as unauthorized entering a hoisting field where workers are not allowed to enter, incorrect wearing of safety helmets and the like in the hoisting process, and realizes the all-round monitoring of the safety of the workers;

(3) The method comprises the steps of carrying out image acquisition on a hoisting field, and identifying obstacles in the hoisting field, so that the predicted worker path and the path of a hoisting object are closer to the actual path, and the accuracy of path prediction is improved;

(4) According to the invention, only the path prediction is carried out on the worker moving relative to the hoisting object, so that the safety monitoring of the worker is ensured, the data processing amount is reduced, and the efficiency of the safety monitoring of the worker is improved;

(5) The LSTM worker path prediction model based on the sliding window predicts the worker path, realizes the position prediction of workers in a long time period, and ensures the accuracy of the position prediction in the long time period.

Drawings

Fig. 1 is a flowchart of a method for monitoring personnel safety in a hoisting process according to an embodiment;

fig. 2 is a structural diagram of a personnel safety monitoring system in a hoisting process according to the second embodiment.

Detailed Description

The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It should be noted that the features in the following embodiments and examples may be combined with each other without conflict.

It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the components related to the present invention are only shown in the drawings rather than drawn according to the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.

The invention is further described with reference to the following drawings and specific examples, which are not intended to be limiting.

Example one

As shown in fig. 1, this embodiment provides a method for monitoring personnel safety in a hoisting process, including:

according to the invention, the personnel accidents in the hoisting process are prevented by predicting the paths of workers and hoisted objects in the hoisting field. In general, the environment in a hoisting site is complicated, and the paths of workers and hoisted objects are affected by obstacles in the hoisting site. Therefore, the method and the device collect the images of the hoisting field, identify the obstacles in the hoisting field and construct the hoisting field model marked with the obstacles.

Specifically, the multiple cameras are arranged at different positions of a hoisting field, and the three-dimensional grid model is constructed by combining images acquired by the multiple cameras. And aiming at each collected hoisting field image, dividing the image into two-dimensional image grids in a transverse and longitudinal equal division mode, and combining a plurality of two-dimensional image grids to integrate and generate a three-dimensional grid model. The obstacle may be identified by a Convolutional Neural Network (CNN) or the like, which is not limited herein. And determining the position of the recognized barrier in the three-dimensional grid model based on the position of the recognized barrier in the image, identifying the barrier in the three-dimensional grid model, and generating a hoisting field model with the barrier. Because the obstacles on the hoisting site are relatively fixed, the invention can collect images of the hoisting site when hoisting construction is not carried out, thereby improving the data processing efficiency.

S2, training to generate a Faster R-CNN worker and hoisting an object detection network;

the safety accidents of workers in the hoisting process are mainly caused by collision between workers and a hoisting object, and the like, so that the worker safety prediction method and the worker safety prediction device can identify and detect the workers and the hoisting object so as to predict the safety of the workers. Because the hoisting object may have a larger size, the invention replaces the detection of the hoisting object with the detection of the hoisting object, so as to improve the accuracy rate of the safety monitoring of the personnel.

In order to improve the efficiency of danger identification in the hoisting process, the invention utilizes Faster R-CNN to detect workers and hoisted objects. The Faster RCNN network is one of the popular universal multi-target detection frameworks at present. The existing fast RCNN algorithm is mainly divided into three parts: the first part is a CNN basic network used for completing the extraction of image features; the second part is an RPN (Region pro-polar Networks) network, which directly generates regional targets mainly using convolutional neural Networks, using a method that is essentially a sliding window. Using fixed-size window sliding on the last layer of feature map of the CNN basic network, outputting a feature vector with fixed-size dimension by each window (the dimension is related to the CNN basic network, for example, VGG16 is 512 dimensions), and judging whether the candidate frames generated by each window are targets and performing coordinate regression; the last part is a discrimination network, namely classification and frame regression, which is used for classifying and coordinate regression correction of the target region extracted by the RPN network.

Specifically, the fast R-CNN network structure is first constructed, preferably the CNN base network selects VGG16, specifically comprising 13 convolutional layers, 4 pooling layers, with 2, 4, 7, 10 convolutional layers followed by maximum pooling. The RPN is a full convolution network that takes as input the convolution signature of the underlying network output. Specifically, a 512-channel, 3 × 3 core convolutional layer is employed, followed by two parallel 1 × 1 core convolutional layers. After the fast R-CNN network structure is constructed, training is carried out based on a training data set. A certain hoisting place is selected for video shooting, a camera is respectively and fixedly placed in the east-west direction and the south-north direction, and horizontal shooting is carried out. The video capture frequency was 30FPS with a resolution of 1920 x 1280 pixels. Note that the shot content includes people of various shapes (front, side, back, crouch, stoop, standing, walking, etc.) as much as possible. Selecting 600 frames of images in a certain video to mark: "r" represents a worker who correctly wears the crash helmet; "b" represents a worker who did not correctly wear the headgear; "w" represents a hoisted object. And taking the marked data set as a training data set, training the established Faster R-CNN network, and generating a fast R-CNN worker and object hoisting detection network. According to the invention, through training of the hoisted object, each angle of the worker and each shape image, the accuracy of the worker and the hoisted object detection network can be improved.

and for the scene needing hoisting process danger identification, video shooting is continuously carried out on the hoisting site through the camera. Specifically, one camera can be fixedly arranged in the east-west direction and the south-north direction respectively, and horizontal shooting is carried out. The shot video comprises a plurality of frames of images, and the invention processes each image frame by frame, for example, the acquisition of the image is carried out at 1 frame per second. The method comprises the steps of selecting images which can simultaneously comprise workers and hoisting objects for a plurality of images in the same scene which are obtained simultaneously, and selecting one image randomly when the plurality of images comprise the hoisting objects and the workers, so that the target detection efficiency is improved. Because the Faster R-CNN worker and the hoisted object detection network generated by training of the invention train the images of all angles, even if one image is selected at will, the accurate detection of the worker and the hoisted object can be realized.

The invention inputs the image collected by the camera into the Faster R-CNN worker and the hoisted object detection network, and identifies and positions the worker and the hoisted object in the image by using the Faster R-CNN worker and the hoisted object detection network. Displaying and outputting a worker who correctly wears the safety helmet, a worker who incorrectly wears the safety helmet and a hoisted object of the current frame image, and marking the workers with 'r', 'b' and 'w' respectively.

The invention monitors the safety of personnel in the hoisting process, so that the invention continuously collects video frame images and does not predict the corresponding safety state until workers appear in the images. When no worker is detected in the image, no safety accident of the worker is naturally generated, and the prediction of the safety of the worker is not carried out.

in the hoisting process, the behavior of endangering the safety of workers comprises the following steps: the safety helmet is not worn when the safety helmet enters a site where workers are not allowed to enter without permission, enters a hoisting site, and the like. Therefore, the invention triggers the alarm after detecting the actions of workers entering a hoisting place where people are not allowed to enter, wearing no safety helmet and the like.

When the alarm is triggered, alarm information can be sent to a manager, the alarm information can be displayed in a short message mode, a telephone mode, a management interface mode and the like, and the manager can process the alarm information in time when receiving the alarm information. In addition, in order to enhance the real-time performance of alarm processing, the invention can install sound amplifying equipment on the hoisting site, and after danger is identified, the sound amplifying equipment is triggered to carry out real-time early warning on personnel in the hoisting site.

during the hoisting process, hoisting accidents usually occur between hoisted objects and workers outside the crane. Workers in the crane are responsible for operating the crane, and safety accidents such as smashing, collision and the like can not happen usually. Therefore, in order to improve the monitoring performance of personnel safety, workers who cannot have hoisting safety accidents are eliminated, and the monitoring efficiency of the personnel safety in the hoisting process is improved. The invention detects whether the worker and the hoisting object move relatively or not so as to eliminate the detected workers such as crane operators. When no relative movement exists between a worker and a hoisting object, the relative position is basically fixed, safety accidents cannot occur, and path prediction is not carried out.

Therefore, after the worker is detected for the first time, the worker is tracked and positioned, and when the distance between the hoisting object and a certain worker is basically unchanged along with time, the fact that relative movement does not exist between the worker and the hoisting object is indicated. Therefore, the distance between the worker and the hoisting object in the two front and back frames of images is compared, and the distance between the worker and the hoisting object is the distance between the center points of the worker and the hoisting object. The invention constructs the hoisting field model marked with the obstacles, so that the positions of workers and hoisting objects are the positions of the workers and the hoisting objects in the three-dimensional grid in the hoisting field model.

Specifically, the present invention judges:

wherein,

、

respectively the position coordinates of the worker in the front and back two frames of images,

、

the preset distance threshold value is set because even a worker in the crane may cause the displacement of the center point due to the worker's operation in the crane, resulting in the distance values of the worker and the hoisting object from the front frame to the rear frame not being exactly equal, and thus the distance values are not exactly equal

The value of (d) is small and can be set as desired. If so, the worker and the hoisting object have relative movement, otherwise, the worker and the hoisting object do not have relative movement.

In addition, when the sampling period of the video frame is short or the moving speed of the worker and the hoisted object is small, the distances between two continuous frames of images may be substantially equal, and therefore, the two frames before and after in the invention are not necessarily two continuous frames, and may be two frames of images with a long time interval. The distance between the worker and the hoisting object can be the pixel distance of the worker and the hoisting object in the image frame, and can also be a spatial distance converted according to the corresponding relation between the image and the spatial position of the hoisting field, and the method is not limited herein.

S6, training and generating an LSTM worker path prediction model based on the detected historical path of the worker, and predicting the worker path by combining the lifting site model marked with the obstacle and using a sliding window method;

in order to effectively prevent the occurrence of lifting accidents, the invention predicts the paths of workers and lifting objects. Specifically, the worker and the hoisting object are tracked and positioned after the worker and the hoisting object are detected for the first time. The central point of the detection target is extracted to be used as the position of the detection target, and the future paths of workers and the hoisting object are predicted on the basis of detecting the positions of a plurality of workers and the hoisting object, so that the occurrence of safety accidents such as collision between the workers and the hoisting object is effectively prevented.

For the detected positions of the worker at a plurality of times, the positions are serially connected into a position coordinate sequence based on a time sequence, and a worker historical path is formed. The essence of the prediction of the worker's path is to predict the worker's location at a future time. The Long Short Term Memory (LSTM) is a specific form of RNN, and shows strong adaptability in time series data analysis, and can better describe the development rule in a time series system and predict the development trend of the time series system. Therefore, the invention utilizes the LSTM neural network to predict the worker path, and concretely, the invention provides sliding window-based LSTM worker path prediction, utilizes the detected historical worker path to train and generate an LSTM worker path prediction model, and utilizes a sliding window method to predict the worker path. As described above, the path of the worker is limited by the obstacles in the lifting site, and thus, the present invention performs the prediction of the path of the worker in conjunction with the lifting site model in which the obstacles are identified.

The LSTM is a typical recurrent neural network, and takes the output of the previous time period as the input of the next time period, and the LSTM mainly comprises an input gate, an output gate, a forgetting gate, an input node and the like. The invention firstly constructs the LSTM network, inputs the historical path of the worker into the LSTM network and predicts the position of the worker at the next moment. The LSTM network is trained by utilizing a large amount of existing worker path data, the existing path data are grouped, each group comprises continuous M +1 position data, the former M position data are used as the input of the LSTM network, the M +1 th data are predicted, the loss function of the LSTM worker path prediction model is calculated to iterate, optimize and update the LSTM worker path prediction model, and the worker path prediction model is trained to obtain the finally used LSTM worker path prediction model.

In order to realize the prediction of the position of a worker for a longer time period, the invention predicts the path of the worker by a sliding window method and predicts the path of the worker by a sliding window method. Assuming that N position sequences included in the worker historical path are available, a sliding window with the length of M is selected. The length of the input training sequence is limited by the length of the sliding window, so that the overlong learning time in the iterative process is avoided. For the medium-long term prediction, the samples in the sliding window can be updated by an iterative method. For example, with each iteration, the predicted location prediction value of the ith (i =1,2,3.., L.) worker replaces the oldest data in the sliding window, and each replacement causes the LSTM network to perform a new learning, update the network structure, and perform the next prediction with the new network structure. For example, the position of the time N +1 is predicted by using the position data of the time N-M to N, and the position of the worker at the time N +1 is predicted and then replaced by the position data of the time N-M, that is, the position of the time N +2 is predicted by using the data of the time N-M +1 to N + 1.

In the position prediction process, the path of the worker is affected by the obstacle, and therefore, the position in the worker path prediction should exclude the position of the obstacle. Therefore, the invention constructs an obstacle position set based on the position of the obstacle, judges whether the predicted position belongs to the obstacle position set, and if so, excludes the position to predict the position again.

S7, predicting a path of the hoisted object by utilizing an ant colony algorithm based on the detected historical path of the hoisted object and the hoisting field model marked with the obstacle;

the safety of personnel in the hoisting process is related to the path of a hoisting object, the randomness of the safety is different from the randomness of the target position of workers, and a hoisting machine usually has a definite target position in the one-time hoisting process, so the method carries out prediction on the path of the hoisting object based on the initial position and the target position of the hoisting object. When a hoisting task is started every time, namely after the hoisting object is detected to move in an image frame for the first time, the target position of the hoisting task is obtained, and path planning of the hoisting object is carried out by combining the position of an obstacle.

Before predicting the path of the hoisted object, the positions of all obstacles in a hoisting field, the starting point of the hoisted object and the position of the target point are determined. And initializing parameters, wherein the parameters comprise maximum iteration times, information elicitation factors, expected elicitation factors, the number of ants and the like, and the elicitation factor at the position of the barrier is 0. And generating ants with preset number at the starting position of the hoisting object at equal time intervals, selecting the next position node by each ant according to the state transition probability, and adding the nodes which pass by the ant into a taboo table. The path pheromone is updated according to the position transition process of the ants, so that the state transition probability is updated. And continuously executing the operation to select the next position node until the ant reaches the target position, updating the pheromone concentration and the iteration times, and outputting an optimal path when the iteration times reach the preset maximum iteration times.

In the hoisting process, when the distance between a worker and a hoisting object is smaller than a certain threshold value, the risk of safety accidents exists. Therefore, the distance between the worker and the hoisting object is predicted based on the predicted worker path and the predicted hoisting object path, and when the distance between the worker and the hoisting object at a certain moment is smaller than the preset distance, an alarm is triggered. Predicted path for worker

Predicted path of hoisted object

Wherein

for the position coordinates of the worker at the i-th time,

the position coordinates of the hoisting object at the ith moment are shown as p, the operation time of a worker on a hoisting field is shown as q, the operation time of the hoisting object is shown as q, and the values of p and q are not limited. Therefore, the distance value between the worker and the hoisted object at the time i is as follows:

specifically, the invention adopts Euclidean distance to calculate the distance between workers and the hoisted object. When present, is

And triggering an alarm.

Example two

As shown in fig. 2, this embodiment provides a safety monitoring system for personnel in a hoisting process, including:

Specifically, the multiple cameras are arranged at different positions of a hoisting field, and the three-dimensional grid model is constructed by combining images acquired by the multiple cameras. And aiming at each collected hoisting field image, dividing the image into two-dimensional image grids in a transverse and longitudinal equal division mode, and combining a plurality of two-dimensional image grids to integrally generate a three-dimensional grid model. The obstacle may be identified by a Convolutional Neural Network (CNN) or the like, which is not limited herein. And determining the position of the recognized barrier in the three-dimensional grid model based on the position of the recognized barrier in the image, identifying the barrier in the three-dimensional grid model, and generating a hoisting field model with the barrier. Because the obstacles on the hoisting site are relatively fixed, the invention can collect images of the hoisting site when hoisting construction is not carried out, thereby improving the data processing efficiency.

in the hoisting process, personnel safety accidents are mainly caused by collision between workers and hoisting objects, and the like, so that the workers and the hoisting objects are identified and detected to predict the safety of the workers. Because the hoisting object may have a larger size, the invention replaces the detection of the hoisting object with the detection of the hoisting object, thereby improving the accuracy of the safety monitoring of the personnel.

In order to improve the efficiency of danger identification in the hoisting process, the invention utilizes the Faster R-CNN to detect workers and hoisted objects. The fast RCNN network is one of the popular universal multi-target detection frameworks at present. The existing Faster RCNN algorithm is mainly divided into three parts: the first part is a CNN basic network used for completing the extraction of image features; the second part is an RPN (Region pro-polar Networks) network, which directly generates regional targets mainly using convolutional neural Networks, using a method that is essentially a sliding window. Using fixed-size window sliding on the last layer of feature map of the CNN basic network, outputting a feature vector with fixed-size dimension by each window (the dimension is related to the CNN basic network, for example, VGG16 is 512 dimensions), and judging whether the candidate frames generated by each window are targets and performing coordinate regression; the last part is a discrimination network, namely classification and frame regression, which is used for carrying out classification and coordinate regression correction on the target region extracted by the RPN.

Specifically, the Faster R-CNN network structure is first constructed, preferably the CNN base network selects the VGG16, specifically comprising 13 convolutional layers, 4 pooling layers, with the 2 nd, 4 th, 7 th, 10 th convolutional layers followed by maximum pooling. The RPN is a full convolution network that takes as input the convolution signature of the underlying network output. Specifically, a 512-channel, 3 × 3 core convolutional layer is employed, followed by two parallel 1 × 1 core convolutional layers. After the fast R-CNN network structure is constructed, training is carried out based on a training data set. A certain hoisting site is selected for video shooting, and a camera is respectively and fixedly placed in the east-west direction and the south-north direction for horizontal shooting. The video capture frequency was 30FPS with a resolution of 1920 x 1280 pixels. Note that the shot content includes people of various shapes (front, side, back, crouch, stoop, standing, walking, etc.) as much as possible. Selecting 600 frames of images in a certain video to be marked: "r" represents a worker who correctly wears the crash helmet; "b" represents a worker who did not correctly wear the headgear; "w" represents a hoisted object. And taking the marked data set as a training data set, training the established Faster R-CNN network, and generating a fast R-CNN worker and object hoisting detection network. According to the invention, through training of the hoisted object, each angle of the worker and each shape image, the accuracy of the worker and the hoisted object detection network can be improved.

The detection module is used for continuously detecting workers who correctly wear the safety helmet, workers who do not correctly wear the safety helmet and hoisted objects by using the frame images in the video shot in the hoisting field by using the detection network until the workers who correctly wear the safety helmet and/or the workers who do not correctly wear the safety helmet are detected;

and for the scene needing hoisting process danger identification, video shooting is continuously carried out on the hoisting site through the camera. Specifically, one camera can be fixedly arranged in the east-west direction and the south-north direction respectively, and horizontal shooting is carried out. The shot video comprises a plurality of frames of images, and the invention processes each image frame by frame, for example, the image is collected at 1 frame per second. The method comprises the steps of selecting images which can simultaneously comprise workers and hoisting objects for a plurality of images in the same scene which are obtained simultaneously, and selecting one image randomly when the plurality of images comprise the hoisting objects and the workers, so that the target detection efficiency is improved. Because the fast R-CNN worker and the hoisted object detection network generated by training of the invention train the images of all angles, even if one image is selected at will, the accurate detection of the worker and the hoisted object can be realized.

The invention inputs the image collected by the camera into the Faster R-CNN worker and the hoisted object detection network, and the Faster R-CNN worker and the hoisted object detection network are used for identifying and positioning the worker and the hoisted object in the image. And displaying and outputting a worker who correctly wears the safety helmet, a worker who incorrectly wears the safety helmet and a hoisted object of the current frame image, and marking the workers with 'r', 'b' and 'w' respectively.

The invention monitors the safety of personnel in the hoisting process, so that the invention continuously collects video frame images and does not predict the corresponding safety state until workers appear in the images. When no worker is detected in the image, no personnel safety accident can be generated naturally, and the personnel safety prediction is not carried out.

The first alarm module is used for triggering an alarm when a worker is detected but the hoisting site does not allow people to enter and/or a worker who does not correctly wear the safety helmet is detected, or calling the relative motion detection module;

in the hoisting process, the actions endangering the safety of workers comprise: the safety helmet is not worn when the safety helmet enters a site where workers are not allowed to enter without permission, enters a hoisting site, and the like. Therefore, the invention triggers the alarm after detecting the actions of workers entering a hoisting place where people are not allowed to enter, wearing no safety helmet and the like.

The relative motion detection module is used for judging whether relative motion exists between the worker and the hoisting object or not based on the distance difference value between the front frame image and the rear frame image of the worker and the hoisting object; if yes, calling a first path prediction module;

during the hoisting process, hoisting accidents usually occur between hoisted objects and workers outside the crane. Workers in the crane are responsible for operating the crane, and safety accidents such as smashing, collision and the like can not happen usually. Therefore, in order to improve the monitoring performance of personnel safety, workers who cannot have hoisting safety accidents are eliminated, and the monitoring efficiency of the personnel safety in the hoisting process is improved. The invention detects whether the worker and the hoisting object move relatively or not so as to eliminate the detected workers such as crane operators. When there is no relative movement between the worker and the hoisting object, the relative position is basically fixed, no safety accident occurs, and no path prediction is performed

Therefore, after the worker is detected for the first time, the worker is tracked and positioned, and when the distance between the hoisting object and a certain worker is basically unchanged along with time, the fact that relative movement does not exist between the worker and the hoisting object is indicated. Therefore, the distance between the worker and the hoisting object in the front frame image and the distance between the worker and the hoisting object in the rear frame image are compared, and the distance between the worker and the hoisting object is the distance between the center points of the worker and the hoisting object. The invention constructs the hoisting field model marked with the obstacles, so that the positions of workers and hoisting objects are the positions of the workers and the hoisting objects in the three-dimensional grid in the hoisting field model.

Specifically, the present invention judges:

wherein,

、

、

In addition, when the sampling period of the video frame is short or the moving speed of the worker and the hoisted object is small, the distances between two continuous frames of images may be substantially equal, so that the two frames before and after in the invention are not necessarily two continuous frames, and may be two images with a long time interval. The distance between the worker and the hoisting object may be a pixel distance between the worker and the hoisting object in the image frame, or may be a spatial distance converted according to a corresponding relationship between the image and the spatial position of the hoisting site, which is not limited herein.

The first path prediction module is used for training and generating an LSTM worker path prediction model based on the detected historical path of the worker, and predicting the worker path by combining the hoisting site model marked with the obstacle and using a sliding window method;

in order to effectively prevent the occurrence of lifting accidents, the method and the system predict the paths of workers and lifting objects. Specifically, the worker and the hoisting object are tracked and positioned after the worker and the hoisting object are detected for the first time. The central point of the detection target is extracted to be used as the position of the detection target, and the future paths of workers and the hoisting object are predicted on the basis of detecting the positions of a plurality of workers and the hoisting object, so that safety accidents such as collision between the workers and the hoisting object are effectively prevented.

For the detected positions of the worker at a plurality of times, the positions are serially connected into a position coordinate sequence based on a time sequence, and a worker historical path is formed. The essence of the prediction of the worker's path is to predict the location of the worker at a future time. The Long Short Term Memory (LSTM) is a specific form of RNN, and shows strong adaptability in time series data analysis, and can better describe the development rule in a time series system and predict the development trend of the time series system. Therefore, the invention utilizes the LSTM neural network to predict the worker path, and concretely, the invention provides sliding window-based LSTM worker path prediction, utilizes the detected historical worker path to train and generate an LSTM worker path prediction model, and utilizes a sliding window method to predict the worker path. As described above, the path of the worker is limited by the obstacles in the hoisting site, and therefore, the present invention performs the prediction of the path of the worker in conjunction with the hoisting site model in which the obstacles are identified.

The LSTM is a typical recurrent neural network, and takes the output of the previous time period as the input of the next time period, and the LSTM mainly comprises an input gate, an output gate, a forgetting gate, an input node and the like. The invention firstly constructs the LSTM network, inputs the historical path of the worker into the LSTM network, and predicts the position of the worker at the next moment. The LSTM network is trained by utilizing a large amount of existing worker path data, the existing path data are grouped, each group comprises continuous M +1 position data, the former M position data are used as the input of the LSTM network, the M +1 th data are predicted, the loss function of the LSTM worker path prediction model is calculated to iterate, optimize and update the LSTM worker path prediction model, and the worker path prediction model is trained to obtain the finally used LSTM worker path prediction model.

In order to realize the prediction of the position of a worker for a longer time period, the invention predicts the path of the worker by a sliding window method and predicts the path of the worker by a sliding window method. Assuming that N position sequences included in the worker historical path are available, a sliding window with the length of M is selected. The length of the input training sequence is limited by the length of the sliding window, so that the overlong learning time in the iterative process is avoided. For the medium-long term prediction, the samples in the sliding window can be updated by an iterative method. For example, with each iteration, the predicted location prediction value of the ith (i =1,2,3.., L.) worker replaces the oldest data in the sliding window, and each replacement causes the LSTM network to perform a new learning, update the network structure, and perform the next prediction with the new network structure. For example, the position of the N +1 th time is predicted by using the position data of the N-M to N, and the position data of the N-M th time is replaced by the position data of the N-M th time after the position of the worker at the N +1 th time is predicted, namely, the position of the N +2 th time is predicted by using the data of the N-M +1 th to N +1 th times.

the safety of personnel in the hoisting process is related to the path of a hoisted object, the safety is different from the randomness of the target position of workers, and a hoisting machine usually has a definite target position in the one-time hoisting process, so the method and the system predict the path of the hoisted object based on the initial position and the target position of the hoisted object. When a hoisting task is started every time, namely after the hoisting object is detected to move in an image frame for the first time, the target position of the hoisting task is obtained, and the path planning of the hoisting object is carried out by combining the position of the obstacle.

Before predicting the path of the hoisted object, the positions of all obstacles in a hoisting field, the starting point of the hoisted object and the position of the target point are determined. And initializing parameters, wherein the parameters comprise maximum iteration times, information elicitation factors, expected elicitation factors, the number of ants and the like, and the elicitation factor at the position of the barrier is 0. Ants with the number preset for ants are generated at equal time intervals at the starting position of the hoisting object, each ant selects a next position node according to the state transition probability, and the nodes which have already passed through are added into a taboo table. The path pheromone is updated according to the position transition process of the ants, so that the state transition probability is updated. And continuously executing the operation to select the next position node until the ant reaches the target position, updating the pheromone concentration and the iteration times, and outputting an optimal path when the iteration times reach the preset maximum iteration times.

Predicted path of hoisted object

Wherein，

As the position coordinates of the worker at the i-th time,

the method is characterized in that the method is carried out by using a lifting object lifting method, and comprises the following steps of (1) taking the position coordinate of the lifting object at the ith moment, p is the operation time of a worker on a lifting site, q is the operation time of the lifting object, and the values of p and q are not limited. Therefore, the distance value between the worker and the hoisted object at the time i is as follows:

And then triggering an alarm.

Therefore, the hoisting process personnel safety monitoring method and system provided by the invention can predict the paths of workers and hoisted objects, find possible personnel safety accidents in the hoisting process in advance, perform safety early warning on personnel in a hoisting field in time, and effectively prevent hoisting accidents; unsafe behaviors such as unauthorized entering of a hoisting site where workers are not allowed to enter, incorrect wearing of safety helmets and the like in the hoisting process are identified, and all-round monitoring of safety of the workers is realized; the method comprises the steps of carrying out image acquisition on a hoisting field, and identifying obstacles in the hoisting field, so that the predicted worker path and the path of a hoisting object are closer to the actual path, and the accuracy of path prediction is improved; only the path of a worker moving relative to the hoisted object is predicted, so that the safety monitoring of the worker is ensured, the data processing amount is reduced, and the efficiency of the safety monitoring of the worker is improved; the LSTM worker path prediction model based on the sliding window predicts the worker path, realizes the position prediction of the worker in a long time period, and ensures the accuracy of the position prediction in the long time period.

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A hoisting process personnel safety monitoring method is characterized by comprising the following steps:

s5, judging whether relative motion exists between the worker and the hoisting object or not based on the distance difference between the front frame image and the rear frame image of the worker and the hoisting object; if yes, executing step S6;

s6, training and generating a long-short term memory network LSTM worker path prediction model based on the detected historical path of the worker, and predicting the worker path by combining the hoisting site model marked with the barrier and utilizing a sliding window method;

s8, judging whether the distance between the worker and the hoisting object at a certain moment is smaller than a preset distance in the predicted path of the worker and the hoisting object, and if so, triggering an alarm;

the step S1 includes:

dividing each collected hoisting field image into two-dimensional image grids in a transverse and longitudinal equal division mode, and integrating a plurality of two-dimensional image grids to generate a three-dimensional grid model; identifying the obstacles through a convolutional neural network, determining the positions of the obstacles in the three-dimensional grid model, and identifying in the three-dimensional grid model to generate a hoisting field model with the obstacles identified;

the step S5 includes:

judgment (loc) ₁₁ -loc ₂₁ )-(loc ₁₂ -loc ₂₂ ) Whether the alpha is greater than or equal to alpha is determined, if yes, relative motion exists between the worker and the hoisting object, and if not, relative motion does not exist; wherein, loc ₁₁ 、loc ₁₂ Position coordinates, loc, of the worker in the two frames of images before and after the worker, respectively ₂₁ 、loc ₂₂ Respectively is the position coordinates of a hoisted object in the front frame image and the rear frame image, and alpha is a preset distance threshold value;

the step S6 includes:

grouping historical path data of workers, wherein each group comprises continuous M +1 position data, taking the former M position data as the input of an LSTM network, predicting the M +1 th data, calculating a loss function of an LSTM worker path prediction model, iterating, optimizing and updating the LSTM worker path prediction model, and training to generate the LSTM worker path prediction model; predicting the position at the (N + 1) th moment by adopting the position data from the (N-M) th to the (N) th moments, judging whether the predicted position belongs to an obstacle position set, if so, excluding the position for predicting the position again, replacing the position data at the (N-M) th moment by the position data at the (N + 1) th moment of a worker after predicting the position at the (N + 1) th moment, and predicting the position at the (N + 2) th moment by adopting the data from the (N-M + 1) th to the (N + 1) th moments;

the step S7 includes:

2. The utility model provides a hoist and mount process personnel safety monitoring system which characterized in that includes:

the hoisting field model building module is used for collecting images of a hoisting field, identifying obstacles in the hoisting field and building a hoisting field model marked with the obstacles;

the first path prediction module is used for training and generating a long-short term memory network LSTM worker path prediction model based on the detected historical path of the worker, and predicting the worker path by combining the hoisting site model marked with the obstacle and utilizing a sliding window method;

the second alarm module is used for judging whether the distance between the worker and the hoisted object at a certain moment is smaller than the preset distance in the predicted path of the worker and the hoisted object or not, and if yes, triggering an alarm;

the hoisting site model building module comprises:

dividing each acquired image of the hoisting site into two-dimensional image grids in a transverse and longitudinal equal division mode, and combining a plurality of two-dimensional image grids to integrate and generate a three-dimensional grid model; identifying the obstacles through a convolutional neural network, determining the positions of the obstacles in the three-dimensional grid model, and identifying in the three-dimensional grid model to generate a hoisting field model with the obstacles identified;

the relative motion detection module includes:

judgment (loc) ₁₁ -loc ₂₁ )-(loc ₁₂ -loc ₂₂ ) Whether the alpha is greater than or equal to alpha is determined, if yes, relative motion exists between the worker and the hoisting object, and if not, relative motion does not exist; wherein, loc ₁₁ 、loc ₁₂ Position coordinates, loc, of the worker in the two frames of images before and after the worker, respectively ₂₁ 、loc ₂₂ Respectively representing the position coordinates of the hoisted object in the front and rear two frames of images, wherein alpha is a preset distance threshold;

the first path prediction module comprises:

the second path prediction module comprises:

determining the positions of all barriers in a hoisting field, the starting point of the hoisting object and the position of the target point, and initializing parameters, wherein the parameters comprise maximum iteration times, information elicitation factors, expected elicitation factors and the number of ants, and the elicitation factors at the barriers are 0; generating ants with preset number at the starting position of the hoisting object at equal time intervals, selecting the next position node by each ant according to the state transition probability, and adding the nodes which pass by the ants into a taboo table; updating path pheromone according to the position transfer process of ants and updating state transfer probability; and continuously executing the operation to select the next position node until the ant reaches the target position, updating the pheromone concentration and the iteration times, and outputting an optimal path when the iteration times reach the preset maximum iteration times.