CN111798516B - Method for detecting running state quantity and analyzing errors of bridge crane equipment - Google Patents

Abstract

The invention discloses a method for detection and error analysis of operating status quantities of bridge crane equipment, which includes: using the deep learning model Mask R-CNN to predict the positioning frame of the crane component in each frame of the camera, the instance segmentation of the component area, and the key components of the component. points and key points of natural calibration objects; use the key points of natural calibration objects on the crane to establish a world coordinate system, and use the pixel coordinates of the corresponding positions identified by Mask R-CNN as the input value to solve the PnP problem, and solve the single unit to obtain the coordinate transformation. The response matrix is to obtain the coordinate transformation matrix from the camera coordinates of the current frame to the world coordinates; combined with the coordinate transformation matrix and the key position pixel coordinates predicted by Mask R-CNN, the camera projection transformation properties are used to obtain the key point world corresponding to the pixel coordinates. coordinates; based on the obtained world coordinates of key points, calculate the operating state quantity of the crane; use the method of differential approximation error to analyze the calculation error of obtaining the world coordinates of key points, and obtain the analytical expression of the calculation error.

Description

A method for detection and error analysis of operating status quantities of bridge crane equipment

Technical field

The present invention relates to the technical field of pattern recognition, and in particular to a method for identifying operating status quantities of an overhead crane based on monocular vision.

Background technique

Existing operating state quantity measurement technologies for bridge cranes mostly rely on direct measurement of state quantities by various sensors. Such sensors need to be installed at key positions of the crane for direct measurement, which results in inconvenience of disassembly and assembly and redundant installation costs and reduces the reusability of the sensor on other cranes. The invention proposes a method for detecting operating status quantities of bridge crane equipment, which combines a deep learning algorithm with a monocular vision camera pose PnP problem solving method, making full use of the deep learning algorithm's advanced semantic understanding of images and Gao Lu's capabilities. With its strong properties and high generalization, it is possible to obtain various operating status quantities of the bridge crane using only monocular vision. Compared with traditional measurement solutions that install a large number of sensors, this method only requires a monocular camera, thus saving equipment costs. It also takes advantage of the high generalization of deep learning, so there is no need to make specific requirements for camera placement, which improves the efficiency of the measurement device. Ease of use.

Contents of the invention

In order to solve the above technical problems, the purpose of the present invention is to provide a method for pattern measurement and error analysis of operating state quantities of an overhead crane.

The purpose of the present invention is achieved through the following technical solutions:

A method for detecting operating status quantities of bridge crane equipment and error analysis, including:

A uses the deep learning model Mask R-CNN to predict the positioning frame of the crane component in each frame of the camera, the instance segmentation of the component area, and the key points of the component and the key points of the natural calibration object;

B uses the key points of natural calibration objects on the crane to establish a world coordinate system, and uses the pixel coordinates of the corresponding positions identified by Mask R-CNN as the input value to solve the PnP problem, and solves to obtain the homography matrix of the coordinate transformation, that is, the current The coordinate transformation matrix from the camera coordinates of the frame to the world coordinates;

C combines the coordinate transformation matrix and the key position pixel coordinates predicted by Mask R-CNN, and uses the camera projection transformation properties to obtain the key point world coordinates corresponding to the pixel coordinates;

D calculates the crane operating status quantity based on the obtained world coordinates of the key points;

E uses the method of differential approximation error to analyze the calculation error of obtaining the world coordinates of key points, and obtains the analytical expression of the calculation error.

Compared with the prior art, one or more embodiments of the present invention may have the following advantages:

Compared with traditional measurement solutions that install a large number of sensors, this method only requires a monocular camera, thus saving equipment costs. It also takes advantage of the high generalization of deep learning, so there is no need to make specific requirements for camera placement, which improves the efficiency of the measurement device. Ease of use.

Description of the drawings

Figure 1 is a flow chart of the detection and error analysis method of operating status quantities of bridge crane equipment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail below with reference to the embodiments and drawings.

As shown in Figure 1, it is the detection and error analysis method flow of operating status quantities of bridge crane equipment, including the sample set production and algorithm training stages; the real-time calculation stage of camera pose; the key point world coordinate calculation stage; and the operating status quantity calculation stage. ;Error calculation stage; specifically includes the following steps:

Step 10 uses the deep learning model Mask R-CNN to predict the positioning frame of the crane component in each frame of the camera, the instance segmentation of the component area, and the key points of the component and the key points of the natural calibration object;

Step 20 uses the key points of the natural calibration objects on the crane to establish a world coordinate system, and uses the pixel coordinates of the corresponding positions identified by Mask R-CNN as the input value to solve the PnP problem, and solves to obtain the homography matrix of the coordinate transformation, that is, we get The coordinate transformation matrix from the camera coordinates of the current frame to the world coordinates;

Step 30 combines the coordinate transformation matrix and the key position pixel coordinates predicted by Mask R-CNN, and uses the camera projection transformation properties to find the world coordinates corresponding to the pixel coordinates;

Step 40: Calculate the operating state quantities of the crane based on the obtained world coordinates of the key points, such as the inclination angle of the crane spreader, the inclination angle of the sling, the position of the trolley, and the position of the trolley;

Step 50 uses the differential approximation error method to analyze the calculation error in obtaining the world coordinates of the key points, and obtains the analytical expression of the calculation error.

The above step 10 specifically includes:

Create a small sample data set of important crane position key points, component positioning frames and component area instance segmentation. The key points include three or more points on the crane whose relative positions are constant and easy to identify, and the world coordinates of these points are measured as natural calibration objects. Use the prepared small sample data set to perform transfer learning on Mask R-CNN. The backbone network of Mask R-CNN is ResNet-50 pre-trained on the ImageNet data set, combined with the Feature Pyramid Networks (FPN) structure to fully extract feature information at different resolutions of the image. The feature map output by the ResNet-50FPN backbone network and the anchor box (anchor) on the feature map together serve as the input of the Region Proposal Network (RPN) layer. This layer uses softmax to determine that the anchor point belongs to the positive class (the region contains the target object) or a negative class (the area does not contain the target object), and then use the vertex coordinate error of the regression anchor box to obtain an accurate regression box. The loss function of the anchor point box vertex regression is shown in the following formula, in the formula φ (A ⁱ ) is the feature vector composed of the anchor box area feature map, W ^T _* is the parameter that needs to be learned, is the real anchor point box vertex coordinate value:

For the feature map of the regression frame part, a feature map with a fixed size of 7×7 is obtained through RoIAlign. This feature map is used as the input of the positioning frame regression and classification branch, the instance segmentation prediction branch and the key point prediction branch in subsequent steps. Among them, the reverse formula of the RoIAlign layer is as follows, x _i represents the pixel point on the feature map before pooling, y _rj represents the j-th point of the r-th candidate area after pooling, and d(.) represents the distance between the two points. Distance, i ^* (r, j) represents the coordinates of the point where the maximum pixel value is selected during maximum pooling, Δh and Δw represent the difference between the horizontal and vertical coordinates of x _i and i ^* (r, j):

The positioning box regression and classification branch consists of two fully connected layers. One outputs an 8-dimensional vector that represents the offset of each vertex coordinate component of the positioning box; the other output vector dimension is the number of categories, which is used to predict the positioning box. The category of the object. The loss function L _box of the sub-branch used to regress the offset is the least squares error between the real value and the predicted value, and the form is the same as the loss function of the anchor box vertex regression; the classification sub-branch loss function uses the following cross-entropy loss Function L _cls :

Among them, p _out represents the output of the classification sub-branch, which is a function of the network weight; N _ap represents the number of samples; p _n represents the real classification label.

The instance segmentation branch and the key point prediction branch are both composed of inverse convolution layers. They output a spatial probability distribution map that corresponds to the original image but has been scaled, indicating the probability that each position is a certain category; its loss function L _mask and L _point is also a cross-entropy loss function, and its form is consistent with the classification sub-branch loss function.

The total loss function of the entire Mask R-CNN network is L=L _box +L _cls +L _mask +L _point . On the labeled bridge crane small sample data set, the optimizer based on the gradient descent algorithm such as Adam or SGD is minimized. This total loss is transformed to obtain the optimal Mask R-CNN network model parameters and complete the model training.

The above step 20 specifically includes:

The pixel coordinates of key points predicted by Mask R-CNN and used as natural calibration objects have the following relationship with the corresponding world coordinates:

Among them (w, h, 1) represents the pixel coordinates of the predicted point, d _x and d _y represent the pixel equivalents of the x-axis and y-axis of the camera sensor respectively, f represents the focal length of the camera image side, (w ₀ , h ₀ ) is the imaging center pixel coordinate, is the transformation matrix of the camera coordinate system with respect to the world coordinate system, z _c represents the z-axis component of the origin of the camera coordinate system in the world coordinate system.

According to the above relationship, when the key point n ≥ 3 is predicted, the transformation matrix of the calibrated camera relative to the world coordinates at any time can be obtained by solving the homography matrix on the right side of the above equation, thus completing a PnP problem. solution. To solve the above equation, we use the native API solvePnPRansac provided by OpenCV. When the number of selected natural calibration object coordinate points is much greater than 3, this solver has strong robustness to noise.

The above step 30 specifically includes:

On the basis of the calculated camera transformation matrix at the current moment, using the camera projection imaging law, the world coordinates of any key point with known motion surface parameters in the world coordinate system can be deduced. At this time, there are only two key points. Freedom of movement. The pixel coordinates of any two-degree-of-freedom key point have the following relationship with the world coordinates. For the sake of simplicity, it may be assumed that the motion surface is a plane:

formula, Represents pixel coordinates,/> Represents the corresponding world coordinates,/> Represents the normal vector of the movement plane of the key point in the world coordinate system, and h represents the intercept of the equation of this plane. The above formula expresses the /> Calculate/> the whole process. The key point positions of the crane predicted by MaskR-CNN include: the upper and lower end points and midpoints of the sling, the upper and lower end points of the spreader, and the vertices of the trolley outline polygon. The predicted pixel coordinates of these key points are combined with the motion plane of these points on the crane in the world coordinate system. By bringing the parameters in into the above series of equations, the world coordinates at any time corresponding to these key points can be obtained.

The above step 40 specifically includes:

Based on the obtained world coordinates of the key points, the operating state quantities of the bridge crane are calculated: trolley position, trolley speed, trolley position, trolley speed, spreader inclination, and sling inclination.

The position of the car is represented by the y-axis of the average point of its outline vertices, and the speed is obtained by taking the difference on the y-axis: v _sc = FPS × Δy, where FPS represents the frame rate, which is determined by the camera sampling rate and algorithm calculation speed. When the predicted test value is missing for the contour vertex, or the prediction error is large, the y-axis component y _m of the centroid world coordinate of the car area Mask R-CNN instance segmentation result is used instead for calculation.

The position of the cart is represented by the z component of the translation transformation vector T, and the running speed is calculated as: v _bc = FPS × Δz.

The calculation method for the inclination angle of spreaders and lifting ropes is:

in, Represents a series of predicted key points on the straight line where the sling is located, or a series of key points on the center line of the spreader. ||.|| represents the vector norm, (.) _x and (.) _y represent the vector x and y components respectively.

The above step 50 specifically includes:

Now we analyze the error corresponding to the world coordinates of the two-degree-of-freedom key points calculated from the pixel coordinates. It is believed that when the input error is small, the output error can be approximately represented by the function differential value:

in, dh is the measurement error of the key point motion plane parameters; dR, /> To calculate the camera pose calculation error, solvers such as solvePnPRansac will give corresponding algorithm solution errors that can be used as pose calculation error values;/> Calibration error for camera internal parameters;/> is the prediction error of the key point, and the value can be the prediction probability of the key point by Mask R-CNN.

Although the embodiments disclosed in the present invention are as above, the described contents are only used to facilitate the understanding of the present invention and are not intended to limit the present invention. Any person skilled in the technical field to which the present invention belongs may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed by the present invention. However, the patent protection scope of the present invention shall not The scope defined by the appended claims shall prevail.

Claims (8)

1. The method for detecting and analyzing the running state quantity of the bridge crane equipment is characterized by comprising the following steps:

a, predicting a positioning frame of a crane component, an example segmentation of a component area and a component key point and a natural calibration object key point in each frame of picture of a camera by using a deep learning model Mask R-CNN;

b, establishing a world coordinate system by utilizing natural calibration object key points on the crane, and taking pixel coordinates of corresponding positions identified by Mask R-CNN as input values for solving a PnP problem, and solving a homography matrix for obtaining coordinate transformation, namely obtaining a coordinate transformation matrix from camera coordinates of a current frame to world coordinates;

c, combining the coordinate transformation matrix and the key position pixel coordinates predicted by Mask R-CNN, and solving the world coordinates of the key points corresponding to the pixel coordinates by utilizing the projection transformation property of the camera;

d, calculating the crane running state quantity according to the calculated world coordinates of the key points;

and E, analyzing and solving the calculation error of the world coordinates of the key points by using a differential approximation error method, and obtaining an analysis expression of the calculation error.

2. The method for detecting and analyzing the operation state quantity and the error of the bridge crane equipment according to claim 1, wherein the step a specifically comprises:

manufacturing a small sample data set of important position key points, component positioning frames and component area example segmentation of a crane; wherein the key points comprise: under the condition that the relative position on the crane is constant, three or more points which are easy to identify are detected, and world coordinates of the points are measured to be used as natural calibration objects;

performing migration learning on Mask R-CNN by using the prepared small sample data set; the backbone network of Mask R-CNN is selected as ResNet-50 pre-trained on an ImageNet data set, and feature information of images under different resolutions is extracted by combining a feature pyramid structure; the feature map output by ResNet-50FPN backbone network and the anchor blocks on the feature map together serve as inputs to the region providing network layer through softmax judges whether the anchor point belongs to positive class or negative class, and then the coordinate error of the vertex of the regression anchor point frame is utilized to obtain an accurate regression frame, wherein the loss function of the vertex regression of the anchor point frame is shown as the following formula, phi (A) ⁱ ) Is the characteristic vector composed of the characteristic map of the anchor point frame region, W ^T _* Is a parameter that needs to be learned and,the vertex coordinate value of the true anchor point frame is as follows:

obtaining a feature map with a fixed size of 7 multiplied by 7 through RoIAlign by using the feature map of the regression frame part as input of a positioning frame regression and classification branch, an example segmentation prediction branch and a key point prediction branch; wherein the roualign layer inversion formula is as follows:

wherein x is _i Representing pixel points on the characteristic diagram before pooling, y _rj The j-th point representing the pooled r-th candidate region, d () represents the distance between the two points, i ^* (r, j) represents the coordinates of the point where the maximum pixel value selected at the time of maximum pooling is located, and Δh and Δw represent x _i And i ^* (r, j) difference in abscissa and ordinate.

3. The method for detecting and analyzing the operation state quantity of the bridge crane equipment according to claim 2, wherein,

the regression and classification branch of the positioning frame consists of two paths of full-connection layers, and one path of output 8-dimensional vector respectively represents the offset of each vertex coordinate component of the positioning frame; the other path of output vector dimension is the number of categories and is used for predicting the categories of the objects in the positioning frame; one-way sub-division for regression offsetIts loss function L _box The least square error of the true value and the predicted value is the same as the loss function of the peak regression of the anchor point frame; the classifying sub-branch loss function adopts the following cross entropy loss function L _cls ：

Wherein p is _out The output representing the classification sub-branch is a function of the network weight; n (N) _ap Representing the number of samples; p is p _n Representing a true class label.

4. The method for detecting and analyzing the operation state quantity of the bridge crane equipment according to claim 2, wherein,

the example segmentation branch and the key point prediction branch are both composed of deconvolution layers, and a space probability distribution diagram corresponding to the original image but scaled is output to represent the probability that each position is of a certain class; loss function L _mask And L _point The cross entropy loss function is the same as the classifying sub-branch loss function in form;

the total loss function of the whole Mask R-CNN network is L=L _box +L _cls +L _mask +L _point And on the marked bridge crane small sample data set, minimizing the total loss function by using an optimizer of Adam based on a gradient descent algorithm to obtain optimal Mask R-CNN network model parameters, and completing training of the model.

5. The method for detecting and analyzing the operation state quantity of the bridge crane according to claim 1, wherein in the step B:

the key point pixel coordinates predicted by Mask R-CNN and serving as natural calibration objects have the following relation with corresponding world coordinates:

wherein (w, H, 1) represents the predicted point pixel coordinates, d _x 、d _y Pixel equivalent of x-axis and y-axis of camera sensor, respectively, f represents camera image Fang Jiaoju, (w) ₀ ,H ₀ ) For the imaging center pixel coordinates,z is a transformation matrix of the camera coordinate system with respect to the world coordinate system _c Representing a z-axis component of the camera coordinate system origin in the world coordinate system;

according to the relation, when the predicted key point n is more than or equal to 3, the homography matrix on the right side of the equation can be solved to obtain the transformation matrix of the calibrated camera relative to the world coordinates at any moment, and then the solution of the PnP problem is completed once.

6. The method for detecting and analyzing the operation state quantity and the error of the bridge crane equipment according to claim 1, wherein the step C specifically comprises:

based on the obtained camera transformation matrix at the current moment, the world coordinates of key points with known motion curved surface parameters in a world coordinate system are reversely deduced by utilizing a camera projection imaging rule, and at the moment, the key points have only two motion degrees of freedom; the pixel coordinates of any two-degree-of-freedom key point and world coordinates have the following relation, and for brevity, it is not necessary to assume that the motion curved surface is a plane:

the above formula expresses thatCalculate->In the formula, ++>Representing pixel coordinates, +.>Representing corresponding world coordinates>The normal vector of the motion plane of the key point under the world coordinate system is represented, and H represents the intercept of the plane equation;

the predicted crane key point position of the Mask R-CNN comprises: the world coordinates of any moment corresponding to the key points can be obtained by combining the predicted pixel coordinates of the key points with the parameters of the motion planes of the points on the crane in the world coordinate system and taking the parameters into the series of equations.

7. The method for detecting and analyzing the operation state quantity and the error of the bridge crane equipment according to claim 1, wherein the step D specifically comprises:

calculating the running state quantity of the bridge crane on the basis of the calculated world coordinates of the key points, wherein the crane running state quantity comprises a trolley position, a trolley speed, a lifting appliance gradient and a lifting rope gradient;

the trolley position, represented by the y-axis of the average point of its profile vertices, the velocity is then obtained by differentiating the y-axis: v _sc =fps×Δy, where FPS represents the frame rate, determined by the camera sampling rate, the algorithm calculation speed; when the outline vertex lacks a pre-test value or the prediction error is larger, the y-axis component y of the centroid world coordinate of the result is segmented by using the trolley region Mask R-CNN example _m Instead of calculation;

the cart position is represented by the z component of the translation transformation vector T, and the running speed is calculated by: v _bc ＝FPS×Δz；

The calculation mode of the inclination angle of the lifting appliance and the lifting rope is as follows:

wherein,representing a series of predicted key points on a straight line where the lifting rope is located or a series of key points on a central line of the lifting appliance; i denote vector norms () _x 、(.) _y Representing the x and y components of the vector, respectively.

8. The method for detecting and analyzing the operation state quantity and the error of the bridge crane equipment according to claim 1, wherein the step E specifically comprises:

analyzing the error of the world coordinates corresponding to the two-degree-of-freedom key points obtained by the pixel coordinates, and considering that when the input error is small, the output error can be approximately represented by a function differential value:

wherein,dh is the measurement error of the motion plane parameters of the key points; dR, & lt + & gt>For the pose calculation errors of the camera, solvers such as a solvePnPRansac and the like give corresponding algorithm solving errors which can be used as pose calculation errors to take values; />Calibrating errors for internal parameters of the camera; />The prediction error of the key point can be the prediction probability of Mask R-CNN on the key point.