Disclosure of Invention
The invention aims to provide a hydraulic support pushing progress detection method and system based on a monocular vision automatic label aiming at the defects of the prior art.
In order to achieve the purpose, the invention adopts the technical scheme that:
the invention provides a hydraulic support propulsion degree detection method based on a monocular vision automatic label, which comprises the following steps:
deployment detection system
Mounting a camera on a hydraulic support, wherein a horizontal plane 0-degree line of the camera is a central line of a top beam of the hydraulic support, a pitching plane of the camera is vertical to the top beam of the hydraulic support, and a pitching plane 0-degree line of the camera is a central line of the top beam of the hydraulic support;
the camera shoots a pushing rod of the hydraulic support in a downward direction, and a hydraulic support base and a scraper conveyor cross head at two ends of the pushing rod are respectively provided with a mark 1 and a mark 2;
constructing a deep neural network model
Constructing ten-layer depth residual error network model composed of one-dimensional convolutional layers Conv1D (M, N), wherein M represents the number of input channels, N represents the number of output channels, the first layer convolutional layer is Conv1D (3, 64), the second layer convolutional layer is Conv1D (64, 128), the third layer convolutional layer is Conv1D (128 ), the fourth layer convolutional layer is Conv1D (128, 64), the fifth layer convolutional layer is Conv1D (64, 64), the sixth layer convolutional layer is Conv1D (64, 32), the seventh layer convolutional layer is Conv1D (32, 32), the eighth layer convolutional layer is Conv1D (32, 4), the ninth layer convolutional layer is Conv1D (4, 2), and the tenth layer is Conv1D (2, 4);
neural network training based on automatic labeling
The camera is arranged on the holder and can record and output the rotation angle information of the horizontal shaft and the pitching shaft.
Based on the above, when training the neural network based on the automatic label, the image information of the camera read by the controller is subjected to enhancement preprocessing of histogram equalization and contrast adjustment, wherein the contrast enhancement adjustment range is between +0.0 to + 1.0; and image information of the current hydraulic frame and the pushing rods of the left and right adjacent hydraulic frames is obtained from the read image information of the camera.
Based on the above, during the neural network training based on the automatic label, the controller only reads the transition distances of the push rods of the current hydraulic frame and the left and right adjacent hydraulic frames, and the transition distances of the push rods are obtained through the detection of the sensors.
Based on the above, the method for extracting the position information of the marker in the image according to the image information of the camera comprises the following steps:
in the image that the camera was shot, need confirm current hydraulic support, control the end information of adjacent hydraulic support, divide into three kinds of situations: the left deviation of the camera, the centering of the camera and the right deviation of the camera;
dividing each mark in the enhanced image, and numbering from top to bottom and from left to right;
according to the positions of the marks in the image, firstly, the marks are divided into an upper row and a lower row, and then mark pairing is carried out from left to right;
the center point is confirmed by the intersection point of the horizontal line and the vertical line in the divided marker, and the coordinate of the spatial position center of the marker pixel on the coordinate axis x0y is obtained.
Based on the above, the camera is a monocular servo camera, provides three primary color images of RGB, and has a resolution of 1920 × 1080.
Based on the above, the data accuracy of the moving distance of the moving rod is 0.01 m.
Based on the above, the GPU of the controller selects the NVIDIA T4 GPU graphics card.
The method comprises the steps that a built deep neural network model is placed in a GPU unit of a controller, and the controller reads image information of a camera, position information of the camera and a pushing distance of a pushing rod at fixed time;
extracting position information of a marker in the image according to the image information of the camera, namely coordinates of a spatial position center of the marker pixel on a coordinate axis x0y, wherein the total number of the position information is 4 data; extracting horizontal angle and pitch angle information of the camera according to the angle information of the camera, wherein the total number of the information is 2;
the 6 data information is used as input signals and sent into a deep neural network model, the transition distance of a transition rod is used as a label of the deep neural network model, the transition distance is used as an output signal and sent into the deep neural network model, and the deep neural network model is trained;
during training, the loss function: log loss function
In the formula, X is a neural network input signal, and Y is a corresponding neural network label; the training method comprises the following steps: a random gradient descent method; end of training marker: the loss function value is less than 0.001;
push progress detection
The controller reads image information of the camera and position information of the camera, extracts coordinates of a space position center of an identification pixel on a coordinate axis x0y according to the image information of the camera, and extracts horizontal angle and pitch angle information of the camera according to angle information of the camera, wherein the total number of the data is 4; and the 6 data information is used as an input signal and is sent into a trained deep neural network model, and the pushing distance of the pushing rod can be output.
Based on the above, the mark 1 and the mark 2 are red and white alternating reflective stickers with 5cm by 5 cm.
Based on the above, the camera is arranged on the holder and can record and output the rotation angle information of the horizontal axis and the pitch axis.
Based on the above, when training the neural network based on the automatic label, the image information of the camera read by the controller is subjected to enhancement preprocessing of histogram equalization and contrast adjustment, wherein the contrast enhancement adjustment range is between +0.0 to + 1.0; and image information of the current hydraulic frame and the pushing rods of the left and right adjacent hydraulic frames is obtained from the read image information of the camera.
Based on the above, during the neural network training based on the automatic label, the controller only reads the transition distances of the push rods of the current hydraulic frame and the left and right adjacent hydraulic frames, and the transition distances of the push rods are obtained through the detection of the sensors.
Based on the above, the method for extracting the position information of the marker in the image according to the image information of the camera comprises the following steps:
in the image that the camera was shot, need confirm current hydraulic support, control the end information of adjacent hydraulic support, divide into three kinds of situations: the left deviation of the camera, the centering of the camera and the right deviation of the camera;
dividing each mark in the enhanced image, and numbering from top to bottom and from left to right;
according to the positions of the marks in the image, firstly, the marks are divided into an upper row and a lower row, and then mark pairing is carried out from left to right;
the center point is confirmed by the intersection point of the horizontal line and the vertical line in the divided marker, and the coordinate of the spatial position center of the marker pixel on the coordinate axis x0y is obtained.
The invention provides a hydraulic support pushing degree detection system based on a monocular vision automatic label, which comprises a hydraulic support, a camera and a controller, wherein the controller is provided with a GPU unit; the hydraulic support, the camera and the controller are used for completing the monocular vision automatic label based hydraulic support propulsion degree detection method of any one of claims 1 to 6.
Based on the above, the camera is a monocular servo camera, provides three primary color images of RGB, and has a resolution of 1920 × 1080.
Based on the above, the data accuracy of the moving distance of the moving rod is 0.01 m.
Based on the above, the GPU of the controller selects the NVIDIA T4 GPU graphics card.
Compared with the prior art, the invention has prominent substantive characteristics and remarkable progress, and particularly has the following beneficial effects:
(1) only one monocular servo camera is employed: the camera is arranged on the hydraulic support, a pushing rod of the hydraulic support is shot in a depression mode, after image information shot by the camera is segmented and matched with the identification according to the identification, if n pairs of identification objects exist in one image, the one image can provide n samples to participate in training, and multiple training samples can be provided by only adopting one monocular servo camera;
(2) calibration is not required: the method comprises the following steps of training image information of a camera, position information of the camera and a pushing distance of a pushing rod by adopting a deep neural network model, and realizing accurate detection of a pushing stroke of the hydraulic support without calibrating a detection position of the camera;
(3) the labels do not need to be provided manually: the label of the deep neural network model corresponds to the pushing distance of the pushing rod, the pushing distance detected by the sensor is automatically read by the controller to serve as the label, and the label is not required to be manually provided;
(4) can be trained on line: the image information of the camera, the position information of the camera and the transition distance of the transition rod are trained by adopting a deep neural network model, wherein the image information of the camera, the position information of the camera and the transition distance of the transition rod are in communication connection with the controller through the camera and the sensor, and the controller can read the GPU on line at regular time for training.
Detailed Description
The technical solution of the present invention is further described in detail by the following embodiments.
Example 1
As shown in fig. 1 to 7, the embodiment provides a hydraulic support pushing progress detection system based on a monocular vision automatic label, and the detection system includes a hydraulic support, a camera and a controller, and the controller is provided with a GPU unit; the hydraulic support, the camera and the controller are used for completing the detection method of the propelling degree of the hydraulic support based on the monocular vision automatic label, and the method comprises the following steps:
deployment detection system
Mounting a camera on a hydraulic support, wherein a horizontal plane 0-degree line of the camera is a central line of a top beam of the hydraulic support, a pitching plane of the camera is vertical to the top beam of the hydraulic support, and a pitching plane 0-degree line of the camera is a central line of the top beam of the hydraulic support;
the camera shoots a pushing rod of the hydraulic support in a downward direction, and a hydraulic support base and a scraper conveyor cross head at two ends of the pushing rod are respectively provided with a mark 1 and a mark 2; specifically, the marks 1 and 2 are 5cm by 5cm red and white reflecting stickers, and in other embodiments, other markers which are used for marking on the base of the hydraulic support and the crosshead of the scraper conveyor can be directly adopted.
Constructing a deep neural network model
Ten deep residual error network models composed of one-dimensional convolutional layers Conv1D (M, N) were constructed, where M represents the number of input channels, N represents the number of output channels, the first layer convolutional layers were Conv1D (3, 64), the second layer convolutional layers were Conv1D (64, 128), the third layer convolutional layers were Conv1D (128 ), the fourth layer convolutional layers were Conv1D (128, 64), the fifth layer convolutional layers were Conv1D (64, 64), the sixth layer convolutional layers were Conv1D (64, 32), the seventh layer convolutional layers were Conv1D (32, 32), the eighth layer convolutional layers were Conv1D (32, 4), the ninth layer convolutional layers were Conv1D (4, 2), and the tenth layer was Conv1D (2, 4).
Neural network training based on automatic labeling
(1) The image acquisition method comprises the following steps: the camera is arranged on the holder and can record and output the rotation angle information of the horizontal shaft and the pitching shaft, and the camera is a monocular servo camera and can be directly purchased;
(2) information acquisition and preprocessing:
to determine the length of the push rod in the video image, the detection system provides three types of information: image information of the camera; the length of the pushing rod is detected by a sensor; the pitch angle and the horizontal angle of rotation of the camera;
the camera provides three primary color images of RGB, and the resolution is 1920 x 1080. The image information of the camera read by the controller is subjected to enhancement preprocessing of histogram equalization and contrast adjustment, wherein the contrast enhancement adjustment range is +0.0 to + 1.0; and image information of the current hydraulic frame and the push rods of the left and right adjacent hydraulic frames is obtained from the read image information of the camera, and the other image information is not considered. The controller only reads the pushing distance of the current hydraulic frame and the pushing rods of the left and right adjacent hydraulic frames, the pushing distance of the pushing rods is obtained through detection of the sensors, and the data precision of the pushing distance of the pushing rods is 0.01 m;
(3) determining end head information of pushing rods of three hydraulic supports:
in the image that the camera was shot, need confirm current hydraulic support, control the end information of adjacent hydraulic support, divide into three kinds of situations: the left deviation of the camera, the centering of the camera and the right deviation of the camera;
dividing each mark in the enhanced image, and numbering from top to bottom and from left to right;
according to the position of the mark in the image, firstly dividing the mark into an upper row and a lower row, and then carrying out mark pairing from left to right, namely realizing the pairing of the image information of the camera, the length of a push rod, the rotating pitch angle and the horizontal angle of the camera;
confirming a central point through the intersection point of a horizontal line and a vertical line in the divided identification to obtain the coordinate of the spatial position center of the identification pixel on a coordinate axis x0y, namely the coordinate of two identification central points on the coordinate axis, wherein the total number of the two identification central points is 4 data;
(4) deep neural network training
The method comprises the steps of placing a built deep neural network model in a GPU unit of a controller, wherein a GPU of the controller selects an NVIDIA T4 GPU display card; the controller reads image information of the camera, position information of the camera and the pushing distance of the pushing rod at fixed time;
extracting position information of the marker in the image according to the image information of the camera, namely coordinates of a spatial position center of the marker pixel on a coordinate axis x0 y; extracting horizontal angle and pitch angle information of the camera according to the angle information of the camera, wherein the total number of the information is 2;
the 6 data information is used as input signals and sent into a deep neural network model, the transition distance of a transition rod is used as a label of the deep neural network model, the transition distance is used as an output signal and sent into the deep neural network model, and the deep neural network model is trained;
the main technical parameters of neural network training include:
loss function: log loss function
In the formula, X is a neural network input signal, and Y is a corresponding neural network label;
the training method comprises the following steps: a random gradient descent method;
end of training marker: the loss function value is less than 0.001.
Push progress detection
The controller reads image information of the camera and position information of the camera, extracts coordinates of a space position center of an identification pixel on a coordinate axis x0y according to the image information of the camera, and extracts horizontal angle and pitch angle information of the camera according to angle information of the camera, wherein the total number of the data is 4; and the 6 data information is used as an input signal and is sent into a trained deep neural network model, and the pushing distance of the pushing rod can be output.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention and not to limit it; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art will understand that: modifications to the specific embodiments of the invention or equivalent substitutions for parts of the technical features may be made; without departing from the spirit of the present invention, it is intended to cover all aspects of the invention as defined by the appended claims.