CN114037703B

CN114037703B - Subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation

Info

Publication number: CN114037703B
Application number: CN202210019235.XA
Authority: CN
Inventors: 秦娜; 周重合; 黄德青; 万字朋; 倪思杰; 蔡重阳
Original assignee: Southwest Jiaotong University
Current assignee: Southwest Jiaotong University
Priority date: 2022-01-10
Filing date: 2022-01-10
Publication date: 2022-04-15
Anticipated expiration: 2042-01-10
Also published as: CN114037703A

Abstract

The invention discloses a subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation, which specifically comprises the following steps: acquiring 2D and 3D data of a subway train valve component to be overhauled by using a subway inspection trolley with an industrial camera; positioning the 2D picture component and the reference object based on a deep learning target detection network; further screening point clouds through an R channel in the point cloud RGB, and fitting a valve base mass center and a valve tail end mass center; calculating a conversion matrix of a current coordinate system and a template coordinate system by taking the coordinates of the valve base as a reference, and obtaining the coordinates of the tail end of the valve under the template coordinate system through coordinate transformation; and resolving the valve attitude, and judging whether the valve is in a normal state or not according to the comparison between the deflection angle and a set threshold value. The invention can provide strong guarantee for the safe operation of the subway train, can save each time of data by replacing a manual mode with the inspection robot, and establishes a more accurate digital model for the train.

Description

Subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation

Technical Field

The invention belongs to the field of subway valve state detection, and particularly relates to a subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation.

Background

With the rapid development of economy and science and technology in China, the intelligent level is improved by means of the assistance of the Internet of things, artificial intelligence, 5G technology and the like in one field and another. The successful landing of the items such as unmanned ticketing, face brushing and station entering, NFC rapid payment and the like greatly improves the intelligent level of the transportation field in China. The shadow of the intelligent subway can be seen everywhere from the arrival of passengers to the departure of passengers. However, on the side that passengers cannot sense, the inspection and maintenance of the subway still mostly depend on the traditional manual detection mode, which obviously does not meet the urban rail transit information construction requirements proposed by the China urban rail transit Association. And the mode of artifical detection has a great deal of shortcoming, and is efficient, with high costs, operational environment is abominable, threatens the maintainer safety easily, has the hidden danger of lou examining to some complicated parts easily simultaneously, influences train safe operation. Therefore, by combining the current leading-edge technologies, such as computer vision, artificial intelligence, anomaly detection and the like, a part of manual work is replaced, and the increase of the intelligent degree of train routing inspection becomes the inevitable trend of the development of intelligent subways.

At present, a large number of applications based on machine vision are successfully applied to the ground, and in the field of civilian life, the projects of traffic violation shooting, key area face brushing access control, AI special effects, content retrieval based on pictures and the like exist. The safety guarantee is provided for the peaceful and happiness industry of people, convenience is provided for people, and the life style of people is enriched. In the industrial field, machine vision-based safety helmet detection, product abnormity detection and the like exist, so that the life safety of workers is guaranteed, and the production efficiency of enterprises is improved. And the application in the fields of medical images, aerospace mapping and the like greatly assists in industrial development. The intellectualization of the rail transit anomaly detection is also promoted at present, and becomes a research hotspot. At present, most of the subway abnormity detection methods for detecting the abnormity of the subway based on sensor signals, such as temperature detection and the like, fall on the ground. The detection has specificity, and although the accuracy is high, the detection does not have expansibility and is difficult to be applied to the detection of other parts. And the anomaly detection based on computer vision is in a development stage, most anomaly detection algorithms are detected based on 2D images, the detection method has strong expandability, can complete detection aiming at most parts, such as bolt looseness and gear box liquid level detection, can realize anomaly detection through the position of a locking line, and has the advantages of simple and quick acquisition and easy transmission and processing. However, the 2D picture cannot acquire spatial information of the component, and is easily affected by the environment and dirt. In a subway train, it is difficult to determine the abnormality of a part of components by using only 2D information, and it is necessary to complete the determination based on spatial information, such as the amount of wear of a consumable part, the detection of the size of the part, and the like. Therefore, 3D-based anomaly detection is also a hot spot of current research.

At present, a subway appearance detection system with high completion degree is mainly characterized in that a line scanning camera is erected at a maintenance station, and a side view and a bottom view of a train are obtained in a scanning imaging mode when the train enters the maintenance station. This kind of system can only acquire surface part single angle information to there is the inside region part that shelters from, can't acquire effective image information, also has the limitation that 2D detected, can only reduce artifical maintenance work load to a certain extent.

A large number of valves, such as a main air valve, an air supply valve, a main air pipe and the like, exist in the train and provide power support for train pneumatic components; such as a sliding plug door, an air spring and the like, and provides guarantee for the stable operation of the train. Its abnormity will cause great harm to the train operation safety.

From the above background, it can be analyzed that the realization of the valve state detection by using 2D +3D requires the following key points to be solved: 1. the algorithm model needs to effectively solve the problem that the 2D picture is easily affected by environment and stains, and has strong robustness. 2. The algorithm model needs to have strong generalization capability, and can accurately acquire valve attitude information under different train numbers, different positions and different environments. 3. The efficiency is high, the train is only overhauled in the outage stage, and subway trains with a plurality of routes are overhauled at the same station. After the detection system finishes detection, the fault needs to be provided for maintenance personnel to realize fixed-point maintenance, so that the algorithm model needs to finish component detection quickly and efficiently.

Disclosure of Invention

Aiming at the problems, the invention provides a subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation.

The invention discloses a subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation, which comprises the following steps of:

step 1: the subway inspection trolley is automatically positioned to a designated position, and 2D and 3D data of a subway train valve component to be overhauled are acquired through a three-dimensional industrial camera carried on a mechanical arm.

Step 2: the deep learning based YOLOV5L target detection network locates the 2D picture component and the reference.

And step 3: further screening point clouds through an R channel in the point cloud RGB, and fitting a valve base mass center and a valve tail end mass center; and resolving a conversion matrix of the current coordinate system and the template coordinate system by taking the coordinates of the valve base as a reference, and obtaining the coordinates of the tail end of the valve under the template coordinate system through coordinate transformation.

And 4, step 4: and resolving the valve attitude, and judging whether the valve is in a normal state or not according to the comparison between the deflection angle and a set threshold value.

Further, step 2 specifically comprises:

s21: and loading the trained YOLOV5L target detection model, and carrying out target detection processing on the 2D picture.

S22: the target detection model selects valve information from the 2D picture frame: valve base coordinates, valve tip coordinates.

S23: and mapping the coordinates to the three-dimensional point cloud data, and acquiring the point cloud data of the valve base and the valve tail end, thereby filtering irrelevant point cloud data and accelerating the model speed.

Further, step 3 specifically comprises:

s31: and traversing the point cloud data of the valve base and the valve tail end in the step 2, and accurately screening out the point clouds belonging to the valve part through the point cloud RGB information.

S32: and (3) utilizing a mass center formula to carry out mass center calculation on the screened point cloud, and obtaining a coordinate which can represent the spatial positions of the valve base and the valve tail end most:

s33: determining coordinates of a valve base centroid and a valve tail end centroid in a normal state and a template coordinate system through first data acquisition; and taking the data as a reference, and calculating the coordinate of the center of mass of the tail end of the valve under a template coordinate system by taking the center of mass of the valve base as a reference through three-dimensional coordinate transformation

。

Further, step 4 specifically includes:

s41: the coordinate of the tail end of the valve arranged under the database template coordinate system is

The center of mass of the end of the valve to be measured is

The coordinate of the valve tail end mass center in the template coordinate system after coordinate conversion is

The length of the valve isl：

Calculated by a space distance formula

Euclidean distance of (c):

s42: calculating valve offset angle by inverse trigonometric functionθ ₁：

S43: according to the angle of deflection of the valveθ ₁And a set threshold value deltaθAnd (5) comparing, judging whether the valve is in a normal state:

if the abnormal condition occurs, an alarm is sent out, the specific position of the fault occurs, and the maintenance personnel is prompted to carry out fixed-point maintenance.

Further, a threshold value Δ is setθThe value is 15 degrees.

The beneficial technical effects of the invention are as follows:

1. according to the invention, the intelligent patrol trolley autonomously and accurately reaches a parking point and carries the 3D camera to realize data acquisition at the optimal shooting angle, the spatial information of the component to be detected can be obtained to the maximum extent, the problem of detection difficulty caused by insufficient information acquisition of an appearance detection system based on fixed-point line erection scanning is solved, and the detection accuracy is effectively improved.

2. The invention designs a preprocessing method based on 2-dimensional image target positioning, which maps the 2-dimensional image target positioning to 3-dimensional point cloud data and secondarily filters the point cloud by utilizing a color gamut, thereby effectively filtering irrelevant point cloud, accelerating the speed of an algorithm model and improving the efficiency. The neural network based on deep learning is utilized to realize target positioning, the robustness of the algorithm model can be effectively improved, the problem of inaccurate target positioning caused by deviation of shooting angles or influence of ambient light and stains is reduced, meanwhile, the data volume can be continuously increased in the later stage of the algorithm model to train the network, and the performance is continuously improved in use.

3. The method comprises the steps of accurately calculating the spatial coordinates of the valve base and the mass center of the tail end of the valve through 2-dimensional positioning and 3-dimensional point cloud data, and obtaining the mass center coordinate of the tail end of the valve under a database template coordinate system by using the valve base as a conversion operator according to an acquisition time coordinate system and the database template coordinate system. The method comprises the steps of establishing a mathematical model by analyzing the relationship between a valve offset angle and chord lengths obtained before and after the offset of the tail end of the valve, accurately calculating the valve offset angle by a chord length formula and an inverse trigonometric function, and accurately judging whether the valve is in a normal working state or not according to a set threshold value.

4. The component state analysis process combining 2-dimensional positioning and 3-dimensional point cloud data provided by the invention does not need additional features for support, and the detection model is simple and efficient. And the mathematical model can be adjusted to realize the detection of other parts, and the expansibility is realized. The whole algorithm model can realize automatic judgment of the valve state, and guarantee is provided for safe operation of the train.

Drawings

FIG. 1 is a flow chart of a subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation.

Fig. 2 is a schematic diagram of a valve reconstruction coordinate system.

FIG. 3 is a schematic diagram of valve attitude calculation.

Fig. 4 is a schematic diagram of the detection result of the same valve by collecting data for multiple times.

Fig. 5 is a schematic diagram of the detection results of different valves for different compartments.

Detailed Description

The invention is described in further detail below with reference to the figures and the detailed description.

According to the invention, during data acquisition, the intelligent inspection trolley of the train accurately arrives at a specified place to stop, and then the mechanical arm of the trolley carries the 3D industrial camera to acquire three-dimensional point cloud information (XYZ) and corresponding RGB information of each part of the train. The train valve attitude detection needs to be completed through the following steps: 1. and training a target positioning network, and positioning the valve and the reference object thereof on the two-dimensional picture. 2. And mapping the target frame on the two-dimensional picture to the three-dimensional point cloud, acquiring a valve and a reference object point cloud block, further screening the point cloud by using the color gamut, and solving the mass center of the corresponding point cloud block. 3. And solving the valve attitude by establishing a mathematical model. 5. And comparing the valve deflection with a threshold value to judge whether the valve is in a normal working state.

The invention discloses a subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation, which is shown in figure 1 and comprises the following steps:

The acquisition mode is that the intelligent inspection robot of the train autonomously locates to the designated position of the subway train, and then the data acquisition of the valve component and the reference object thereof is completed through a three-dimensional industrial camera carried on a mechanical arm of the robot, and the camera acquires XYZ and RGB information of component point cloud in a structured light mode. And then transmitting the data to an algorithm server for real-time detection.

The number of originally collected 3D point clouds is up to 100 ten thousand, and direct operation and retrieval of the point clouds consume a large amount of computing power and time, and if the point clouds are directly utilized for component positioning and segmentation, the algorithm efficiency is greatly influenced. Therefore, the parts and the reference objects are positioned on the 2D picture, the coordinate information of the parts and the reference objects is mapped into the point cloud data to obtain the parts and the reference object point cloud blocks, and the point cloud blocks are further screened through the color gamut, so that the detection cost is reduced, and the detection efficiency is improved. The YOLOV5L target detection network is adopted for positioning on the 2D picture, the current target detection based on the convolutional neural network is mature on the 2D picture, but most networks still have difficulty in achieving ideal effects on the detection of small targets. The YOLOV5 network performs Mosaic enhancement during data loading, the accuracy of small target detection is effectively improved through scaling, color space adjustment and Mosaic enhancement, the idea of learning an anchor frame is learned by introducing the distribution of a boundary frame in a custom data set, and the size of the anchor frame can be automatically adjusted by a model according to the data set, so that a more accurate target frame can be generated during target positioning. This provides assurance for subsequent accurate acquisition of the target point cloud.

And step 3: the point cloud is further screened through an R channel in the point cloud RGB, and in a subway train, a valve component is red and is obviously different from surrounding backgrounds and components, so that the color of the point cloud in the point cloud block can be further screened by utilizing the characteristic, and the fitting degree of the point cloud block is improved. And fitting a valve base mass center and a valve tail end mass center by using a mass center formula after obtaining a target point cloud block:

。

when the 3D camera collects data, all point cloud coordinates are obtained by using a three-dimensional coordinate system established by taking a camera lens as an origin. Because the position of the trolley at each stop cannot be guaranteed to be free from errors in data acquisition, the position of the offset valve centroid coordinate in the original template coordinate system needs to be obtained through coordinate transformation, and comparison and offset angle calculation can be carried out.

In the conversion of the three-dimensional coordinate system, two operators, namely a rotation conversion operator and a translation conversion operator, need to be solved, wherein the rotation conversion operator is used for correcting the shaft body deflection in the two coordinate systems, and the translation conversion operator is used for correcting the space positions of the two coordinate systems. In the project, the positioning error mainly comprises two parts, namely an error (translation error) caused by the parking position of the trolley and a positioning error (rotation error + translation error) of the pose of the mechanical arm.

Because the adopted UR5e cooperative mechanical arm repeated positioning precision is +/-0.03 mm, the error caused by the repeated positioning of the mechanical arm can be ignored under the precision, so that the rotation transformation operator can be ignored when the coordinate system conversion is carried out, and only the translation error caused by the trolley positioning error is considered.

The coordinates of the mass center of the valve base arranged in the reference template database are

The center of mass coordinate of the valve tail end is

. The coordinates of the mass center of the valve base are acquired in the detection process

Offset rear valve end centroid of

The center of mass coordinate of the valve tail end under the template coordinate system is obtained through coordinate transformation

。

The algorithm idea is as follows:

(1) translation transformation operator is solved through corresponding relation of mass center coordinates of valve base

. As shown in fig. 2.

(2) Coordinate of valve tail end centroid coordinate in template coordinate system is obtained through translation operator

：

As shown in FIG. 3, the coordinates of the center of mass of the valve end in the template coordinate system after the coordinate transformation are as

The coordinate of the center of mass of the valve end in the normal state in the template reference is

The length of the valve isl：

Calculated by a space distance formula

Euclidean distance of (c):

calculating valve offset angle by inverse trigonometric functionθ ₁：

The invention provides a perfect detection process for subway train valve detection, obtains greater freedom degree by comparing the data acquisition mode of the autonomous positioning shooting component of the train inspection robot with the mode of scanning trains by fixed cameras on the market to acquire data, can acquire more comprehensive data of equipment components, and can adjust the shooting angle according to the component detection requirement, thereby being more beneficial to subsequent algorithm detection and improving the algorithm accuracy. Meanwhile, the three-dimensional industrial camera is used for collecting 2D and 3D data of the component in a structured light mode, so that not only can 2D data (RGB) obtained in a traditional mode be obtained, but also accurate spatial information (XYZ) of the component can be collected, more solutions are provided for subsequent algorithm detection, and the false alarm rate is reduced.

According to the method, a deep learning model is introduced, the robustness and the generalization of an algorithm model are improved, the Yolov5L target detection model can still accurately position a target part when the shooting angle is deviated or certain environmental influence and stain influence exist, and the robustness is greatly improved compared with the traditional template matching. Meanwhile, with continuous data acquisition, a larger data set can be obtained, and data are sent into the positioning model for training after the label is printed, so that the model is more suitable for the working environment, and the detection accuracy is improved. Meanwhile, the network is an One-Stage model, namely the network directly detects a target area in a picture without generating a candidate frame and classifying the candidate frame, so that time consumption is greatly reduced, and through actual test, the detection of a 2D picture (1944 multiplied by 1200) of a valve on a 1080Ti 8G hardware platform only needs about 0.02s, and the requirement of real-time detection can be met.

According to the valve offset angle calculating method, the spatial relationship between the valve and the reference object is obtained by positioning, screening and solving the mass center and the fitting plane by utilizing point cloud data acquired from a high-precision three-dimensional camera. Therefore, a mathematical model is established, the deviation angle of the valve can be accurately calculated by combining the Euclidean distance and chord length formula with the inverse trigonometric function, and intuitive and rigorous mathematical extrapolation is achieved.

According to the invention, through a large amount of actual data tests of the Jinggang subway trains, all test cases can accurately judge whether the train is abnormal or not. Wherein some valves are manually measured. FIG. 4 is a schematic diagram of data test results and manual measurement results collected for multiple stops of the same valve. Where the straight line is the angle calculated from the manual measurement. It can be seen that the measured values of the algorithm model are always above and below the manually measured values, no obvious deviation occurs, and the deviation of the result is less than 1.5 degrees. As shown in fig. 5, the offset angle is calculated by model solution of the pictures taken by different types of valves in different vehicle compartments, the black star points are manual measurement values, and the black dots are model measurement values. And the detection deviation of the algorithm model is within 3 degrees according to the experimental comparison result. Accurate valve gesture detects can provide powerful guarantee for subway train's safe operation, and the difficult problem that must solve is patrolled and examined in the intellectuality of train more. The inspection robot can save data every time in a mode of replacing manpower, and a more accurate digital model is established for the train.

Claims

1. A subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation is characterized by comprising the following steps:

step 1: the subway inspection trolley is automatically positioned to a designated position, and 2D and 3D data of a subway train valve component to be overhauled are acquired through a three-dimensional industrial camera carried on a mechanical arm;

step 2: the method comprises the steps that a deep learning-based YOLOV5L target detection network positions a 2D picture component and a reference object;

s21: loading a trained YOLOV5L target detection model, and carrying out target detection processing on the 2D picture;

s22: the target detection model selects valve information from the 2D picture frame: valve base coordinates, valve end coordinates;

s23: mapping the coordinates to three-dimensional point cloud data, and acquiring point cloud data of a valve base and a valve tail end, thereby filtering irrelevant point cloud data and accelerating the model speed;

and step 3: further screening point clouds through an R channel in the point cloud RGB, and fitting a valve base mass center and a valve tail end mass center; calculating a conversion matrix of a current coordinate system and a template coordinate system by taking the coordinates of the valve base as a reference, and obtaining the coordinates of the tail end of the valve under the template coordinate system through coordinate transformation;

s31: traversing the point cloud data of the valve base and the valve end in the step 2, and accurately screening out point clouds belonging to the valve part through point cloud RGB information;

s32: and (3) utilizing a mass center formula to carry out mass center calculation on the screened point cloud to obtain a coordinate representing the spatial positions of the valve base and the valve tail end:

s33: determining coordinates of a valve base centroid and a valve tail end centroid in a normal state and a template coordinate system through first data acquisition; and taking the data as a reference, and calculating the coordinate b ' (x ') of the center of mass of the tail end of the valve under a template coordinate system through three-dimensional coordinate transformation by taking the center of mass of the valve base as a reference '₂，y′₂，z′₂)；

And 4, step 4: resolving the valve attitude, and judging whether the valve is in a normal state or not according to the comparison between the deflection angle and a set threshold value;

s41: the coordinate of the tail end of the valve is B (x) under the database template coordinate system_b，y_b，z_b) The center of mass of the tail end of the valve to be measured is b (x)₂，y₂，z₂) And the coordinate of the center of mass of the tail end of the valve in the template coordinate system after coordinate conversion is b '(x'₂，y′₂，z′₂) The valve length is l:

calculating the Euclidean distance of Bb' by a space distance formula:

s42: solving out valve offset angle theta through inverse trigonometric function₁：

S43: according to valve deflection angle theta₁And comparing with a set threshold value delta theta, and judging whether the valve is in a normal state:

2. A method for detecting the state of a subway valve based on two-dimensional positioning and three-dimensional attitude calculation according to claim 1, wherein said set threshold value Δ θ takes the value of 15 °.